Learn about Centmin Mod LEMP Stack today
Register Now

Nginx PHP-FPM nginx crash in a single website empty server

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Kintaro, Nov 9, 2018.

Tags:
  1. Kintaro

    Kintaro Premium Member Premium Member

    47
    2
    8
    Dec 2, 2016
    Ratings:
    +9
    Local Time:
    1:45 PM
    1.9.10
    MariaDB 10
    Please fill in any relevant information that applies to you:
    • CentOS Version: CentOS 7 64bit ?
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed:
      Code:
      nginx/1.15.6 (071118-181110)
    • PHP Version Installed:
      PHP 7.1.23 (cli) (built: Oct 16 2018 15:13:58) ( NTS )
    • MariaDB MySQL Version Installed: MariaDB 10.1.37
    • When was last time updated Centmin Mod code base ? : uhmmmm november the 7th 2018 (when 1.15.6 was released)
    • Persistent Config: no

    after updating nginx to 1.15.6 is find nginx crashed once a day, (it's a sandbox)
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:45 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Can you elaborate on crash, symptoms, errors on browser front end and in nginx logs ?

    If you downgrade to 1.15.5 does issue go away ?

    To troubleshoot Nginx and PHP-FPM issues you'd want to check the domain site's vhost access.log and error.log logs located within directory at /home/nginx/domains/yourdomain.com/logs. You can see a full overview at centminmod.com/configfiles.html

    FAQ item 19 has more info on all Centmin Mod relevant log files locations and how to use tail command to view a sample of the entries.

    Also post the contents of your site's nginx vhost http /usr/local/nginx/conf/conf.d/yourdomain.com.conf and/or /usr/local/nginx/conf/conf.d/yourdomain.com.ssl.conf in BBCODE CODE tags as outlined at How to use forum BBCODE code tags | Centmin Mod Community

    Nginx Debug Mode



    If you have a lot of free disk space, you can enable debug nginx version and compile a nginx debug build of nginx via centmin.sh option NGINX_DEBUG=y centminmod/centmin.sh at 123.08stable · centminmod/centminmod · GitHub

    You can place the option in persistent config /etc/centminmod/custom_config.inc outlined at centminmod.com/upgrade.html#persistent so place in file /etc/centminmod/custom_config.inc
    Code:
    NGINX_DEBUG=y
    then recompile nginx via centmin.sh menu option 4 and then in your nginx error_log directive add debug option A debugging log and restart nginx and check logs.

    You can also backup nginx binary with debug and then again without debug once and switch between them without recompiling Beta Branch - Nginx binary + modules backup/restore - nginx-binary-backup.sh. So to backup existing nginx binary run
    Code (Text):
    cmupdate
    /usr/local/src/centminmod/tools/nginx-binary-backup.sh backup
    

    Do this once before you recompile for nginx with debug mode and then do this again for after you recompile again with nginx debug mode. You will have 2 nginx binary backups and can then switch via restore method between them without needing to recompile again.

    They will be very very very verbose and lot alot to disk usage logged to error logs, so ensure you have a lot of disk free space.

    You can minimise this by limiting it to specific ip based client connections too
    After debugging is done, reverse the debug now by setting NGINX_DEBUG=n and recompile Nginx again or restore Nginx binary without debug mode if you used tools/nginx-binary-backup.sh.
     
  3. Kintaro

    Kintaro Premium Member Premium Member

    47
    2
    8
    Dec 2, 2016
    Ratings:
    +9
    Local Time:
    1:45 PM
    1.9.10
    MariaDB 10
    I found it down every time cron run :
    Code:
    19 02 * * * /root/tools/phpmyadmin_update.sh
    Infacts the crash time is 2:19.
    I haven't found errors on logs files.
    if I try to start it, it takes more than 30 seconds to start when in a similar server is instant... and sometimes it crashes.

    It crashes if I restart it while is running.
    If I start while it is in stop nginx goes up correctly (but it start in 30 sec)

    Code:
    journalctl -xe
    output is saying:
    Code:
    nov 12 18:12:55 subdomain.domain.com kernel: [10459]     0 10459    29150      155      14      237             0 bash
    nov 12 18:12:55 subdomain.domain.com kernel: [10614]    89 10614    22611       16      45      262             0 pickup
    nov 12 18:12:55 subdomain.domain.com kernel: [12294]     0 12294    22637       37      47      253             0 local
    nov 12 18:12:55 subdomain.domain.com kernel: [13845]     0 13845    50861        1      47      313             0 pure-ftpd
    nov 12 18:12:55 subdomain.domain.com kernel: [14240]     0 14240    42940      861      40     4553             0 lfd - sleeping
    nov 12 18:12:55 subdomain.domain.com kernel: [14616]    89 14616    22612       13      43      264             0 trivial-rewrite
    nov 12 18:12:55 subdomain.domain.com kernel: [14675]    89 14675    22648       31      45      254             0 cleanup
    nov 12 18:12:55 subdomain.domain.com kernel: [14854]     0 14854   425176   156546     819   242398             0 nginx
    nov 12 18:12:55 subdomain.domain.com kernel: [14967]     0 14967    28363      105      13        0             0 service
    nov 12 18:12:55 subdomain.domain.com kernel: [14974]     0 14974    28891      112      13        0             0 nginx
    nov 12 18:12:55 subdomain.domain.com kernel: [14980]     0 14980    33717       64      26        0             0 systemctl
    nov 12 18:12:55 subdomain.domain.com kernel: [14981]     0 14981     3869       36      13        0             0 systemd-tty-ask
    nov 12 18:12:55 subdomain.domain.com kernel: [14982]     0 14982    73667      147      45        0             0 pkttyagent
    nov 12 18:12:55 subdomain.domain.com kernel: [14986]     0 14986    28891      118      14        0             0 nginx
    nov 12 18:12:55 subdomain.domain.com kernel: [14994]     0 14994    52903    34309     105        0             0 nginx
    nov 12 18:12:55 subdomain.domain.com kernel: Out of memory: Kill process 14854 (nginx) score 666 or sacrifice child
    nov 12 18:12:55 subdomain.domain.com kernel: Killed process 14854 (nginx) total-vm:1700704kB, anon-rss:626172kB, file-rss:0kB, shmem-rss:12kB
    nov 12 18:12:55 subdomain.domain.com systemd[1]: nginx.service: main process exited, code=killed, status=9/KILL
    nov 12 18:13:02 subdomain.domain.com systemd[1]: Started Session 45382 of user root.
    -- Subject: L'unità session-45382.scope termina la fase di avvio
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- L'unità session-45382.scope ha terminato la fase di avvio.
    --
    -- La fase di avvio è done.
    nov 12 18:13:02 subdomain.domain.com systemd[1]: Starting Session 45382 of user root.
    -- Subject: L'unità session-45382.scope inizia la fase di avvio
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- L'unità session-45382.scope ha iniziato la fase di avvio.
    nov 12 18:13:02 subdomain.domain.com CROND[15007]: (root) CMD (php /home/nginx/domains/domain.com/public/modules/cronjobs/controllers/front/cron.php --token=$2y$10$n.
    nov 12 18:13:03 subdomain.domain.com postfix/pickup[10614]: 2920E3FEE9: uid=0 from=<root>
    nov 12 18:13:03 subdomain.domain.com postfix/cleanup[14675]: 2920E3FEE9: message-id=<[email protected]>
    nov 12 18:13:03 subdomain.domain.com postfix/qmgr[16533]: 2920E3FEE9: from=<[email protected]>, size=911, nrcpt=1 (queue active)
    nov 12 18:13:03 subdomain.domain.com postfix/local[12294]: 2920E3FEE9: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.28, delays=0.17/0.02/0/0.09,
    nov 12 18:13:03 subdomain.domain.com postfix/qmgr[16533]: 2920E3FEE9: removed
    nov 12 18:13:04 subdomain.domain.com nginx[14986]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nov 12 18:13:10 subdomain.domain.com nginx[14986]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    nov 12 18:13:10 subdomain.domain.com nginx[14986]: Reloading nginx: [FAILED]
    nov 12 18:13:10 subdomain.domain.com systemd[1]: PID 14854 read from file /usr/local/nginx/logs/nginx.pid does not exist or is a zombie.
    nov 12 18:13:10 subdomain.domain.com systemd[1]: Reload failed for SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
    -- Subject: L'unità nginx.service termina il caricamento della propria configurazione
    -- Defined-By: systemd
    -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
    --
    -- L'unità nginx.service è terminata ricaricando la propria configurazione
    --
    -- Il risultato è failed.
    nov 12 18:13:10 subdomain.domain.com systemd[1]: Unit nginx.service entered failed state.
    nov 12 18:13:10 subdomain.domain.com systemd[1]: nginx.service failed.
    nov 12 18:13:10 subdomain.domain.com polkitd[456]: Unregistered Authentication Agent for unix-process:14980:234176005 (system bus name :1.90928, object path /org/freedesktop/P
    lines 1225-1277/1277 (END)
    
    So it seems a memory related issue.

    I deleted the only vhost config I have (besides the phpmyadmin one) and the "restart" now is fast.

    this is the configuration:
    Code:
    #x# HTTPS-DEFAULT
     server {
      
       server_name domain.com www.domain.com;
       return 302 https://domain.com$request_uri;
       include /usr/local/nginx/conf/staticfiles.conf;
     }
    
    server {
      listen 443 ssl http2 reuseport;
      server_name domain.com www.domain.com;
    
      include /usr/local/nginx/conf/ssl/domain.com/domain.com.crt.key.conf;
      include /usr/local/nginx/conf/ssl_include.conf;
    
     
      http2_max_field_size 16k;
      http2_max_header_size 32k;
      # mozilla recommended
      ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
    
      # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
      #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
      #add_header X-Frame-Options SAMEORIGIN;
      add_header X-Xss-Protection "1; mode=block" always;
      add_header X-Content-Type-Options "nosniff" always;
      #add_header Referrer-Policy "strict-origin-when-cross-origin";
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
     
      # enable ocsp stapling
      resolver 8.8.8.8 8.8.4.4 valid=10m;
      resolver_timeout 10s;
      ssl_stapling on;
      ssl_stapling_verify on;
    
    
    ## per thirtybees
       index index.html index.htm index.php;
    
      rewrite ^/([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$1$2$3.jpg last;
      rewrite ^/([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$1$2$3$4.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$1$2$3$4$5.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$1$2$3$4$5$6.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$1$2$3$4$5$6$7.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$1$2$3$4$5$6$7$8.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$1$2$3$4$5$6$7$8$9.jpg last;
      rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$8/$1$2$3$4$5$6$7$8$9$10.jpg last;
    
      rewrite ^/c/([0-9]+)(-[_a-zA-Z0-9-]*)/[_a-zA-Z0-9-]*.jpg$ /img/c/$1$2.jpg last;
      rewrite ^/c/([a-zA-Z-]+)/[a-zA-Z0-9-]+.jpg$ /img/c/$1.jpg last;
      rewrite ^/([0-9]+)(-[_a-zA-Z0-9-]*)/[_a-zA-Z0-9-]*.jpg$ /img/c/$1$2.jpg last;
    
    
      rewrite ^/order$ /index.php?controller=order last;
      if (!-e $request_filename){
      rewrite ^(.*)$ /index.php last;
      }
    
      # Redirect needed to "hide" index.php
    
      location ~* \.tpl$ {
        deny all;
      }
    
    ## fine per thirtybees
    
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf;
      root /home/nginx/domains/domain.com/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
    
    # block common exploits, sql injections etc
    #include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Wordpress Permalinks example
      #try_files $uri $uri/ /index.php?q=$uri&$args;
    
      }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    
    }
    I then restored this vhost config file to trying to isolate the problem commenting single includes one by one and the issue suddenly disappeared (with all the standard includes activated!!)... I'm going to monitor this system because these things make me mad!
     
    • Informative Informative x 1
  4. Kintaro

    Kintaro Premium Member Premium Member

    47
    2
    8
    Dec 2, 2016
    Ratings:
    +9
    Local Time:
    1:45 PM
    1.9.10
    MariaDB 10
    and of course the server is right... it was the autoprotect conf. The vhost is a thirtybees (fork of prestashop). a 3rd party plugin add .htaccess in every subdirectory of it's "internal data" dir (12806 dirs and counting!).

    as if internal data of xenforo would include an .htaccess in each sub dir. :dead:
     
    Last edited: Nov 13, 2018
  5. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:45 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    weird indeed !
     
  6. Kintaro

    Kintaro Premium Member Premium Member

    47
    2
    8
    Dec 2, 2016
    Ratings:
    +9
    Local Time:
    1:45 PM
    1.9.10
    MariaDB 10
    it was a combination of autoprotect and a 3rd party plugin:
     
    • Informative Informative x 1
  7. eva2000

    eva2000 Administrator Staff Member

    36,915
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,438
    Local Time:
    10:45 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    ah missed that 2nd post you made.
     
..