Nginx PHP-FPM nginx crash in a single website empty server

Please fill in any relevant information that applies to you:

• CentOS Version: CentOS 7 64bit ?
• Centmin Mod Version Installed: 123.09beta01
• Nginx Version Installed:
  Code:

  nginx/1.15.6 (071118-181110)

• PHP Version Installed:
  PHP 7.1.23 (cli) (built: Oct 16 2018 15:13:58) ( NTS )
• MariaDB MySQL Version Installed: MariaDB 10.1.37
• When was last time updated Centmin Mod code base ? : uhmmmm november the 7th 2018 (when 1.15.6 was released)
• Persistent Config: no

after updating nginx to 1.15.6 is find nginx crashed once a day, (it's a sandbox)

---

 

I found it down every time cron run :

Code:

19 02 * * * /root/tools/phpmyadmin_update.sh

Infacts the crash time is 2:19.
I haven't found errors on logs files.
if I try to start it, it takes more than 30 seconds to start when in a similar server is instant... and sometimes it crashes.

It crashes if I restart it while is running.
If I start while it is in stop nginx goes up correctly (but it start in 30 sec)

Code:

journalctl -xe

output is saying:

Code:

nov 12 18:12:55 subdomain.domain.com kernel: [10459]     0 10459    29150      155      14      237             0 bash
nov 12 18:12:55 subdomain.domain.com kernel: [10614]    89 10614    22611       16      45      262             0 pickup
nov 12 18:12:55 subdomain.domain.com kernel: [12294]     0 12294    22637       37      47      253             0 local
nov 12 18:12:55 subdomain.domain.com kernel: [13845]     0 13845    50861        1      47      313             0 pure-ftpd
nov 12 18:12:55 subdomain.domain.com kernel: [14240]     0 14240    42940      861      40     4553             0 lfd - sleeping
nov 12 18:12:55 subdomain.domain.com kernel: [14616]    89 14616    22612       13      43      264             0 trivial-rewrite
nov 12 18:12:55 subdomain.domain.com kernel: [14675]    89 14675    22648       31      45      254             0 cleanup
nov 12 18:12:55 subdomain.domain.com kernel: [14854]     0 14854   425176   156546     819   242398             0 nginx
nov 12 18:12:55 subdomain.domain.com kernel: [14967]     0 14967    28363      105      13        0             0 service
nov 12 18:12:55 subdomain.domain.com kernel: [14974]     0 14974    28891      112      13        0             0 nginx
nov 12 18:12:55 subdomain.domain.com kernel: [14980]     0 14980    33717       64      26        0             0 systemctl
nov 12 18:12:55 subdomain.domain.com kernel: [14981]     0 14981     3869       36      13        0             0 systemd-tty-ask
nov 12 18:12:55 subdomain.domain.com kernel: [14982]     0 14982    73667      147      45        0             0 pkttyagent
nov 12 18:12:55 subdomain.domain.com kernel: [14986]     0 14986    28891      118      14        0             0 nginx
nov 12 18:12:55 subdomain.domain.com kernel: [14994]     0 14994    52903    34309     105        0             0 nginx
nov 12 18:12:55 subdomain.domain.com kernel: Out of memory: Kill process 14854 (nginx) score 666 or sacrifice child
nov 12 18:12:55 subdomain.domain.com kernel: Killed process 14854 (nginx) total-vm:1700704kB, anon-rss:626172kB, file-rss:0kB, shmem-rss:12kB
nov 12 18:12:55 subdomain.domain.com systemd[1]: nginx.service: main process exited, code=killed, status=9/KILL
nov 12 18:13:02 subdomain.domain.com systemd[1]: Started Session 45382 of user root.
-- Subject: L'unità session-45382.scope termina la fase di avvio
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- L'unità session-45382.scope ha terminato la fase di avvio.
--
-- La fase di avvio è done.
nov 12 18:13:02 subdomain.domain.com systemd[1]: Starting Session 45382 of user root.
-- Subject: L'unità session-45382.scope inizia la fase di avvio
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- L'unità session-45382.scope ha iniziato la fase di avvio.
nov 12 18:13:02 subdomain.domain.com CROND[15007]: (root) CMD (php /home/nginx/domains/domain.com/public/modules/cronjobs/controllers/front/cron.php --token=$2y$10$n.
nov 12 18:13:03 subdomain.domain.com postfix/pickup[10614]: 2920E3FEE9: uid=0 from=<root>
nov 12 18:13:03 subdomain.domain.com postfix/cleanup[14675]: 2920E3FEE9: message-id=<20181112171303.2920E3FEE9@subdomain.domain.com>
nov 12 18:13:03 subdomain.domain.com postfix/qmgr[16533]: 2920E3FEE9: from=<root@subdomain.domain.com>, size=911, nrcpt=1 (queue active)
nov 12 18:13:03 subdomain.domain.com postfix/local[12294]: 2920E3FEE9: to=<root@subdomain.domain.com>, orig_to=<root>, relay=local, delay=0.28, delays=0.17/0.02/0/0.09,
nov 12 18:13:03 subdomain.domain.com postfix/qmgr[16533]: 2920E3FEE9: removed
nov 12 18:13:04 subdomain.domain.com nginx[14986]: nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nov 12 18:13:10 subdomain.domain.com nginx[14986]: nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
nov 12 18:13:10 subdomain.domain.com nginx[14986]: Reloading nginx: [FAILED]
nov 12 18:13:10 subdomain.domain.com systemd[1]: PID 14854 read from file /usr/local/nginx/logs/nginx.pid does not exist or is a zombie.
nov 12 18:13:10 subdomain.domain.com systemd[1]: Reload failed for SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
-- Subject: L'unità nginx.service termina il caricamento della propria configurazione
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- L'unità nginx.service è terminata ricaricando la propria configurazione
--
-- Il risultato è failed.
nov 12 18:13:10 subdomain.domain.com systemd[1]: Unit nginx.service entered failed state.
nov 12 18:13:10 subdomain.domain.com systemd[1]: nginx.service failed.
nov 12 18:13:10 subdomain.domain.com polkitd[456]: Unregistered Authentication Agent for unix-process:14980:234176005 (system bus name :1.90928, object path /org/freedesktop/P
lines 1225-1277/1277 (END)

So it seems a memory related issue.

I deleted the only vhost config I have (besides the phpmyadmin one) and the "restart" now is fast.

this is the configuration:

Code:

#x# HTTPS-DEFAULT
 server {
  
   server_name domain.com www.domain.com;
   return 302 https://domain.com$request_uri;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

server {
  listen 443 ssl http2 reuseport;
  server_name domain.com www.domain.com;

  include /usr/local/nginx/conf/ssl/domain.com/domain.com.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

 
  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;


## per thirtybees
   index index.html index.htm index.php;

  rewrite ^/([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$1$2$3.jpg last;
  rewrite ^/([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$1$2$3$4.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$1$2$3$4$5.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$1$2$3$4$5$6.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$1$2$3$4$5$6$7.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$1$2$3$4$5$6$7$8.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$1$2$3$4$5$6$7$8$9.jpg last;
  rewrite ^/([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])([0-9])(\-[_a-zA-Z0-9-]*)?(-[0-9]+)?/.+\.jpg$ /img/p/$1/$2/$3/$4/$5/$6/$7/$8/$1$2$3$4$5$6$7$8$9$10.jpg last;

  rewrite ^/c/([0-9]+)(-[_a-zA-Z0-9-]*)/[_a-zA-Z0-9-]*.jpg$ /img/c/$1$2.jpg last;
  rewrite ^/c/([a-zA-Z-]+)/[a-zA-Z0-9-]+.jpg$ /img/c/$1.jpg last;
  rewrite ^/([0-9]+)(-[_a-zA-Z0-9-]*)/[_a-zA-Z0-9-]*.jpg$ /img/c/$1$2.jpg last;


  rewrite ^/order$ /index.php?controller=order last;
  if (!-e $request_filename){
  rewrite ^(.*)$ /index.php last;
  }

  # Redirect needed to "hide" index.php

  location ~* \.tpl$ {
    deny all;
  }

## fine per thirtybees


# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/domain.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf;
  root /home/nginx/domains/domain.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  location / {
  include /usr/local/nginx/conf/503include-only.conf;

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Wordpress Permalinks example
  #try_files $uri $uri/ /index.php?q=$uri&$args;

  }

  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;

}

I then restored this vhost config file to trying to isolate the problem commenting single includes one by one and the issue suddenly disappeared (with all the standard includes activated!!)... I'm going to monitor this system because these things make me mad!

 

and of course the server is right... it was the autoprotect conf. The vhost is a thirtybees (fork of prestashop). a 3rd party plugin add .htaccess in every subdirectory of it's "internal data" dir (12806 dirs and counting!).

as if internal data of xenforo would include an .htaccess in each sub dir.

 

it was a combination of autoprotect and a 3rd party plugin: