PHP-FPM Recent update causes PHP-FPM spikes and WP crashes

HELP! I have no idea what to do anymore. I know tweaking is up to me, and I'm interested in hiring someone to fix this.

On June 25 or 26, I ran "yum update" and rebuilt Nginx, both for the latest security updates. Since then, my server has been extremely unstable. PHP-FPM is spiking on CPU usage and making my WordPress websites crash.

Here's my setup:

• Centminmod .07
• PHP 5.4.42 (also tried 5.5 and 5.6 versions)
• Nginx 1.8.0 (also tried 1.7.9 and 1.9.1)
• Followed WordPress + Super Cache plugin for Centmin Mod Nginx | wp1.centmincache.com to setup WP Super Cache for my WordPress sites
• 2 core, 2 GB RAM VPS on Vultr
• Four WordPress sites: 3 single-installations and 1 Multisite—all have their Nginx and WPSC configured

I've tried re-adjusting the php-fpm.conf settings as follows (brackets indicate values I've tried).

---

Code:

emergency_restart_threshold = [3, 5, 10]
emergency_restart_interval = [30s, 1m]
process_control_timeout = [5s, 10s]
listen.backlog = [128 to 8150]
pm = ondemand
pm.max_children = [2 to 50]
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.max_requests = [100, 200, 500, 1000, 10000, 50000]
pm.process_idle_timeout = [5s, 10s]

Last night, when I reduced pm.max_requests down to 100, I thought everything was golden again. But I woke up to see PHP-FPM had locked up and crashed my sites 8 sites.

When PHP-FPM is locked up, sometimes I can get a phpstatus result that will look something like this (brackets indicate numbers I've seen):

Code:

pool:                 www
process manager:      ondemand
start time:           02/Jul/2015:17:58:19 -0400
start since:          2175
accepted conn:        2795
listen queue:         1153
max listen queue:     1393
listen queue len:     [sometimes 0, sometimes my backlog number of 128 or 8150]
idle processes:       0
active processes:     10
total processes:      10
max active processes: 10
max children reached: 1
slow requests:        1

I think I know, in theory, what's happening. PHP-FPM is given some process that hangs and causes more tasks to pile up, eventually filling up the queue. PHP-FPM tries to process all of these but can't because the CPU is maxed out. Is that right?

The strangest thing about all of this is that shouldn't my use of WP Super Cache mean that Nginx shouldn't be sending this many processes to PHP-FPM? I can verify that WPSC is working, but this behavior is as if hundreds of requests are bypassing the cache.

When I've enabled the FPM slow log, nothing in particular stands out as causing the problem.

For reference, here are my site confs.

Single site:

Code:

# redirect from non-www to www
# uncomment, save file and restart Nginx to enable
#server {
#            listen   80;
#            server_name danieljlewis.net;
#            return 301 $scheme://www.danieljlewis.net$request_uri;
#       }

server {
  server_name danieljlewis.net www.danieljlewis.net;
  if ($host ~* www\.(.*)) {
      set $host_without_www $1;
      #   rewrite ^(.*)$ $scheme://$host_without_www$1 permanent; #1
      rewrite ^ $scheme://$host_without_www$1$request_uri permanent; #2
  }

# ngx_pagespeed & ngx_pagespeed handler
# removed 2015-03-16 include /usr/local/nginx/conf/pagespeed.conf;
# removed 2015-03-16 include /usr/local/nginx/conf/pagespeedhandler.conf;
# removed 2015-03-16 include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/danieljlewis.net/log/access.log combined buffer=32k;
  # removed 2015-03-16 access_log /var/log/nginx/access.cache.log cache;
  error_log /home/nginx/domains/danieljlewis.net/log/error.log;

  root /home/nginx/domains/danieljlewis.net/public;

  #Yoast sitemap
    location ~ ([^/]*)sitemap(.*)\.x(m|s)l$ {
    rewrite ^/sitemap\.xml$ /sitemap_index.xml permanent;
    rewrite ^/([a-z]+)?-?sitemap\.xsl$ /index.php?xsl=$1 last;
    rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
    rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;

    ## following lines are options. Needed for wordpress-seo addons
    rewrite ^/video-sitemap\.xsl$ /index.php?xsl=video last;
    access_log off;
  }

  location / {

    if (-f $request_filename/index.html){
      rewrite (.*) $1/index.html break;
    }

    if (-f $request_filename/index.php){
      rewrite (.*) $1/index.php;
    }

    if (!-f $request_filename){
      rewrite (.*) /index.php;
    }

    # block common exploits, sql injections etc
    include /usr/local/nginx/conf/block.conf;
  }

  include /usr/local/nginx/conf/wpsupercache.conf;
  include /usr/local/nginx/conf/wpsecure.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  #include /usr/local/nginx/conf/phpstatus.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
}

Multisite:

Code:

map $uri $blogname{
  ~^(?<blogpath>/[^/]+/)files/(.*)    $blogpath ;
}
map $blogname $blogid {
  default -999;
  include /home/nginx/domains/noodle.mx/public/wp-content/uploads/nginx-helper/map.conf ;
}

server {
  listen 80;
  server_name noodle.mx *.com www.noodle.mx;
  root /home/nginx/domains/noodle.mx/public;
  #Uncomment to debug rewrite rules
  #error_log    logs/rewrite.log notice;


  if ($host ~* www\.(.*)) {
      set $host_without_www $1;
    #   rewrite ^(.*)$ $scheme://$host_without_www$1 permanent; #1
      rewrite ^ $scheme://$host_without_www$1$request_uri permanent; #2
  }

  # SSL
  listen 443 ssl;
  ssl on;
  ssl_certificate /usr/local/nginx/conf/ssl/djosephdesign.com.crt-combined;
  ssl_certificate_key /usr/local/nginx/conf/ssl/djosephdesign.com.key;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ecdh_curve  secp384r1;
  ssl_session_cache      shared:SSL:10m;
  ssl_session_timeout  10m;
  ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
  # add_header Alternate-Protocol  443:npn-spdy/3;
  add_header Strict-Transport-Security "max-age=0; includeSubdomains";
  # added by Tiago - UCC cert ####
  set $ssl 0;
  if ($scheme = https) {
    set $ssl 1;
  }
  if ($host ~ "(noodle.mx|oncepodcast.com|theaudacitytopodcast.com)") {
    set $ssl 0;
  }
  if ($ssl = 1) {
    rewrite "^/(.*)" http://$host/$1 permanent;
  }
  # /SSL

  # ngx_pagespeed & ngx_pagespeed handler
  # removed 2015-03-16 include /usr/local/nginx/conf/pagespeed.conf;
  # removed 2015-03-16 include /usr/local/nginx/conf/pagespeedhandler.conf;
  # removed 2015-03-16 include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/noodle.mx/log/access.log combined buffer=32k;
  # removed 2015-03-16 access_log /var/log/nginx/access.cache.log cache;
  error_log /home/nginx/domains/noodle.mx/log/error.log;

  #Yoast sitemap
  location ~ ([^/]*)sitemap(.*)\.x(m|s)l$ {
    rewrite ^/sitemap\.xml$ /sitemap_index.xml permanent;
    rewrite ^/([a-z]+)?-?sitemap\.xsl$ /index.php?xsl=$1 last;
    rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
    rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;

    ## following lines are options. Needed for wordpress-seo addons
    rewrite ^/video-sitemap\.xsl$ /index.php?xsl=video last;
    access_log off;
  }

  location ~ ^(/[^/]+/)?files/(?<rt_file>.+) {
    try_files /wp-content/blogs.dir/$blogid/files/$rt_file /wp-includes/ms-files.php?file=$rt_file ;
    access_log off;
    log_not_found off;
    expires max;
  }

  location / {

    if (-f $request_filename/index.html){
      rewrite (.*) $1/index.html break;
    }

    if (-f $request_filename/index.php){
      rewrite (.*) $1/index.php;
    }

    if (!-f $request_filename){
      rewrite (.*) /index.php;
    }

    # block common exploits, sql injections etc
    include /usr/local/nginx/conf/block.conf;
  }

  # Pass uploaded files to wp-includes/ms-files.php.
  rewrite /files/$ /index.php last;

  if ($uri !~ wp-content/plugins) {
    rewrite /files/(.+)$ /wp-includes/ms-files.php?file=$1 last;
  }

  # Rewrite multisite '.../wp-.' and '.../.php'.
  if (!-e $request_filename) {
    rewrite ^/[_0-9a-zA-Z-]+(/wp-.*) $1 last;
    rewrite ^/[_0-9a-zA-Z-]+.(/wp-admin/.\.php)$ $1 last;
    rewrite ^/[_0-9a-zA-Z-]+(/.*\.php)$ $1 last;
  }

  include /usr/local/nginx/conf/noodlemx_redirects.conf;
 include /usr/local/nginx/conf/wpsupercache.conf;
  # include /usr/local/nginx/conf/wpffpc.conf;
  include /usr/local/nginx/conf/wpsecure.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/php.conf;
  include /usr/local/nginx/conf/drop.conf;
}

Please help! This has be driving me crazy for a week and I've wasted too much time trying to fix this. I'm willing to hire someone at a reasonable price to fix this for me.

 

Code:

pool:                 www
process manager:      ondemand
start time:           03/Jul/2015:07:59:45 -0400
start since:          5509
accepted conn:        6733
listen queue:         902
max listen queue:     902
listen queue len:     8150
idle processes:       0
active processes:     10
total processes:      10
max active processes: 10
max children reached: 7
slow requests:        0

************************
pid:                  6163
state:                Running
start time:           03/Jul/2015:09:29:08 -0400
start since:          146
requests:             59
request duration:     927917
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6164
state:                Running
start time:           03/Jul/2015:09:29:08 -0400
start since:          146
requests:             57
request duration:     1452416
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6136
state:                Running
start time:           03/Jul/2015:09:28:59 -0400
start since:          155
requests:             65
request duration:     1477431
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6107
state:                Running
start time:           03/Jul/2015:09:27:38 -0400
start since:          236
requests:             89
request duration:     12346
request method:       GET
request URI:          /phpstatus?full
content length:       0
user:                 -
script:               -
last request cpu:     0.00
last request memory:  0

************************
pid:                  6165
state:                Running
start time:           03/Jul/2015:09:29:11 -0400
start since:          143
requests:             55
request duration:     1554383
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6169
state:                Running
start time:           03/Jul/2015:09:29:15 -0400
start since:          139
requests:             48
request duration:     1144879
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6170
state:                Running
start time:           03/Jul/2015:09:29:16 -0400
start since:          138
requests:             48
request duration:     631853
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6171
state:                Running
start time:           03/Jul/2015:09:29:16 -0400
start since:          138
requests:             52
request duration:     1268572
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6172
state:                Running
start time:           03/Jul/2015:09:29:16 -0400
start since:          138
requests:             51
request duration:     1386855
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

************************
pid:                  6173
state:                Running
start time:           03/Jul/2015:09:29:16 -0400
start since:          138
requests:             48
request duration:     830838
request method:       POST
request URI:          /xmlrpc.php
content length:       309
user:                 -
script:               /home/nginx/domains/noodle.mx/public/xmlrpc.php
last request cpu:     0.00
last request memory:  0

It's strange to see XML RPC in here, because I installed Disable XML-RPC.

 

Thank you! I implemented for xmlrpc.php and wp-login.php and could verify that's working from siege.

I'll report back with my results either for further help or to let others know.

 

THANK YOU! Since implementing those lines of code, my server has not crashed in 24 hours!

Either my update last week reset some security setting to prevent these attacks, or it was pure coincidence that the attacks around the same time as I updated. In either case, I'm considering this fixed!

As an expression of my appreciation for your help, I just upgraded to premium forum membership.

 

Yeah, I thought I had Cloudflare setup for all of my sites, but my assistant didn't finished that project.

 

Oh. Then it's not affordable for me with my 15+ sites.