PHP-FPM How to troubleshoot & optimize PHP-FPM server?

eva2000 · Aug 2, 2018

Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such. However, below steps can help - starting with enabling PHP-FPM status page output. There's also an accompanying guide for high CPU load troubleshooting.

FYI, if your web app supports it, you should update to latest PHP 7.4, PHP 8.0 or PHP 8.1 version for performance. See Wordpress benchmarks here (for reference PHP 7.0.x is 2x faster than PHP 5.6.x already) and PHP 8.1.0 vs 8.0.13 vs 7.4.26 vs 7.3.33 vs 7.2.34 benchmarks.

Nginx 502 or 504 Bad Gateway Errors

Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable PHP-FPM status page output outlined at

Enable PHP-FPM status page output http://centminmod.com/phpfpm.html#browserstatus. Update: Centmin Mod 123.09beta01 and newer have it enabled by default for PHP 7+ via cminfo phpstats command.

and https://community.centminmod.com/threads/pm-max_children.479/ and

https://community.centminmod.com/threads/warning-pool-www-server-reached-max_children-setting-50-consider-raising-it.791/ which outlines the official PHP-FPM config documentation as well.

PHP: Configuration - Manual

How To Guide For Nginx & PHP-FPM Debugging

Enabling PHP-FPM status also allows setting up 3rd party PHP-FPM status metric monitoring from services like:

Nixststats PHP-FPM monitoring Monitor PHP-FPM with Nixstats

Nginx Amplify PHP-FPM monitoring Monitoring PHP Applications with NGINX Amplify

Checking PHP-FPM etc logs

You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

In SSH2 telnet you can use tail command to view the last X number of lines in the file.

For example for viewing last 10 lines in the file for:

For Nginx access and error logs:
Code:
  tail -10 /usr/local/nginx/logs/access.log
  tail -10 /usr/local/nginx/logs/error.log
For specific domainname.com access and error log:
Code:
  tail -10 /home/nginx/domains/domainname.com/log/access.log
  tail -10 /home/nginx/domains/domainname.com/log/error.log
For other system error logs located at /var/log:

list /var/log files in ascending time order so the most recently modified files are at the bottom
Code:
  ls -lhrt /var/log
Code:
total 2.7M
-rw------- 1 root  root    0 Aug 29 15:33 tallylog
-rw------- 1 root  root    0 Aug 29 15:33 spooler
drwx------ 3 root  root 4.0K Aug 29 15:35 samba
drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
-rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
-rw------- 1 root  500     0 Oct  8 18:13 boot.log
-rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
-rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
-rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
-rw------- 1 root  utmp  94K Nov  7 22:59 btmp
drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
-rw------- 1 root  root 269K Nov  8 21:39 messages
-rw------- 1 root  root 110K Nov  8 23:08 secure
-rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
-rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
-rw------- 1 root  root  69K Nov  8 23:08 lfd.log
-rw------- 1 root  root 332K Nov  8 23:08 maillog
-rw------- 1 root  500  1.6M Nov  8 23:10 cron
For PHP-FPM error log:
Code:
  tail -10 /var/log/php-fpm/www-error.log
and/or
Code:
  /var/log/php-fpm/www-php.error.log
For MySQL / MariaDB error log:
Code:
  tail -10 /var/log/mysqld.log
For CSF firewall LFD log:
Code:
  tail -10 /var/log/lfd.log
For Mail log:
Code:
  tail -10 /var/log/maillog
For Cron job logs:
Code:
  tail -10 /var/log/cron
How to edit php.ini and php-fpm configuration files ?

Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:

Edit php.ini = phpedit ( /usr/local/lib/php.ini )

Edit my.cnf = mycnf ( /etc/my.cnf )

Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )

Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )

Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later

Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )

Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )

Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )

nginx stop/start/restart = ngxstop/ngxstart/ngxrestart

php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart

mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart

nginx + php-fpm stop/start/restart = npstop/npstart/nprestart

memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart

csf stop/start/restart = csfstop/csfstart/csfrestart

Troubleshooting Tools

However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.

20 Command Line Tools to Monitor Linux Performance

13 Linux Performance Monitoring Tools - Part 2

80 Linux Monitoring Tools for SysAdmins - Server Density Blog

Amazing ! 25 Linux Performance Monitoring Tools

20 Linux System Monitoring Tools Every SysAdmin Should Know

How to monitor disk I/O in Linux from command line - Xmodulo

10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring

Generate CPU, Memory & I/O report using SAR command

My mysqlmymonlite.sh script can help with that mysqlmymonlite.sh README text file has instructions for setting up a cronjob to email you reports i.e. every 10/15min or every hour.

Centmin Mod cminfo top command etc

Centmin Mod 123.09beta01 and newer also have cminfo top command outlined here Insight Guide - cminfo command explained along with new phpstats, phpmem, sar-cpu and sar-mem.

Full stats via cminfo top command can output info that might be sensitive like server location, database usernames/database names and query info so you would need to mask or hide that if if posting on public forums. The info is meant for your eyes only or for whoever you hire to work on your server most of the time.
Code (Text):
cminfo top
Example output for cminfo phpstats command if PHP-FPM process manager is set to either non-default static or dynamic, pm = ondemand and not pm = static or pm = dynamic
Code (Text):
cminfo phpstats

------------------------------------------------------------------
Total PHP-FPM Master Processes: 1
PHP-FPM Pool Names:
www
------------------------------------------------------------------
27335 php-fpm: master /usr/local/etc/php-fpm.conf
------------------------------------------------------------------
Current Free Memory (KB): 26181960
PHP-FPM Available Memory (KB): 26423228
Estimated Max PHP Children: 3285.54
Estimated Max PHP Children To CPU Thread Ratio: 410.693
PHP-FPM Total Children: 30 from 1 PHP-FPM master(s)
PHP-FPM Total Used Memory (KB): swap:0 uss:7824 pss:12832 rss:241268
PHP-FPM Average Per Child (KB): swap:0 uss:260.8 pss:427.733 rss:8042.27
uss = user set size
pss = process set size
rss = resident set size

Ready to handle connections
pool:                 www
process manager:      static
start time:           12/Jan/2020:15:22:22 +0000
start since:          9
accepted conn:        2
listen queue:         0
max listen queue:     0
listen queue len:     65535
idle processes:       29
active processes:     1
total processes:      30
max active processes: 1
max children reached: 0
slow requests:        0
Notes:

Centmin Mod already installed Glances tool. It can be invoked from SSH command glances or top2. Official Glances site at nicolargo.github.io/glances/ and documentation here.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

eva2000 · May 17, 2019

Additional Information

Online guides for optimal php-fpm config are predominantly incorrect as it isn't just about memory. Optimizing php-fpm configurations is based on monitoring and measuring php-fpm cpu and memory usage and your PHP web apps response time/process time and understanding what your php-fpm statistics output metrics mean as outlined at

https://community.centminmod.com/threads/how-to-troubleshoot-optimize-php-fpm-server.15317/

http://centminmod.com/phpfpm.html#browserstatus

PHP: Configuration - Manual

php-fpm.conf config options

php-fpm.conf config options are located in /usr/local/etc/php-fpm.conf settings file for main php-fpm pool as outlined at https://centminmod.com/configfiles.html

From above php.net php-fpm configuration manual

pm string

Choose how the process manager will control the number of child processes. Possible values: static, ondemand, dynamic. This option is mandatory.

static - the number of child processes is fixed (pm.max_children).

ondemand - the processes spawn on demand (when requested, as opposed to dynamic, where pm.start_servers are started when the service is started.

dynamic - the number of child processes is set dynamically based on the following directives: pm.max_children, pm.start_servers,pm.min_spare_servers, pm.max_spare_servers.

pm.max_children int
The number of child processes to be created when pm is set to static and the maximum number of child processes to be created when pm is set to dynamic. This option is mandatory.

This option sets the limit on the number of simultaneous requests that will be served. Equivalent to the ApacheMaxClients directive with mpm_prefork and to the PHP_FCGI_CHILDREN environment variable in the original PHP FastCGI.

pm.start_servers int
The number of child processes created on startup. Used only when pm is set to dynamic. Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2.

pm.min_spare_servers int
The desired minimum number of idle server processes. Used only when pm is set to dynamic. Also mandatory in this case.

pm.max_spare_servers int
The desired maximum number of idle server processes. Used only when pm is set to dynamic. Also mandatory in this case.

pm.process_idle_timeout mixed
The number of seconds after which an idle process will be killed. Used only when pm is set to ondemand. Available units: s(econds)(default), m(inutes), h(ours), or d(ays). Default value: 10s.

pm.max_requests int
The number of requests each child process should execute before respawning. This can be useful to work around memory leaks in 3rd party libraries. For endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS. Default value: 0.
Click to expand...

php-fpm statistics status page

From official centminmod.com site link above for php-fpm status output enabling https://centminmod.com/phpfpm.html#browserstatus. You can also monitor php-fpm status via setting up 3rd party PHP-FPM status metric monitoring from services like:

Nixststats PHP-FPM monitoring Monitor PHP-FPM with Nixstats

Nginx Amplify PHP-FPM monitoring Monitoring PHP Applications with NGINX Amplify
Code (Text):
lynx --dump  http://127.0.0.1/phpstatus
pool:                 www
process manager:      static
start time:           13/May/2019:09:42:33 +0000
start since:          306419
accepted conn:        325
listen queue:         0
max listen queue:     0
listen queue len:     511
idle processes:       29
active processes:     1
total processes:      30
max active processes: 12
max children reached: 0
slow requests:        0
Centmin Mod 123.09beta01 on CentOS 7 also has command shortcut for php-fpm statistics output = fpmstats. Available due to the work I did here.
Code (Text):
fpmstats
Processes active: 0, idle: 30, Requests: 325, slow: 0, Traffic: 0req/sec
pool:                 www
process manager:      static
start time:           13/May/2019:09:42:33 +0000
start since:          306740
accepted conn:        327
listen queue:         0
max listen queue:     0
listen queue len:     511
idle processes:       29
active processes:     1
total processes:      30
max active processes: 12
max children reached: 0
slow requests:        0
PHP Status Explained:

pool - the name of the pool that is listening on the connected socket, as defined in the php-fpm config.

process manager - the method used by the process manager to control the number of child processes - either ondemand, dynamic or static - set on a per pool basis (in the php-fpm config) by the pm parameter.

start time - the date, time, and UTC offset corresponding to when the PHP-FPM server was started.

start since - the number of seconds that have elapsed since the PHP-FPM server was started (i.e. uptime).

accepted conn - the number of incoming requests that the PHP-FPM server has accepted; when a connection is accepted it is removed from the listen queue (displayed in real time).

listen queue - the current number of connections that have been initiated, but not yet accepted. If this value is non-zero it typically means that all the available server processes are currently busy, and there are no processes available to serve the next request. Raising pm.max_children (provided the server can handle it) should help keep this number low. This property follows from the fact that PHP-FPM listens via a socket (TCP or file based), and thus inherits some of the characteristics of sockets.

max listen queue - the maximum value the listen queue has reached since the server was started.

listen queue len - the upper limit on the number of connections that will be queued Once this limit is reached, subsequent connections will either be refused, or ignored. This value is set by the php-fpm per pool configuration option 'listen.backlog', which defaults to -1 (unlimited). However, this value is also limited by the system (sysctl) value 'net.core.somaxconn', which defaults to 128 on many Linux systems.

idle processes - the number of servers in the 'waiting to process' state (i.e. not currently serving a page). This value should fall between the pm.min_spare_servers and pm.max_spare_servers values when the process manager is dynamic. (updated once per second)

active processes - the number of servers current processing a page - the minimum is 1 (so even on a fully idle server, the result will be not read 0). (updated once per second)

total processes - the total number of server processes currently running; the sum of idle processes + active processes. If the process manager is static, this number will match pm.max_children. (updated once per second)

max active processes - the highest value that 'active processes' has reached since the php-fpm server started. This value should not exceed pm.max_children.

max children reached - the number of times that pm.max_children has been reached since the php-fpm server started (only applicable if the process manager is ondemand or dynamic)

Click to expand...

Centmin Mod 123.09beta01 CentOS 7 Unique PHP-FPM Statistics Info

FYI 123.09beta01 on CentOS 7 only that have been recently updated all switch to systemd php-fpm statistics supported service management so have additional statistics too. Notice the extra line in status output available only to CentOS 7 based 123.09beta01 and newer Centmin Mod installs.
Code (Text):
   Status: "Processes active: 0, idle: 30, Requests: 325, slow: 0, Traffic: 0req/sec"
full output
Code (Text):
systemctl status php-fpm
● php-fpm.service - PHP FastCGI Process Manager
   Loaded: loaded (/etc/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/php-fpm.service.d
           └─50-CPUAccounting.conf, 50-CPUShares.conf, limit.conf
   Active: active (running) since Mon 2019-05-13 09:42:34 UTC; 3 days ago
 Main PID: 6821 (php-fpm)
   Status: "Processes active: 0, idle: 30, Requests: 325, slow: 0, Traffic: 0req/sec"
    Tasks: 32
   Memory: 275.2M
   CGroup: /system.slice/php-fpm.service
           ├─6821 php-fpm: master process (/usr/local/etc/php-fpm.conf)
           ├─6822 php-fpm: pool www
           ├─6823 php-fpm: pool www
           ├─6824 php-fpm: pool www
           ├─6825 php-fpm: pool www
           ├─6826 php-fpm: pool www
           ├─6827 php-fpm: pool www
           ├─6828 php-fpm: pool www
           ├─6829 php-fpm: pool www
           ├─6830 php-fpm: pool www
           ├─6831 php-fpm: pool www
           ├─6832 php-fpm: pool www
           ├─6833 php-fpm: pool www
           ├─6834 php-fpm: pool www
           ├─6835 php-fpm: pool www
           ├─6836 php-fpm: pool www
           ├─6837 php-fpm: pool www
           ├─6838 php-fpm: pool www
           ├─6839 php-fpm: pool www
           ├─6840 php-fpm: pool www
           ├─6841 php-fpm: pool www
           ├─6842 php-fpm: pool www
           ├─6843 php-fpm: pool www
           ├─6844 php-fpm: pool www
           ├─6845 php-fpm: pool www
           ├─6846 php-fpm: pool www
           ├─6847 php-fpm: pool www
           ├─6848 php-fpm: pool www
           ├─6849 php-fpm: pool www
           ├─6850 php-fpm: pool www
           └─6851 php-fpm: pool www

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Centmin Mod 123.09beta01 on CentOS 7 also has command shortcut for php-fpm service status output = fpmstatus. Available due to the work I did here.
Code (Text):
fpmstatus
● php-fpm.service - PHP FastCGI Process Manager
   Loaded: loaded (/etc/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/php-fpm.service.d
           └─50-CPUAccounting.conf, 50-CPUShares.conf, limit.conf
   Active: active (running) since Mon 2019-05-13 09:42:34 UTC; 3 days ago
 Main PID: 6821 (php-fpm)
   Status: "Processes active: 0, idle: 30, Requests: 325, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php-fpm.service
           ├─6821 php-fpm: master process (/usr/local/etc/php-fpm.conf)
           ├─6822 php-fpm: pool www
           ├─6823 php-fpm: pool www
           ├─6824 php-fpm: pool www
           ├─6825 php-fpm: pool www
           ├─6826 php-fpm: pool www
           ├─6827 php-fpm: pool www
           ├─6828 php-fpm: pool www
           ├─6829 php-fpm: pool www
           ├─6830 php-fpm: pool www
           ├─6831 php-fpm: pool www
           ├─6832 php-fpm: pool www
           ├─6833 php-fpm: pool www
           ├─6834 php-fpm: pool www
           ├─6835 php-fpm: pool www
           ├─6836 php-fpm: pool www
           ├─6837 php-fpm: pool www
           ├─6838 php-fpm: pool www
           ├─6839 php-fpm: pool www
           ├─6840 php-fpm: pool www
           ├─6841 php-fpm: pool www
           ├─6842 php-fpm: pool www
           ├─6843 php-fpm: pool www
           ├─6844 php-fpm: pool www
           ├─6845 php-fpm: pool www
           ├─6846 php-fpm: pool www
           ├─6847 php-fpm: pool www
           ├─6848 php-fpm: pool www
           ├─6849 php-fpm: pool www
           ├─6850 php-fpm: pool www
           └─6851 php-fpm: pool www

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
But basic gist is PHP-FPM optimization is calculated from continuous monitoring of php-fpm statistics and resource usage and make informed decisions on php-fpm options based on understanding of those options and how they play out in php-fpm statistics status output/monitored metrics.

Log in / Register

Forums

Want to subscribe to topics you're interested in?

PHP-FPM How to troubleshoot & optimize PHP-FPM server?

eva2000 Administrator Staff Member

Nginx 502 or 504 Bad Gateway Errors

Checking PHP-FPM etc logs

How to edit php.ini and php-fpm configuration files ?

Troubleshooting Tools

Centmin Mod cminfo top command etc

eva2000 Administrator Staff Member

Additional Information

php-fpm.conf config options

php-fpm statistics status page

Centmin Mod 123.09beta01 CentOS 7 Unique PHP-FPM Statistics Info

.

Log in / Register

Useful Searches

Want to subscribe to topics you're interested in?

PHP-FPM How to troubleshoot & optimize PHP-FPM server?

eva2000 Administrator Staff Member

Nginx 502 or 504 Bad Gateway Errors

Checking PHP-FPM etc logs

How to edit php.ini and php-fpm configuration files ?

Troubleshooting Tools

Centmin Mod cminfo top command etc

eva2000 Administrator Staff Member

Additional Information

php-fpm.conf config options

php-fpm statistics status page

Centmin Mod 123.09beta01 CentOS 7 Unique PHP-FPM Statistics Info

.