PHP-FPM Random 500 server error (Magento)

Afaq · Mar 11, 2016

I am running 3 Magento website on a dedicated server with Centminmod and CentOS 7. Since a few days, I am getting random 500 server errors several times everyday. The error stays for 1-2 minutes and then the website starts working again.
I've looked into Magento's error logs but didnt find anything which could be related to this however upon looking at PHP-FPM logs I found some errors which might be related to this. Below are some lines for the log file. Any help will be highly appreciated.

Code:

[10-Mar-2016 15:57:49] WARNING: [pool www] child 31481 exited on signal 11 (SIGSEGV) after 149.897018 seconds from start
[10-Mar-2016 16:15:31] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 16:24:10] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 16:37:49] WARNING: [pool www] child 4510 exited on signal 11 (SIGSEGV) after 243.748941 seconds from start
[10-Mar-2016 17:02:49] WARNING: [pool www] child 7971 exited on signal 11 (SIGSEGV) after 149.235433 seconds from start
[10-Mar-2016 17:37:49] WARNING: [pool www] child 12610 exited on signal 11 (SIGSEGV) after 73.613595 seconds from start
[10-Mar-2016 18:07:49] WARNING: [pool www] child 16488 exited on signal 11 (SIGSEGV) after 124.660663 seconds from start
[10-Mar-2016 18:42:49] WARNING: [pool www] child 20975 exited on signal 11 (SIGSEGV) after 129.574268 seconds from start
[10-Mar-2016 18:49:31] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 19:10:57] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 19:37:49] WARNING: [pool www] child 28075 exited on signal 11 (SIGSEGV) after 161.653891 seconds from start
[10-Mar-2016 20:07:49] WARNING: [pool www] child 31979 exited on signal 11 (SIGSEGV) after 149.307583 seconds from start
[10-Mar-2016 20:27:50] WARNING: [pool www] child 2298 exited on signal 11 (SIGSEGV) after 139.921016 seconds from start
[10-Mar-2016 20:37:49] WARNING: [pool www] child 3612 exited on signal 11 (SIGSEGV) after 130.615810 seconds from start

eva2000 · Mar 11, 2016

What's output for these commands - post output wrapped in CODE tags
Code (Text):
php -v
Code (Text):
php -m
Code (Text):
php-config --configure-options
PHP is segfaulting and hitting it's limits with your specific php web application so you need to debug your php server. What are segfaults = Why Does The Segmentation Fault Occur on Linux / UNIX Systems?

Examples with strace below:

Debugging Stuck PHP-FPM Process With Strace - Coffee Coder

Why you should be using strace – Brandon Wamboldt

Linux application/script debugging with 'strace'

markus-perl/php-strace · GitHub

With gdb backtrace and PHP debug compiled mode where centmin.sh has a PHPDEBUGMODE variable which you can set to PHPDEBUGMODE=y and recompile php via centmin.sh menu option 5 to enable debug mode for PHP-FPM. After troubleshooting set PHPDEBUGMODE=n and recompile php via centmin.sh menu option 5 again to disable debug mode.
Code:
PHPDEBUGMODE=n # --enable-debug PHP compile flag
PHP :: Generating a gdb backtrace

Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.

Afaq · Mar 11, 2016

Thanks for the detailed reply.

I've run the commands you said. Here are their output in order:

Code:

PHP Warning:  Version warning: Imagick was compiled against Image Magick version 1682 but version 1683 is loaded. Imagick will run but may behave surprisingly in Unknown on line 0
PHP 5.4.45 (cli) (built: Dec 20 2015 09:03:36)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.5, Copyright (c) 1999-2015, by Zend Technologies

Code:

PHP Warning:  Version warning: Imagick was compiled against Image Magick version 1682 but version 1683 is loaded. Imagick will run but may behave surprisingly in Unknown on line 0
[PHP Modules]
bcmath
bz2
calendar
Core
ctype
curl
date
dom
enchant
ereg
exif
filter
ftp
gd
geoip
gettext
gmp
hash
iconv
igbinary
imagick
imap
intl
json
libxml
mbstring
mcrypt
memcache
memcached
mhash
mongo
mysql
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
pspell
readline
redis
Reflection
session
shmop
SimpleXML
snmp
soap
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

Code:

--enable-fpm --enable-intl --enable-pcntl --with-mcrypt --with-snmp --with-mhash --with-zlib --with-gettext --enable-exif --enable-zip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --with-gd --with-xmlrpc --enable-bcmath --enable-calendar --enable-ftp --enable-gd-native-ttf --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr --with-xpm-dir=/usr --with-vpx-dir=/usr --with-t1lib=/usr --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --disable-fileinfo --with-config-file-scan-dir=/etc/centminmod/php.d

eva2000 · Mar 11, 2016

For imagick message, just run centmin.sh menu option 15 to recompile imagick PHP extension. For segfault, could be as simple as PHP 5.4 being buggy as it's EOL and no longer updated for security or bug fixes. I'd try updating to PHP 5.5.33 or PHP 5.6.19 via centmin.sh menu option 5.

I'd update to latest version of Centmin Mod 123.08stable or 123.09beta01 prior to running centmin.sh menu option 5 to ensure you have the latest version of the Centmin Mod code and routines which update PHP via source compile.

Upgrading Centmin Mod Code to Latest Version

Getting Started Guide step 19 outlines also how to keep Centmin Mod code updated or how to switch version branches.

Centmin Mod LEMP stack's script code is constantly updated for improvements, bug fixes and security fixes so keeping the Centmin Mod code up to date is important. With Centmin Mod 1.2.3-eva2000.08) (123.08stable) and higher releases, a newly added centmin.sh menu option 23 allows much easier code updates and version branch swicthing via Git backed environment you can setup. For full details read the following links:

New centmin.sh menu option 23 updating

Centmin Mod Upgrades

Cronjob Auto Updating Centmin Mod Code Guide

Upgrading Centmin Mod involves 2 parts.

Upgrading the actual Centmin Mod code outlined at Upgrade Centmin Mod. This is heart of Centmin Mod where the code is the engine that runs centmin.sh shell based menu and all the automation you're accustomed to. You can easily update within a Centmin Mod version branch or switch version branches via centmin.sh menu option 23 outlined here.

Upgrade software that Centmin Mod installed or manages. For this part following outline at How to upgrade Centmin Mod software installed on your server.

So essentially, you can upgrade from .07 to .08 in place, but not everything is upgraded as some things like server initial environment setup isn't changed i.e. how swap, tmp setup and allocation are created etc. The main parts from part 2 above are what in place upgrades do i.e. Nginx and PHP-FPM compilation and config/settings parameters and MariaDB version from 5.5 to 10.0.x. If you want the full environment changed including tmp and swap setup to .08's configuration, then you would need a fresh OS install and fresh .08 initial install. You can think of it like upgrading Windows 7 to Windows 8. An in place upgrade will upgrade code but won't change your computer environment from when you installed Windows 7 i.e. disk configuration and partition sizes won't change from when you initially installed Windows 7. Only way to change that would be fresh Windows 8 install.

Afaq · Mar 11, 2016

I am already running the latest stable release of the centmin (ie 1.2.3-eva2000.08) so I didnt update that but I've not updated the Nginx from 1.9.9 to 1.9.12 and PHP from 5.4 to the 5.6.19. I will observe for a while to see the 500 errors are gone. I hope they are. Otherwise, I'll be debugging the php configurations as you mentioned above.

Edit: Ok so as soon as I opened one of the website after doing the update...500 error. Now they seem to be more frequent but go away after a refresh.

eva2000 · Mar 11, 2016

error messages in logs ?

What's output for these commands on updated PHP version - post output wrapped in CODE tags
Code (Text):
php -v
Code (Text):
php -m
Code (Text):
php-config --configure-options

Afaq · Mar 11, 2016

Here are a few of the recent errors. BTW these are from the www-error.log file, the www-php.error.log is empty.

Code:

[10-Mar-2016 21:58:14] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 22:02:49] WARNING: [pool www] child 14721 exited on signal 11 (SIGSEGV) after 160.784828 seconds from start
[10-Mar-2016 22:12:49] WARNING: [pool www] child 15892 exited on signal 11 (SIGSEGV) after 223.622404 seconds from start
[10-Mar-2016 22:17:49] WARNING: [pool www] child 16682 exited on signal 11 (SIGSEGV) after 125.344318 seconds from start
[10-Mar-2016 22:27:44] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 22:37:49] WARNING: [pool www] child 19382 exited on signal 11 (SIGSEGV) after 127.200666 seconds from start
[10-Mar-2016 22:57:49] WARNING: [pool www] child 22300 exited on signal 11 (SIGSEGV) after 88.513249 seconds from start
[10-Mar-2016 23:32:49] WARNING: [pool www] child 18980 exited on signal 11 (SIGSEGV) after 163.343618 seconds from start
[10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:36:27] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 23:36:41] WARNING: [pool www] server reached max_children setting (16), consider raising it
[10-Mar-2016 23:36:49] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:36:49] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
[10-Mar-2016 23:43:35] WARNING: [pool www] server reached max_children setting (16), consider raising it

And here is the output of the commands:

Code:

PHP 5.6.19 (cli) (built: Mar 10 2016 23:34:46)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Code:

[PHP Modules]
bcmath
bz2
calendar
Core
ctype
curl
date
dom
enchant
ereg
exif
filter
ftp
gd
geoip
gettext
gmp
hash
iconv
igbinary
imagick
imap
intl
json
libxml
mbstring
mcrypt
memcache
memcached
mhash
mongo
mysql
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
pspell
readline
redis
Reflection
session
shmop
SimpleXML
snmp
soap
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

Code:

--enable-fpm --enable-opcache --enable-intl --enable-pcntl --with-mcrypt --with-snmp --with-mhash --with-zlib --with-gettext --enable-exif --enable-zip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --with-gd --with-xmlrpc --enable-bcmath --enable-calendar --enable-ftp --enable-gd-native-ttf --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr --with-xpm-dir=/usr --with-vpx-dir=/usr --with-t1lib=/usr --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --disable-fileinfo --with-config-file-scan-dir=/etc/centminmod/php.d --with-xsl CC=ccache CXX=ccache

eva2000 · Mar 11, 2016

the above php debug/strace outlined options are still applicable. You can also do it tediously to check if there's a problematic php extension loaded that is causing segfaults crashes with your magento php code.

php --ini command outputs a list of php extension .ini settings files as well as custom php settings. For some of the php -m listed modules, you can temporarily disable the php extension by commenting out the zend_extension or extension load line with semi-colon and restart php-fpm. You can do it one by one until you don't see segfault (signal 11 errors in php error log). You can see an example of this when i disabled and unloaded newrelic-php extension which was causing segfaults for me on this forum at PHP-FPM - newrelic-php extension causing segfault errors in PHP-FPM | Centmin Mod Community

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

PHP-FPM Random 500 server error (Magento)

Afaq Member

eva2000 Administrator Staff Member

Afaq Member

eva2000 Administrator Staff Member

Upgrading Centmin Mod Code to Latest Version

Afaq Member

eva2000 Administrator Staff Member

Afaq Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

PHP-FPM Random 500 server error (Magento)

Afaq Member

eva2000 Administrator Staff Member

Afaq Member

eva2000 Administrator Staff Member

Upgrading Centmin Mod Code to Latest Version

Afaq Member

eva2000 Administrator Staff Member

Afaq Member

eva2000 Administrator Staff Member

.