Learn about Centmin Mod LEMP Stack today
Register Now

PHP-FPM Random 500 server error (Magento)

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Afaq, Mar 11, 2016.

  1. Afaq

    Afaq Member

    46
    1
    8
    Aug 5, 2015
    Ratings:
    +2
    Local Time:
    8:04 AM
    I am running 3 Magento website on a dedicated server with Centminmod and CentOS 7. Since a few days, I am getting random 500 server errors several times everyday. The error stays for 1-2 minutes and then the website starts working again.
    I've looked into Magento's error logs but didnt find anything which could be related to this however upon looking at PHP-FPM logs I found some errors which might be related to this. Below are some lines for the log file. Any help will be highly appreciated.

    Code:
    [10-Mar-2016 15:57:49] WARNING: [pool www] child 31481 exited on signal 11 (SIGSEGV) after 149.897018 seconds from start
    [10-Mar-2016 16:15:31] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 16:24:10] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 16:37:49] WARNING: [pool www] child 4510 exited on signal 11 (SIGSEGV) after 243.748941 seconds from start
    [10-Mar-2016 17:02:49] WARNING: [pool www] child 7971 exited on signal 11 (SIGSEGV) after 149.235433 seconds from start
    [10-Mar-2016 17:37:49] WARNING: [pool www] child 12610 exited on signal 11 (SIGSEGV) after 73.613595 seconds from start
    [10-Mar-2016 18:07:49] WARNING: [pool www] child 16488 exited on signal 11 (SIGSEGV) after 124.660663 seconds from start
    [10-Mar-2016 18:42:49] WARNING: [pool www] child 20975 exited on signal 11 (SIGSEGV) after 129.574268 seconds from start
    [10-Mar-2016 18:49:31] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 19:10:57] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 19:37:49] WARNING: [pool www] child 28075 exited on signal 11 (SIGSEGV) after 161.653891 seconds from start
    [10-Mar-2016 20:07:49] WARNING: [pool www] child 31979 exited on signal 11 (SIGSEGV) after 149.307583 seconds from start
    [10-Mar-2016 20:27:50] WARNING: [pool www] child 2298 exited on signal 11 (SIGSEGV) after 139.921016 seconds from start
    [10-Mar-2016 20:37:49] WARNING: [pool www] child 3612 exited on signal 11 (SIGSEGV) after 130.615810 seconds from start
     
  2. eva2000

    eva2000 Administrator Staff Member

    30,166
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    1:04 PM
    Nginx 1.13.x
    MariaDB 5.5
    What's output for these commands - post output wrapped in CODE tags
    Code (Text):
    php -v

    Code (Text):
    php -m

    Code (Text):
    php-config --configure-options


    PHP is segfaulting and hitting it's limits with your specific php web application so you need to debug your php server. What are segfaults = Why Does The Segmentation Fault Occur on Linux / UNIX Systems?

    Examples with strace below:
    With gdb backtrace and PHP debug compiled mode where centmin.sh has a PHPDEBUGMODE variable which you can set to PHPDEBUGMODE=y and recompile php via centmin.sh menu option 5 to enable debug mode for PHP-FPM. After troubleshooting set PHPDEBUGMODE=n and recompile php via centmin.sh menu option 5 again to disable debug mode.
    Code:
    PHPDEBUGMODE=n # --enable-debug PHP compile flag
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.
     
  3. Afaq

    Afaq Member

    46
    1
    8
    Aug 5, 2015
    Ratings:
    +2
    Local Time:
    8:04 AM
    Thanks for the detailed reply.

    I've run the commands you said. Here are their output in order:
    Code:
    PHP Warning:  Version warning: Imagick was compiled against Image Magick version 1682 but version 1683 is loaded. Imagick will run but may behave surprisingly in Unknown on line 0
    PHP 5.4.45 (cli) (built: Dec 20 2015 09:03:36)
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
        with Zend OPcache v7.0.5, Copyright (c) 1999-2015, by Zend Technologies
    Code:
    PHP Warning:  Version warning: Imagick was compiled against Image Magick version 1682 but version 1683 is loaded. Imagick will run but may behave surprisingly in Unknown on line 0
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    libxml
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mongo
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    Code:
    --enable-fpm --enable-intl --enable-pcntl --with-mcrypt --with-snmp --with-mhash --with-zlib --with-gettext --enable-exif --enable-zip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --with-gd --with-xmlrpc --enable-bcmath --enable-calendar --enable-ftp --enable-gd-native-ttf --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr --with-xpm-dir=/usr --with-vpx-dir=/usr --with-t1lib=/usr --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --disable-fileinfo --with-config-file-scan-dir=/etc/centminmod/php.d
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,166
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    1:04 PM
    Nginx 1.13.x
    MariaDB 5.5
    For imagick message, just run centmin.sh menu option 15 to recompile imagick PHP extension. For segfault, could be as simple as PHP 5.4 being buggy as it's EOL and no longer updated for security or bug fixes. I'd try updating to PHP 5.5.33 or PHP 5.6.19 via centmin.sh menu option 5.

    I'd update to latest version of Centmin Mod 123.08stable or 123.09beta01 prior to running centmin.sh menu option 5 to ensure you have the latest version of the Centmin Mod code and routines which update PHP via source compile.

    Upgrading Centmin Mod Code to Latest Version



    Getting Started Guide step 19 outlines also how to keep Centmin Mod code updated or how to switch version branches.

    Centmin Mod LEMP stack's script code is constantly updated for improvements, bug fixes and security fixes so keeping the Centmin Mod code up to date is important. With Centmin Mod 1.2.3-eva2000.08) (123.08stable) and higher releases, a newly added centmin.sh menu option 23 allows much easier code updates and version branch swicthing via Git backed environment you can setup. For full details read the following links:
    Upgrading Centmin Mod involves 2 parts.
    1. Upgrading the actual Centmin Mod code outlined at Upgrade Centmin Mod. This is heart of Centmin Mod where the code is the engine that runs centmin.sh shell based menu and all the automation you're accustomed to. You can easily update within a Centmin Mod version branch or switch version branches via centmin.sh menu option 23 outlined here.
    2. Upgrade software that Centmin Mod installed or manages. For this part following outline at How to upgrade Centmin Mod software installed on your server.
    So essentially, you can upgrade from .07 to .08 in place, but not everything is upgraded as some things like server initial environment setup isn't changed i.e. how swap, tmp setup and allocation are created etc. The main parts from part 2 above are what in place upgrades do i.e. Nginx and PHP-FPM compilation and config/settings parameters and MariaDB version from 5.5 to 10.0.x. If you want the full environment changed including tmp and swap setup to .08's configuration, then you would need a fresh OS install and fresh .08 initial install. You can think of it like upgrading Windows 7 to Windows 8. An in place upgrade will upgrade code but won't change your computer environment from when you installed Windows 7 i.e. disk configuration and partition sizes won't change from when you initially installed Windows 7. Only way to change that would be fresh Windows 8 install.
     
  5. Afaq

    Afaq Member

    46
    1
    8
    Aug 5, 2015
    Ratings:
    +2
    Local Time:
    8:04 AM
    I am already running the latest stable release of the centmin (ie 1.2.3-eva2000.08) so I didnt update that but I've not updated the Nginx from 1.9.9 to 1.9.12 and PHP from 5.4 to the 5.6.19. I will observe for a while to see the 500 errors are gone. I hope they are. Otherwise, I'll be debugging the php configurations as you mentioned above.

    Edit: Ok so as soon as I opened one of the website after doing the update...500 error. Now they seem to be more frequent but go away after a refresh.
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,166
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    1:04 PM
    Nginx 1.13.x
    MariaDB 5.5
    error messages in logs ?

    What's output for these commands on updated PHP version - post output wrapped in CODE tags
    Code (Text):
    php -v

    Code (Text):
    php -m

    Code (Text):
    php-config --configure-options
     
  7. Afaq

    Afaq Member

    46
    1
    8
    Aug 5, 2015
    Ratings:
    +2
    Local Time:
    8:04 AM
    Here are a few of the recent errors. BTW these are from the www-error.log file, the www-php.error.log is empty.
    Code:
    [10-Mar-2016 21:58:14] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 22:02:49] WARNING: [pool www] child 14721 exited on signal 11 (SIGSEGV) after 160.784828 seconds from start
    [10-Mar-2016 22:12:49] WARNING: [pool www] child 15892 exited on signal 11 (SIGSEGV) after 223.622404 seconds from start
    [10-Mar-2016 22:17:49] WARNING: [pool www] child 16682 exited on signal 11 (SIGSEGV) after 125.344318 seconds from start
    [10-Mar-2016 22:27:44] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 22:37:49] WARNING: [pool www] child 19382 exited on signal 11 (SIGSEGV) after 127.200666 seconds from start
    [10-Mar-2016 22:57:49] WARNING: [pool www] child 22300 exited on signal 11 (SIGSEGV) after 88.513249 seconds from start
    [10-Mar-2016 23:32:49] WARNING: [pool www] child 18980 exited on signal 11 (SIGSEGV) after 163.343618 seconds from start
    [10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:35:58] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:36:00] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/redis.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:36:27] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 23:36:41] WARNING: [pool www] server reached max_children setting (16), consider raising it
    [10-Mar-2016 23:36:49] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:36:49] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mongo.so: cannot open shared object file: No such file or directory in Unknown on line 0
    [10-Mar-2016 23:43:35] WARNING: [pool www] server reached max_children setting (16), consider raising it
    And here is the output of the commands:

    Code:
    PHP 5.6.19 (cli) (built: Mar 10 2016 23:34:46)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    
    Code:
    [PHP Modules]
    bcmath
    bz2
    calendar
    Core
    ctype
    curl
    date
    dom
    enchant
    ereg
    exif
    filter
    ftp
    gd
    geoip
    gettext
    gmp
    hash
    iconv
    igbinary
    imagick
    imap
    intl
    json
    libxml
    mbstring
    mcrypt
    memcache
    memcached
    mhash
    mongo
    mysql
    mysqli
    mysqlnd
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    Phar
    posix
    pspell
    readline
    redis
    Reflection
    session
    shmop
    SimpleXML
    snmp
    soap
    sockets
    SPL
    sqlite3
    standard
    sysvmsg
    sysvsem
    sysvshm
    tidy
    tokenizer
    xml
    xmlreader
    xmlrpc
    xmlwriter
    xsl
    Zend OPcache
    zip
    zlib
    
    [Zend Modules]
    Zend OPcache
    
    
    Code:
    --enable-fpm --enable-opcache --enable-intl --enable-pcntl --with-mcrypt --with-snmp --with-mhash --with-zlib --with-gettext --enable-exif --enable-zip --with-bz2 --enable-soap --enable-sockets --enable-sysvmsg --enable-sysvsem --enable-sysvshm --enable-shmop --with-pear --enable-mbstring --with-openssl --with-mysql=mysqlnd --with-libdir=lib64 --with-mysqli=mysqlnd --with-mysql-sock=/var/lib/mysql/mysql.sock --with-curl --with-gd --with-xmlrpc --enable-bcmath --enable-calendar --enable-ftp --enable-gd-native-ttf --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr --with-xpm-dir=/usr --with-vpx-dir=/usr --with-t1lib=/usr --enable-pdo --with-pdo-sqlite --with-pdo-mysql=mysqlnd --enable-inline-optimization --with-imap --with-imap-ssl --with-kerberos --with-readline --with-libedit --with-gmp --with-pspell --with-tidy --with-enchant --with-fpm-user=nginx --with-fpm-group=nginx --disable-fileinfo --with-config-file-scan-dir=/etc/centminmod/php.d --with-xsl CC=ccache CXX=ccache
    
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,166
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    1:04 PM
    Nginx 1.13.x
    MariaDB 5.5
    the above php debug/strace outlined options are still applicable. You can also do it tediously to check if there's a problematic php extension loaded that is causing segfaults crashes with your magento php code.

    php --ini command outputs a list of php extension .ini settings files as well as custom php settings. For some of the php -m listed modules, you can temporarily disable the php extension by commenting out the zend_extension or extension load line with semi-colon and restart php-fpm. You can do it one by one until you don't see segfault (signal 11 errors in php error log). You can see an example of this when i disabled and unloaded newrelic-php extension which was causing segfaults for me on this forum at PHP-FPM - newrelic-php extension causing segfault errors in PHP-FPM | Centmin Mod Community