Get the most out of your Centmin Mod LEMP stack
Become a Member

php-fpm? 502 Gateaway

Discussion in 'Bug Reports' started by Oxide, Aug 24, 2015.

Tags:
  1. Oxide

    Oxide Active Member

    502
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    9:22 PM
    People have been reporting random 502 Gateaway issues, also when I came online today I had to restart php-fpm before it worked, all my sites wouldn't load.

    How can I research this, and find out what caused it?

    www-error.log
    Code:
    [31-Jul-2015 03:41:47] WARNING: [pool www] server reached max_children setting (5), consider raising it
    [09-Aug-2015 05:26:44] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [09-Aug-2015 05:34:40] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [10-Aug-2015 03:42:34] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [10-Aug-2015 17:02:01] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [11-Aug-2015 03:25:24] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [11-Aug-2015 17:53:46] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [11-Aug-2015 17:55:46] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [12-Aug-2015 03:30:06] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [12-Aug-2015 04:04:12] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [13-Aug-2015 03:33:01] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [13-Aug-2015 03:33:39] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [13-Aug-2015 20:26:40] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [14-Aug-2015 03:51:53] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [15-Aug-2015 03:08:43] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [16-Aug-2015 03:46:31] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [16-Aug-2015 04:09:13] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [17-Aug-2015 04:03:34] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [17-Aug-2015 17:28:42] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [18-Aug-2015 04:17:34] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [19-Aug-2015 01:18:38] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [19-Aug-2015 02:30:08] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [19-Aug-2015 03:15:11] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [19-Aug-2015 03:42:07] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [20-Aug-2015 03:33:51] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [21-Aug-2015 03:26:59] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [22-Aug-2015 04:45:37] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [23-Aug-2015 06:46:52] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [24-Aug-2015 03:43:36] WARNING: [pool www] server reached max_children setting (4), consider raising it
    [24-Aug-2015 12:32:08] ERROR: unable to bind listening socket for address '127.0.0.1:9000': Address already in use (98)
    [24-Aug-2015 12:32:08] ERROR: FPM initialization failed
    [24-Aug-2015 12:32:22] WARNING: [pool www] server reached max_children setting (4), consider raising it
    Increased max children from 4 to 84, is that too much? i got told to run
    echo "pm.max_children = $(( $(awk '/MemTotal:/ { printf "%d\n", ($2*0.80) }' /proc/meminfo) / $(ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR) }') ))"
    to figure..

    returns 84
     
    Last edited: Aug 24, 2015
  2. eva2000

    eva2000 Administrator Staff Member

    28,971
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,759
    Local Time:
    9:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Nginx 502 or 504 Bad Gateway Errors



    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
    tail -10 /usr/local/nginx/logs/access.log
    tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
    tail -10 /home/nginx/domains/domainname.com/log/access.log
    tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
    ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root root 0 Aug 29 15:33 tallylog
    -rw------- 1 root root 0 Aug 29 15:33 spooler
    drwx------ 3 root root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root 500 0 Oct 8 18:13 dmesg.old
    -rw------- 1 root 500 0 Oct 8 18:13 boot.log
    -rw-r--r-- 1 root 500 0 Oct 8 18:14 dmesg
    drwx------ 2 root root 4.0K Oct 8 18:14 httpd
    drwxr-xr-x 2 root root 4.0K Oct 8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct 9 12:38 mysqld.log
    -rw------- 1 root root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root utmp 94K Nov 7 22:59 btmp
    drwxr-xr-x 2 root root 4.0K Nov 8 00:00 sa
    -rw------- 1 root root 269K Nov 8 21:39 messages
    -rw------- 1 root root 110K Nov 8 23:08 secure
    -rw-rw-r-- 1 root utmp 43K Nov 8 23:08 wtmp
    -rw-r--r-- 1 root root 144K Nov 8 23:08 lastlog
    -rw------- 1 root root 69K Nov 8 23:08 lfd.log
    -rw------- 1 root root 332K Nov 8 23:08 maillog
    -rw------- 1 root 500 1.6M Nov 8 23:10 cron
    
    For PHP-FPM error log:
    Code:
    tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
    /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
    tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
    tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
    tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
    tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  3. RoldanLT

    RoldanLT Well-Known Member

    3,829
    929
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,258
    Local Time:
    7:22 PM
    1.11
    10.2
    I hope your server doesn't suffer layer 7 DDOS attacks.
    Average Visitors based on Google Analytics?
    Server specs?
     
  4. Oxide

    Oxide Active Member

    502
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    9:22 PM
    are all your replies preset replies because they contain so much text and makes me feel not special
     
  5. eva2000

    eva2000 Administrator Staff Member

    28,971
    6,575
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,759
    Local Time:
    9:22 PM
    Nginx 1.13.x
    MariaDB 5.5
    some text is preset if it applies to the topic.. do you expect me to hand type all above everytime someone asks about php-fpm 502/504 errors ? :)