Welcome to Centmin Mod Community
Register Now

Nginx PHP-FPM Nginx keeps crashing?

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by joshuah, Jun 2, 2017.

  1. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Hello,

    As of this evening, my server keeps crashing out (I keep getting bad gateway 502 errors for NGINX) and also "refused to connect" error messages.

    Any idea where I can start looking as to why it's crashing?
     
  2. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    Nginx 502 or 504 Bad Gateway Errors



    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
      tail -10 /usr/local/nginx/logs/access.log
      tail -10 /usr/local/nginx/logs/error.log
    
    For specific domainname.com access and error log:
    Code:
      tail -10 /home/nginx/domains/domainname.com/log/access.log
      tail -10 /home/nginx/domains/domainname.com/log/error.log
    
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
      ls -lhrt /var/log
    
    Code:
    total 2.7M
    -rw------- 1 root  root    0 Aug 29 15:33 tallylog
    -rw------- 1 root  root    0 Aug 29 15:33 spooler
    drwx------ 3 root  root 4.0K Aug 29 15:35 samba
    drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
    -rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
    -rw------- 1 root  500     0 Oct  8 18:13 boot.log
    -rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
    drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
    drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
    -rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
    -rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
    -rw------- 1 root  utmp  94K Nov  7 22:59 btmp
    drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
    -rw------- 1 root  root 269K Nov  8 21:39 messages
    -rw------- 1 root  root 110K Nov  8 23:08 secure
    -rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
    -rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
    -rw------- 1 root  root  69K Nov  8 23:08 lfd.log
    -rw------- 1 root  root 332K Nov  8 23:08 maillog
    -rw------- 1 root  500  1.6M Nov  8 23:10 cron
    
    For PHP-FPM error log:
    Code:
      tail -10 /var/log/php-fpm/www-error.log
    
    and/or
    Code:
      /var/log/php-fpm/www-php.error.log
    
    For MySQL / MariaDB error log:
    Code:
      tail -10 /var/log/mysqld.log
    
    For CSF firewall LFD log:
    Code:
      tail -10 /var/log/lfd.log
    
    For Mail log:
    Code:
      tail -10 /var/log/maillog
    
    For Cron job logs:
    Code:
      tail -10 /var/log/cron
    

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
  3. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Thanks for the reply.

    This is interesting:

    Code:
    tailf /var/log/php-fpm/www-error.log-20170602
    [01-Jun-2017 08:04:35] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [01-Jun-2017 08:04:35] ERROR: FPM initialization failed
    [01-Jun-2017 16:04:35] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [01-Jun-2017 16:04:35] ERROR: FPM initialization failed
    [02-Jun-2017 00:02:04] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [02-Jun-2017 00:02:04] ERROR: FPM initialization failed
    [02-Jun-2017 00:04:37] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [02-Jun-2017 00:04:37] ERROR: FPM initialization failed
     
  4. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    That unix socket is for phpmyadmin.sh addon installed phpmyadmin install with it's own separate php-fpm pool. It would happen when you try to start php-fpm service when there is already a running php-fpm service so might not be related to your 502 bad gateway errors

    example running php-fpm service
    Code (Text):
    service php-fpm status    
    php-fpm (pid 13547) is running...
    

    Code (Text):
    ps aufxw | grep php-fpm | grep -v grep
    root     13547  0.0  0.4 534204  9092 ?        Ss   04:56   0:00 php-fpm: master process (/usr/local/etc/php-fpm.conf)
    

    now try starting php-fpm service when it's already running
    Code (Text):
    service php-fpm start  
    Starting php-fpm [02-Jun-2017 04:59:24] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [02-Jun-2017 04:59:24] ERROR: FPM initialization failed
     failed
    

    proper restart
    Code (Text):
    service php-fpm restart
    Gracefully shutting down php-fpm . done
    Starting php-fpm  done
    

    or proper stop and start
    Code (Text):
    service php-fpm stop
    Gracefully shutting down php-fpm . done
    
    service php-fpm start
    Starting php-fpm  done
    

    Code (Text):
    tail -100 /var/log/php-fpm/www-error.log
    
    [02-Jun-2017 04:56:37] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [02-Jun-2017 04:56:37] ERROR: FPM initialization failed
    [02-Jun-2017 04:59:24] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [02-Jun-2017 04:59:24] ERROR: FPM initialization failed
    
     
  5. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Sorry to bring up this old thread.

    I had two outages today which required me to manually restart nginx to get it back up and running.

    It was interesting because when I look at the

    Code:
    tailf /var/log/php-fpm/www-error.log*
    [29-Jun-2017 08:06:50] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [29-Jun-2017 08:06:50] ERROR: FPM initialization failed
    [29-Jun-2017 16:01:59] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [29-Jun-2017 16:01:59] ERROR: FPM initialization failed
    [29-Jun-2017 16:04:41] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [29-Jun-2017 16:04:41] ERROR: FPM initialization failed
    [29-Jun-2017 16:05:38] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [29-Jun-2017 16:05:38] ERROR: FPM initialization failed
    [29-Jun-2017 16:06:50] ERROR: An another FPM instance seems to already listen on /tmp/phpfpm_myadmin.sock
    [29-Jun-2017 16:06:50] ERROR: FPM initialization failed
    
    Both of these are exactly the time when I had outages....
     
  6. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    Timestamps look the same as previously posted ones so any cronjobs running around that time? Check other logs around that time too

    what's output for cronjob listing command
    Code (Text):
    crontab -l
    

    could be wordpress auto installer's auto update cronjob random sleep formula at play for centmin.sh menu option 22 based wordpress installs where you have more than one wordpress install and the wp auto update cronjob's sleep random 3 digit seconds is very close together. If it is, then you can just change the sleep interval to not be so close together. Or change the cronjob schdule completely.

    also added fix for 123.09beta01 for new centmin.sh menu option 22 installed wordpress instances Beta Branch - update inc/wpsetup.inc cronjob sleep time fix
     
    Last edited: Jun 30, 2017
  7. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Interesting.. do you mind if I were to private message you the crontab? There is quite a few entries in there that i dont want published publically.
     
  8. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah can pm me or to check just wp updater related, just filter for sleep entries using command
    Code (Text):
    crontab -l | grep wp_updater
    

    i.e.
    Code (Text):
    crontab -l | grep wp_updater
    0 */8 * * * sleep 268s ;/root/tools/wp_updater_domain.com.sh >/dev/null 2>&1
    
     
  9. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    for privacy, you can post results for command
    Code (Text):
    crontab -l | grep wp_updater | awk '{print $1,$2,$3,$4,$5,$6,$7}'
    

    which only prints first 7 fields up to sleep XXXs
     
  10. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Hey mate,

    Code:
    crontab -l | grep wp_updater | awk '{print $1,$2,$3,$4,$5,$6,$7}'
    0 */8 * * * sleep 3202s
    0 */8 * * * sleep 109s
    0 */8 * * * sleep 316s
    0 */8 * * * sleep 204s
    0 */8 * * * sleep 728s
    0 */8 * * * sleep 220s
    0 */8 * * * sleep 330s
    0 */8 * * * sleep 398s
    0 */8 * * * sleep 365s
    0 */8 * * * sleep 269s
    0 */8 * * * sleep 272s
    0 */8 * * * sleep 167s
    0 */8 * * * sleep 108s
    0 */8 * * * sleep 326s
    0 */8 * * * sleep 301s
    0 */8 * * * sleep 401s
    0 */8 * * * sleep 145s
    
     
  11. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Also, here is a screenshot of my outage report from my monitoring (1 minute checks).

    Two of these outages I required to manually run "nprestart" hence the 15minute and 20 minute outage.. otherwise, it automatically restarted by itself.

    [​IMG]

    Generally speaking, it is just a "connection refused" error.
     
  12. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    That's ALOT of wordpress installs there ! Can see why there'd be overlap for wordpress auto updating ! The fix I made, checks that the first 2 characters for each XXXs sleep time so make sure the inserted new cronjob doesn't match any of existing cronjob's first 2 characters. So sleep 326s and sleep 3202s or 108s and 109s won't be possible on updated installs as first 2 characters of 32 and 10 would match.

    Guess I might need to randomise the /8 (every 8hrs) too. For now manual fix would be just to manually change the cronjob schedules so they don't overlap.
     
  13. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    So ideally, just change the /8 to make it unique?

    Edit: also, just confirming, this script will basically just update the wordpress core to the latest version? In theory, this could be disabled all together, right? As long as I manually update the plugins and core?
     
  14. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    yes make it unique but changing /8 might not be best as /7 means every 7hrs, /4 means every 4 hrs which would make matters worse as it would be too frequent.

    For coding, need to come up with a better way as clearly this wouldn't scale for dozens of wordpress installs heh
     
  15. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    What's wrong with running it every 8, 10, 12, 14, 16, 18, 20, 22 (each new crontab entry it will increase by 2 hours?)
     
  16. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    ah increasing the value should be fine though it would mean some wp instances don't get updated every 24hrs i.e. /30 would = every 30 hrs
     
  17. joshuah

    joshuah Member

    115
    14
    18
    Apr 3, 2017
    Ratings:
    +16
    Local Time:
    11:57 PM
    Yes, this is true... however, it's going to be rare that someone runs this number of websites, and will suffice for someone who runs 1-5 websites per install.

    Just confirming, would there be any benefit from running the wordpress updater if I manually update all plugins and core myself manually? Looks like at the end of each update, it will do a nginx restart... so may benefit me all together to disable the update script and do it myself manually?
     
  18. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    Actually incrementing still has chance of overlap i.e. /6 and /8 so every 6 and 8 hrs would overlap at 24th hour mark

    It's nginx + php-fpm restart using nprestart command to clear wp caching and zend opcache caching after wp updates.

    though for you you could incremant by 1hr each entry - this command would print out the new cronjob listing for wp_updater related commands
    Code (Text):
    crontab -l | awk '/wp_updater/' | xargs -n10 | while read m hr a b c d e f g h; do i=$(($i+1)) ;HR=$(echo $hr| sed -e 's|\*\/||'); HRR=$(($HR+$i)); echo "$m "*/"$HRR $a $b $c $d $e $f $g $h"; done
    

    for folks looking on, public snipped for privacy version would be like
    Code (Text):
    0 */9 * * * sleep 3202s
    0 */10 * * * sleep 109s
    0 */11 * * * sleep 316s
    0 */12 * * * sleep 204s
    0 */13 * * * sleep 728s
    0 */14 * * * sleep 220s
    0 */15 * * * sleep 330s
    0 */16 * * * sleep 398s
    0 */17 * * * sleep 365s
    0 */18 * * * sleep 269s
    0 */19 * * * sleep 272s
    0 */20 * * * sleep 167s
    0 */21 * * * sleep 108s
    0 */22 * * * sleep 326s
    0 */23 * * * sleep 301s
    0 */24 * * * sleep 401s
    0 */25 * * * sleep 145s
    

    of course the command you run would have full cronjob entries included

    that should suffice for your situation, though need to come up with better way in 123.09beta01
     
  19. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    instead of manually make changes, you can use some SSH commands to do it by backing up existing crontab list to file named cronjoblist, the using cronjoblist to build the new cronjobs in new-cronjoblist file then using sed to remove all lines containing the word wp_updater. Then concatenate the new-cronjoblist generated lines into cronjoblist backup file and then setup as cronjob overwriting existing cronjob list
    Code (Text):
    crontab -l > cronjoblist
    touch new-cronjoblist
    cat cronjoblist | awk '/wp_updater/' | xargs -n10 | while read m hr a b c d e f g h; do i=$(($i+1)) ;HR=$(echo $hr| sed -e 's|\*\/||'); HRR=$(($HR+$i)); echo "$m "*/"$HRR $a $b $c $d $e $f $g $h"; done >> new-cronjoblist
    sed -i '/wp_updater/d' cronjoblist
    cat new-cronjoblist >> cronjoblist
    crontab cronjoblist
    rm -rf cronjoblist
    crontab -l
    

    might look into revising wp_updater to use flock
    or something similar
     
    Last edited: Jun 30, 2017
  20. eva2000

    eva2000 Administrator Staff Member

    29,024
    6,587
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,778
    Local Time:
    11:57 PM
    Nginx 1.13.x
    MariaDB 5.5
    Thinking of using flock for wp_updater's nprestart command. Did a quick experimental test creating 3 scripts restartphp1.sh, restartphp2.sh and restartphp3.sh with following code to just run nprestart command. If the locked file is still locked, report already detected running process
    Code (Text):
    #!/bin/bash
    {
    exec 99>/tmp/wp_updater_phpfpm.lock
    if flock -n -x 99; then
    /usr/bin/nprestart
    rm -rf /tmp/wp_updater_phpfpm.lock
    else
    echo "already detected running process"
    fi
    echo
    echo "---"
    echo done
    echo "---"
    } 2>&1 | tee "$0.log"
    

    setup the 3 files to run at exact same scheduled 18:02 times
    Code (Text):
    02 18 * * * /root/tools/cronjob/restartphp1.sh >/dev/null 2>&1
    02 18 * * * /root/tools/cronjob/restartphp2.sh >/dev/null 2>&1
    02 18 * * * /root/tools/cronjob/restartphp3.sh >/dev/null 2>&1
    

    so they ran at same time
    Code (Text):
    tail -l /var/log/cron | grep restart
    Jun 29 18:02:01 centos7 CROND[9580]: (root) CMD (/root/tools/cronjob/restartphp2.sh >/dev/null 2>&1)
    Jun 29 18:02:01 centos7 CROND[9581]: (root) CMD (/root/tools/cronjob/restartphp1.sh >/dev/null 2>&1)
    Jun 29 18:02:01 centos7 CROND[9582]: (root) CMD (/root/tools/cronjob/restartphp3.sh >/dev/null 2>&1)
    

    Each script logged to a file named same as script
    Code (Text):
    cat restartphp2.sh.log 
    Restarting nginx (via systemctl):  [  OK  ]
    Gracefully shutting down php-fpm . done
    Starting php-fpm  done
    
    ---
    done
    ---
    

    Code (Text):
    cat restartphp1.sh.log             
    already detected running process
    
    ---
    done
    ---
    

    Code (Text):
     cat restartphp3.sh.log 
    already detected running process
    
    ---
    done
    ---
    

    Double check /var/log/php-fpm/www-error.log php log file and it's clear of errors for Address already in use / unable to bind listening socket for address messages.

    So essentially nprestart command was skipped for restartphp1.sh and restartphp3.sh as restartphp2.sh had exclusive lock on /tmp/wp_updater_phpfpm.lock. Guess could of just used normal lockfile too.