Nginx 502 Bad Gateway

Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

Nginx 502 or 504 Bad Gateway Errors

Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at

• http://centminmod.com/phpfpm.html#browserstatus and
• https://community.centminmod.com/threads/pm-max_children.479/ and
• https://community.centminmod.com/threads/warning-pool-www-server-reached-max_children-setting-50-consider-raising-it.791/ which outlines the official PHP-FPM config documentation as well.

Checking PHP-FPM etc logs

You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

In SSH2 telnet you can use tail command to view the last X number of lines in the file.

For example for viewing last 10 lines in the file for:

For Nginx access and error logs:

Code:

  tail -10 /usr/local/nginx/logs/access.log
  tail -10 /usr/local/nginx/logs/error.log

For specific domainname.com access and error log:

Code:

  tail -10 /home/nginx/domains/domainname.com/log/access.log
  tail -10 /home/nginx/domains/domainname.com/log/error.log

For other system error logs located at /var/log:

list /var/log files in ascending time order so the most recently modified files are at the bottom

Code:

  ls -lhrt /var/log

Code:

total 2.7M
-rw------- 1 root  root    0 Aug 29 15:33 tallylog
-rw------- 1 root  root    0 Aug 29 15:33 spooler
drwx------ 3 root  root 4.0K Aug 29 15:35 samba
drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
-rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
-rw------- 1 root  500     0 Oct  8 18:13 boot.log
-rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
-rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
-rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
-rw------- 1 root  utmp  94K Nov  7 22:59 btmp
drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
-rw------- 1 root  root 269K Nov  8 21:39 messages
-rw------- 1 root  root 110K Nov  8 23:08 secure
-rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
-rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
-rw------- 1 root  root  69K Nov  8 23:08 lfd.log
-rw------- 1 root  root 332K Nov  8 23:08 maillog
-rw------- 1 root  500  1.6M Nov  8 23:10 cron

For PHP-FPM error log:

Code:

  tail -10 /var/log/php-fpm/www-error.log

and/or

Code:

  /var/log/php-fpm/www-php.error.log

For MySQL / MariaDB error log:

Code:

  tail -10 /var/log/mysqld.log

For CSF firewall LFD log:

Code:

  tail -10 /var/log/lfd.log

For Mail log:

Code:

  tail -10 /var/log/maillog

For Cron job logs:

Code:

  tail -10 /var/log/cron

How to edit php.ini and php-fpm configuration files ?

Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:

• Edit php.ini = phpedit ( /usr/local/lib/php.ini )
• Edit my.cnf = mycnf ( /etc/my.cnf )
• Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
• Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
• Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
• Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
• Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
• Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
• nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
• php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
• mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
• nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
• memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
• csf stop/start/restart = csfstop/csfstart/csfrestart

Troubleshooting Tools

However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.

• 20 Command Line Tools to Monitor Linux Performance
• 13 Linux Performance Monitoring Tools - Part 2
• 80 Linux Monitoring Tools for SysAdmins - Server Density Blog
• Amazing ! 25 Linux Performance Monitoring Tools
• 20 Linux System Monitoring Tools Every SysAdmin Should Know
• How to monitor disk I/O in Linux from command line - Xmodulo
  
• 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
  
• Generate CPU, Memory & I/O report using SAR command
  
• My mysqlmymonlite.sh script can help with that mysqlmymonlite.sh README text file has instructions for setting up a cronjob to email you reports i.e. every 10/15min or every hour.

Notes:

• Centmin Mod already installed Glances tool. It can be invoked from SSH command glances or top2. Official Glances site at nicolargo.github.io/glances/ and documentation here.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

 

oh well, you just missed an opportunity to learn more and improve your PHP tuning skills !

 

literally the 3rd paragraph in above post of my has all the info for PHP-FPM side tuning for the issue including using PHP status statistics to delve deeper into figuring out your current state of PHP-FPM performance. But that doesn't rule out MySQL side being part of the problem as outlined in 2nd paragraph

3rd paragraph

 

Have you read the links i posted in 1st reply ? I suggest you read them fully first to understand PHP-FPM status metrics output and what they mean first as the 3rd paragraph's linked discussion threads shed light on your issue and most common PHP-FPM tuning/optimisation that folks may eventually need to do.

 

Key to PHP-FPM tuning is PHP-FPM Status output + the inspecting logs

It's not just logs for nginx but php-fpm and also maybe inspecting mysql/database backend slow query logs as it can be a chain reaction and in your case a likely case considering you're doing data import from/to a MySQL database server. Usually when I do Xenforo forum imports for clients, I monitor and measure the server's statistics and through several test import runs, optimise the server for the best import speed and performance so that final import is blazing fast.

 

Centmin Mod is auto tuned to your servers hardware specs detected but that isn't same as tuned to your web app/scripts usage requirements. For the latter, you need to optimise the software for your usage requirements. Centmin Mod auto tuned gives you an optimised baseline to build upon for your own further web app specific requirements. Tuning for web apps is rarely a set and forget affair but involves constant monitoring and re-tuning based on your usage requirements

• What version of Centmin Mod ? 123.08stable or 123.09beta01 ? Was it fresh install or upgrade ? How long ago did you install Centmin Mod ? There's numerous code changes, bug fixes over time, so ensure you have latest Centmin Mod code installed by upgrading your Centmin Mod code as instructed below.
• What version of PHP used ? Did you upgrade or downgrade PHP recently ?
• If you upgraded and you get some PHP warnings that some PHP extensions are unable to load, all you need to do is delete the relevant *.ini files for them and then reinstall if you require the PHP extension

Example of error on restart PHP-FPM

Code (Text):

fpmrestart
Gracefully shutting down php-fpm . done
Starting php-fpm [06-Nov-2014 01:25:37] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20140815/geoip.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20140815/geoip.so: cannot open shared object file: No such file or directory in Unknown on line 0
[06-Nov-2014 01:25:37] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20140815/igbinary.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20140815/igbinary.so: cannot open shared object file: No such file or directory in Unknown on line 0
[06-Nov-2014 01:25:37] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20140815/imagick.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20140815/imagick.so: cannot open shared object file: No such file or directory in Unknown on line 0
[06-Nov-2014 01:25:37] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20140815/memcache.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20140815/memcache.so: cannot open shared object file: No such file or directory in Unknown on line 0
[06-Nov-2014 01:25:37] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/extensions/no-debug-non-zts-20140815/memcached.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20140815/memcached.so: cannot open shared object file: No such file or directory in Unknown on line 0
done

find the paths to the *.ini files

Code (Text):

php --ini 2>/devnull
Configuration File (php.ini) Path: /usr/local/lib
Loaded Configuration File:         /usr/local/lib/php.ini
Scan for additional .ini files in: /etc/centminmod/php.d
Additional .ini files parsed:      /etc/centminmod/php.d/curlcainfo.ini,
/etc/centminmod/php.d/custom_php.ini,
/etc/centminmod/php.d/geoip.ini,
/etc/centminmod/php.d/igbinary.ini,
/etc/centminmod/php.d/imagick.ini,
/etc/centminmod/php.d/memcache.ini,
/etc/centminmod/php.d/memcached.ini,
/etc/centminmod/php.d/zendopcache.ini

delete ones that are reported as unable to load so if geoip.so, igbinary.so, imagick.so, memcache.so and memcached.so are reported as unable to load, remove their respective *.ini files

Code (Text):

rm -rf /etc/centminmod/php.d/geoip.ini
rm -rf /etc/centminmod/php.d/igbinary.ini
rm -rf /etc/centminmod/php.d/imagick.ini
rm -rf /etc/centminmod/php.d/memcache.ini
rm -rf /etc/centminmod/php.d/memcached.ini

then restart PHP-FPM again

Code (Text):

fpmrestart
Gracefully shutting down php-fpm . done
Starting php-fpm  done

Code (Text):

php -v
PHP 7.0.0-dev (cli) (built: Nov  6 2014 00:54:07)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.8.0-dev, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies

Manually reinstall PHP extension

For reported PHP extensions that do not load, i.e. "PHP Startup: Unable to load dynamic" and references PHPEXTENSIONNAME.so, some can be installed again via centmin.sh menu. For instance menu options 7, 9, 10, 12, 15, 18, and 19.

for 123.08stable

Code (Text):

--------------------------------------------------------
Centmin Mod 1.2.3-eva2000.08
--------------------------------------------------------
                   Centmin Mod Menu       
--------------------------------------------------------
1).  Centmin Install
2).  Add Nginx vhost domain
3).  NSD setup domain name DNS
4).  Nginx Upgrade / Downgrade
5).  PHP Upgrade / Downgrade
6).  XCache Re-install
7).  APC Cache Re-install
8).  XCache Install
9).  APC Cache Install
10). Memcached Server Re-install
11). MariaDB 5.2, 5.5, 10 Upgrade Sub-Menu
12). Zend OpCache Install/Re-install
13). Install ioping.sh vbtechsupport.com/1239/
14). SELinux disable
15). Install/Re-install ImageMagick PHP Extension
16). Change SSHD Port Number
17). Multi-thread compression: pigz,pbzip2,lbzip2,p7zip etc
18). Suhosin PHP Extension install
19). Install FFMPEG and FFMPEG PHP Extension
20). NSD Re-install
21). Update - Nginx + PHP-FPM + Siege
22). Add Wordpress Nginx vhost + WP Super Cache
23). Exit
--------------------------------------------------------
Enter option [ 1 - 23 ]
--------------------------------------------------------

for 123.09beta01 - menu options 7, 9, 10, 12, 13 (for redis), 15, 18, and 19 are relevant menu options that can be used only if the specific PHP extension is reporting as unable to load

Code (Text):

--------------------------------------------------------
     Centmin Mod Menu 123.09beta01 centminmod.com
--------------------------------------------------------
1).  Centmin Install
2).  Add Nginx vhost domain
3).  NSD setup domain name DNS
4).  Nginx Upgrade / Downgrade
5).  PHP Upgrade / Downgrade
6).  XCache Re-install
7).  APC Cache Re-install
8).  XCache Install
9).  APC Cache Install
10). Memcached Server Re-install
11). MariaDB 5.2/5.5 & 10.x Upgrade Sub-Menu
12). Zend OpCache Install/Re-install
13). Install/Reinstall Redis PHP Extension
14). SELinux disable
15). Install/Reinstall ImagicK PHP Extension
16). Change SSHD Port Number
17). Multi-thread compression: pigz,pbzip2,lbzip2...
18). Suhosin PHP Extension install
19). Install FFMPEG and FFMPEG PHP Extension
20). NSD Install/Re-Install
21). Update - Nginx + PHP-FPM + Siege
22). Add Wordpress Nginx vhost + Cache Plugin
23). Update Centmin Mod Code Base
24). Exit
--------------------------------------------------------
Enter option [ 1 - 24 ]

Upgrading Centmin Mod Code to Latest Version

Getting Started Guide step 19 outlines also how to keep Centmin Mod code updated or how to switch version branches.

Centmin Mod LEMP stack's script code is constantly updated for improvements, bug fixes and security fixes so keeping the Centmin Mod code up to date is important. With Centmin Mod 1.2.3-eva2000.08) (123.08stable) and higher releases, a newly added centmin.sh menu option 23 allows much easier code updates and version branch swicthing via Git backed environment you can setup. For full details read the following links:

• New centmin.sh menu option 23 updating
• Centmin Mod Upgrades
• Cronjob Auto Updating Centmin Mod Code Guide
• How to upgrade Centmin Mod + backing up customisations

Upgrading Centmin Mod involves 2 parts.

1. Upgrading the actual Centmin Mod code outlined at Upgrade Centmin Mod. This is heart of Centmin Mod where the code is the engine that runs centmin.sh shell based menu and all the automation you're accustomed to. You can easily update within a Centmin Mod version branch or switch version branches via centmin.sh menu option 23 outlined here.
2. Upgrade software that Centmin Mod installed or manages. For this part following outline at How to upgrade Centmin Mod software installed on your server.

That's why it's always good idea to have a test VPS or clone of your live server for testing changes in a controlled environment as per tip at Guide to learning more about Centmin Mod. Also always backup your settings before making changes and always document the changes you make so you can revert them. This is where monitoring server stats is key as to be able to measure and see whether changes you make have a positive or negative effect - which is why my initial reply in this thread listed alot of server monitoring and statistics gathering articles/guides.

That's what server monitoring is for. I usually write my own scripts for some of this which monitor php-fpm status output and can email me at set intervals the stats output when they reach certain preset thresholds. Nixststats can do some form of visual monitoring Monitor PHP-FPM with Nixstats as can Nginx Amplify Monitoring PHP Applications with NGINX Amplify. I should add these 2 to the list I usually quote.

 

You generally do all this is you want to monitor your server software usage statistics to really KNOW what's going on on your server. So if you monitor everyday, you know what is normal and what is not normal - that is probably 75% of optimisation tuning process as how else would you know whether or not settings changes actually do anything positive or not ?

Oh and for Nixstats PHP-FPM - How to configure PHP-FPM Status Monitoring with Nixstats

 

Yeah you'd need to custom script for that i.e. i send alerts to email and push notifications to mobile via pushbullet. There's probably other 3rd party monitoring that can do such alerts out there but I haven't really looked.

 

Learning is really 80% reading and practising what you read. Becoming an expert is about reading and practising more than normal folks and making more mistakes than normal folks and learning from them So don't be afraid of reading alot and making alot of mistakes

As to where to focus, the PHP-FPM status output metrics are explained fully at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS so I'll let you learn through trial and error - best way to learn !

Nagios monitoring has such I believe but probably a more complicated monitoring setup for noob folks compared to simple standalone scripts.