Nginx PHP-FPM 502 Bad Gateway - portscanner?

you scheduled a nprestart run ?

Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

Nginx 502 or 504 Bad Gateway Errors

Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

You'll need to tune your PHP-FPM settings with php-fpm main pool config file at /usr/local/etc/php-fpm.conf (overview of config files) and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at

• http://centminmod.com/phpfpm.html#browserstatus and
• https://community.centminmod.com/threads/pm-max_children.479/ and
• https://community.centminmod.com/threads/warning-pool-www-server-reached-max_children-setting-50-consider-raising-it.791/ which outlines the official PHP-FPM config documentation as well.

Checking PHP-FPM etc logs

You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

In SSH2 telnet you can use tail command to view the last X number of lines in the file.

For example for viewing last 10 lines in the file for:

For Nginx access and error logs:

Code:

  tail -10 /usr/local/nginx/logs/access.log
  tail -10 /usr/local/nginx/logs/error.log

For specific domainname.com access and error log:

Code:

  tail -10 /home/nginx/domains/domainname.com/log/access.log
  tail -10 /home/nginx/domains/domainname.com/log/error.log

For other system error logs located at /var/log:

list /var/log files in ascending time order so the most recently modified files are at the bottom

Code:

  ls -lhrt /var/log

Code:

total 2.7M
-rw------- 1 root  root    0 Aug 29 15:33 tallylog
-rw------- 1 root  root    0 Aug 29 15:33 spooler
drwx------ 3 root  root 4.0K Aug 29 15:35 samba
drwxr-xr-x 2 root  root 4.0K Aug 29 15:35 mail
-rw-r--r-- 1 root  500     0 Oct  8 18:13 dmesg.old
-rw------- 1 root  500     0 Oct  8 18:13 boot.log
-rw-r--r-- 1 root  500     0 Oct  8 18:14 dmesg
drwx------ 2 root  root 4.0K Oct  8 18:14 httpd
drwxr-xr-x 2 root  root 4.0K Oct  8 19:08 php-fpm
-rw-rw---- 1 mysql root 2.3K Oct  9 12:38 mysqld.log
-rw------- 1 root  root 9.2K Oct 26 10:48 yum.log
-rw------- 1 root  utmp  94K Nov  7 22:59 btmp
drwxr-xr-x 2 root  root 4.0K Nov  8 00:00 sa
-rw------- 1 root  root 269K Nov  8 21:39 messages
-rw------- 1 root  root 110K Nov  8 23:08 secure
-rw-rw-r-- 1 root  utmp  43K Nov  8 23:08 wtmp
-rw-r--r-- 1 root  root 144K Nov  8 23:08 lastlog
-rw------- 1 root  root  69K Nov  8 23:08 lfd.log
-rw------- 1 root  root 332K Nov  8 23:08 maillog
-rw------- 1 root  500  1.6M Nov  8 23:10 cron

For PHP-FPM error log:

Code:

  tail -10 /var/log/php-fpm/www-error.log

and/or

Code:

  /var/log/php-fpm/www-php.error.log

For MySQL / MariaDB error log:

Code:

  tail -10 /var/log/mysqld.log

For CSF firewall LFD log:

Code:

  tail -10 /var/log/lfd.log

For Mail log:

Code:

  tail -10 /var/log/maillog

For Cron job logs:

Code:

  tail -10 /var/log/cron

How to edit php.ini and php-fpm configuration files ?

Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:

• Edit php.ini = phpedit ( /usr/local/lib/php.ini )
• Edit my.cnf = mycnf ( /etc/my.cnf )
• Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
• Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
• Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
• Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
• Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
• Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
• nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
• php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
• mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
• nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
• memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
• csf stop/start/restart = csfstop/csfstart/csfrestart

Troubleshooting Tools

However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.

• 20 Command Line Tools to Monitor Linux Performance
• 13 Linux Performance Monitoring Tools - Part 2
• 80 Linux Monitoring Tools for SysAdmins - Server Density Blog
• Amazing ! 25 Linux Performance Monitoring Tools
• 20 Linux System Monitoring Tools Every SysAdmin Should Know
• How to monitor disk I/O in Linux from command line - Xmodulo
  
• 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
  
• Generate CPU, Memory & I/O report using SAR command
  
• My mysqlmymonlite.sh script can help with that mysqlmymonlite.sh README text file has instructions for setting up a cronjob to email you reports i.e. every 10/15min or every hour.

Notes:

• Centmin Mod already installed Glances tool. It can be invoked from SSH command glances or top2. Official Glances site at nicolargo.github.io/glances/ and documentation here.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

 

where you seeing these logs as the format you're showing isn't standard log format

check your /var/log/secure, /var/log/message and /var/log/lfd.log for csf firewall login failure daemon for clues and if you have the sshd login user's IP address should grep all logs to get an idea

Code (Text):

grep -R 'ipaddress' /var/log/*

where ipaddress is the ip address

install malware detect , addons/maldet.sh addon and run it to check for malware Maldet - Linux Malware Detect Addon (discussion)

if you had installed tools/auditd.sh you would be able to somewhat track user logins and actions too Centmin Mod Auditd Support Added In Latest 123.09beta01 but if you haven't you wouldn't have as many auditd rules to utilise but would still have some idea

 

best to check your actual logs as directed above

 

that is failed login as pure-ftpd virtual ftp users are TLS encrypted connections only and do not connect via clear plain text sessions

if that isn't your ip, then you can search and filter logs by that ip to get an idea of what the server is doing

Example of tools/auditd.sh installed auditd logging on server Centmin Mod Auditd Support Added In Latest 123.09beta01

checking logins

check all successful + failed logins (2nd last column yes = success, no = failed)

Code (Text):

aureport -au -i

Code (Text):

aureport -au -i

Authentication Report
============================================
# date time acct host term exe success event
============================================
1. 07/08/2014 03:32:38 root ? tty1 /usr/bin/login yes 338
2. 07/08/2014 03:33:53 ? ? tty1 /usr/bin/login no 352
3. 12/12/2016 23:50:31 root 192.168.0.xxx ssh /usr/sbin/sshd yes 342
4. 12/12/2016 23:50:31 root 192.168.0.xxx ssh /usr/sbin/sshd yes 345
5. 07/22/2017 16:43:58 root 192.168.0.xxx ssh /usr/sbin/sshd yes 92
6. 07/22/2017 16:43:58 root 192.168.0.xxx ssh /usr/sbin/sshd yes 95
7. 07/22/2017 16:49:33 root 192.168.0.xxx ssh /usr/sbin/sshd yes 94
8. 07/22/2017 16:49:33 root 192.168.0.xxx ssh /usr/sbin/sshd yes 97
9. 07/22/2017 18:23:36 root 192.168.0.xxx ssh /usr/sbin/sshd yes 94
10. 07/22/2017 18:23:36 root 192.168.0.xxx ssh /usr/sbin/sshd yes 97
11. 07/30/2017 04:02:02 root 192.168.0.xxx ssh /usr/sbin/sshd yes 98
12. 07/30/2017 04:02:02 root 192.168.0.xxx ssh /usr/sbin/sshd yes 101
13. 08/04/2017 13:10:43 root 192.168.0.xxx ssh /usr/sbin/sshd yes 86
14. 08/04/2017 13:10:43 root 192.168.0.xxx ssh /usr/sbin/sshd yes 89
15. 08/12/2017 19:30:13 root 192.168.0.xxx ssh /usr/sbin/sshd yes 205
16. 08/12/2017 19:30:13 root 192.168.0.xxx ssh /usr/sbin/sshd yes 208
17. 08/12/2017 19:31:13 root 192.168.0.xxx ssh /usr/sbin/sshd no 238
18. 08/12/2017 19:31:16 root 192.168.0.xxx ssh /usr/sbin/sshd no 239

check successful logins only

Code (Text):

aureport -au -i --success

Code (Text):

aureport -au -i --success

Authentication Report
============================================
# date time acct host term exe success event
============================================
1. 07/08/2014 03:32:38 root ? tty1 /usr/bin/login yes 338
2. 12/12/2016 23:50:31 root 192.168.0.xxx ssh /usr/sbin/sshd yes 342
3. 12/12/2016 23:50:31 root 192.168.0.xxx ssh /usr/sbin/sshd yes 345
4. 07/22/2017 16:43:58 root 192.168.0.xxx ssh /usr/sbin/sshd yes 92
5. 07/22/2017 16:43:58 root 192.168.0.xxx ssh /usr/sbin/sshd yes 95
6. 07/22/2017 16:49:33 root 192.168.0.xxx ssh /usr/sbin/sshd yes 94
7. 07/22/2017 16:49:33 root 192.168.0.xxx ssh /usr/sbin/sshd yes 97
8. 07/22/2017 18:23:36 root 192.168.0.xxx ssh /usr/sbin/sshd yes 94
9. 07/22/2017 18:23:36 root 192.168.0.xxx ssh /usr/sbin/sshd yes 97
10. 07/30/2017 04:02:02 root 192.168.0.xxx ssh /usr/sbin/sshd yes 98
11. 07/30/2017 04:02:02 root 192.168.0.xxx ssh /usr/sbin/sshd yes 101
12. 08/04/2017 13:10:43 root 192.168.0.xxx ssh /usr/sbin/sshd yes 86
13. 08/04/2017 13:10:43 root 192.168.0.xxx ssh /usr/sbin/sshd yes 89
14. 08/12/2017 19:30:13 root 192.168.0.xxx ssh /usr/sbin/sshd yes 205
15. 08/12/2017 19:30:13 root 192.168.0.xxx ssh /usr/sbin/sshd yes 208

check failed logins only

Code (Text):

aureport -au -i --failed

Code (Text):

aureport -au -i --failed

Authentication Report
============================================
# date time acct host term exe success event
============================================
1. 07/08/2014 03:33:53 ? ? tty1 /usr/bin/login no 352
2. 08/12/2017 19:31:13 root 192.168.0.xxx ssh /usr/sbin/sshd no 238
3. 08/12/2017 19:31:16 root 192.168.0.xxx ssh /usr/sbin/sshd no 239

 

seems ip was from a known port scanner fishing for vulnerabilities 158.85.81.114 | Administrative Contact | AbuseIPDB

 

pure-ftpd logs to system /var/log/messages

Code (Text):

grep 'pure-ftpd' /var/log/messages  | tail -100
Aug 12 19:39:20 host pure-ftpd: (?@192.168.0.xxx) [INFO] New connection from 192.168.0.xxx
Aug 12 19:39:21 host pure-ftpd: (?@192.168.0.xxx) [INFO] SSL/TLS: Enabled TLSv1/SSLv3 with ECDHE-RSA-AES128-GCM-SHA256, 128 secret bits cipher
Aug 12 19:39:26 host pure-ftpd: (?@192.168.0.xxx) [WARNING] Authentication failed for user [ftpuser1]
Aug 12 19:39:32 host pure-ftpd: (?@192.168.0.xxx) [INFO] Logout.

Aug 12 19:39:47 host pure-ftpd: (?@192.168.0.xxx) [INFO] New connection from 192.168.0.xxx
Aug 12 19:39:47 host pure-ftpd: (?@192.168.0.xxx) [INFO] SSL/TLS: Enabled TLSv1/SSLv3 with ECDHE-RSA-AES128-GCM-SHA256, 128 secret bits cipher
Aug 12 19:39:47 host pure-ftpd: (?@192.168.0.xxx) [INFO] ftpuser1 is now logged in
Aug 12 19:39:47 host pure-ftpd: (ftpuser1@192.168.0.xxx) [INFO] SSL/TLS: Enabled TLSv1/SSLv3 with ECDHE-RSA-AES128-GCM-SHA256, 128 secret bits cipher

Aug 12 19:48:40 host pure-ftpd: (?@192.168.0.xxx) [INFO] New connection from 192.168.0.xxx
Aug 12 19:48:40 host pure-ftpd: (?@192.168.0.xxx) [WARNING] Sorry, cleartext sessions and weak ciphers are not accepted on this server.#012Please reconnect using SSL/TLS security mechanisms.

above i grouped them into 3 separate pure-ftpd login attempts for pure-ftpd virtual ftp user = ftpuser1

1st login failed due to incorrect password

2nd login succeeded

3rd login failed due to trying to connect via plain clear text session instead of required explicit TLS encrypted connection that Centmin Mod configures and requires for pure-ftpd

seems the portscanner on that ip trying to fish for vulnerabilities 158.85.81.114 | Administrative Contact | AbuseIPDB

and probably overwhelmed your php-fpm server with requests so it died leading to 502 bad gateway errors so you need to follow and track that ip address or ip addresses you see doing similar access/scan patterns to figure it out. As posted before a quick start is to grep filter your system logs in /var/log for the ip address

Code (Text):

grep -R 'ipaddress' /var/log/

Centmin Mod is provided as is, so alot of this would be on you and/or assistance from other forum members here.

 

Centmin Mod installs and configures CSF Firewall + LFD protection CSF Firewall - CentminMod.com LEMP Nginx web stack for CentOS for failed sshd logins so partially protected. Example from LFD /var/log/lfd.log of failed sshd login attempts blocked at CSF Firewall level

Code (Text):

tail -100 /var/log/lfd.log

Aug 12 13:44:49 host lfd[26448]: (sshd) Failed SSH login from 5.101.40.17 (RU/Russian Federation/-): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Aug 12 14:26:09 host lfd[26712]: (sshd) Failed SSH login from 59.96.226.109 (IN/India/-): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Aug 12 17:17:30 host lfd[27689]: (sshd) Failed SSH login from 35.202.162.85 (US/United States/85.162.202.35.bc.googleusercontent.com): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Aug 12 17:32:31 host lfd[27786]: (sshd) Failed SSH login from 190.214.92.158 (EC/Ecuador/158.92.214.190.static.anycast.cnt-grms.ec): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Aug 12 18:13:11 host lfd[28039]: 181.211.224.233 (EC/Ecuador/233.224.211.181.static.anycast.cnt-grms.ec), 5 distributed sshd attacks on account [support] in the last 3600 secs - *Blocked in csf* [LF_DISTATTACK]
Aug 12 18:13:11 host lfd[28039]: 103.89.88.111 (VN/Vietnam/-), 5 distributed sshd attacks on account [support] in the last 3600 secs - *Blocked in csf* [LF_DISTATTACK]
Aug 12 18:46:51 host lfd[28250]: (sshd) Failed SSH login from 181.174.24.128 (CO/Colombia/181-174-24-128.telebucaramanga.net.co): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Aug 12 18:52:11 host lfd[28304]: 60.2.76.134 (CN/China/-), 7 distributed sshd attacks on account [root] in the last 3600 secs - *Blocked in csf* [LF_DISTATTACK]
Aug 12 18:57:51 host lfd[28354]: (sshd) Failed SSH login from 125.25.22.10 (TH/Thailand/node-4cq.pool-125-25.dynamic.totbb.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]

There's no layer 7 app level protection for overwhelming php-fpm server with php requests though there are a few things I am working on for this

• fail2ban + csf Security - Cloudflare - fail2ban for Centmin Mod + CSF Firewall / Cloudflare API
• bad bot blocking/rate limiting Security - Blocking bad or aggressive bots
• modsecurity support Beta Branch - update prep for ModSecurity v3.0
• enabling CSF Firewall blocklists and advance blocklists selectively if you have non-OpenVZ system that supports IPSET

You can also natively rate limit requests to php locations via supported nginx rate limiting as seen in bad bot blocking thread Nginx - Nginx rate limiting new article

Cloudflare WAF would help but it's paid Pro plan and higher as would Sucuri Cloudproxy (more tricky to get right for web apps). The forums here use Sucuri Cloudproxy as I can not change my DNS provider to Cloudflare as I use AWS Route53 GeoDNS and Latenacy based DNS.

and as my 1st reply in this thread mentions, still optimising and tuning PHP-FPM to best handle the load is also important