/ Register

Welcome to Centmin Mod Community
Register Now

PHP-FPM Mysterious Layer 7 flooding

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by M.P., Mar 26, 2016.

Tags:
Previous Thread Next Thread
Loading...
  1. Mar 26, 2016 #1
    M.P.

    M.P. New Member

    5
    1
    3
    Mar 5, 2016
    Ratings:
    +1
    Local Time:
    8:25 AM
    1.9.11
    MariaDB 10
    I'm experiencing what I think is a layer 7 flooding/DoS attack targeting php-fpm.
    The problem is that I can't seem to identify the exact attack vector.

    Firstly, here's a pic.

    [​IMG]

    Here are some things I tried to identify the source of the attack / what I know:
    1. Randomly sampled tens of IPs from the logs within the timeframe of the attack and performed some checks - whether they belong to a server/proxy, 'abnormal' country, previous visits before the attack has started etc. Nothing seemed out of the ordinary.
    2. Access log didn't seem abnormal in general, as I'm seeing 500-700 requests/min, which is average. Requests are being logged live as opposed to being buffered.
    3. Just max_children warnings in the php logs, also nothing abnormal in the slow log.
    4. No abnormal useragents, exploit-seeking urls, etc.
    5. Requests limit to php per ip didn't help
    6. Connection rate limit / max connection limit in iptables didn't help
    7. Tried some anti-flooding protections via CSF (such as CT), didn't help.
    8. Cloudflare (I'm under attack mode w/ pro plan) kind of worked, but ended up causing more issues such as ~1.5 times slower load speeds, and a, quote, "SSL error" that caused hours of downtime while I was away, so I got rid of it.
    9. Tried various settings with limit_req_zone and limit_conn_zone, didn't help, and mostly just fed 444 to innocent users on shared networks.
    10. Tried tens of different configs with nginx / fpm, didn't help - i.e., I'm almost confident that this isn't caused by bad configs. The only thing that seemingly helped at all is raising the max_children limit, but this isn't really a solution as I can't raise it infinitely.
    11. Double checked cron to see if anything hogging resources, got nothing
    12. Checked the number of connections during the attack, two or three suspicious IPs with 10+ connections, but this is normal, and all of them belonged to shared mobile networks.
    13. Enabled several block lists in CSF, didn't help.
    14. The server has been set up to pretty much block every known hosting providers on the planet, so it's unlikely that it's a "D"Dos, but just one person doing this. However, this (the protection) is done at the nginx layer - if I understood correctly how php-fpm works, this should still effective against attacks targeting php.
    15. Bot detection via checking cookie availability, useful, but didn't stop the attacks.

    Would it be possible to spawn fpm processes via maliciously crafted packets, without going through nginx, thus being undetected in the logs / protections? If so, how would I go about investigating further and mitigating it, and if my theory is inaccurate, is there anything I can do/check to stop this? Any input would be appreciated.
     
    Last edited: Mar 26, 2016
  2. Mar 26, 2016 #2
    eva2000

    eva2000 Administrator Staff Member

    36,860
    8,068
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,426
    Local Time:
    11:25 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such.

    I know 500 max_children is too high unless you have a 128+ cpu thread/core based server to service that config ! What's the web app using php-fpm ? forum software ? cms ? wordpress ? Server hardware configuration ?

    Shared networks ? where more than one mobile user users the same IP simultaneously ? Is that even possible/the norm for simultaneously having more than one user on same IP ?

    Tried using ngxtop Nginx - ngxtop real time metrics for Nginx | Centmin Mod Community ? See the later posts in the trhead for more examples of usage along with Blocking bad or aggressive bots | Centmin Mod Community

    Nginx 502 or 504 Bad Gateway Errors



    Bad gateway 502 /504 timeouts are usually related to Nginx timing out waiting on PHP-FPM to respond as PHP-FPM is overloaded or overwhelmed with requests, so may need to tune PHP-FPM values. It also maybe due to PHP-FPM in turn being queued and backed up waiting on MariaDB MySQL server to respond - so also need to look at MySQL.

    You'll need to tune your PHP-FPM settings and this is left up to end user to do but here's a thread for starters to enable php status page output outlined at PHP-FPM - CentminMod.com LEMP Nginx web stack for CentOS and PHP-FPM - pm.max_children & PHP-FPM - WARNING: [pool www] server reached max_children setting (50), consider raising it | Centmin Mod Community which outlines the official PHP-FPM config documentation as well.

    Checking PHP-FPM etc logs



    You'll also need to check into your PHP-FPM, Nginx and MariaDB logs which you can find as outlined at How to troubleshoot Centmin Mod initial install issues

    Server logs include Nginx, PHP-FPM, MariaDB MySQL error logs as well as others. You can find your Centmin Mod install/menu logs at FAQ 7 and server logs at FAQ 19 at Centmin Mod FAQ (most up to date info in FAQ so always read that first). Spoiler tag below has info too but may not be up to date.

    Some of Centmin Mod's installed software will have their own access and error logs which maybe useful for diagnosing errors or give info, notes, or warning notices.

    Note: There's no support provided by me for diagnosing such errors which may occur for various reasons including misconfiguration of installed php/mysql scripts or applications.

    In SSH2 telnet you can use tail command to view the last X number of lines in the file.

    For example for viewing last 10 lines in the file for:

    For Nginx access and error logs:
    Code:
    tail -10 /usr/local/nginx/logs/access.log
tail -10 /usr/local/nginx/logs/error.log
    For specific domainname.com access and error log:
    Code:
    tail -10 /home/nginx/domains/domainname.com/log/access.log
tail -10 /home/nginx/domains/domainname.com/log/error.log
    For other system error logs located at /var/log:

    list /var/log files in ascending time order so the most recently modified files are at the bottom
    Code:
    ls -lhrt /var/log
    Code:
    total 2.7M
-rw------- 1 root root 0 Aug 29 15:33 tallylog
-rw------- 1 root root 0 Aug 29 15:33 spooler
drwx------ 3 root root 4.0K Aug 29 15:35 samba
drwxr-xr-x 2 root root 4.0K Aug 29 15:35 mail
-rw-r--r-- 1 root 500 0 Oct 8 18:13 dmesg.old
-rw------- 1 root 500 0 Oct 8 18:13 boot.log
-rw-r--r-- 1 root 500 0 Oct 8 18:14 dmesg
drwx------ 2 root root 4.0K Oct 8 18:14 httpd
drwxr-xr-x 2 root root 4.0K Oct 8 19:08 php-fpm
-rw-rw---- 1 mysql root 2.3K Oct 9 12:38 mysqld.log
-rw------- 1 root root 9.2K Oct 26 10:48 yum.log
-rw------- 1 root utmp 94K Nov 7 22:59 btmp
drwxr-xr-x 2 root root 4.0K Nov 8 00:00 sa
-rw------- 1 root root 269K Nov 8 21:39 messages
-rw------- 1 root root 110K Nov 8 23:08 secure
-rw-rw-r-- 1 root utmp 43K Nov 8 23:08 wtmp
-rw-r--r-- 1 root root 144K Nov 8 23:08 lastlog
-rw------- 1 root root 69K Nov 8 23:08 lfd.log
-rw------- 1 root root 332K Nov 8 23:08 maillog
-rw------- 1 root 500 1.6M Nov 8 23:10 cron
    For PHP-FPM error log:
    Code:
    tail -10 /var/log/php-fpm/www-error.log
    and/or
    Code:
    /var/log/php-fpm/www-php.error.log
    For MySQL / MariaDB error log:
    Code:
    tail -10 /var/log/mysqld.log
    For CSF firewall LFD log:
    Code:
    tail -10 /var/log/lfd.log
    For Mail log:
    Code:
    tail -10 /var/log/maillog
    For Cron job logs:
    Code:
    tail -10 /var/log/cron

    How to edit php.ini and php-fpm configuration files ?



    Centmin Mod install created command short cuts outlined here to allow you to quickly edit your /usr/local/lib/php.ini file and your /usr/local/etc/php-fpm.conf file. Full list of command shortcuts below:
    • Edit php.ini = phpedit ( /usr/local/lib/php.ini )
    • Edit my.cnf = mycnf ( /etc/my.cnf )
    • Edit php-fpm.conf = fpmconf ( /usr/local/etc/php-fpm.conf )
    • Edit nginx.conf = nginxconf ( /usr/local/nginx/conf/nginx.conf )
    • Edit (nginx) virtual.conf = vhostconf - only edits /usr/local/nginx/conf/conf.d/virtual.conf not the additional vhost domain.com.conf files added later
    • Edit (nginx) php.conf = phpinc ( /usr/local/nginx/conf/php.conf )
    • Edit (nginx) drop.conf = dropinc ( /usr/local/nginx/conf/drop.conf )
    • Edit (nginx) staticfiles.conf = statfilesinc ( /usr/local/nginx/conf/staticfiles.conf )
    • nginx stop/start/restart = ngxstop/ngxstart/ngxrestart
    • php-fpm stop/start/restart = fpmstop/fpmstart/fpmrestart
    • mysql stop/start/restart = mysqlstop/mysqlstart/mysqlrestart
    • nginx + php-fpm stop/start/restart = npstop/npstart/nprestart
    • memcached stop/start/restart =memcachedstop/memcachedstart/memcachedrestart
    • csf stop/start/restart = csfstop/csfstart/csfrestart

    Troubleshooting Tools



    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
    Notes:
    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)
     
    Last edited: Mar 26, 2016
  3. Mar 26, 2016 #3
    rdan

    rdan Premium Member Premium Member

    4,367
    1,053
    113
    May 25, 2014
    Ratings:
    +1,523
    Local Time:
    9:25 PM
    Mainline
    10.2
    My short advice is try Sucuri Firewall (y) :).
     
  4. Mar 26, 2016 #4
    M.P.

    M.P. New Member

    5
    1
    3
    Mar 5, 2016
    Ratings:
    +1
    Local Time:
    8:25 AM
    1.9.11
    MariaDB 10
    Server is running fine with ~200 children. Specs are i7 4790k and 32GB 1333MHz RAM
    I have one xenforo forum and one media wiki, which I applied stricter rules to block any potential attack vectors -- and the same thing has been going on before I added the wiki.

    Yes, I believe it's very common in the SEA regions, and the same holds true for mobile 3G/4G networks.

    I haven't tried ngxtop, I'll give it a go, but I've already checked the logs thoroughly with bash scripts and also manually.

    I have already set up nginx to deny malicious useragents that comes bundled with centmin, blocked http/1.0, and cookie detection.
     
  5. Mar 26, 2016 #5
    M.P.

    M.P. New Member

    5
    1
    3
    Mar 5, 2016
    Ratings:
    +1
    Local Time:
    8:25 AM
    1.9.11
    MariaDB 10
    I was trying to decide between Sucuri and Cloudflare the other day, and went with CF.
    Cloudflare has way more data centres so I assumed it'd be faster, but the site still took a major performance hit.
    I'll give Sucuri a try, but probably as a last resort.
     
  6. Mar 26, 2016 #6
    rdan

    rdan Premium Member Premium Member

    4,367
    1,053
    113
    May 25, 2014
    Ratings:
    +1,523
    Local Time:
    9:25 PM
    Mainline
    10.2
    Sucuri is the most powerful Layer 7 protection I know and tested.
     
    • Like Like x 1
  7. Mar 26, 2016 #7
    eva2000

    eva2000 Administrator Staff Member

    36,860
    8,068
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,426
    Local Time:
    11:25 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    take a full read of thread at Redis - How to install Redis server on Centmin Mod LEMP stack and post regarding installing Xenforo Redis Cache addons for redis caching. I am using Redis cache for my Xenforo forums here as well :)

    Also as posted above, PHP-FPM - WARNING: [pool www] server reached max_children setting (50), consider raising it | Centmin Mod Community is relevant too - full read
     
    Last edited: Mar 26, 2016
  8. Mar 26, 2016 #8
    Revenge

    Revenge Active Member

    421
    86
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +311
    Local Time:
    1:25 PM
    1.9.x
    10.1.x
    If its someone with a good booter making a layer 7 ddos attack, its not easy to find the source.
     
    • Like Like x 1
  9. Mar 28, 2016 #9
    Revenge

    Revenge Active Member

    421
    86
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +311
    Local Time:
    1:25 PM
    1.9.x
    10.1.x
    Today one of my servers received a massive layer 7 attack, using the XMLRPC Wordpress flaw.

    Looking at the logs i could see something like this:
    Code:
    173.245.49.210 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 200 18146 "-" "WordPress/3.4.1; http://udi-immo.com"
108.162.216.150 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 200 18146 "-" "WordPress/3.4.1; http://dunescalumetaudubon.org"
173.245.56.134 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 499 0 "-" "WordPress/4.3; http://pfsbrands.com; verifying pingback from 185.130.5.195"
141.101.80.87 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 200 18145 "-" "WordPress/4.4.2; http://ir.folksam.se; verifying pingback from 185.130.5.195"
141.101.105.225 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 200 18145 "-" "WordPress/4.4.2; http://www.jlmdsystem.com; verifying pingback from 185.130.5.195"
162.158.56.131 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 499 0 "-" "WordPress/3.3.2; http://scooters4life.com"
173.245.50.130 - - [28/Mar/2016:11:15:35 +0200] "GET / HTTP/1.1" 200 18144 "-" "WordPress/4.3.3; http://www.qumashstudios.com; verifying pingback from 185.130.5.195"
    I created a rule in Fail2ban to block all requests that contains the word "WordPress".

    This was the result:
    1288 server IP's banned... WTF...
     
  10. Mar 28, 2016 #10
    eva2000

    eva2000 Administrator Staff Member

    36,860
    8,068
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,426
    Local Time:
    11:25 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    well centmin.sh menu option 22 auto installed wordpress auto disables XMLRPC so shouldn't be affected by the ping back reflection attacks :)
     
Previous Thread Next Thread
Loading...
..

.