Join the community today
Register Now

Nginx Nginx down when DDoS

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Oxide, Apr 18, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    41,080
    9,185
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,093
    Local Time:
    12:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    are all other affected centmin mod servers on OVH too ? tried non-OVH server ?
     
  2. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    i will order a digitalocean vps and install same installation and confirm this

    i cant remember i had this issue - before i moved to ovh.. OR centos 7.
     
  3. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    1 server uses centmin
    1 server uses "yum install nginx"

    the centmin one has the issue, the yum install nginx (no lempstack) works fine.

    both on ovh..? what does that mean ? lol
     
  4. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    @eva2000

    I confirm this issue is caused by Centmin somewhere.

    I installed nginx on the same DigitalOcean VPS. It worked fine during attack.
    However, soon as I installed centmin - the cpu went straight to 0% idle.

    Something is going wrong, when recieveing a large amount of packets. What? I don't know..

    I've now tested it in four servers.

    2 x DigitalOcean (Centminmod & Non-centmin w/ nginx)
    2 x OVH (Centminmod & Non-centmin w/ nginx)
    1 x SoYouStart (Centminmod)

    All the servers with centminmod seems to have the same issue. However, if i install nginx my self manually (example yum install nginx) it works completely fine.

    By that, I tested the same install procedure on four servers. Two centmin and two yum install nginx. Both centmin had same issue, but the default nginx install did not........

    Since this is a optimization issue, or issue in general with centminmod I hope you will join the research to find out what's going wrong here.
     
    • Informative Informative x 1
  5. eva2000

    eva2000 Administrator Staff Member

    41,080
    9,185
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,093
    Local Time:
    12:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    No free time to spare with my mother heart surgery and related health matters and own server DDOS protection deployment plans in the works and centmin mod is provided as is and I only work on centmin mod in my free spare time when I have it.
     
  6. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    Of course, there is no rush. Maybe when you have time, we can take a look at it together. I can help initating those attacks, if you want to see it your self - even give ssh to my server. Not expecting you to fix it for me, but maybe find a solution for the community.

    Good luck with your mom, hope everything goes OK.
     
  7. eva2000

    eva2000 Administrator Staff Member

    41,080
    9,185
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,093
    Local Time:
    12:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    btw what version nginx was this with ? 1.9.15 had a few possible related bug fixes Nginx - [nginx-announce] nginx-1.9.15 | Centmin Mod Community

    1.9.13 had a few too Nginx - [nginx-announce] nginx-1.9.13 | Centmin Mod Community
     
  8. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    The server that i did "yum install nginx" on was nginx/1.6.3 (it worked fine)

    All the Centmin servers was (that it stopped accepting connections):
    nginx/1.9.13
    nginx/1.9.11
    nginx/1.9.9
    nginx/1.9.14

    it looks like the issue is not with nginx, but some other app within centmin that are blocking connections.. I am not sure what, because nginx would've been burning my cpu before it would even consider crashing right?

    OVH said it looks like something are blocking connections, before it even manage to impact Nginx. That's why, there are no cpu usage within nginx.. It's like no requests even reaches nginx. However when I did tcpdump,. they appeared there. So it's really hard to tell.

    When I use siege to test concurrency on my server, i can get over 50k r/s - fine. The CPU goes very high, this is normal how i want it to work. But when it's under attack, XML-RPC/PingBack it uses no cpu and stops responding.

    I tried disabling csf:
    csf -x

    Could even something like "
    include /usr/local/nginx/conf/geoip.conf;
    " have any impact?
     
    Last edited: Apr 27, 2016
  9. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    Hope to see a solution's for this soon, really don't want to change lemp stack. Don't know about any else tbh.
     
  10. eva2000

    eva2000 Administrator Staff Member

    41,080
    9,185
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,093
    Local Time:
    12:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Probably not
    As I said with my current lack of free time won't be any time soon, could be weeks if not months with my current real life obligations.
     
  11. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    ya that sucks, sucks that nobody else knows what to do..
     
  12. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    3:27 PM
    1.9.x
    10.1.x
    Its not possible to change to the Stock Kernel after the installation? Searching in Google returns some results to that, but i never tried something like that before.
     
  13. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    I understand that it's a lot of replies to go through, but as i said earlier here - it has nothing to do with OVH Kernel.

    I rented a DigitalOcean VPS - and installed Centmin. And on another one, i installed "yumi install nginx".. The centmin had same issue, on a html page - however the non-centmin one worked just fine.

    It's like there are something within centmin locking nginx down, under certain amount of requests. I've tested and confirmed this on multiply servers, i have over 5 servers with centmin and they all have the exact same issue - from DigitalOcean, RamNode and OVH VPS/DEDI.

    Also about kernel - no there are no way to do this - official ways anyways. I followed some tutorials, but they crashed my server - so i rather not touch it again. Since it has nothing to do with the issue, even i thought it had to do.. OVH also confirmed them self there is nothing able to do this.

    IT seems like something is "blocking" the traffic from coming in, before Nginx even gets impacted. I've disabled CSF, same issue still.

    It's mind blowing lol.

    When i run a tool like Siege (directly to http://127.0.0.1/index.html) it works just fine. Nginx gets impacted, and the CPU raises a lot. However when it's under attack, it's like something is cutting out all connections to the server.

    What could it be?

    We've tried the attacks on a domain name, and IP Directly. On PHP Files, and on HTML Files.
     
    Last edited: May 1, 2016
  14. Ahmad

    Ahmad Premium Member Premium Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    4:27 PM
    1.9.9
    10.1.10
    This seems interesting to me. Do you have skype? If yes, do you mind PMing it to me? I want you to launch an attack at one of my sites to see if the same thing happens.
     
    • Like Like x 2
  15. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    3:27 PM
    1.9.x
    10.1.x
    I can't help a lot either, because like i said in my introduction, i use my own centos/nginx/php-fpm/mariadb installation. I just came here, because i use a lot of configuration from centmin ;)

    Talking about attacks, i received last night a new attack. This time with almost 8000 servers. I can assume they were using more than 1 booter at the same time(or it was a fuckin big booter).

    Code:
     Actions
      |- Currently banned: 7798
    Fail2ban banned all of them. While it was banning, site never went down, but it took all the 8 cpu threads to 100%. Site slowed down to 5 seconds in the main page for some minutes, but it never went down, which was good.

    After that i permanently ban all those IP's, using ipset. If i use only iptables, is more resource intensive. I saw a benchmark where having 50,000 ip's banned in iptables, can lead to a 12ms increase in every request. That seems low, but with ipset, its less than 1ms.

    So basically i created a list in ipset called blacklist:
    Code:
    ipset create blacklist hash:ip
    Then i tell iptables do drop all the connections to the ip's inside that ipset list:
    Code:
    iptables -I INPUT -m set --match-set blacklist src -j DROP
    To insert ip's in that list:
    Code:
    ipset add blacklist 5.2.134.226
    Here is the list of all the ip's with the command added: Blacklist - Pastebin.com
    If you want to ban that ip's, confirmed by me of being hacked wordpress servers that attacked me, you just need to put it on a bash script or just paste it in your ssh terminal.

    After that just run "ipset list" and check if all the ip's are there.
     
    • Informative Informative x 3
  16. Ahmad

    Ahmad Premium Member Premium Member

    209
    80
    28
    Apr 13, 2015
    Ratings:
    +150
    Local Time:
    4:27 PM
    1.9.9
    10.1.10
    @Oxide @Revenge just launched an attack at my site (tried to revert configs to default as much as possible) and CPU went straight to 100% and I could see the attack in logs.
    Maybe it's attack type specific? Maybe it's a bug in nginx as @eva2000 mentioned (I'm using 1.9.15). Did you do any configuration yourself? Maybe there's an error.
    Would probably need to debug on a server that has this specific 'issue'.

    Edit: above behaviour is without useragent blocking. With useragent blocking CPU does not go over 10% and everything works as it should. Access logs shows that the pingback attack is being "blocked": http://i.imgur.com/NVioCIG.png
     
    Last edited: May 2, 2016
    • Like Like x 1
  17. eva2000

    eva2000 Administrator Staff Member

    41,080
    9,185
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,093
    Local Time:
    12:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    @Ahmad thanks for offering to help @Oxide while I'm too busy. Interesting results and yes could be tied to my previously posted <1.9.13 and lower closed connection bug nginx fixed in nginx 1.9.15. Great to see user agent blocking also doing it's job.

    @Revenge - Centmin Mod's auto installed CSF Firewall will auto use IPSET for iptables if Centmin Mod initial install routine detects that your server's linux kernel supports IPSET as CSF Firewall is just a wrapper to iptables. If the linux kernel doesn't support IPSET, CSF Firewall auto setup routine will just not enable IPSET support. Basically banned or whitelisted IPs in CSF Firewall will use IPSET if it's supported by server.

    So Centmin Mod users don't need to setup IPSET themselves as Centmin Mod and CSF firewall auto configures it if your server supports IPSET :)
     
    • Like Like x 2
  18. pamamolf

    pamamolf Premium Member Premium Member

    3,404
    320
    83
    May 31, 2014
    Ratings:
    +607
    Local Time:
    5:27 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    @Revenge

    What fail2ban rules do you use and you are able to catch the attacks?

    Can you post the config?
     
  19. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
  20. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    12:27 AM
    This is 100% the default installation that it happens. I've tested it on 2 x DigitalOcean VPS also. One had centmin and one did not, the centmin one seemed to have same issue.

    So it does not look like a "OVH ONLY" issue. And honestly, I've never had this issue my self - but with this specific attack it happens.. Most likely since it's massive.

    Updating nginx now as I am typing this ^^