Join the community today
Register Now

Nginx Nginx down when DDoS

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by Oxide, Apr 18, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    41,088
    9,194
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,104
    Local Time:
    10:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    no restarts. all auto taken care of for new tcp settings once you re-run centmin.sh after updating to latest 123.09beta01 code :)

    as to connection limiting it's a feature of CSF Firewall too see Nginx - Nginx down when DDoS | Page 5 | Centmin Mod Community but not request limiting
     
    • Informative Informative x 1
  2. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Filter:
    Code:
    [Definition]
    
    failregex = ^<HOST> -.*GET.*/
    
    ignoreregex = ^<HOST> -.*GET.*/uploads
                  ^<HOST> -.*GET.*/applications
    Here you can notice that it will catch every GET, unless the GET if for Uploads and Applications folders. I do this because i have a Invision Board, and sometimes a user can make a lot of GET's in that folders to get all the css, js etc etc. So i will ignore those ones.

    Jail:
    Code:
    [http-get-dos]
    enabled = true
    port = http,https
    filter = http-get-dos
    logpath = /path/to/your/access.log
    backend  = polling
    journalmatch =
    maxretry = 50
    findtime = 5
    bantime = 300
    action = iptables-allports
    This Jail will ban for 300 seconds, if an IP makes 50 requests in 5 or less seconds. You can changes this values off course. I just did this if someone hits the F5 at my site, he will simple get banned for 300 seconds.
     
    Last edited: May 3, 2016
    • Informative Informative x 2
  3. pamamolf

    pamamolf Premium Member Premium Member

    3,405
    320
    83
    May 31, 2014
    Ratings:
    +608
    Local Time:
    3:32 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    What about banning many connections also and not only many requests?
     
  4. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    What is the diference between requests and connections?
     
  5. eva2000

    eva2000 Administrator Staff Member

    41,088
    9,194
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,104
    Local Time:
    10:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  6. pamamolf

    pamamolf Premium Member Premium Member

    3,405
    320
    83
    May 31, 2014
    Ratings:
    +608
    Local Time:
    3:32 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Why not just use the CT_LIMIT = "0" option on csf ?

    Isn't the same?Why fail2ban is better?
     
  7. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Because with fail2ban you can creates regex rules. I can say to fail2ban to ban an IP that make 50 connections/requests, but the requests to css and js don't count. Or i can say to fail2ban to just ban IP's that make requests with WordPress user agent. I never used CSF, but i think it can't do that, at least by itself. It will ban if an IP makes x connections, and just that.
     
    • Informative Informative x 1
  8. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Ok people... New situation here.

    The booter that i have access, its from a client i have. We use the booter to test the protection of is own server. The booter that he have, now have 3 new Layer 7 attacks(although its almost the same), GET, HEAD and POST.
    [​IMG]

    I tried the GET attack against his site, and its impossible to defend... There is no UserAgent now. We can't tell the diference between a normal request, and this requests.

    Code:
    5.189.205.193 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
    5.189.205.229 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Linux; U; Android 2.2; fr-fr; Desire_A8181 Build/FRF91) App3leWebKit/53.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
    5.101.217.153 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.0) Opera 7.02 Bork-edition [en]"
    185.89.100.34 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Linux; U; Android 2.2; fr-fr; Desire_A8181 Build/FRF91) App3leWebKit/53.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
    5.101.220.191 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/534.57.5 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.4"
    185.14.194.122 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11"
    146.185.204.248 - - [03/May/2016:02:09:59 +0100] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
    What the hell can we do against this?
     
  9. pamamolf

    pamamolf Premium Member Premium Member

    3,405
    320
    83
    May 31, 2014
    Ratings:
    +608
    Local Time:
    3:32 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    If the requests per ip are many and fast you can use csf and CT_LIMIT = "0" to ban the ip's :)
    If the requests are not many per ip don't know :(
    If the requests are very slow per ip you can try to close connections faster on Nginx as you can check on the above link that eva2000 posts:

    DDos Mitigation - Using NGINX to Prevent DDoS Attacks | NGINX
     
  10. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Well, you have like 8000 to 10000 servers making less than 1 request per second each. Its impossible to put rules under that situation, or we would also be limiting good requests.
     
  11. eva2000

    eva2000 Administrator Staff Member

    41,088
    9,194
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,104
    Local Time:
    10:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    true CSF Firewall doesn't have that fine grain control as to which http request elements it tracks like css and js exclusion

    a caching layer on the front end probably can help but depends on your web app too
     
  12. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    New kind of DDOS attack using Facebook? o_O
    I stopped it with 444 response to facebookexternalhit.
    But i need to disable it after the attack stops.

    Code:
    173.252.123.129 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485223866 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233582 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.137 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233583 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.144 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233609 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233797 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233799 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.132 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224190 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224191 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224227 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.129 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224240 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.146 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224295 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.129 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233851 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.137 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485233803 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.137 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485234077 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.129 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224371 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224524 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224528 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.132 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485234086 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.128 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485224530 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.128 - - [17/May/2016:13:44:00 +0200] "GET /?id=1463485234200 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485224577 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.146 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234249 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.146 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234395 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.129 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234199 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.137 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485224847 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.137 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234401 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.144 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234452 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234465 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.132 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234497 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.128 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485224851 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.144 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485224865 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.146 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234728 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485225082 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234729 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.136 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485225084 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.144 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485234743 HTTP/1.1" 444 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.147 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485225091 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
    173.252.123.128 - - [17/May/2016:13:44:01 +0200] "GET /?id=1463485225094 HTTP/1.1" 499 0 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
     
    • Informative Informative x 1
  13. eva2000

    eva2000 Administrator Staff Member

    41,088
    9,194
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,104
    Local Time:
    10:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    interesting will need to add that as an option to my botlimit routine as a rate limitable setting Blocking bad or aggressive bots | Centmin Mod Community (optional commented out)

    edit: doh already been added as a rate limited option :D
     
    • Like Like x 2
  14. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    I don't know if someone can give a help here.

    One site of a friend is being targeted just now, and nginx is not being able to block it with a a 444 response. It responds with a 200.

    The site is a IPB 3.4.

    If i do this:
    curl -A "WordPress" https://site.com/
    I get an empty response. 444

    But if i do this(its what the attacker is doing:
    curl -A "WordPress" https://site.com/index.php?app=core&module=global&section=register
    I get a response 200.

    Why nginx does not block it?
     
  15. eva2000

    eva2000 Administrator Staff Member

    41,088
    9,194
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,104
    Local Time:
    10:32 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    what does the nginx vhost contexts look like for 444 ?
     
  16. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Code:
    if ($http_user_agent ~ "WordPress") {
            set $block_user_agents 1;
     }
    
    if ($block_user_agents = 1) {
            return 444;
    }
     
  17. Oxide

    Oxide Active Member

    516
    29
    28
    Mar 19, 2015
    Ratings:
    +54
    Local Time:
    10:32 PM
    try to use this instead

    if ($http_user_agent ~* "PHP|curl|Wget|HTTrack|Nmap|Verifying|PingBack|Pingdom|Joomla|Wordpress") { return 444; }
     
    Last edited: Sep 6, 2016
  18. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    Fail2ban took care of the attack:
    The attacked continued all day but without affecting the site since all server's ip's were blocked.
    Too many fat kids goes to hackforums to buy a booter... Thats the world we leave today.
     
    • Like Like x 1
  19. pamamolf

    pamamolf Premium Member Premium Member

    3,405
    320
    83
    May 31, 2014
    Ratings:
    +608
    Local Time:
    3:32 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    What rule did you use for it?
     
  20. Revenge

    Revenge Active Member

    442
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    1:32 PM
    1.9.x
    10.1.x
    The rule is very simple.
    Code:
    [Definition]
    
    failregex = ^<HOST> -.*WordPress.*/
    
    It does the same as the nginx rule to return 444 to the user agent WordPress. In this case, it bans the IP using iptables.

    Now what i do is permanently ban all those IP's using ipset.
     
    • Informative Informative x 2