Get the most out of your Centmin Mod LEMP stack
Become a Member

Cloudflare Cloudflare Announces Unmetered DDOS Mitigation For All Plans

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, Sep 26, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Don't forget Cloudflare now also has unmetered rate limiting too https://blog.cloudflare.com/unmetered-ratelimiting/. Though from your analytics, looks like a lot of IPs involved in the attack.
    Not bad :)

     
  2. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    Yeah I utilize that too.

    But with 3.5 million request per second peak seems some just hit my server for few seconds until Cloudflare kick in.
     
  3. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
  4. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    :eek: Someone must love your site/forums :)
     
  5. duy13

    duy13 New Member

    16
    3
    3
    Oct 22, 2016
    California, USA
    Ratings:
    +16
    Local Time:
    5:12 PM
    1.13.x
    MariaDB 10.x
    I don't think your attacker is smart, he is making an impressive big attack but ineffective, if the attack still can't bypass the challenge then no matter how much it sends it won't be enough.
    Smart attackers only need 10 minutes to know if their hit is enough to take your site down, they'll move on if it crashes, they'll find another way if it doesn't. I'm just seeing an obstinate person sending a huge amount of disabling attacks to your site (seems like he's hoping cloudflare will remove you from the system if you don't use the 200usd/month enterprise plan)
    I still fear attackers with a very small resource but still able to bring down any website, they bypass every challenge we know: javascript, captcha...
     
  6. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    Those 2 attacks earlier, totaling 20 minutes, successfully reached my server for 2 minutes (every 1st minute during the attack hit my server), resulting in a total of 780 million requests and causing my server to hang or halt.
    But Cloudflare blocked the remaining 1.27 billion requests.

    I don't think they can totally bypass Cloudflare; I just think Cloudflare has a few seconds' delay when the attack is too strong.

    If someone can really bypass Cloudflare, a lot of sites will be in total trouble.
     
  7. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    I don't even use CF UAM, cause it's to problematic with legit users.
    I just utilize the custom firewall rules, rate limiting, caching, etc.
     
  8. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Cloudflare has had unmetered DDOS protection for a few years, even non-enterprise plans won't get kicked off anymore. But US$200/month is Business plan. Enterprise plan is at least US$3-5K/month with negotiable rates on billable usages/quotas etc :)

    But some attackers are finding other ways to disrupt i.e. if they know a Cloudflare protected site is using Cloudflare Workers, they will try to drive up the site's CF worker costs as it's billed on a per request basis (US$0.15-0.50/1 million) and/or try to hit the threshold on the CF worker free plan's 100K requests/day quota to knock a site offline. I've had such attacks for 8+ months where the attacker only made ~100K requests/day which coincides with CF worker free plan quota.

    +1 on that Cloudflare WAF/Firewall rules, rate limiting and even Transform and Cache rules when properly used will help :)

    Depends on the CF user's configuration though. Almost 90% of my paid consult clients who I work with for the first time have no Cloudflare Firewall/Rate Limiting rules setup at all for their site and only rely on the default CF WAF rules inherited on CF free plan or do not even touch the default CF WAF rules on Pro or higher plans. Not really sufficient for layer 7 application level attack mitigation :)
     
  9. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    With much changes on the cf firewall and rate limit, this time it didn't touch my server :)
    Still I'm not using UAM.

    upload_2022-11-5_16-55-30.png
     
  10. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Excellente :D
     
  11. duy13

    duy13 New Member

    16
    3
    3
    Oct 22, 2016
    California, USA
    Ratings:
    +16
    Local Time:
    5:12 PM
    1.13.x
    MariaDB 10.x
    In my experience a perfect L7 attack would look like this:
    1. Bypass cache, using random query 404 /asrgasrfdg /345fawsef /randomxxx... will cause the query to go straight to the backend and not be served by the cache, the backend (database server) will have to process and issue "404" result
    2. bypass ratelimit, with 1 ip the attacker will send packets below the ratelimit you set, he sends an unbelievably small number of packets and use of a large number of IPs
    3. bypass javascript check, using attack engine written in python, nodejs... use browser headless to handle javascript challenges
    4. bypass captcha, use the api sold from the "captcha solver farm" for example 2captcha.com. they solve almost all the captcha we know: reCAPTCHA V2-V3, hCaptcha, GeeTest CAPTCHA...

    Check netstat we only see about 500 to 1000con/s (too little to die:confused:), but a 20core server can die, because of cpu overload (PHP server, Mysql Server...):dead:
     
  12. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    Yes, they do this; I believe they have between 20 and 40 thousand distinct IP addresses at any given time.
    And they know which page isn't cached at all.
    This can be easily mitigated with CF rate limit rules and a CF custom firewall.
    My average number of HTTPS/443 connected visitors per second is 1,500 to 3,500, and 1,000 to 2,000 for web sockets.
    And I'm just using 6 CPU cores and 12 threads on a dedicated server :D.
    I'll share again here once the attacker resumes.
     
  13. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    I got a 3 minutes attack again today.

    upload_2022-11-19_9-22-35.png

    My custom firewall mitigated 2.2M and CF DDOS protection mitigated 1.6M.
     
  14. cloud9

    cloud9 Premium Member Premium Member

    431
    117
    43
    Oct 6, 2015
    England
    Ratings:
    +217
    Local Time:
    10:12 AM
    1.25.3
    10.6.x
    @rdan Are you on the free or paid plan ?
     
  15. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    Pro plan.
     
  16. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    4 minutes attack.
    upload_2022-11-23_2-18-20.png

    I think "Browser integrity check" and "Security Level" isn't that useful anymore.
     
  17. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    I believe if you have Firewall rule that overlap with BIC/Security Level, the Firewall rule takes priority so takes some of the count/numbers away from BIC/Security Level.

    upload_2022-11-23_7-15-51.png
     
  18. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    Yes, the CF security level is still good to have, but the "Browser integrity check" is prone to false positives, resulting in an immediate block rather than a request challenge.

    I have browser integrity checking disabled now because I had a user with a new Ubuntu Desktop install who was using the Edge browser and was falsely blocked by BIC a few days ago.
    and there's no way to enter the site.
     
  19. rdan

    rdan Well-Known Member

    5,446
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    6:12 PM
    Mainline
    10.2
    This is so true, and I'm curious how to bypass the PHP/MySQL backend, Nginx alone can't decide if it's a 200 or 404 response.

    This month, the low attacks I got still used 10–15 thousand unique IPs.
    Even if they only run 5 requests per IP per second, it will cause the site to crash.
    I have to disable some custom CF-managed challenges as they cause some issues for a few VPN users.

    Never ending battle :|
     
  20. eva2000

    eva2000 Administrator Staff Member

    54,529
    12,212
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,781
    Local Time:
    8:12 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Well depending on Cloudflare plan, rate limiting by response status count is a thing. Not perfect but better than nothing I guess.

    Cloudflare Free plan no such support and only supports Block action

    cf-rate-limiting-unmetered-no404-count-00.png

    Cloudflare Business supports counting by extra fields like response status code via custom count expression with other actions like Managed Challenge and supports applying rate limits against cached requests too.

    cf-business-rate-limiting-unmetered-404-count-00.png

    As does Cloudflare Enterprise with Advanced Rate Limiting feature supports counting by extra fields like response status code via custom count expression with other actions like Managed Challenge but also via different fields other than IP address i.e. Host name :)Also supports applying rate limits against cached requests too.

    cf-advanced-rate-limiting-unmetered-404-count-00.png