/ Register

Learn about Centmin Mod LEMP Stack today
Become a Member

What do you use to fight bots, scrapper and ddos

Discussion in 'System Administration' started by Jay Chen, Apr 22, 2025.

Previous Thread Next Thread
Loading...
  1. Apr 22, 2025 #1
    Jay Chen

    Jay Chen Active Member

    184
    60
    28
    Sep 10, 2017
    Ratings:
    +116
    Local Time:
    1:24 AM
    My server was recently brought offline due to a very aggressive scrappers, so I implemented a rate limiting on the Cloudflare side.

    Just curious to see what other people are doing.
     
  2. Apr 22, 2025 #2
    eva2000

    eva2000 Administrator Staff Member

    55,458
    12,257
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,841
    Local Time:
    3:24 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Cloudflare Pro, Business or Enterprise based Cloudflare WAF, Transform and Rate Limit Rules FTW :D. Free plan doesn't have WAF analytics detailed enough to help you investigate bad actors and you don't have enough quota rules to cover all bad actors, you really need Pro, Business and if you can afford Enterprise plans.

    For scrappers i.e. AI bots, also check out Cloudflare AI Audit Start auditing and controlling the AI models accessing your content

    Cloudflare Enterprise plans also have Bot Management Enterprise that highly integrated into Cloudflare WAF Bot Management · Cloudflare bot solutions docs unlike Pro and Business plans.
     
  3. Apr 22, 2025 #3
    buik

    buik “The best traveler is one without a camera.”

    2,037
    527
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,684
    Local Time:
    7:24 AM
    When budget is not adequate and you are stuck with a free Cloudflare package.
    There are still some options available.

    For example. You could use the free Cloudflare option to block AI bots, scrapers and crawlers. Only decent services with a proper reputation are then permitted.

    You could also block ASN networks with the free package. If the above does not work adequately.

    I did help out at a volunteer website once. Blocking AWS and Azure (where most of the boredom is). Ensures right away that the most aggressive scrappers and bots are excluded.

    Real visitors do not use an AWS or Azure internet line to visit website's, so these visitors are not excluded. And if in doubt or the golden mean, block ASN's and place a challenge to allow any legetime visitors who slip through, to still able to visit the site.
     
    Last edited: Apr 22, 2025
  4. Apr 23, 2025 #4
    Jay Chen

    Jay Chen Active Member

    184
    60
    28
    Sep 10, 2017
    Ratings:
    +116
    Local Time:
    1:24 AM
    Can't afford to put all sites under Pro, most of them don't generate money, they all hobby projects. Will look into putting some of them under Pro and see how it goes. Enterprise is too expensive for me.
     
  5. Apr 23, 2025 #5
    Jay Chen

    Jay Chen Active Member

    184
    60
    28
    Sep 10, 2017
    Ratings:
    +116
    Local Time:
    1:24 AM
    Will look into the free Cloudflare option to block AI bots and also blocking ASN networks. Lots of reading and research to do over the weekend.
     
Previous Thread Next Thread
Loading...

.