Security Blocking bad or aggressive bots

Hi i've got a question:
Nothing stops the bot from faking the agent right? He can say it's google bot and bypass the block, can't it?

---

 

Hi George, any particular reason to block Majestic?
I participate/help in such project by crawling and we are no evil. Is bandwidth consumption that sets you back?
Also If you like google to crawl your website I'm aware of people who fake such user agent to scrape data and fake browser also.
With "us" you can check if the bot that pretends to say is legit or not.

BTW: I'm about to reach 7 PB of backlinks crawled since 2008 ( been crawling since then)

EDIT: I've not read the full thread but saw majestic being listed. Forgot to mention we obey robots.txt so if you really don't want us it's pretty simple.

 

I've just edited my post. But do you happen to block google bot for example? ( just curious and to try to understand why people block us)

 

Cool! The software I have installed on my server allows to "block" bots from contaminating visitor data, but this is more preferable because you can block them from crawling your sites in the first place

 

I tried implementing this behind Cloudflare, and rate limiting the bots causes Cloudflare to show multiple legitimate visitors errors and showing the site is offline (they see the response codes and mark the site offline).

 

But maximum for map_hash_bucket_size is only 128.
http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size

 

I redirect bad bots to my FB page .
Instead of:

Code:

if ($bot_agent = '3') {
  return 444;
}

So a legit user can report or PM me on FB if they will encounter it instead of dropping it directly.

Code:

if ($bot_agent = '3') {
###return 444;
return https://www.facebook.com/myfbpage;
}

 

I use this extension: https://chrome.google.com/webstore/...ke?utm_source=chrome-app-launcher-info-dialog

and I manually added custom Bots to test (level 2 and level 3).



And I can confirm @eva2000 is using different config on this forum vs the config posted on first post .

 

I created a specific note for it .
Phrase credits to Google

 

Another one that we must have on block bots is Netsparker...

I think this is the user agent:

Code:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; Netsparker)

So can we use this one?

Code:

    if ($http_user_agent ~ "Netsparker") {
        set $block_user_agents 1;
    }

 

So is it better to use:

Code:

botlimit.conf

or

Code:

block.conf

I think we need to totally block such requests and not just limit them ....