Security Blocking bad or aggressive bots

Maybe is time to have a decision about this and add it as default ?

I would like to see it ready to use as default but with a comment # so if anyone want to use it to just uncomment the include line and restart Nginx.

So we will avoid the all the custom edits to enable it....

Thank you

---

 

Yeah, that the only thing that you have to change when following the manual, it needs a little update;

 

Ok George i got your point !

 

Hello,

Nice work on this, works quite well.
I do have a question since I've been having some hard time making both zones work with different rates per second at the same time.

How does one actually make it work so that 1 is also rate-limited not just option 2 and all those requests go through both either bota if set to 1 or botb zone if set to 2 respectively?

Looking at config (with default 1; in the user_agent/bot_agent map. I skipped pasting here)
map $bot_agent $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
}
limit_conn_zone $bot_iplimit zone=bota_connlimit:10m;
limit_req_zone $bot_iplimit zone=bota_reqlimitip:10m rate=100r/s;
limit_conn_zone $bot_iplimit zone=botb_connlimit:10m;
limit_req_zone $bot_iplimit zone=botb_reqlimitip:10m rate=10r/s;

so 1 is not being rated limited and 2 is caught in botb zone.
If I update bot_iplimit map and set like follows:

map $bot_agent $bot_iplimit {
0 "";
1 $binary_remote_addr;
2 $binary_remote_addr;
}

1 and 2 are both being rated limited however by the same one same zone, whichever has a lower rate of request per second set, regardless of the default set to 1 or 2, which kinda does make sense with 1 and being mapped above the same.

Anyone got it working with both limit_req zones? Any thoughts?
Thanks. Much appreciated.

 

Thanks for the tip

I understand it needs to be enabled first, so inside virtual-host config I enable the use of both bota and botb zones with

Code (Text):

 server {
...             
                limit_conn bota_connlimit 200;
                limit_conn botb_connlimit 10;
                limit_req zone=bota_reqlimitip burst=10;
                limit_req zone=botb_reqlimitip burst=5;
                if ($bot_agent = '3') {
                        return 403;
                }

and in nginx.conf

Code (Text):

http {
...
map $http_user_agent $bot_agent {
  default                    1;
......
 
      map $bot_agent $bot_iplimit {
            0    "";
            1    "";
            2    $binary_remote_addr;
        }
        limit_conn_zone $bot_iplimit zone=bota_connlimit:15m;
        limit_req_zone $bot_iplimit zone=bota_reqlimitip:15m rate=50r/s;
        limit_conn_zone $bot_iplimit zone=botb_connlimit:15m;
        limit_req_zone $bot_iplimit zone=botb_reqlimitip:15m rate=3r/s;

both bot_agent= 1 and 2 should be enabled right? But my findings in this configuration is that bot_agent=1 is not being rate-limited (not seeing bota zone references in the logs), can only see botb zone limits being hit for bot_agent = 2 connections.

If I change bot_iplimit map to

Code (Text):

      map $bot_agent $bot_iplimit {
            0    "";
            1    $binary_remote_addr;
            2    $binary_remote_addr;
        }

Then bota zones do seem to actually be enabled however there seems to be an issue, while connections with bot_agent set to 1 (default 1; right) that should fall into bota zone, are showing up as in botb zone limits, (that one has a lower rate per second limit).

Now if I change bota limits to be lower than botb (bot agent 1 has a lower limit than bot _agent 2 right other way around as before) as follows:

Code (Text):

        limit_conn_zone $bot_iplimit zone=bota_connlimit:15m;
        limit_req_zone $bot_iplimit zone=bota_reqlimitip:15m rate=2r/s;
        limit_conn_zone $bot_iplimit zone=botb_connlimit:15m;
        limit_req_zone $bot_iplimit zone=botb_reqlimitip:15m rate=3r/s;

Then I do see bota zone doing the limiting for bot_agent 1 as expected right? but also botb connections bot_agent = 2 are falling into this while they should be falling into botb zone, nothing is falling into botb zone anymore and appears bota is catching connections for bot_agnet = 2 as well.

It appears that only one zone is being used at the time, whichever one has lower rate limit.

I am unsure how Nginx should be configured to make use of rate limiting for both bot_agent = 1 to fall into bota zone limits and bot_agent = 2 into botb zone it doesn't appear to be the case in my tests, and only one zone can be used at the time, but I'd like to use both with different limits for different user agents.

Suggestions much appreciated. Thanks.