Learn about Centmin Mod LEMP Stack today
Register Now

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. pamamolf

    pamamolf Premium Member Premium Member

    3,774
    366
    83
    May 31, 2014
    Ratings:
    +706
    Local Time:
    7:45 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Maybe is time to have a decision about this and add it as default ?

    I would like to see it ready to use as default but with a comment # so if anyone want to use it to just uncomment the include line and restart Nginx.

    So we will avoid the all the custom edits to enable it....

    Thank you :)
     
    • Agree Agree x 1
  2. eva2000

    eva2000 Administrator Staff Member

    44,180
    10,071
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,567
    Local Time:
    2:45 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    you mean ultimate bad blocker version at Security - Nginx Ultimate Bad Block Blocker and not version in 1st post of this thread ? right now decision is the same ultimate bad blocker is up to end users to config as it automates modifications of nginx vhosts in a way Centmin Mod Nginx isn't natively structured to support out of the box which can potentially break Centmin Mod Nginx functionality and ability to restart Nginx if Nginx vhost is broken.

    As there is no way for me to anticipate and guess every possible incorrect auto modification of Nginx vhost and config files by Ultimate Bad Blocker, you have a situation in the future where Ultimate Bad Blocker changed automated modification could break 1000s of Centmin Mod Nginx installs overnight if it was included by default for folks who have no clue how Ultimate Bad Blocker was setup and their got-cha issues.

    Personally, as I am using Cloudflare I don't need to block all bots just ones I don't want via Cloudflare User Agent blocking rules

    cf-user-agent-blocks.png
     
    Last edited: Jul 12, 2019
  3. EckyBrazzz

    EckyBrazzz Active Member

    864
    180
    43
    Mar 28, 2018
    >>>>Click here<<<< i'm nearby......
    Ratings:
    +331
    Local Time:
    1:45 AM
    Latest
    Latest
    Yeah, that the only thing that you have to change when following the manual, it needs a little update;
     
  4. pamamolf

    pamamolf Premium Member Premium Member

    3,774
    366
    83
    May 31, 2014
    Ratings:
    +706
    Local Time:
    7:45 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok George i got your point !
     
  5. SkylineServers

    SkylineServers New Member

    3
    0
    1
    May 8, 2020
    Ratings:
    +0
    Local Time:
    11:45 PM
    Hello,

    Nice work on this, works quite well.
    I do have a question since I've been having some hard time making both zones work with different rates per second at the same time.

    How does one actually make it work so that 1 is also rate-limited not just option 2 and all those requests go through both either bota if set to 1 or botb zone if set to 2 respectively?

    Looking at config (with default 1; in the user_agent/bot_agent map. I skipped pasting here)
    map $bot_agent $bot_iplimit {
    0 "";
    1 "";
    2 $binary_remote_addr;
    }
    limit_conn_zone $bot_iplimit zone=bota_connlimit:10m;
    limit_req_zone $bot_iplimit zone=bota_reqlimitip:10m rate=100r/s;
    limit_conn_zone $bot_iplimit zone=botb_connlimit:10m;
    limit_req_zone $bot_iplimit zone=botb_reqlimitip:10m rate=10r/s;

    so 1 is not being rated limited and 2 is caught in botb zone.
    If I update bot_iplimit map and set like follows:

    map $bot_agent $bot_iplimit {
    0 "";
    1 $binary_remote_addr;
    2 $binary_remote_addr;
    }

    1 and 2 are both being rated limited however by the same one same zone, whichever has a lower rate of request per second set, regardless of the default set to 1 or 2, which kinda does make sense with 1 and being mapped above the same.

    Anyone got it working with both limit_req zones? Any thoughts?
    Thanks. Much appreciated.
     
  6. eva2000

    eva2000 Administrator Staff Member

    44,180
    10,071
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,567
    Local Time:
    2:45 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    For posting code or output from commands to keep the formatting, you might want to use CODE tags for code How to use forum BBCODE code tags :)

    by default botlimit.conf map setup is disabled for $bot_agent = 1 so commented out. The mapping enables limit_conn_zone and limit_req_zone usage while /usr/local/nginx/conf/blockbots.conf entries tell which zone name to use.
    Code (Text):
    # limits for googlebot and $bot_agent = 1
    #limit_conn_zone $bot_iplimit zone=bota_connlimit:16m;
    #limit_req_zone  $bot_iplimit zone=bota_reqlimitip:16m  rate=50r/s;
    # limits for $bot_agent = 2
    limit_conn_zone $bot_iplimit zone=botb_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=botb_reqlimitip:16m  rate=2r/s;
    

    when enabled via uncommenting the $bot_agent = 1 lines
    Code (Text):
    # limits for googlebot and $bot_agent = 1
    limit_conn_zone $bot_iplimit zone=bota_connlimit:16m;
    limit_req_zone  $bot_iplimit zone=bota_reqlimitip:16m  rate=50r/s;
    

    what controls which is used is the /usr/local/nginx/conf/blockbots.conf entries you add via include file in your nginx vhost which reference the zone name
    Code (Text):
    #######################################################################
    # add this to your nginx vhost domain's config file within
    # the location contexts you want to rate limit and/or bot
    # block with 403 permission denied error or change return 403
    # to return 444 to just drop the connection completely
    #limit_conn bota_connlimit 100;
    limit_conn botb_connlimit 10;
    #limit_req  zone=bota_reqlimitip burst=50;
    limit_req  zone=botb_reqlimitip burst=10;
    if ($bot_agent = '3') {
      return 444;
    }
    #######################################################################
    

    limits for googlebot and $bot_agent = 1 are commented out
    Code (Text):
    #limit_conn bota_connlimit 100;
    #limit_req  zone=bota_reqlimitip burst=50;

    and would need uncommenting to enable
    Code (Text):
    limit_conn bota_connlimit 100;
    limit_req  zone=bota_reqlimitip burst=50;
     
    • Like Like x 1
  7. SkylineServers

    SkylineServers New Member

    3
    0
    1
    May 8, 2020
    Ratings:
    +0
    Local Time:
    11:45 PM
    Thanks for the tip :)

    I understand it needs to be enabled first, so inside virtual-host config I enable the use of both bota and botb zones with
    Code (Text):
     server {
    ...             
                    limit_conn bota_connlimit 200;
                    limit_conn botb_connlimit 10;
                    limit_req zone=bota_reqlimitip burst=10;
                    limit_req zone=botb_reqlimitip burst=5;
                    if ($bot_agent = '3') {
                            return 403;
                    }
    

    and in nginx.conf
    Code (Text):
    http {
    ...
    map $http_user_agent $bot_agent {
      default                    1;
    ......
     
          map $bot_agent $bot_iplimit {
                0    "";
                1    "";
                2    $binary_remote_addr;
            }
            limit_conn_zone $bot_iplimit zone=bota_connlimit:15m;
            limit_req_zone $bot_iplimit zone=bota_reqlimitip:15m rate=50r/s;
            limit_conn_zone $bot_iplimit zone=botb_connlimit:15m;
            limit_req_zone $bot_iplimit zone=botb_reqlimitip:15m rate=3r/s;
    

    both bot_agent= 1 and 2 should be enabled right? But my findings in this configuration is that bot_agent=1 is not being rate-limited (not seeing bota zone references in the logs), can only see botb zone limits being hit for bot_agent = 2 connections.

    If I change bot_iplimit map to
    Code (Text):
          map $bot_agent $bot_iplimit {
                0    "";
                1    $binary_remote_addr;
                2    $binary_remote_addr;
            }
    

    Then bota zones do seem to actually be enabled however there seems to be an issue, while connections with bot_agent set to 1 (default 1; right) that should fall into bota zone, are showing up as in botb zone limits, (that one has a lower rate per second limit).

    Now if I change bota limits to be lower than botb (bot agent 1 has a lower limit than bot _agent 2 right other way around as before) as follows:
    Code (Text):
            limit_conn_zone $bot_iplimit zone=bota_connlimit:15m;
            limit_req_zone $bot_iplimit zone=bota_reqlimitip:15m rate=2r/s;
            limit_conn_zone $bot_iplimit zone=botb_connlimit:15m;
            limit_req_zone $bot_iplimit zone=botb_reqlimitip:15m rate=3r/s;
    

    Then I do see bota zone doing the limiting for bot_agent 1 as expected right? but also botb connections bot_agent = 2 are falling into this while they should be falling into botb zone, nothing is falling into botb zone anymore and appears bota is catching connections for bot_agnet = 2 as well.

    It appears that only one zone is being used at the time, whichever one has lower rate limit.

    I am unsure how Nginx should be configured to make use of rate limiting for both bot_agent = 1 to fall into bota zone limits and bot_agent = 2 into botb zone it doesn't appear to be the case in my tests, and only one zone can be used at the time, but I'd like to use both with different limits for different user agents.

    Suggestions much appreciated. Thanks.
     
  8. eva2000

    eva2000 Administrator Staff Member

    44,180
    10,071
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,567
    Local Time:
    2:45 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    that would be the correct mapping if you want bot_agent = 1 included in limiting

    but do you have example commands or user agents you are testing with ? might want to set a larger difference in rate limits than 2r/s and 3r/s as I don't think there that easy to differentiate between them.

    But I haven't honestly tested them much these days, I just move all bot management to Cloudflare to handle + Cloudflare Firewall/Rate Limiting :D