Learn about Centmin Mod LEMP Stack today
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. pamamolf

    pamamolf Premium Member Premium Member

    3,581
    345
    83
    May 31, 2014
    Ratings:
    +667
    Local Time:
    7:52 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Is block.conf something that is not needed to use anymore as botlimit.conf can do that also?

    Is it also possible to ban the ip of the user with that bad agent?

    Thanks
     
  2. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    • Like Like x 1
  3. pamamolf

    pamamolf Premium Member Premium Member

    3,581
    345
    83
    May 31, 2014
    Ratings:
    +667
    Local Time:
    7:52 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok so in my opinion there is no need to have at block.conf the ## Block user agents block and transfer the user agents to the botlimit.conf as there is more easy to control and with many options :)

    In my opinion :) Also that will not confuse users with two files to handle user agents .....

    Thanks !
     
    • Agree Agree x 1
  4. pamamolf

    pamamolf Premium Member Premium Member

    3,581
    345
    83
    May 31, 2014
    Ratings:
    +667
    Local Time:
    7:52 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    It seems that Netsparker doesn't use the Netsparker as user agent and instead they add on the link the Netsparker tag....

    So no idea how to totally block that:

    Code:
    "GET /public/Netsparkere93a37d429134af38fd61a4a0fb02f63/ HTTP/1.1" 200 15268 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.16 Safari/537.36"
    All requests has on the link the Netsparker tag....
     
  5. rdan

    rdan Well-Known Member

    4,741
    1,144
    113
    May 25, 2014
    Ratings:
    +1,711
    Local Time:
    1:52 PM
    Mainline
    10.2
    What I got now:
    upload_2017-8-25_21-1-18.png
    116K bots redirected.


    By the way, any idea how to block rssing.com from scraping RSS content?
    Edit: I think it's RSSingBot.
     
  6. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    add user agent to blocking config
     
  7. rdan

    rdan Well-Known Member

    4,741
    1,144
    113
    May 25, 2014
    Ratings:
    +1,711
    Local Time:
    1:52 PM
    Mainline
    10.2
    Any way we can set this to off/do not log at all on error log?
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    3,581
    345
    83
    May 31, 2014
    Ratings:
    +667
    Local Time:
    7:52 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    I need that function on :)

    As it is good for fail2ban also....

    Also it is good to know when it is activated or in general to have that info at the logs..... !
     
  9. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    that doesn't control logging.. you control logging of those by NOT referencing it in your nginx vhosts at all i.e. old discussion Nginx - # limit_conn limit_per_ip 16; ssi on;

    centmin mod comments out connection limit out of the box in vhosts
    Code (Text):
      # limit_conn limit_per_ip 16;
    

    so only time it's logged would be if you are using connection limiting within your nginx vhost
     
  10. rdan

    rdan Well-Known Member

    4,741
    1,144
    113
    May 25, 2014
    Ratings:
    +1,711
    Local Time:
    1:52 PM
    Mainline
    10.2
    Yes I was using this config for Bad BOT Limiting/Blocking.

    But my domain error_log file was filled with this.
    I want to eliminate it to fucos on some other important logs.
     
  11. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    • Like Like x 1
  12. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  13. BamaStangGuy

    BamaStangGuy Active Member

    608
    179
    43
    May 25, 2014
    Ratings:
    +244
    Local Time:
    11:52 PM
  14. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yes that's expected as Cloudflare isn't able to connect to Centmin Mod Nginx servers since they return 444 status and closes the connection. Doesn't really matter what status code Cloudflare returns as the point is to not let those bad bots to even hit your Centmin Mod Nginx server, so in case of Cloudflare the bad bot's access stops on Cloudflare edge with 520 status code.
     
  15. Jon Snow

    Jon Snow Active Member

    452
    66
    28
    Jun 30, 2017
    Ratings:
    +104
    Local Time:
    1:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    What about OVH data scrapers?
     
  16. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    If they have their own user agents, just add it to the list to be either blocked or rate limited
     
  17. Jon Snow

    Jon Snow Active Member

    452
    66
    28
    Jun 30, 2017
    Ratings:
    +104
    Local Time:
    1:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Code (Text):
    54.36.149.60 - - [28/Apr/2018:06:40:24 +0000] "GET /blahblahblah/ HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

    So I guess it'll be blocked?
     
  18. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    shows 301 permanent redirect status so not blocked which would be 403 permission denied or 444 return empty response or 503 service not available.
     
  19. Jon Snow

    Jon Snow Active Member

    452
    66
    28
    Jun 30, 2017
    Ratings:
    +104
    Local Time:
    1:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    I'm not using the file in my config yet so they're getting through. Asking before if this will block these OVH bots.
     
  20. eva2000

    eva2000 Administrator Staff Member

    42,359
    9,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,741
    Local Time:
    3:52 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yes if you setup this thread's outlined badbot config, it will block that useragent = AhrefsBot/5.2