/ Register

Get the most out of your Centmin Mod LEMP stack
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

Tags:
Previous Thread Next Thread
Loading...
Page 5 of 8
  1. Jun 24, 2017 #81
    pamamolf

    pamamolf Premium Member Premium Member

    4,077
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    10:54 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Is block.conf something that is not needed to use anymore as botlimit.conf can do that also?

    Is it also possible to ban the ip of the user with that bad agent?

    Thanks
     
  2. Jun 24, 2017 #82
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. Jun 24, 2017 #83
    pamamolf

    pamamolf Premium Member Premium Member

    4,077
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    10:54 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    Ok so in my opinion there is no need to have at block.conf the ## Block user agents block and transfer the user agents to the botlimit.conf as there is more easy to control and with many options :)

    In my opinion :) Also that will not confuse users with two files to handle user agents .....

    Thanks !
     
  4. Jun 25, 2017 #84
    pamamolf

    pamamolf Premium Member Premium Member

    4,077
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    10:54 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    It seems that Netsparker doesn't use the Netsparker as user agent and instead they add on the link the Netsparker tag....

    So no idea how to totally block that:

    Code:
    "GET /public/Netsparkere93a37d429134af38fd61a4a0fb02f63/ HTTP/1.1" 200 15268 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.16 Safari/537.36"
    All requests has on the link the Netsparker tag....
     
  5. Aug 25, 2017 #85
    rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    4:54 PM
    Mainline
    10.2
    What I got now:
    upload_2017-8-25_21-1-18.png
    116K bots redirected.


    By the way, any idea how to block rssing.com from scraping RSS content?
    Edit: I think it's RSSingBot.
     
  6. Aug 25, 2017 #86
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    add user agent to blocking config
     
  7. Oct 25, 2017 #87
    rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    4:54 PM
    Mainline
    10.2
    Any way we can set this to off/do not log at all on error log?
     
  8. Oct 25, 2017 #88
    pamamolf

    pamamolf Premium Member Premium Member

    4,077
    427
    83
    May 31, 2014
    Ratings:
    +833
    Local Time:
    10:54 AM
    Nginx-1.25.x
    MariaDB 10.3.x
    I need that function on :)

    As it is good for fail2ban also....

    Also it is good to know when it is activated or in general to have that info at the logs..... !
     
  9. Oct 25, 2017 #89
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    that doesn't control logging.. you control logging of those by NOT referencing it in your nginx vhosts at all i.e. old discussion Nginx - # limit_conn limit_per_ip 16; ssi on;

    centmin mod comments out connection limit out of the box in vhosts
    Code (Text):
      # limit_conn limit_per_ip 16;

    so only time it's logged would be if you are using connection limiting within your nginx vhost
     
  10. Oct 26, 2017 #90
    rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    4:54 PM
    Mainline
    10.2
    Yes I was using this config for Bad BOT Limiting/Blocking.

    But my domain error_log file was filled with this.
    I want to eliminate it to fucos on some other important logs.
     
  11. Oct 26, 2017 #91
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  12. Apr 3, 2018 #92
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  13. Apr 4, 2018 #93
    BamaStangGuy

    BamaStangGuy Active Member

    668
    192
    43
    May 25, 2014
    Ratings:
    +272
    Local Time:
    2:54 AM
  14. Apr 4, 2018 #94
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yes that's expected as Cloudflare isn't able to connect to Centmin Mod Nginx servers since they return 444 status and closes the connection. Doesn't really matter what status code Cloudflare returns as the point is to not let those bad bots to even hit your Centmin Mod Nginx server, so in case of Cloudflare the bad bot's access stops on Cloudflare edge with 520 status code.
     
  15. Apr 28, 2018 #95
    Jon Snow

    Jon Snow Active Member

    827
    169
    43
    Jun 30, 2017
    Ratings:
    +253
    Local Time:
    4:54 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    What about OVH data scrapers?
     
  16. Apr 28, 2018 #96
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    If they have their own user agents, just add it to the list to be either blocked or rate limited
     
  17. Apr 29, 2018 #97
    Jon Snow

    Jon Snow Active Member

    827
    169
    43
    Jun 30, 2017
    Ratings:
    +253
    Local Time:
    4:54 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Code (Text):
    54.36.149.60 - - [28/Apr/2018:06:40:24 +0000] "GET /blahblahblah/ HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

    So I guess it'll be blocked?
     
  18. Apr 29, 2018 #98
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    shows 301 permanent redirect status so not blocked which would be 403 permission denied or 444 return empty response or 503 service not available.
     
  19. Apr 29, 2018 #99
    Jon Snow

    Jon Snow Active Member

    827
    169
    43
    Jun 30, 2017
    Ratings:
    +253
    Local Time:
    4:54 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    I'm not using the file in my config yet so they're getting through. Asking before if this will block these OVH bots.
     
  20. Apr 29, 2018 #100
    eva2000

    eva2000 Administrator Staff Member

    54,336
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    6:54 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yes if you setup this thread's outlined badbot config, it will block that useragent = AhrefsBot/5.2
     
Previous Thread Next Thread
Loading...
Page 5 of 8

.