Learn about Centmin Mod LEMP Stack today
Become a Member

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. Jon Snow

    Jon Snow Active Member

    390
    61
    28
    Jun 30, 2017
    Ratings:
    +92
    Local Time:
    5:36 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    They're still getting through:
    Code (Text):
    54.36.148.156 - - [28/Apr/2018:22:45:24 +0000] "GET /forum/threads/blahblah/page-3 HTTP/1.1" 200 8581 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

    I added a "3" to AhrefsBot. SemrushBot was set to 3 by default from your file contents but it's still getting through too. I restarted nginx after adding everything.

    A China IP was blocked though:
    Code (Text):
    IP address - - [28/Apr/2018:21:52:12 +0000] "GET /forum/threads/blahblah... HTTP/1.1" 404 17095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"

     
    Last edited: Apr 29, 2018
  2. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Double check the timestamp in access logs versus system time as Centmin Mod by default uses buffered access log writes in memory for better performance and is flushed and written to disk once the default buffer memory size is full or when flush time is reached or until nginx is reload/restarted. So the entry in access log might have been an entry already in memory buffer prior to the block being in place.

    So simpling doing an nginx reload or restart before inspecting access log will ensure you have flushed the memory buffered entries and written them to access log first
    Code (Text):
    ngxreload
    

    or
    Code (Text):
    ngxrestart
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Also make sure include file include /usr/local/nginx/conf/blockbots.conf is within each location context you want it to apply to ie. if you have site off a subdirectory like /forum instead of public web root /, then need include file in both places
    Code (Text):
    location / {
      include /usr/local/nginx/conf/blockbots.conf;
    
     <your code>
    }
    
    location /forum {
      include /usr/local/nginx/conf/blockbots.conf;
    
     <your code>
    }
    
     
  4. Jon Snow

    Jon Snow Active Member

    390
    61
    28
    Jun 30, 2017
    Ratings:
    +92
    Local Time:
    5:36 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Adding it to /forum worked. Thanks!
     
    • Like Like x 1
  5. Colin

    Colin Active Member

    139
    41
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +102
    Local Time:
    9:36 AM
    1.13.#
    MariaDB 10.1.#
    Just applied the ultimate bad-bot config - amazing :)

    Traded a stack of traffic spikes for a stack of 444 response states. Whilst everything coped under the bad bots, this gives me a little more headroom before working all weekend on a server upgrade :D So yet another value add!

    The only conn I've seen so far, is a near doubling of nginx ram., circa 238 to 476. I can cope with that give mysql isn't trying so hard any more :D and my request time is heading back towards the 0.5sec line :D
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    excellent to hear :) Folks sometimes estimate the value of having more memory on a system - it can be useful for stuff like this and/or caching as well as advanced CSF Firewall configurations like extending CSF Firewall blocklists Beta Branch - revise inc/csftweaks.inc :)
     
  7. Meirami

    Meirami Member

    124
    14
    18
    Dec 21, 2017
    Ratings:
    +38
    Local Time:
    11:36 AM
  8. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    both versions work with or without IPSET support just that without IPSET = less performance and less number of total IPs you can block so if on OpenVZ with no IPSET support, you get less performance and less number of total IPs you can block.
     
    • Informative Informative x 1
  9. dcg

    dcg Member

    53
    18
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +34
    Local Time:
    4:36 AM
    1.15.x
    10.2.x
    Are you running this on community.centminmod ?
     
  10. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    not anymore since I now have Cloudflare in front of Nginx but i do have some bad bot user agents added to Cloudflare firewall too now
     
    • Informative Informative x 1
  11. Jimmy

    Jimmy Premium Member Premium Member

    1,444
    297
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +732
    Local Time:
    4:36 AM
    1.13.x
    MariaDB 10.1.x
    Hey @eva2000 every time I try to check I get the error that ngxtop command not found.

    Code:
    cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '11/Jan' | ngxtop --no-follow --group-by http_user_agent
    
    Code:
    -bash: ngxtop: command not found
     
  12. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  13. Jimmy

    Jimmy Premium Member Premium Member

    1,444
    297
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +732
    Local Time:
    4:36 AM
    1.13.x
    MariaDB 10.1.x
    Nope. I'll have to do that. Thanks!
     
..