Join the community today
Register Now

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. Jon Snow

    Jon Snow Active Member

    423
    64
    28
    Jun 30, 2017
    Ratings:
    +100
    Local Time:
    9:35 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    They're still getting through:
    Code (Text):
    54.36.148.156 - - [28/Apr/2018:22:45:24 +0000] "GET /forum/threads/blahblah/page-3 HTTP/1.1" 200 8581 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

    I added a "3" to AhrefsBot. SemrushBot was set to 3 by default from your file contents but it's still getting through too. I restarted nginx after adding everything.

    A China IP was blocked though:
    Code (Text):
    IP address - - [28/Apr/2018:21:52:12 +0000] "GET /forum/threads/blahblah... HTTP/1.1" 404 17095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"
     
    Last edited: Apr 29, 2018
  2. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Double check the timestamp in access logs versus system time as Centmin Mod by default uses buffered access log writes in memory for better performance and is flushed and written to disk once the default buffer memory size is full or when flush time is reached or until nginx is reload/restarted. So the entry in access log might have been an entry already in memory buffer prior to the block being in place.

    So simpling doing an nginx reload or restart before inspecting access log will ensure you have flushed the memory buffered entries and written them to access log first
    Code (Text):
    ngxreload
    

    or
    Code (Text):
    ngxrestart
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Also make sure include file include /usr/local/nginx/conf/blockbots.conf is within each location context you want it to apply to ie. if you have site off a subdirectory like /forum instead of public web root /, then need include file in both places
    Code (Text):
    location / {
      include /usr/local/nginx/conf/blockbots.conf;
    
     <your code>
    }
    
    location /forum {
      include /usr/local/nginx/conf/blockbots.conf;
    
     <your code>
    }
    
     
  4. Jon Snow

    Jon Snow Active Member

    423
    64
    28
    Jun 30, 2017
    Ratings:
    +100
    Local Time:
    9:35 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Adding it to /forum worked. Thanks!
     
    • Like Like x 1
  5. Colin

    Colin Active Member

    139
    41
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +102
    Local Time:
    1:35 PM
    1.13.#
    MariaDB 10.1.#
    Just applied the ultimate bad-bot config - amazing :)

    Traded a stack of traffic spikes for a stack of 444 response states. Whilst everything coped under the bad bots, this gives me a little more headroom before working all weekend on a server upgrade :D So yet another value add!

    The only conn I've seen so far, is a near doubling of nginx ram., circa 238 to 476. I can cope with that give mysql isn't trying so hard any more :D and my request time is heading back towards the 0.5sec line :D
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    excellent to hear :) Folks sometimes estimate the value of having more memory on a system - it can be useful for stuff like this and/or caching as well as advanced CSF Firewall configurations like extending CSF Firewall blocklists Beta Branch - revise inc/csftweaks.inc :)
     
  7. Meirami

    Meirami Member

    128
    15
    18
    Dec 21, 2017
    Ratings:
    +41
    Local Time:
    3:35 PM
  8. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    both versions work with or without IPSET support just that without IPSET = less performance and less number of total IPs you can block so if on OpenVZ with no IPSET support, you get less performance and less number of total IPs you can block.
     
    • Informative Informative x 1
  9. dcg

    dcg Member

    53
    18
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +34
    Local Time:
    8:35 AM
    1.15.x
    10.2.x
    Are you running this on community.centminmod ?
     
  10. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    not anymore since I now have Cloudflare in front of Nginx but i do have some bad bot user agents added to Cloudflare firewall too now
     
    • Informative Informative x 1
  11. Jimmy

    Jimmy Premium Member Premium Member

    1,553
    321
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +796
    Local Time:
    8:35 AM
    1.15.x
    MariaDB 10.3.x
    Hey @eva2000 every time I try to check I get the error that ngxtop command not found.

    Code:
    cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '11/Jan' | ngxtop --no-follow --group-by http_user_agent
    
    Code:
    -bash: ngxtop: command not found
     
  12. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Informative Informative x 1
  13. Jimmy

    Jimmy Premium Member Premium Member

    1,553
    321
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +796
    Local Time:
    8:35 AM
    1.15.x
    MariaDB 10.3.x
    Nope. I'll have to do that. Thanks!
     
  14. Jimmy

    Jimmy Premium Member Premium Member

    1,553
    321
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +796
    Local Time:
    8:35 AM
    1.15.x
    MariaDB 10.3.x
    If behind cloudflare, should I remove block.conf also?
     
    • Like Like x 1
  15. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    optional.. leaving them there just means cloudflare gives the blocked bot a cloudflare error page which is essentially what blocking at cloudflare level does anyway.
     
    • Informative Informative x 1
  16. Jimmy

    Jimmy Premium Member Premium Member

    1,553
    321
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +796
    Local Time:
    8:35 AM
    1.15.x
    MariaDB 10.3.x
    Should the blockbots.conf be before or after the try_files line? I put mine below the block.conf which I commented out but a post on page 3 had it after the try files, just wanted clarification. Thanks!
     
    Last edited: Jun 12, 2019
  17. eva2000

    eva2000 Administrator Staff Member

    40,276
    8,926
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,746
    Local Time:
    10:35 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    either is fine i usually put include files before though myself
     
    • Like Like x 1
  18. fabianski

    fabianski Member

    74
    7
    8
    Feb 20, 2019
    Brazil
    Ratings:
    +22
    Local Time:
    9:35 AM
    I followed the steps to install the tutorial in github but it is not yet blocking


    Code:
    curl -I https://domain.org -e http://100dollars-seo.com
    HTTP/1.1 200 OK
    Date: Thu, 13 Jun 2019 18:21:09 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 120667
    Last-Modified: Thu, 13 Jun 2019 17:10:43 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5d028393-1d75b"
    Server: nginx
    Vary: Accept-Encoding, Cookie
    Cache-Control: no-cache, no-store, must-revalidate
    X-Rocket-Nginx-Serving-Static: Yes
    Strict-Transport-Security: max-age=31536000
    X-Frame-Options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Accept-Ranges: bytes
    
    Install log
    botblocker-nginx-settings.conf
    virtual.conf
    nginx.conf
    domain.ssl.conf
     
  19. EckyBrazzz

    EckyBrazzz Active Member

    391
    73
    28
    Mar 28, 2018
    Brazil
    Ratings:
    +143
    Local Time:
    9:35 AM
    1.17.x
    10.3.x
    Noticed you removed https://pastebin.com/JyNE7rCc
    Better to use a private gist, it's free and you have more space to post

    But I guess you forgot to add this to your domain.ssl.conf

    Code (Text):
      # if using the ultimate badbot blocker guide after testing
      # https://community.centminmod.com/threads/nginx-ultimate-bad-block-blocker.14413/
      include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
      include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
    
     
    Last edited: Jun 15, 2019
    • Informative Informative x 1
  20. fabianski

    fabianski Member

    74
    7
    8
    Feb 20, 2019
    Brazil
    Ratings:
    +22
    Local Time:
    9:35 AM
    This code was automatically added after following the installation tutorial
     
..