/ Register

Join the community today
Register Now

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

Tags:
Previous Thread Next Thread
Loading...
Page 6 of 8
  1. Apr 29, 2018 #101
    Jon Snow

    Jon Snow Active Member

    811
    163
    43
    Jun 30, 2017
    Ratings:
    +240
    Local Time:
    12:14 PM
    Nginx 1.13.9
    MariaDB 10.1.31
    They're still getting through:
    Code (Text):
    54.36.148.156 - - [28/Apr/2018:22:45:24 +0000] "GET /forum/threads/blahblah/page-3 HTTP/1.1" 200 8581 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

    I added a "3" to AhrefsBot. SemrushBot was set to 3 by default from your file contents but it's still getting through too. I restarted nginx after adding everything.

    A China IP was blocked though:
    Code (Text):
    IP address - - [28/Apr/2018:21:52:12 +0000] "GET /forum/threads/blahblah... HTTP/1.1" 404 17095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"

     
    Last edited: Apr 29, 2018
  2. Apr 29, 2018 #102
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Double check the timestamp in access logs versus system time as Centmin Mod by default uses buffered access log writes in memory for better performance and is flushed and written to disk once the default buffer memory size is full or when flush time is reached or until nginx is reload/restarted. So the entry in access log might have been an entry already in memory buffer prior to the block being in place.

    So simpling doing an nginx reload or restart before inspecting access log will ensure you have flushed the memory buffered entries and written them to access log first
    Code (Text):
    ngxreload

    or
    Code (Text):
    ngxrestart
     
  3. Apr 29, 2018 #103
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Also make sure include file include /usr/local/nginx/conf/blockbots.conf is within each location context you want it to apply to ie. if you have site off a subdirectory like /forum instead of public web root /, then need include file in both places
    Code (Text):
    location / {
  include /usr/local/nginx/conf/blockbots.conf;

 <your code>
}

location /forum {
  include /usr/local/nginx/conf/blockbots.conf;

 <your code>
}
     
  4. Apr 29, 2018 #104
    Jon Snow

    Jon Snow Active Member

    811
    163
    43
    Jun 30, 2017
    Ratings:
    +240
    Local Time:
    12:14 PM
    Nginx 1.13.9
    MariaDB 10.1.31
    Adding it to /forum worked. Thanks!
     
  5. Apr 29, 2018 #105
    Colin

    Colin Premium Member Premium Member

    191
    59
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +153
    Local Time:
    4:14 PM
    1.19.#
    MariaDB 10.1.#
    Just applied the ultimate bad-bot config - amazing :)

    Traded a stack of traffic spikes for a stack of 444 response states. Whilst everything coped under the bad bots, this gives me a little more headroom before working all weekend on a server upgrade :D So yet another value add!

    The only conn I've seen so far, is a near doubling of nginx ram., circa 238 to 476. I can cope with that give mysql isn't trying so hard any more :D and my request time is heading back towards the 0.5sec line :D
     
  6. Apr 30, 2018 #106
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    excellent to hear :) Folks sometimes estimate the value of having more memory on a system - it can be useful for stuff like this and/or caching as well as advanced CSF Firewall configurations like extending CSF Firewall blocklists Beta Branch - revise inc/csftweaks.inc :)
     
  7. Aug 2, 2018 #107
    Meirami

    Meirami Active Member

    154
    28
    28
    Dec 21, 2017
    Ratings:
    +63
    Local Time:
    6:14 PM
  8. Aug 2, 2018 #108
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    both versions work with or without IPSET support just that without IPSET = less performance and less number of total IPs you can block so if on OpenVZ with no IPSET support, you get less performance and less number of total IPs you can block.
     
  9. Aug 2, 2018 #109
    dcg

    dcg Member

    67
    24
    8
    Oct 17, 2015
    Florida, USA
    Ratings:
    +49
    Local Time:
    11:14 AM
    1.21.x
    10.3.x
    Are you running this on community.centminmod ?
     
  10. Aug 2, 2018 #110
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    not anymore since I now have Cloudflare in front of Nginx but i do have some bad bot user agents added to Cloudflare firewall too now
     
  11. Jan 12, 2019 #111
    Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    11:14 AM
    Hey @eva2000 every time I try to check I get the error that ngxtop command not found.

    Code:
    cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '11/Jan' | ngxtop --no-follow --group-by http_user_agent
    Code:
    -bash: ngxtop: command not found
     
  12. Jan 12, 2019 #112
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  13. Jan 12, 2019 #113
    Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    11:14 AM
    Nope. I'll have to do that. Thanks!
     
  14. Mar 22, 2019 #114
    Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    11:14 AM
    If behind cloudflare, should I remove block.conf also?
     
  15. Mar 22, 2019 #115
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    optional.. leaving them there just means cloudflare gives the blocked bot a cloudflare error page which is essentially what blocking at cloudflare level does anyway.
     
  16. Jun 12, 2019 #116
    Jimmy

    Jimmy Well-Known Member

    1,788
    390
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +990
    Local Time:
    11:14 AM
    Should the blockbots.conf be before or after the try_files line? I put mine below the block.conf which I commented out but a post on page 3 had it after the try files, just wanted clarification. Thanks!
     
    Last edited: Jun 12, 2019
  17. Jun 13, 2019 #117
    eva2000

    eva2000 Administrator Staff Member

    54,070
    12,176
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,734
    Local Time:
    2:14 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    either is fine i usually put include files before though myself
     
  18. Jun 14, 2019 #118
    fabianski

    fabianski Member

    102
    14
    18
    Feb 20, 2019
    Ratings:
    +36
    Local Time:
    1:14 PM
    I followed the steps to install the tutorial in github but it is not yet blocking


    Code:
    curl -I https://domain.org -e http://100dollars-seo.com
HTTP/1.1 200 OK
Date: Thu, 13 Jun 2019 18:21:09 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 120667
Last-Modified: Thu, 13 Jun 2019 17:10:43 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5d028393-1d75b"
Server: nginx
Vary: Accept-Encoding, Cookie
Cache-Control: no-cache, no-store, must-revalidate
X-Rocket-Nginx-Serving-Static: Yes
Strict-Transport-Security: max-age=31536000
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Accept-Ranges: bytes
    Install log
    botblocker-nginx-settings.conf
    virtual.conf
    nginx.conf
    domain.ssl.conf
     
  19. Jun 15, 2019 #119
    EckyBrazzz

    EckyBrazzz Active Member

    916
    189
    43
    Mar 28, 2018
    >>>>Click here<<<< i'm nearby......
    Ratings:
    +362
    Local Time:
    1:14 PM
    Latest
    Latest
    Noticed you removed https://pastebin.com/JyNE7rCc
    Better to use a private gist, it's free and you have more space to post

    But I guess you forgot to add this to your domain.ssl.conf

    Code (Text):
      # if using the ultimate badbot blocker guide after testing
  # https://community.centminmod.com/threads/nginx-ultimate-bad-block-blocker.14413/
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
     
    Last edited: Jun 15, 2019
  20. Jun 16, 2019 #120
    fabianski

    fabianski Member

    102
    14
    18
    Feb 20, 2019
    Ratings:
    +36
    Local Time:
    1:14 PM
    This code was automatically added after following the installation tutorial
     
Previous Thread Next Thread
Loading...
Page 6 of 8

.