Join the community today
Register Now

Sysadmin Not blocking bad bots

Discussion in 'System Administration' started by gawk, Apr 2, 2018.

  1. gawk

    gawk New Member

    12
    7
    3
    Feb 24, 2018
    Ratings:
    +9
    Local Time:
    5:07 AM
    nginx-1.13.9
    MariaDB 10.3.4
    • CentOS Version: CentOS 7 64-bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.13.9
    • PHP Version Installed: 7.1.14
    • MariaDB MySQL Version Installed: 10.1.31
    • When was last time updated Centmin Mod code base? : Today
    • Persistent Config: Yes

    Hi,

    I edited my nginx.conf and domain.conf file to include the botlimit.conf and blockbots.conf files. This worked for some time, but today I tested different user agents with curl, and none of them gave a 444 error on the request (despite me using "ahrefsbot" for example as my user agent).

    How can I troubleshoot the problem?

     
  2. eva2000

    eva2000 Administrator Staff Member

    54,906
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    2:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    I assume you mean the bad bot blocking and rate limiting implementation outlined here ?

    Using BBCODE CODE tags post contents of your config files for
    • /usr/local/nginx/conf/conf.d/nginx.conf
    • /usr/local/nginx/conf/conf.d/domain.com.conf and/or /usr/local/nginx/conf/conf.d/domain.com.ssl.conf
    • /usr/local/nginx/conf/botlimit.conf
    • /usr/local/nginx/conf/blockbots.conf
    You can mask your real domain name if you want in text editor just do a mass search and replace in the files.

    Also post examples of how and what command you use to test and the output from those commands. Again use CODE tags for output
     
  3. wmtech

    wmtech Active Member

    175
    44
    28
    Jul 22, 2017
    Ratings:
    +132
    Local Time:
    5:07 AM
  4. eva2000

    eva2000 Administrator Staff Member

    54,906
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    2:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Thanks for sharing (y)

    edit: just had a closer look at this and very well scripted and developed ! I couldn't have done a better job myself if I had time so definitely checking it out as you have to change their default assumed paths as Centmin Mod uses different default nginx directory structure and there's a few manual changes needed to Centmin Mod nginx config files to get it working :)
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,906
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    2:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    where are you checking for 444 error ? in your access.log i.e. /home/nginx/domains/domain1.com/log/access.log ? Centmin Mod by default buffers log writes to memory for a certain amount of entries until the buffer is full and then is flushed and written to disk in access.log so you may need to restart nginx to flush the buffer to see access.log entry for 444 status
     
  6. eva2000

    eva2000 Administrator Staff Member

    54,906
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    2:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  7. gawk

    gawk New Member

    12
    7
    3
    Feb 24, 2018
    Ratings:
    +9
    Local Time:
    5:07 AM
    nginx-1.13.9
    MariaDB 10.3.4
    You're right, I was checking my access.log, and was wondering why all logs were from a few days ago :D

    I'm glad my thread helped bring this addon to life :p

    Anyway, my Cloudflare account was messed up, which is what caused the bad bots not to be blocked/rate limited by Centmin. I fixed my settings in Cloudflare (so the user agent gets properly reported) and everything is working fine.

    Since my server is in production use, I'll wait a few months or so for reports on Krog's plugin, as your method works fine for my usage needs.
     
  8. eva2000

    eva2000 Administrator Staff Member

    54,906
    12,240
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,811
    Local Time:
    2:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah buffered access log writes is to improve Nginx performance :)
    Glad to hear you resolved your issue :)
     
  9. JJC84

    JJC84 Ad astra per aspera

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +169
    Local Time:
    11:07 PM
    1.15.x
    10.x.x
    Three cheers to Gawk for bringing this to our attention. Hip hip hooray!x3 Great find!
     
  10. Colin

    Colin Premium Member Premium Member

    192
    59
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +154
    Local Time:
    4:07 AM
    1.19.#
    MariaDB 10.1.#
    Totally agree. Well worth the hour to test and install :D Im not sure I've spent a better one this year with my terminal :D Thanks for the GitHub write up too, that saved me a few hours :D