Get the most out of your Centmin Mod LEMP stack
Become a Member

Sysadmin Not blocking bad bots

Discussion in 'System Administration' started by gawk, Apr 2, 2018.

  1. gawk

    gawk New Member

    7
    5
    3
    Feb 24, 2018
    Ratings:
    +7
    Local Time:
    11:08 PM
    nginx-1.13.9
    MariaDB 10.3.4
    • CentOS Version: CentOS 7 64-bit
    • Centmin Mod Version Installed: 123.09beta01
    • Nginx Version Installed: 1.13.9
    • PHP Version Installed: 7.1.14
    • MariaDB MySQL Version Installed: 10.1.31
    • When was last time updated Centmin Mod code base? : Today
    • Persistent Config: Yes

    Hi,

    I edited my nginx.conf and domain.conf file to include the botlimit.conf and blockbots.conf files. This worked for some time, but today I tested different user agents with curl, and none of them gave a 444 error on the request (despite me using "ahrefsbot" for example as my user agent).

    How can I troubleshoot the problem?
     
    • Useful Useful x 1
  2. eva2000

    eva2000 Administrator Staff Member

    34,253
    7,581
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,656
    Local Time:
    7:08 AM
    Nginx 1.13.x
    MariaDB 5.5
    I assume you mean the bad bot blocking and rate limiting implementation outlined here ?

    Using BBCODE CODE tags post contents of your config files for
    • /usr/local/nginx/conf/conf.d/nginx.conf
    • /usr/local/nginx/conf/conf.d/domain.com.conf and/or /usr/local/nginx/conf/conf.d/domain.com.ssl.conf
    • /usr/local/nginx/conf/botlimit.conf
    • /usr/local/nginx/conf/blockbots.conf
    You can mask your real domain name if you want in text editor just do a mass search and replace in the files.

    Also post examples of how and what command you use to test and the output from those commands. Again use CODE tags for output
     
    • Like Like x 1
  3. wmtech

    wmtech Member

    51
    15
    8
    Jul 22, 2017
    Ratings:
    +32
    Local Time:
    11:08 PM
    • Informative Informative x 3
    • Like Like x 1
  4. eva2000

    eva2000 Administrator Staff Member

    34,253
    7,581
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,656
    Local Time:
    7:08 AM
    Nginx 1.13.x
    MariaDB 5.5
    Thanks for sharing (y)

    edit: just had a closer look at this and very well scripted and developed ! I couldn't have done a better job myself if I had time so definitely checking it out as you have to change their default assumed paths as Centmin Mod uses different default nginx directory structure and there's a few manual changes needed to Centmin Mod nginx config files to get it working :)
     
    • Informative Informative x 1
  5. eva2000

    eva2000 Administrator Staff Member

    34,253
    7,581
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,656
    Local Time:
    7:08 AM
    Nginx 1.13.x
    MariaDB 5.5
    where are you checking for 444 error ? in your access.log i.e. /home/nginx/domains/domain1.com/log/access.log ? Centmin Mod by default buffers log writes to memory for a certain amount of entries until the buffer is full and then is flushed and written to disk in access.log so you may need to restart nginx to flush the buffer to see access.log entry for 444 status
     
  6. eva2000

    eva2000 Administrator Staff Member

    34,253
    7,581
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,656
    Local Time:
    7:08 AM
    Nginx 1.13.x
    MariaDB 5.5
  7. gawk

    gawk New Member

    7
    5
    3
    Feb 24, 2018
    Ratings:
    +7
    Local Time:
    11:08 PM
    nginx-1.13.9
    MariaDB 10.3.4
    You're right, I was checking my access.log, and was wondering why all logs were from a few days ago :D

    I'm glad my thread helped bring this addon to life :p

    Anyway, my Cloudflare account was messed up, which is what caused the bad bots not to be blocked/rate limited by Centmin. I fixed my settings in Cloudflare (so the user agent gets properly reported) and everything is working fine.

    Since my server is in production use, I'll wait a few months or so for reports on Krog's plugin, as your method works fine for my usage needs.
     
    • Like Like x 2
    • Informative Informative x 1
  8. eva2000

    eva2000 Administrator Staff Member

    34,253
    7,581
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,656
    Local Time:
    7:08 AM
    Nginx 1.13.x
    MariaDB 5.5
    Yeah buffered access log writes is to improve Nginx performance :)
    Glad to hear you resolved your issue :)
     
    • Like Like x 1
  9. JJC84

    JJC84 Premium Member Premium Member

    129
    49
    28
    Jan 31, 2018
    Ratings:
    +71
    Local Time:
    4:08 PM
    1.13.9
    10.1
    Three cheers to Gawk for bringing this to our attention. Hip hip hooray!x3 Great find!
     
    • Like Like x 2
  10. Colin

    Colin Premium Member Premium Member

    136
    39
    28
    Oct 7, 2015
    Sheffield UK
    Ratings:
    +96
    Local Time:
    10:08 PM
    1.11.#
    MariaDB 10.1.#
    Totally agree. Well worth the hour to test and install :D Im not sure I've spent a better one this year with my terminal :D Thanks for the GitHub write up too, that saved me a few hours :D
     
    • Like Like x 2
..