Get the most out of your Centmin Mod LEMP stack
Become a Member

Nginx nginx ultimate bad-bot blocker for with Centmin & FIX auto update

Discussion in 'Centmin Mod User Tutorials & Guides' started by EckyBrazzz, May 20, 2020.

  1. EckyBrazzz

    EckyBrazzz Active Member

    916
    189
    43
    Mar 28, 2018
    >>>>Click here<<<< i'm nearby......
    Ratings:
    +362
    Local Time:
    10:45 AM
    Latest
    Latest
    Here follows the correct way to install the nginx ultimate bad-bot blocker. No hassle, just follow the steps & always check that it's working on another server with your CDN (Cloudflare) disabled

    Step 1
    Code (Text):
    sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker
    sudo chmod +x /usr/local/sbin/install-ngxblocker
    


    Step 2 : Backup
    Code (Text):
    mkdir -p /usr/local/nginx/conf/ultimate-badbot-blocker
    cp -a /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf-backup-b4-badbot
    cp -a /usr/local/nginx/conf/conf.d/ /usr/local/nginx/conf/conf.d-backup-b4-badbot
    


    Step 3: Install (Dry run. no files are changed)
    Code (Text):
    cd /usr/local/sbin
    sudo ./install-ngxblocker -c /usr/local/nginx/conf -b /usr/local/nginx/conf/ultimate-badbot-blocker
    

    Output: (Check configuration files destination)
    Code:
    ** Dry Run ** | not updating files | run  as 'install-ngxblocker -x' to install files.
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /usr/local/nginx/conf/globalblacklist.conf
    Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /usr/local/nginx/conf/botblocker-nginx-settings.conf
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blockbots.conf
    Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/ddos.conf
    Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/custom-bad-referrers.conf
    Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bad-referrer-words.conf
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blacklist-ips.conf
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blacklist-user-agents.conf
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/whitelist-domains.conf
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/whitelist-ips.conf
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/setup-ngxblocker      [TO]=>  /usr/local/sbin/setup-ngxblocker
    Downloading [FROM]=>  [REPO]/update-ngxblocker     [TO]=>  /usr/local/sbin/update-ngxblocker
    Real Live install with -x (execute)
    Code (Text):
    sudo ./install-ngxblocker -c /usr/local/nginx/conf -b /usr/local/nginx/conf/ultimate-badbot-blocker -x
    

    Output:
    Code:
    Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /usr/local/nginx/conf/globalblacklist.conf...OK
    Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /usr/local/nginx/conf/botblocker-nginx-settings.conf...OK
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blockbots.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/ddos.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/custom-bad-referrers.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bad-referrer-words.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blacklist-ips.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/blacklist-user-agents.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/whitelist-domains.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/whitelist-ips.conf...OK
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/setup-ngxblocker      [TO]=>  /usr/local/sbin/setup-ngxblocker...OK
    Downloading [FROM]=>  [REPO]/update-ngxblocker     [TO]=>  /usr/local/sbin/update-ngxblocker...OK
    Setting mode: 700 => /usr/local/sbin/install-ngxblocker
    Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
    Setting mode: 700 => /usr/local/sbin/update-ngxblocker
    Manual Fix Bad bot Updater:
    Code (Text):
    nano /usr/local/sbin/update-ngxblocker
    
    ]
    Line #39 & 40 comment out and change to
    Code (Text):
    CONF_DIR=/usr/local/nginx/conf/ultimate-badbot-blocker/conf.d
    BOTS_DIR=/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
    


    MAKE SURE you set your setup and update scripts to be executable by running the following two commands.
    Code (Text):
    sudo chmod +x /usr/local/sbin/setup-ngxblocker
    sudo chmod +x /usr/local/sbin/update-ngxblocker
    

    Step 5 Real Install
    Dry Run
    Code (Text):
    setup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
    

    Output:
    Code:
    ** Dry Run ** | not updating files | run  as 'setup-ngxblocker -x' to setup files.
    
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;            => /usr/local/nginx/conf/nginx.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;  => /usr/local/nginx/conf/nginx.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           => /usr/local/nginx/conf/conf.d/demodomain.com.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                => /usr/local/nginx/conf/conf.d/demodomain.com.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           => /usr/local/nginx/conf/conf.d/virtual.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                => /usr/local/nginx/conf/conf.d/virtual.conf
    
    setup will fix conflict from: 'server_names_hash_bucket_size' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    setup will fix conflict from: 'server_names_hash_max_size' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    setup will fix conflict from: 'limit_req_zone' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    
    Whitelisting ip:  111.222.333.444   => /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
    
    Web directory not found ('/var/www'): not automatically whitelisting domains.
    
    Checking for missing includes:
    
    Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf...OK
    Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf...OK
    
    REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master
    
    Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/custom-bad-referrers.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/bad-referrer-words.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-ips.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blacklist-user-agents.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-domains.conf...OK
    Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf...OK
    Nothing to update for directory: /usr/local/sbin
    Setting mode: 700 => /usr/local/sbin/install-ngxblocker
    Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
    Setting mode: 700 => /usr/local/sbin/update-ngxblocker
    Updating bots.d path: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
    Live RUN (INSTALL)
    Code (Text):
    setup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
    

    Output:
    Code:
    Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
    
    
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;            => /usr/local/nginx/conf/nginx.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;  => /usr/local/nginx/conf/nginx.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           => /usr/local/nginx/conf/conf.d/demodomain.com.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                => /usr/local/nginx/conf/conf.d/demodomain.com.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;           => /usr/local/nginx/conf/conf.d/virtual.conf
    inserting: include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;                => /usr/local/nginx/conf/conf.d/virtual.conf
    
    setup will fix conflict from: 'server_names_hash_bucket_size' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabling 'server_names_hash_bucket_size' in: /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabled OK
    
    
    setup will fix conflict from: 'server_names_hash_max_size' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabling 'server_names_hash_max_size' in: /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabled OK
    
    setup will fix conflict from: 'limit_req_zone' in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabling 'limit_req_zone' in: /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    disabled OK
    
    Whitelisting ip:  111.222.333.444   => /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf
    
    Web directory not found ('/var/www'): not automatically whitelisting domains.
    
    Checking for missing includes:
    
    Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
    
    Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker
    Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
    Nothing to update for directory: /usr/local/sbin
    Setting mode: 700 => /usr/local/sbin/install-ngxblocker
    Setting mode: 700 => /usr/local/sbin/setup-ngxblocker
    Setting mode: 700 => /usr/local/sbin/update-ngxblocker
    SOME MANUAL FIXES::
    Code (Text):
    nano /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
    

    Comment out lines 18&19
    Code:
    #server_names_hash_bucket_size 256;
    #server_names_hash_max_size 4096;
    #variables_hash_max_size 4096;
    #variables_hash_bucket_size 4096;
    #limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
    limit_conn_zone $binary_remote_addr zone=addr:50m;
    
    Edit nginx.conf
    Code (Text):
    nano /usr/local/nginx/conf/nginx.conf
    

    Make it look like this:
    Code (Text):
    #nginx.conf add line 24 /25 add: Setting to lower value will brake Wordpress sites
    limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
    
    
    include /usr/local/nginx/conf/brotli_inc.conf;
    map_hash_bucket_size 256;
    map_hash_max_size 4096;
    server_names_hash_bucket_size 256;
    server_names_hash_max_size 4096;
    variables_hash_max_size 4096;
    variables_hash_bucket_size 4096;
    

    Code:
    nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    Code:
    nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    Code (Text):
    nprestart

    Keep bad bot blocker up to date with cronjob
    Code (Text):
    00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -e your@email.com
    


    Follow step 10 on mitchellkrogza/nginx-ultimate-bad-bot-blocker , test that it works.... PLEASE USE OTHER SERVER TO TEST!! DISABLE CDN (CLOUDFLARE)

     
  2. Simon Brown

    Simon Brown Member

    55
    6
    8
    Feb 9, 2017
    Ratings:
    +18
    Local Time:
    1:45 PM
    1.11.9
    Is this a better solution than the Centmin bad bot blocker I'm currently using?
     
  3. EckyBrazzz

    EckyBrazzz Active Member

    916
    189
    43
    Mar 28, 2018
    >>>>Click here<<<< i'm nearby......
    Ratings:
    +362
    Local Time:
    10:45 AM
    Latest
    Latest
    This one updates new bots when you enable the cronjob.

    Example mail.
    Code:
    LOCAL Version:  4.2020.05.2068 
    Updated: Thu May 21 22:48:29 SAST 2020
    
    REMOTE Version:  4.2020.05.2069 
    Updated: Sat May 23 22:49:13 SAST 2020
    
    Update Available =>  4.2020.05.2069
    
    Downloading: globalblacklist.conf ... [OK] 
    
    Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
    
    Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/conf.d
    Nothing to update for directory: /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
    Nothing to update for directory: /usr/local/sbin Setting mode: 700 => /usr/local/sbin/install-ngxblocker
    Setting mode: 700 => /usr/local/sbin/setup-ngxblocker Setting mode: 700 => /usr/local/sbin/update-ngxblocker  Updating bots.d path /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d => /usr/local/nginx/conf/ultimate-badbot-blocker/conf.d/globalblacklist.conf [0m
    
     
  4. EckyBrazzz

    EckyBrazzz Active Member

    916
    189
    43
    Mar 28, 2018
    >>>>Click here<<<< i'm nearby......
    Ratings:
    +362
    Local Time:
    10:45 AM
    Latest
    Latest
    UPDATE!!

    Correct path to updater. Default is set to /etc/nginx, while our config files live on another place

    Line 40
    Code (Text):
    nano /usr/local/sbin/update-ngxblocker
    #CONF_DIR=/etc/nginx/conf.d
    CONF_DIR=/usr/local/nginx/conf/ultimate-badbot-blocker/conf.d
    #BOTS_DIR=/etc/nginx/bots.d
    BOTS_DIR=/usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,548
    12,221
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,790
    Local Time:
    11:45 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Thanks for the heads up!