Security Blocking bad or aggressive bots

That's new to me. Already asked @eva2000 to add it by default when creating a new domain, but until now I have to add it manually.
Hope we are talking about the same files because when I installed it I had to add it manually to all vhosts.

---

This is what I have in my notepad++ log when I installed it.

Code (Text):

<----- START ultimate badbot blocker
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/update-ngxblocker -O /usr/local/sbin/update-ngxblocker && chmod 700 /usr/local/sbin/update-ngxblocker
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker
chmod +x /usr/local/sbin/install-ngxblocker
mkdir -p /usr/local/nginx/conf/ultimate-badbot-blocker
cp -a /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf-backup-b4-badbot
cp -a /usr/local/nginx/conf/conf.d/ /usr/local/nginx/conf/conf.d-backup-b4-badbot
install-ngxblocker -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
install-ngxblocker -x -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d
sed -i 's|^server_names_hash_|#server_names_hash_|g' /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
setup-ngxblocker -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
setup-ngxblocker -x -e conf -c /usr/local/nginx/conf/ultimate-badbot-blocker -b /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d -v /usr/local/nginx/conf/conf.d -m /usr/local/nginx/conf/nginx.conf
cd /usr/local/nginx/conf/
nano /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
nano nginx.conf
diff -qr /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d 
diff -r -U4 /usr/local/nginx/conf/conf.d-backup-b4-badbot /usr/local/nginx/conf/conf.d
ngxrestart
nano /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/whitelist-ips.conf

Add to each vhost
  # uncomment ddos.conf and blockbots.conf include
  # if using the   # uncomment ddos.conf and blockbots.conf include
  # if using the ultimate badbot blocker guide after testing
  # https://community.centminmod.com/threads/nginx-ultimate-bad-block-blocker.14413/
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;
<----- END ultimate badbot blocker

And the config files

Code (Text):

 cat /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf
##############################################################################                                                               
#       _  __     _                                                          #
#      / |/ /__ _(_)__ __ __                                                 #
#     /    / _ `/ / _ \\ \ /                                                 #
#    /_/|_/\_, /_/_//_/_\_\                                                  #
#       __/___/      __   ___       __     ___  __         __                #
#      / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____      #
#     / _  / _ `/ _  /  / _  / _ \/ __/  / _  / / _ \/ __/  '_/ -_) __/      #
#    /____/\_,_/\_,_/  /____/\___/\__/  /____/_/\___/\__/_/\_\\__/_/         #
#                                                                            #
##############################################################################                                                               

# Version 1.1
# ! new directives also to be added to include_filelist.txt ! #

#server_names_hash_bucket_size 128;
#server_names_hash_max_size 4096;
limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
limit_conn_zone $binary_remote_addr zone=addr:50m;

# ****************************************************************************
# NOTE: IF you are using a system like Nginx-Proxy from @JWilder
# ****************************************************************************
# Repo URL: https://github.com/jwilder/nginx-proxy
# You will need to comment out the first line here as follows.
#     #server_names_hash_bucket_size 128;
# You will also need to modify the nginx.tmpl file to add the default include
#     include /etc/nginx/conf.d/*
# ****************************************************************************

and

Code (Text):

cat /usr/local/nginx/conf/nginx.conf
user              nginx nginx;
worker_processes 2;
worker_priority -10;

worker_rlimit_nofile 520000;
timer_resolution 100ms;

pcre_jit on;
include /usr/local/nginx/conf/dynamic-modules.conf;


pid         logs/nginx.pid;

events {
    worker_connections  80000;
    accept_mutex off;
    accept_mutex_delay 200ms;
    use epoll;
    #multi_accept on;
}

http {
 include /usr/local/nginx/conf/brotli_inc.conf;
limit_req_zone $binary_remote_addr zone=xwprpc:10m rate=30r/s;

 map_hash_bucket_size 128;
 map_hash_max_size 4096;
 server_names_hash_bucket_size 128;
 server_names_hash_max_size 4096;
 variables_hash_max_size 2048;
# limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
# limit_conn_zone $binary_remote_addr zone=addr:50m;


 # Bad Bot Blocker
include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;
include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;

 resolver 127.0.0.1 valid=10m;
 resolver_timeout 10s;

limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;

# sets Centmin Mod headers via headers more nginx module
# https://github.com/openresty/headers-more-nginx-module
# don't remove the first 2 lines as centmin mod checks to see if they're
# missing and re-adds them anyway. Just uncomment the 3rd & 4th lines
# which is used to override the Server header to what you want = nginx
# and remove the X-Powered-By header + restart nginx service
# do not disable headers more nginx module itself as it's required for
# other centmin mod features like redis nginx level caching & letsencrypt
# integration in vhosts created by addons/acmetool.sh
more_set_headers "Server: nginx centminmod";
more_set_headers "X-Powered-By: centminmod";
#more_set_headers "Server: nginx";
#more_clear_headers "X-Powered-By";

# uncomment cloudflare.conf include if using cloudflare for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfcf.sh auto
# run the auto command once to populate cloudflare ips
#include /usr/local/nginx/conf/cloudflare.conf;
# uncomment incapsula.conf include if using incapsula for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfincapsula.sh auto
# run the auto command once to popular incapsula ips
#include /usr/local/nginx/conf/incapsula.conf;
include /usr/local/nginx/conf/maintenance.conf;
#include /usr/local/nginx/conf/vts_http.conf;
include /usr/local/nginx/conf/geoip.conf;
include /usr/local/nginx/conf/webp.conf;
include /usr/local/nginx/conf/ssl_include.conf;
#include /usr/local/nginx/conf/pagespeedadmin.conf;
include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
include /usr/local/nginx/conf/wpfastcgi_cache_map.conf;
include /usr/local/nginx/conf/wpfastcgi_cache_path_includes.conf;
include /usr/local/nginx/conf/redisupstream.conf;
include /usr/local/nginx/conf/wpcacheenabler_map.conf;
include /usr/local/nginx/conf/default_phpupstream.conf;
#include /usr/local/nginx/conf/botlimit.conf;


log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  ddos-proxy '$remote_addr for $http_x_real_ip - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$host" sn="$server_name" '
                      'rt=$request_time '
                      'ua="$upstream_addr" us="$upstream_status" '
                      'ut="$upstream_response_time" ul="$upstream_response_length" '
                      'cs=$upstream_cache_status' ;

# only uncomment include line to enable it you have enabled ngx_brotli module
# which is disabled by default https://community.centminmod.com/threads/10688/
#include /usr/local/nginx/conf/log_format_brotli.conf;

access_log  off;
error_log   logs/error.log warn;

    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

sendfile on;
sendfile_max_chunk 512k;
tcp_nopush  on;
tcp_nodelay on;
server_tokens off;
server_name_in_redirect off;

keepalive_timeout  5s;
keepalive_requests  1000;
lingering_time 20s;
lingering_timeout 5s;
keepalive_disable msie6;

gzip on;
gzip_vary   on;
gzip_disable msie6;
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

client_body_buffer_size 256k;
client_body_in_file_only off;
client_body_timeout 10s;
client_header_buffer_size 64k;
## how long a connection has to complete sending
## it's headers for request to be processed
client_header_timeout  10s;
client_max_body_size 1024m;
connection_pool_size  512;
directio  4m;
directio_alignment 4096;
ignore_invalid_headers on;       
large_client_header_buffers 8 64k;
output_buffers   1 512k;
postpone_output  1460;
proxy_temp_path  /tmp/nginx_proxy/;
request_pool_size  32k;
reset_timedout_connection on;
send_timeout     60s;
types_hash_max_size 2048;

# for nginx proxy backends to prevent redirects to backend port
# port_in_redirect off;

open_file_cache max=50000 inactive=60s;
open_file_cache_valid 120s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
open_log_file_cache max=10000 inactive=30s min_uses=2;

## limit number of concurrency connections per ip to 16
## add to your server {} section the next line
## limit_conn limit_per_ip 16;
## uncomment below line allows 500K sessions
# limit_conn_log_level error;
#######################################
# use limit_zone for Nginx <v1.1.7 and lower
# limit_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################
# use limit_conn_zone for Nginx >v1.1.8 and higher
# limit_conn_zone $binary_remote_addr zone=limit_per_ip:16m;
#######################################

 include /usr/local/nginx/conf/conf.d/*.conf;
}

Hope it helps.

 

Thank you for your help.

In the file domain.org.ssl.conf there are two server{}, the Bat bot blocker files were inside the first tag, I switched to the second one and it finally worked.

https://www.diffchecker.com/dl9I9KUx

What happens differently is that when I execute

Code:

curl -I http://domain.org -e http://100dollars-seo.com

I receive this curl: (56) TCP connection reset by peer rather than what the tutorial shows Should respond with: curl: (52) Empty reply from server

With you, too?

 

Don't test with http://100dollars-seo.com, that domain is for sale.

 

@fabianski sorry that I did not provide another domain, but I'm helping and supporting other users in my "Free Time", and that time is a little limited.

But check the list and pick another domain to test.

I already notified Mitchell, the author of the blocker that he has to change the domains in step 10 as on mitchellkrogza/nginx-ultimate-bad-bot-blocker

 

For testing, I redirected all blocks to my facebook business page

Code (Text):

if ($bad_bot = '3') {
  return 444;
  #return 301 $scheme://www.facebook.com/YourPage;
}

# BAD REFER WORDS
# ***************
if ($bad_words) {
  return 444;
  #return 301 $scheme://www.facebook.com/YourPage;
}


# REFERERS
# ********
if ($bad_referer) {
  #return 301 $scheme://www.facebook.com/YourPage;
  return 444;
}

# IP BLOCKS
# *********
if ($validate_client) {
  #return 301 $scheme://www.facebook.com/YourPage;
  return 444;
}

#######################################################################

And afterwards checked the insides statics.

 

It works (old picture cause now I have a # in front of the redirect)

 

I've tested with other newer domains, it's working

 

@fabianski
On what domains you tested it ??

 

http://wesharepics.info and https://buzzsumo.com

 

Just got a response from Mitchel the author of the blocker about the domains. Guess he knows it better than anybody else because it's his daily project.

Price for the domain is only a little high

 

Hi,

I have experienced an issue with Badbots when follwoing Security - Blocking bad or aggressive bots

and adding:
include /usr/local/nginx/conf/blockbots.conf;

in vhost's location context on wordpress installs, I get errors in Wordpress' new site health checker

Site Health Status
The site health check shows critical information about your WordPress configuration and items that require your attention.

2 Critical issues
The REST API is one way WordPress, and other applications, communicate with the server. One example is the block editor screen, which relies on this to display, and save, your posts and pages.

The REST API request failed due to an error.
Error: [] cURL error 56: TCP connection reset by peer



Loopback requests are used to run scheduled events, and are also used by the built-in editors for themes and plugins to verify code stability.

The loopback request to your site failed, this means features relying on them are not currently working as expected.
Error: [] cURL error 56: TCP connection reset by peer

It disappears when I remove:
include /usr/local/nginx/conf/blockbots.conf;

Not sure why.

I tried a test with switching to the Ultimate Bad Block Blocker and the error didn't arise.

No biggy for me, but thought I'd see if anyone else has experienced the same.

 

Hope you did the test for another linux machine as step 10 indicates.

 

Hi EckyBrazzz, thanks for the heads up. Yes I did. so far so good

 

Just to know if there are people with Cloudflare that run into trouble with this. Disabling Cloudflare caching and it should be fine. Testing should also done different.

See new README.md at GitHub or the ticket that I opened today [CLOUDFLARE] Wrong Curl -I results, differ from test. · Issue #276 · mitchellkrogza/nginx-ultimate-bad-bot-blocker

 

@eva2000, placed your comment on the GitHub issue. Hope its use full.

 

Just so I get up to speed, the currently recommended method is ultimate bad bot blocker? Even though it is external resource to centminmod it integrates well and there are no issues?

 

Update: I think I managed to install it correctly.

I can confirm that the automatic inclusion in the domain.com.ssl.conf is somewhat finicky. I had only one of the two includes added and it was under the wrong server directive. Moved them both under the second clause like @fabianski did and it seemed fine.

As I am behind cloudflare, I tried to disabled it completely for the tests.

The first test was passed succesfully as it should:

Code (Text):

~$ curl -I -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://mydomain.com
HTTP/1.1 200 OK
Date: Wed, 10 Jul 2019 20:57:33 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 147497
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: ips4_IPSSessionFront=c53huil84e8n6hr71h4nqmnipj; path=/; secure; HttpOnly
Set-Cookie: ips4_guestTime=1562792252; path=/; secure; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-IPS-LoggedIn: 0
Vary: cookie
X-XSS-Protection: 0
X-Frame-Options: sameorigin
Server: nginx centminmod
X-Powered-By: centminmod

But my SSL certificate started playing and I got some ssl error when running the second one:

Code (Text):

~$ curl -I https://mydomain.com -e https://buzzsumo.com
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

So I reenabled cloudflare and tried again.

First test:

Code (Text):

~$ curl -I -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://mydomain.com
HTTP/1.1 200 OK
Date: Wed, 10 Jul 2019 21:00:56 GMT
Content-Type: text/html;charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d0d894b3359f1cfd409f039d01de685a41562792455; expires=Thu, 09-Jul-20 21:00:55 GMT; path=/; domain=.mydomain.com; HttpOnly; Secure
Set-Cookie: ips4_IPSSessionFront=u5bnhefbcgdtprijvbj686737l; path=/; secure; HttpOnly
Set-Cookie: ips4_guestTime=1562792455; path=/; secure; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-IPS-LoggedIn: 0
Vary: cookie, Accept-Encoding
X-XSS-Protection: 0
X-Frame-Options: sameorigin
X-Powered-By: centminmod
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4f4578511b3cd90d-AMS

Second one:

Code (Text):

~$ curl -I https://mydomain.com -e http://wesharepics.info                                                                                         HTTP/1.1 520 Origin Error
Date: Wed, 10 Jul 2019 21:01:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d3c326c6a7ceb4bc5a22c112f8b6b6aae1562792469; expires=Thu, 09-Jul-20 21:01:09 GMT; path=/; domain=.mydomain.com; HttpOnly; Secure
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: cf_use_ob=0; path=/; expires=Wed, 10-Jul-19 21:01:39 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Server: cloudflare
CF-RAY: 4f4578a3cef99d66-AMS

I guess this means I am good to go.

Oh...also, there was another duplicate clause that needed commenting in ultimate-bad-bot config and editing from 2048 to 4096 in nginx.conf. It is variables_hash_max_size and is located right below server_names_hash_max_size:

Code (Text):

[root@concorde conf.d]# nginx -t
nginx: [emerg] "variables_hash_max_size" directive is duplicate in /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf:18
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed

 

It works very well and without problems. Use it since more than a year now. ;-)