Security Nginx Ultimate Bad Block Blocker

Folks may have known I developed my own Bad Bot Blocker & Rate Limiting routine for folks to test. But it has been brought to my attention by @wmtech that there's a more mature and developed version of what I was developing already called Nginx Ultimate Bad Block Blocker written by Mitchell Krog.

I haven't tested this out yet in production, but at least wrote up a quick guide for install and setup on Centmin Mod 123.09beta01 or higher Nginx servers. There is no support provided by me right now seeing as I have no practical experience with this setup yet and any issues with false postives blocks etc, you will need to contact the official developer on their Ultimate Bad Bot Blocker issue tracker

---

Nginx Ultimate Bad Bot Blocker Install Guide

The quick guide for install is located at https://github.com/centminmod/centminmod-ultimate-bad-bot-blocker. Read it thoroughly and maybe test on a test VPS with test Centmin Mod 123.09beta01 install and learn the ropes etc before trying on live production servers.

Notes

• General Notes

Features

From features list:

• Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily)
• Blocking of Spam Referrer Domains and Web Sites
• Blocking of SEO data collection companies like Semalt.com, Builtwith.com, WooRank.com and many others (updated regularly)
• Blocking of clickjacking Sites linked to Adware, Malware and Ransomware
• Blocking of Porn and Gambling Web Sites who use Lucrative Ways to Earn Money through Serving Ads by hopping off your domain names and web sites.
• Blocking of Bad Domains and IP's that you cannot even see in your Nginx Logs. Thanks to the Content Security Policy (CSP) on all my SSL sites I can see things trying to pull resources off my sites before they even get to Nginx and get blocked by the CSP.
• Anti DDOS Filter and Rate Limiting of Agressive Bots
• Alphabetically ordered for easier maintenance (Pull Requests Welcomed)
• Commented sections of certain important bots to be sure of before blocking
• Includes the IP range of Cyveillance who are known to ignore robots.txt rules and snoop around all over the Internet.
• Whitelisting of Google, Bing and Cloudflare IP Ranges
• Whitelisting of your own IP Ranges that you want to avoid blocking by mistake.
• Ability to add other IP ranges and IP blocks that you want to block out.
• If its out there and it's bad it's already in here and BLOCKED !!

 

Example of bad referrer blocking for a domain 100dollars-seo via curl will give a Nginx HTTP 444 Status code and thus return a empty reply curl 52 error

Code (Text):

curl -I http://domain1.com -e http://100dollars-seo.com
curl: (52) Empty reply from server

Bad referrer domain is listed in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

Code (Text):

grep 100dollars /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
        "~*\b100dollars\-seo\.com\b"    1;

You can use ngxtop to analyse your Centmin Mod Nginx access logs and keep an eye on Nginx HTTP 444 status codes and also other user agent strings, HTTP status codes etc as outlined here.

Check for HTTP 444 status codes /home/nginx/domains/domain1.com/log/access.log access log

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow

Only one entry for 444 exists due to test domain1.com test command curl -I http://domain1.com -e 100dollars-seo.com only ran once

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
running for 0 seconds, 1 records processed: 2434.30 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |       1 |            0.000 |     0 |     0 |     1 |     0 |

alternative use native -i 'status == 444' flag

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'
running for 0 seconds, 1 records processed: 369.97 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |       1 |            0.000 |     0 |     0 |     1 |     0 |

print request url, HTTP Status code and http user agent

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
running for 0 seconds, 1 records processed: 335.92 req/sec

request, status, http_user_agent:
| request         |   status | http_user_agent   |
|-----------------+----------+-------------------|
| HEAD / HTTP/1.1 |      444 | curl/7.29.0       |

group by remote IP address via --group-by remote_addr

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr

output where 192.168.0.1 is the visitor ip address

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr
running for 0 seconds, 1 records processed: 1597.22 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| remote_addr   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------------+---------+------------------+-------+-------+-------+-------|
| 192.168.0.1 |       1 |            0.000 |     0 |     0 |     1 |     0 |

group by user agent string via --group-by http_user_agent

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent

the test command was run via curl hence curl user agent string

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
running for 0 seconds, 1 records processed: 1402.31 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |

filter by specific date using grep i.e. April 2nd would filter by '02/Apr'

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent
running for 0 seconds, 1 records processed: 1742.54 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |

 

Test the rate limiting of a bot with user agent = Baidu which is on the aggressive rate limiting profile of 2 requests/second specified in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

Code (Text):

# limits for Zone $bad_bot = 2
# this rate limiting will only take effect if you change any of the bots and change
# their block value from 1 to 2.
        limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
        limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;

Siege load test with user agent Baidu rate limited with a HTTP 503 status code served up to rated limited requests

Code (Text):

siege -b -c10 -r100 -A "Baidu" http://domain1.com/

Lifting the server siege...
Transactions:                    277 hits
Availability:                  51.87 %
Elapsed time:                 138.37 secs
Data transferred:               1.10 MB
Response time:                  4.90 secs
Transaction rate:               2.00 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                    9.80
Successful transactions:         277
Failed transactions:             257
Longest transaction:            5.01
Shortest transaction:           0.00

ngxtop with Baidu user agent string rate limited results for HTTP 503 status codes in acces log

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
running for 1 seconds, 6081 records processed: 7868.07 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    6081 |         2060.460 |   373 |     0 |    81 |  5627 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |    6081 |         2060.460 |   373 |     0 |    81 |  5627 |

Code (Text):

cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent   
running for 1 seconds, 6153 records processed: 7625.33 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    6153 |         2056.997 |   432 |     0 |    87 |  5634 |

Detailed:
| http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-------------------+---------+------------------+-------+-------+-------+-------|
| Baidu             |    6151 |         2057.666 |   431 |     0 |    86 |  5634 |
| curl/7.29.0       |       2 |            0.000 |     1 |     0 |     1 |     0 |