Get the most out of your Centmin Mod LEMP stack
Become a Member

Security Nginx Ultimate Bad Block Blocker

Discussion in 'System Administration' started by eva2000, Apr 3, 2018.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    35,135
    7,753
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,956
    Local Time:
    3:38 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Folks may have known I developed my own Bad Bot Blocker & Rate Limiting routine for folks to test. But it has been brought to my attention by @wmtech that there's a more mature and developed version of what I was developing already called Nginx Ultimate Bad Block Blocker written by Mitchell Krog.

    I haven't tested this out yet in production, but at least wrote up a quick guide for install and setup on Centmin Mod 123.09beta01 or higher Nginx servers. There is no support provided by me right now seeing as I have no practical experience with this setup yet and any issues with false postives blocks etc, you will need to contact the official developer on their Ultimate Bad Bot Blocker issue tracker

    Nginx Ultimate Bad Bot Blocker Install Guide



    The quick guide for install is located at https://github.com/centminmod/centminmod-ultimate-bad-bot-blocker. Read it thoroughly and maybe test on a test VPS with test Centmin Mod 123.09beta01 install and learn the ropes etc before trying on live production servers.

    Notes
    Features

    From features list:
    • Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily)
    • Blocking of Spam Referrer Domains and Web Sites
    • Blocking of SEO data collection companies like Semalt.com, Builtwith.com, WooRank.com and many others (updated regularly)
    • Blocking of clickjacking Sites linked to Adware, Malware and Ransomware
    • Blocking of Porn and Gambling Web Sites who use Lucrative Ways to Earn Money through Serving Ads by hopping off your domain names and web sites.
    • Blocking of Bad Domains and IP's that you cannot even see in your Nginx Logs. Thanks to the Content Security Policy (CSP) on all my SSL sites I can see things trying to pull resources off my sites before they even get to Nginx and get blocked by the CSP.
    • Anti DDOS Filter and Rate Limiting of Agressive Bots
    • Alphabetically ordered for easier maintenance (Pull Requests Welcomed)
    • Commented sections of certain important bots to be sure of before blocking
    • Includes the IP range of Cyveillance who are known to ignore robots.txt rules and snoop around all over the Internet.
    • Whitelisting of Google, Bing and Cloudflare IP Ranges
    • Whitelisting of your own IP Ranges that you want to avoid blocking by mistake.
    • Ability to add other IP ranges and IP blocks that you want to block out.
    • If its out there and it's bad it's already in here and BLOCKED !!
     
    • Like Like x 3
  2. eva2000

    eva2000 Administrator Staff Member

    35,135
    7,753
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,956
    Local Time:
    3:38 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Example of bad referrer blocking for a domain 100dollars-seo via curl will give a Nginx HTTP 444 Status code and thus return a empty reply curl 52 error
    Code (Text):
    curl -I http://domain1.com -e http://100dollars-seo.com
    curl: (52) Empty reply from server
    

    Bad referrer domain is listed in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
    Code (Text):
    grep 100dollars /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
            "~*\b100dollars\-seo\.com\b"    1;
    

    You can use ngxtop to analyse your Centmin Mod Nginx access logs and keep an eye on Nginx HTTP 444 status codes and also other user agent strings, HTTP status codes etc as outlined here.

    Check for HTTP 444 status codes /home/nginx/domains/domain1.com/log/access.log access log
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
    

    Only one entry for 444 exists due to test domain1.com test command curl -I http://domain1.com -e 100dollars-seo.com only ran once
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
    running for 0 seconds, 1 records processed: 2434.30 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
    Detailed:
    | request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |----------------+---------+------------------+-------+-------+-------+-------|
    | /              |       1 |            0.000 |     0 |     0 |     1 |     0 |
    

    alternative use native -i 'status == 444' flag
    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'
    

    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'
    running for 0 seconds, 1 records processed: 369.97 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
    Detailed:
    | request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |----------------+---------+------------------+-------+-------+-------+-------|
    | /              |       1 |            0.000 |     0 |     0 |     1 |     0 |
    

    print request url, HTTP Status code and http user agent
    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
    

    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
    running for 0 seconds, 1 records processed: 335.92 req/sec
    
    request, status, http_user_agent:
    | request         |   status | http_user_agent   |
    |-----------------+----------+-------------------|
    | HEAD / HTTP/1.1 |      444 | curl/7.29.0       |
    

    group by remote IP address via --group-by remote_addr
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr
    

    output where 192.168.0.1 is the visitor ip address
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by remote_addr
    running for 0 seconds, 1 records processed: 1597.22 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
    Detailed:
    | remote_addr   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------------+---------+------------------+-------+-------+-------+-------|
    | 192.168.0.1 |       1 |            0.000 |     0 |     0 |     1 |     0 |
    

    group by user agent string via --group-by http_user_agent
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
    

    the test command was run via curl hence curl user agent string
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
    running for 0 seconds, 1 records processed: 1402.31 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
    Detailed:
    | http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |-------------------+---------+------------------+-------+-------+-------+-------|
    | curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |
    

    filter by specific date using grep i.e. April 2nd would filter by '02/Apr'
    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent
    

    Code (Text):
    grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | grep '02/Apr' | ngxtop --no-follow --group-by http_user_agent
    running for 0 seconds, 1 records processed: 1742.54 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
    Detailed:
    | http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |-------------------+---------+------------------+-------+-------+-------+-------|
    | curl/7.29.0       |       1 |            0.000 |     0 |     0 |     1 |     0 |
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    35,135
    7,753
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,956
    Local Time:
    3:38 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Test the rate limiting of a bot with user agent = Baidu which is on the aggressive rate limiting profile of 2 requests/second specified in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf

    in /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf
    Code (Text):
    # limits for Zone $bad_bot = 2
    # this rate limiting will only take effect if you change any of the bots and change
    # their block value from 1 to 2.
            limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
            limit_req_zone  $bot_iplimit zone=bot2_reqlimitip:16m  rate=2r/s;
    

    Siege load test with user agent Baidu rate limited with a HTTP 503 status code served up to rated limited requests
    Code (Text):
    siege -b -c10 -r100 -A "Baidu" http://domain1.com/
    
    Lifting the server siege...
    Transactions:                    277 hits
    Availability:                  51.87 %
    Elapsed time:                 138.37 secs
    Data transferred:               1.10 MB
    Response time:                  4.90 secs
    Transaction rate:               2.00 trans/sec
    Throughput:                     0.01 MB/sec
    Concurrency:                    9.80
    Successful transactions:         277
    Failed transactions:             257
    Longest transaction:            5.01
    Shortest transaction:           0.00
    

    ngxtop with Baidu user agent string rate limited results for HTTP 503 status codes in acces log
    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
    running for 1 seconds, 6081 records processed: 7868.07 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |    6081 |         2060.460 |   373 |     0 |    81 |  5627 |
    
    Detailed:
    | request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |----------------+---------+------------------+-------+-------+-------+-------|
    | /              |    6081 |         2060.460 |   373 |     0 |    81 |  5627 |
    

    Code (Text):
    cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow --group-by http_user_agent   
    running for 1 seconds, 6153 records processed: 7625.33 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |    6153 |         2056.997 |   432 |     0 |    87 |  5634 |
    
    Detailed:
    | http_user_agent   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |-------------------+---------+------------------+-------+-------+-------+-------|
    | Baidu             |    6151 |         2057.666 |   431 |     0 |    86 |  5634 |
    | curl/7.29.0       |       2 |            0.000 |     1 |     0 |     1 |     0 |
    
     
..
Thread Status:
Not open for further replies.