Learn about Centmin Mod LEMP Stack today
Register Now

Security Blocking bad or aggressive bots

Discussion in 'System Administration' started by eva2000, Feb 28, 2016.

  1. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  2. Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    10:28 PM
    1.9.x
    10.1.x
  3. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    probably some kind of library like lib perl which can be used for such an attack
     
  4. Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    10:28 PM
    1.9.x
    10.1.x
    Yes, it can be that. For now i will leave Pcore-HTTP blocked in Nginx. A normal user don't use that user agent.
     
    • Like Like x 1
  5. Revenge

    Revenge Active Member

    443
    92
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +333
    Local Time:
    10:28 PM
    1.9.x
    10.1.x
    CPU and Load usage when the attack was happening and after i blocked it with Nginx.

    [​IMG]
     
    • Like Like x 1
  6. narji

    narji Member

    69
    6
    8
    Feb 4, 2016
    Ratings:
    +12
    Local Time:
    5:28 AM
    hallo ,
    i have change nginx.conf and vhosdomain.conf
    set baiduspider to 3

    first test using command
    siege -b -c10 -r1 -A "baiduspider" "http://..Domain.com"
    ** SIEGE 4.0.2
    ** Preparing 10 concurrent users for battle.
    The server is now under siege...

    Transactions: 0 hits
    Availability: 0.00 %
    Elapsed time: 0.02 secs
    Data transferred: 0.00 MB
    Response time: 0.00 secs
    Transaction rate: 0.00 trans/sec
    Throughput: 0.00 MB/sec
    Concurrency: 0.00
    Successful transactions: 0
    Failed transactions: 10
    Longest transaction: 0.00
    Shortest transaction: 0.00

    succes not able to penetrate

    then i look in access.log from vhost domain
    180.76.15.137 - - [11/Sep/2016:14:02:12 +0700] "GET /wp-content/plugins/woocommerce/assets/css/woocommerce.css HTTP/1.1" 200 7801 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
    180.76.15.157 - - [11/Sep/2016:14:34:48 +0700] "GET /wp-content/plugins/woocommerce/assets/js/select2/select2.min.js HTTP/1.1" 200 19226 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
    180.76.15.9 - - [11/Sep/2016:14:37:35 +0700] "GET /wp-content/plugins/woocommerce/assets/js/select2/select2.min.js HTTP/1.1" 200 18923 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
    180.76.15.11 - - [11/Sep/2016:15:04:27 +0700] "GET /wp-content/plugins/woocommerce/assets/css/woocommerce-layout.css HTTP/1.1" 200 2012 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-

    test again with command from shell
    siege -b -c10 -r1 -A "baiduspider" http://domain.com/wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js

    ** SIEGE 4.0.2
    ** Preparing 10 concurrent users for battle.
    The server is now under siege...
    HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.04 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
    HTTP/1.1 200 0.04 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js

    Transactions: 10 hits
    Availability: 100.00 %
    Elapsed time: 0.04 secs
    Data transferred: 0.00 MB
    Response time: 0.02 secs
    Transaction rate: 250.00 trans/sec
    Throughput: 0.07 MB/sec
    Concurrency: 4.75
    Successful transactions: 10
    Failed transactions: 0
    Longest transaction: 0.04
    Shortest transaction: 0.01

    what do i miss here.
    the bot still able to penetrate plugin directory in public
     
    Last edited: Sep 11, 2016
  7. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    1. Static files are processed by /usr/local/nginx/conf/staticfiles.conf include file in your nginx vhost, so if you want to rate limit static files you need add the 1st post created include file /usr/local/nginx/conf/blockbots.conf to those location matches add
      Code (Text):
      include /usr/local/nginx/conf/blockbots.conf;
      However, static files shouldn't be a problem for most folks as Nginx eats static file process requests for breakfast and handle them with ease :)
    2. If you used centmin mod 123.09beta01 centmin.sh menu option 22 to auto install wordpress, you would also have white listed location matches for common wp plugins in auto created file at /usr/local/nginx/conf/wpincludes/${vhostname}/wpsecure_${vhostname}.conf where ${vhostname} is yourdomain.com name. Similar to staticfiles.conf, you would need to edit wpsecure_${vhostname}.conf each location match for white listed wp plugins with
      Code (Text):
      include /usr/local/nginx/conf/blockbots.conf;
      . With latest 123.09beta01 update also added to a common include file /usr/local/nginx/conf/wpincludes/${vhostname}/wpwhitelist_common.conf in generated /usr/local/nginx/conf/wpincludes/${vhostname}/wpsecure_${vhostname}.conf which you can add the include file /usr/local/nginx/conf/blockbots.conf to so it is shared with all whitelisted location matches.
      Code (Text):
        # below include file needs to be manually created at that path and to be uncommented
        # by removing the hash # in front of below line to take effect. This wpwhitelist_common.conf
        # allows you to add commonly shared settings to all wp plugin location matches which
        # whitelist php processing access at /usr/local/nginx/conf/wpincludes/${vhostname}/wpsecure_${vhostname}.conf
        #include /usr/local/nginx/conf/wpincludes/${vhostname}/wpwhitelist_common.conf;
      
     
    Last edited: Sep 19, 2016
  8. rdan

    rdan Well-Known Member

    4,740
    1,144
    113
    May 25, 2014
    Ratings:
    +1,711
    Local Time:
    6:28 AM
    Mainline
    10.2
    I am using this approach now on one of the site I manage.
    Thanks a lot Eva and everyone. :)
     
  9. cloud9

    cloud9 Active Member

    140
    54
    28
    Oct 6, 2015
    England
    Ratings:
    +75
    Local Time:
    10:28 PM
    1.11.x
    10.x
    For one forum board (UK Based) we use IPSet to block whole countries, we have no posters ever (5 years) from these countries other than spammers. And for bots we use the conf eva posted above.
     
    • Like Like x 1
  10. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Glad to hear this bad bot blocking as been useful to you folks :D
     
  11. Sunka

    Sunka Well-Known Member

    1,107
    301
    83
    Oct 31, 2015
    Rijeka, Croatia
    Ratings:
    +488
    Local Time:
    11:28 PM
    Nginx 1.15.8
    MariaDB 10.3.12
    @eva2000 I have problems understanding your first post tutorial.

    Inside nginx.conf within http{} context add include file
    Code:
    include /usr/local/nginx/conf/botlimit.conf;
    (also change map_hash_bucket_size 128; to 512)
    So it will be like this:
    Code:
    http {
     include /usr/local/nginx/conf/brotli_inc.conf;
     include /usr/local/nginx/conf/botlimit.conf;
     map_hash_bucket_size 128;
     map_hash_max_size 4096;
     server_names_hash_bucket_size 128;
     server_names_hash_max_size 2048;
     variables_hash_max_size 2048;
    You wrote - within the include file /usr/local/nginx/conf/botlimit.conf add:
    /content from first post/

    but there is difference between that you wrote in post and what I have alredy in file.

    Next, you wrote:
    in your nginx vhost's location context add an include file
    Code:
    include /usr/local/nginx/conf/blockbots.conf;
    So that would be in my case /usr/local/nginx/conf/conf.d/pijanitvor.com.ssl.conf ??
    If yes, that include part would be there:
    Code:
    root /home/nginx/domains/pijanitvor.com/public;
    
      location / {
            index index.php index.html index.htm;
            try_files $uri $uri/ /index.php?$uri&$args;
            include /usr/local/nginx/conf/blockbots.conf;
        }
    
        location /internal_data/ {
            internal;
            allow 127.0.0.1;
    Next you wrote in include file /usr/local/nginx/conf/blockbots.conf add
    ....

    But that is already included there, so nothing to add I think.

    And after that, just restart nginx.

    What about /usr/local/nginx/conf/block.conf
    leave commented or not?
     
  12. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    move it below variables_hash_max_size 2048; line

    what's differing ? use the contents that already exists in 123.09beta01 included version of /usr/local/nginx/conf/botlimit.conf

    yes
    yeah in each location match you want i.e. web root / or maybe if you have separate location match for /search etc etc

    blockbots.conf doesn't exist in any vhost by default only block.conf does and it's commented out by default
    leave it commented out or replace with blockbot.conf
     
    • Like Like x 1
  13. Sunka

    Sunka Well-Known Member

    1,107
    301
    83
    Oct 31, 2015
    Rijeka, Croatia
    Ratings:
    +488
    Local Time:
    11:28 PM
    Nginx 1.15.8
    MariaDB 10.3.12
    File exist by default, or I do not understand what you mean
    Code:
    [[email protected] ~]# cd /usr/local/nginx/conf/
    
    [[email protected] conf]# ls -la
    total 264
    drwxr-xr-x  6 root root 4096 Nov 15 17:02 .
    drwxr-xr-x 10 root root  128 Nov 11 12:27 ..
    -rw-r--r--  1 root root  362 Nov 11 12:21 503include-main.conf
    -rw-r--r--  1 root root   37 Nov 11 12:21 503include-only.conf
    drwxr-xr-x  4 root root   48 Nov 11 17:33 autoprotect
    -rw-r--r--  1 root root  564 Nov 11 12:21 blockbots.conf
    -rw-r--r--  1 root root 4414 Nov 11 12:21 block.conf
    -rw-r--r--  1 root root 5108 Nov 11 12:21 botlimit.conf
    -rw-r--r--  1 root root  277 Nov 15 17:02 brotli_inc.conf
    -rw-r--r--  1 root root  705 Nov 11 12:21 cloudflare.conf
    drwxr-xr-x  2 root root 4096 Nov 14 03:13 conf.d
    -rw-r--r--  1 root root  626 Nov 11 12:21 drop.conf
    -rw-r--r--  1 root root  264 Nov 15 17:02 dynamic-modules.conf
    -rw-r--r--  1 root root 1219 Nov 11 12:21 errorpage.conf
    -rw-r--r--  1 root root 1077 Nov 11 12:21 fastcgi.conf
    -rw-r--r--  1 root root 1077 Nov 14 03:13 fastcgi.conf.default
    -rw-r--r--  1 root root  140 Nov 11 12:21 fastcgi_param_https_map.conf
    -rw-r--r--  1 root root 1272 Nov 11 12:21 fastcgi_params
    -rw-r--r--  1 root root 1007 Nov 14 03:13 fastcgi_params.default
    -rw-r--r--  1 root root  283 Nov 11 12:21 geoip.conf
    -rw-r--r--  1 root root   32 Nov 11 23:18 htpassphpmyadmin
    -rw-r--r--  1 root root    0 Nov 11 12:21 htpasswd
    -rw-r--r--  1 root root   46 Nov 11 12:26 htpasswd_opcache
    -rw-r--r--  1 root root 4045 Nov 11 12:21 htpasswd.py
    -rwxr-xr-x  1 root root 1459 Nov 15 17:01 htpasswd.sh
    -rw-r--r--  1 root root  212 Nov 11 12:26 include_opcache.conf
    -rw-r--r--  1 root root 2837 Nov 14 03:13 koi-utf
    -rw-r--r--  1 root root 2223 Nov 14 03:13 koi-win
    -rw-r--r--  1 root root  645 Nov 11 12:21 maintenance.conf
    -rw-r--r--  1 root root 4189 Nov 11 12:21 mime.types
    -rw-r--r--  1 root root 3957 Nov 14 03:13 mime.types.default
    -rw-r--r--  1 root root 4653 Nov 15 17:02 nginx.conf
    -rw-r--r--  1 root root 2656 Nov 14 03:13 nginx.conf.default
    -rw-r--r--  1 root root 1740 Nov 11 12:21 pagespeedadmin.conf
    -rw-r--r--  1 root root 7152 Nov 11 12:21 pagespeed.conf
    -rw-r--r--  1 root root  669 Nov 15 17:01 pagespeedhandler.conf
    -rw-r--r--  1 root root  131 Nov 11 15:57 pagespeedstatslog.conf
    -rw-r--r--  1 root root 3376 Nov 11 23:18 php_50_mysqladmin1214.conf
    -rw-r--r--  1 root root 2683 Nov 11 12:21 phpalt.conf
    -rw-r--r--  1 root root 3368 Nov 14 03:13 php.conf
    drwxr-xr-x  2 root root  145 Nov 11 23:18 phpfpmd
    -rw-r--r--  1 root root   23 Nov 11 23:18 phpmyadmin_check
    -rw-r--r--  1 root root   97 Nov 11 23:18 phpmyadmin.conf
    -rw-r--r--  1 root root  307 Nov 11 23:18 phpmyadmin_https.conf
    -rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool2.conf
    -rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool3.conf
    -rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool4.conf
    -rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool5.conf
    -rw-r--r--  1 root root 2461 Nov 11 12:21 phpssl.conf
    -rw-r--r--  1 root root  362 Nov 11 12:21 phpstatus.conf
    -rw-r--r--  1 root root  884 Nov 11 12:21 proxy.conf
    -rw-r--r--  1 root root 2441 Nov 11 12:21 pscontrol
    -rw-r--r--  1 root root  636 Nov 11 12:21 scgi_params
    -rw-r--r--  1 root root  636 Nov 14 03:13 scgi_params.default
    -rw-r--r--  1 root root   10 Nov 11 12:21 sitestatus.conf
    drwxr-xr-x  3 root root  143 Nov 12 02:28 ssl
    -rw-r--r--  1 root root  105 Nov 12 02:26 ssl_include.conf
    -rw-r--r--  1 root root  110 Nov 11 12:21 ssl-include.conf
    -rw-r--r--  1 root root 2236 Nov 11 12:21 staticfiles.conf
    -rw-r--r--  1 root root  664 Nov 11 12:21 uwsgi_params
    -rw-r--r--  1 root root  664 Nov 14 03:13 uwsgi_params.default
    -rw-r--r--  1 root root   26 Nov 11 12:21 vts_http.conf
    -rw-r--r--  1 root root  292 Nov 11 12:21 vts_mainserver.conf
    -rw-r--r--  1 root root   24 Nov 11 12:21 vts_server.conf
    -rw-r--r--  1 root root 3610 Nov 14 03:13 win-utf
    -rw-r--r--  1 root root  888 Nov 11 12:21 wpffpc.conf
    
    
    [[email protected]loud conf]# cat /usr/local/nginx/conf/blockbots.conf
    #######################################################################
    # add this to your nginx vhost domain's config file within
    # the location contexts you want to rate limit and/or bot
    # block with 403 permission denied error or change return 403
    # to return 444 to just drop the connection completely
    #limit_conn bota_connlimit 100;
    limit_conn botb_connlimit 10;
    #limit_req  zone=bota_reqlimitip burst=50;
    limit_req  zone=botb_reqlimitip burst=10;
    if ($bot_agent = '3') {
      return 444;
    }
    #######################################################################

    I only have this three location
    Code:
    root /home/nginx/domains/pijanitvor.com/public;
    
      location / {
            index index.php index.html index.htm;
            try_files $uri $uri/ /index.php?$uri&$args;
        }
    
        location /internal_data/ {
            internal;
            allow 127.0.0.1;
            allow 94.237.29.18;
            deny all;
        }
        
        location /library/ {
            internal;
            allow 127.0.0.1;
            allow 94.237.29.18;
            deny all;
        }
    So on first one would be for all forum?
    Code:
    location / {
            index index.php index.html index.htm;
            try_files $uri $uri/ /index.php?$uri&$args;
            include /usr/local/nginx/conf/blockbots.conf;
        }
     
  14. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    I mean it's not listed as an include file in your actual domain vhost so you need to add the include file to your domain vhost file for domain.com.ssl.conf in your case

    yup
     
    • Like Like x 1
  15. Sunka

    Sunka Well-Known Member

    1,107
    301
    83
    Oct 31, 2015
    Rijeka, Croatia
    Ratings:
    +488
    Local Time:
    11:28 PM
    Nginx 1.15.8
    MariaDB 10.3.12
    Done.

    Step 1
    add include part in /usr/local/nginx/conf/nginx.conf
    and change map_hash_bucket_size from 128 to 512
    Code:
    http {
     include /usr/local/nginx/conf/brotli_inc.conf;
     map_hash_bucket_size 512;
     map_hash_max_size 4096;
     server_names_hash_bucket_size 128;
     server_names_hash_max_size 2048;
     variables_hash_max_size 2048;
     include /usr/local/nginx/conf/botlimit.conf;
    And add include part to /usr/local/nginx/conf/conf.d/pijanitvor.com.ssl.conf under location /
    Code:
    location / {
            index index.php index.html index.htm;
            try_files $uri $uri/ /index.php?$uri&$args;
            include /usr/local/nginx/conf/blockbots.conf;
        }
    and then
    Code:
    [[email protected] ~]# nginx -t
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    [[email protected] ~]# ngxrestart
    Restarting nginx (via systemctl):                          [  OK  ]

    Did I miss something?
     
  16. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    looks right
     
    • Winner Winner x 1
  17. Mak Adang

    Mak Adang Member

    60
    20
    8
    Jan 4, 2015
    Ratings:
    +23
    Local Time:
    5:28 AM
    1.9.0
    10.0.18-MariaDB
    Very helpfull @eva2000 baidu is my firs priority to block.
     
    • Like Like x 1
  18. pamamolf

    pamamolf Premium Member Premium Member

    3,581
    345
    83
    May 31, 2014
    Ratings:
    +666
    Local Time:
    12:28 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Baidu and yandex are very aggresive and i think they ignore also robots.txt :(
     
  19. Sunka

    Sunka Well-Known Member

    1,107
    301
    83
    Oct 31, 2015
    Rijeka, Croatia
    Ratings:
    +488
    Local Time:
    11:28 PM
    Nginx 1.15.8
    MariaDB 10.3.12
    I did as I wrote, but still do not know is that working or not?
    Also, after any change in /usr/local/nginx/conf/botlimit.conf (example from 2 to 3 for some bots) shoud I restart nginx or something else?
     
  20. eva2000

    eva2000 Administrator Staff Member

    42,352
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    8:28 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    1st post as examples of testing
     
    • Informative Informative x 1