Security Blocking bad or aggressive bots

But it was definitely a ddos attack using that Pcore-HTTP. All the ip's i checked were all from Buffalo Hosting in the United States. My forum is in Portuguese and all my users are from Portugal.

 

Yes, it can be that. For now i will leave Pcore-HTTP blocked in Nginx. A normal user don't use that user agent.

 

CPU and Load usage when the attack was happening and after i blocked it with Nginx.

 

hallo ,
i have change nginx.conf and vhosdomain.conf
set baiduspider to 3

first test using command
siege -b -c10 -r1 -A "baiduspider" "http://..Domain.com"
** SIEGE 4.0.2
** Preparing 10 concurrent users for battle.
The server is now under siege...

Transactions: 0 hits
Availability: 0.00 %
Elapsed time: 0.02 secs
Data transferred: 0.00 MB
Response time: 0.00 secs
Transaction rate: 0.00 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 0.00
Successful transactions: 0
Failed transactions: 10
Longest transaction: 0.00
Shortest transaction: 0.00

succes not able to penetrate

then i look in access.log from vhost domain
180.76.15.137 - - [11/Sep/2016:14:02:12 +0700] "GET /wp-content/plugins/woocommerce/assets/css/woocommerce.css HTTP/1.1" 200 7801 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
180.76.15.157 - - [11/Sep/2016:14:34:48 +0700] "GET /wp-content/plugins/woocommerce/assets/js/select2/select2.min.js HTTP/1.1" 200 19226 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
180.76.15.9 - - [11/Sep/2016:14:37:35 +0700] "GET /wp-content/plugins/woocommerce/assets/js/select2/select2.min.js HTTP/1.1" 200 18923 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-
180.76.15.11 - - [11/Sep/2016:15:04:27 +0700] "GET /wp-content/plugins/woocommerce/assets/css/woocommerce-layout.css HTTP/1.1" 200 2012 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +百度用户服务中心-站长平台" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-

test again with command from shell
siege -b -c10 -r1 -A "baiduspider" http://domain.com/wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js

** SIEGE 4.0.2
** Preparing 10 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.01 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.02 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.04 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js
HTTP/1.1 200 0.04 secs: 277 bytes ==> GET /wp-content/plugins/woocommerce/assets/js/frontend/woocommerce.min.js

Transactions: 10 hits
Availability: 100.00 %
Elapsed time: 0.04 secs
Data transferred: 0.00 MB
Response time: 0.02 secs
Transaction rate: 250.00 trans/sec
Throughput: 0.07 MB/sec
Concurrency: 4.75
Successful transactions: 10
Failed transactions: 0
Longest transaction: 0.04
Shortest transaction: 0.01

what do i miss here.
the bot still able to penetrate plugin directory in public

 

I am using this approach now on one of the site I manage.
Thanks a lot Eva and everyone.

 

For one forum board (UK Based) we use IPSet to block whole countries, we have no posters ever (5 years) from these countries other than spammers. And for bots we use the conf eva posted above.

 

@eva2000 I have problems understanding your first post tutorial.

Inside nginx.conf within http{} context add include file

Code:

include /usr/local/nginx/conf/botlimit.conf;

(also change map_hash_bucket_size 128; to 512)
So it will be like this:

Code:

http {
 include /usr/local/nginx/conf/brotli_inc.conf;
 include /usr/local/nginx/conf/botlimit.conf;
 map_hash_bucket_size 128;
 map_hash_max_size 4096;
 server_names_hash_bucket_size 128;
 server_names_hash_max_size 2048;
 variables_hash_max_size 2048;

You wrote - within the include file /usr/local/nginx/conf/botlimit.conf add:
/content from first post/

but there is difference between that you wrote in post and what I have alredy in file.

Next, you wrote:
in your nginx vhost's location context add an include file

Code:

include /usr/local/nginx/conf/blockbots.conf;

So that would be in my case /usr/local/nginx/conf/conf.d/pijanitvor.com.ssl.conf ??
If yes, that include part would be there:

Code:

root /home/nginx/domains/pijanitvor.com/public;

  location / {
        index index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$uri&$args;
        include /usr/local/nginx/conf/blockbots.conf;
    }

    location /internal_data/ {
        internal;
        allow 127.0.0.1;

Next you wrote in include file /usr/local/nginx/conf/blockbots.conf add
....

But that is already included there, so nothing to add I think.

And after that, just restart nginx.

What about /usr/local/nginx/conf/block.conf
leave commented or not?

 

File exist by default, or I do not understand what you mean

Code:

[root@upcloud ~]# cd /usr/local/nginx/conf/

[root@upcloud conf]# ls -la
total 264
drwxr-xr-x  6 root root 4096 Nov 15 17:02 .
drwxr-xr-x 10 root root  128 Nov 11 12:27 ..
-rw-r--r--  1 root root  362 Nov 11 12:21 503include-main.conf
-rw-r--r--  1 root root   37 Nov 11 12:21 503include-only.conf
drwxr-xr-x  4 root root   48 Nov 11 17:33 autoprotect
-rw-r--r--  1 root root  564 Nov 11 12:21 blockbots.conf
-rw-r--r--  1 root root 4414 Nov 11 12:21 block.conf
-rw-r--r--  1 root root 5108 Nov 11 12:21 botlimit.conf
-rw-r--r--  1 root root  277 Nov 15 17:02 brotli_inc.conf
-rw-r--r--  1 root root  705 Nov 11 12:21 cloudflare.conf
drwxr-xr-x  2 root root 4096 Nov 14 03:13 conf.d
-rw-r--r--  1 root root  626 Nov 11 12:21 drop.conf
-rw-r--r--  1 root root  264 Nov 15 17:02 dynamic-modules.conf
-rw-r--r--  1 root root 1219 Nov 11 12:21 errorpage.conf
-rw-r--r--  1 root root 1077 Nov 11 12:21 fastcgi.conf
-rw-r--r--  1 root root 1077 Nov 14 03:13 fastcgi.conf.default
-rw-r--r--  1 root root  140 Nov 11 12:21 fastcgi_param_https_map.conf
-rw-r--r--  1 root root 1272 Nov 11 12:21 fastcgi_params
-rw-r--r--  1 root root 1007 Nov 14 03:13 fastcgi_params.default
-rw-r--r--  1 root root  283 Nov 11 12:21 geoip.conf
-rw-r--r--  1 root root   32 Nov 11 23:18 htpassphpmyadmin
-rw-r--r--  1 root root    0 Nov 11 12:21 htpasswd
-rw-r--r--  1 root root   46 Nov 11 12:26 htpasswd_opcache
-rw-r--r--  1 root root 4045 Nov 11 12:21 htpasswd.py
-rwxr-xr-x  1 root root 1459 Nov 15 17:01 htpasswd.sh
-rw-r--r--  1 root root  212 Nov 11 12:26 include_opcache.conf
-rw-r--r--  1 root root 2837 Nov 14 03:13 koi-utf
-rw-r--r--  1 root root 2223 Nov 14 03:13 koi-win
-rw-r--r--  1 root root  645 Nov 11 12:21 maintenance.conf
-rw-r--r--  1 root root 4189 Nov 11 12:21 mime.types
-rw-r--r--  1 root root 3957 Nov 14 03:13 mime.types.default
-rw-r--r--  1 root root 4653 Nov 15 17:02 nginx.conf
-rw-r--r--  1 root root 2656 Nov 14 03:13 nginx.conf.default
-rw-r--r--  1 root root 1740 Nov 11 12:21 pagespeedadmin.conf
-rw-r--r--  1 root root 7152 Nov 11 12:21 pagespeed.conf
-rw-r--r--  1 root root  669 Nov 15 17:01 pagespeedhandler.conf
-rw-r--r--  1 root root  131 Nov 11 15:57 pagespeedstatslog.conf
-rw-r--r--  1 root root 3376 Nov 11 23:18 php_50_mysqladmin1214.conf
-rw-r--r--  1 root root 2683 Nov 11 12:21 phpalt.conf
-rw-r--r--  1 root root 3368 Nov 14 03:13 php.conf
drwxr-xr-x  2 root root  145 Nov 11 23:18 phpfpmd
-rw-r--r--  1 root root   23 Nov 11 23:18 phpmyadmin_check
-rw-r--r--  1 root root   97 Nov 11 23:18 phpmyadmin.conf
-rw-r--r--  1 root root  307 Nov 11 23:18 phpmyadmin_https.conf
-rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool2.conf
-rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool3.conf
-rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool4.conf
-rw-r--r--  1 root root 3372 Nov 11 12:21 php-pool5.conf
-rw-r--r--  1 root root 2461 Nov 11 12:21 phpssl.conf
-rw-r--r--  1 root root  362 Nov 11 12:21 phpstatus.conf
-rw-r--r--  1 root root  884 Nov 11 12:21 proxy.conf
-rw-r--r--  1 root root 2441 Nov 11 12:21 pscontrol
-rw-r--r--  1 root root  636 Nov 11 12:21 scgi_params
-rw-r--r--  1 root root  636 Nov 14 03:13 scgi_params.default
-rw-r--r--  1 root root   10 Nov 11 12:21 sitestatus.conf
drwxr-xr-x  3 root root  143 Nov 12 02:28 ssl
-rw-r--r--  1 root root  105 Nov 12 02:26 ssl_include.conf
-rw-r--r--  1 root root  110 Nov 11 12:21 ssl-include.conf
-rw-r--r--  1 root root 2236 Nov 11 12:21 staticfiles.conf
-rw-r--r--  1 root root  664 Nov 11 12:21 uwsgi_params
-rw-r--r--  1 root root  664 Nov 14 03:13 uwsgi_params.default
-rw-r--r--  1 root root   26 Nov 11 12:21 vts_http.conf
-rw-r--r--  1 root root  292 Nov 11 12:21 vts_mainserver.conf
-rw-r--r--  1 root root   24 Nov 11 12:21 vts_server.conf
-rw-r--r--  1 root root 3610 Nov 14 03:13 win-utf
-rw-r--r--  1 root root  888 Nov 11 12:21 wpffpc.conf


[root@upcloud conf]# cat /usr/local/nginx/conf/blockbots.conf
#######################################################################
# add this to your nginx vhost domain's config file within
# the location contexts you want to rate limit and/or bot
# block with 403 permission denied error or change return 403
# to return 444 to just drop the connection completely
#limit_conn bota_connlimit 100;
limit_conn botb_connlimit 10;
#limit_req  zone=bota_reqlimitip burst=50;
limit_req  zone=botb_reqlimitip burst=10;
if ($bot_agent = '3') {
  return 444;
}
#######################################################################

I only have this three location

Code:

root /home/nginx/domains/pijanitvor.com/public;

  location / {
        index index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$uri&$args;
    }

    location /internal_data/ {
        internal;
        allow 127.0.0.1;
        allow 94.237.29.18;
        deny all;
    }
    
    location /library/ {
        internal;
        allow 127.0.0.1;
        allow 94.237.29.18;
        deny all;
    }

So on first one would be for all forum?

Code:

location / {
        index index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$uri&$args;
        include /usr/local/nginx/conf/blockbots.conf;
    }

 

Done.

Step 1
add include part in /usr/local/nginx/conf/nginx.conf
and change map_hash_bucket_size from 128 to 512

Code:

http {
 include /usr/local/nginx/conf/brotli_inc.conf;
 map_hash_bucket_size 512;
 map_hash_max_size 4096;
 server_names_hash_bucket_size 128;
 server_names_hash_max_size 2048;
 variables_hash_max_size 2048;
 include /usr/local/nginx/conf/botlimit.conf;

And add include part to /usr/local/nginx/conf/conf.d/pijanitvor.com.ssl.conf under location /

Code:

location / {
        index index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$uri&$args;
        include /usr/local/nginx/conf/blockbots.conf;
    }

and then

Code:

[root@upcloud ~]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@upcloud ~]# ngxrestart
Restarting nginx (via systemctl):                          [  OK  ]

Did I miss something?

 

Very helpfull @eva2000 baidu is my firs priority to block.

 

Baidu and yandex are very aggresive and i think they ignore also robots.txt

 

I did as I wrote, but still do not know is that working or not?
Also, after any change in /usr/local/nginx/conf/botlimit.conf (example from 2 to 3 for some bots) shoud I restart nginx or something else?