Easy way to find bots and block them from logs

Hello,

The closest I can get it at the moment is using which will extract them into the bots.txt file

Code:

grep 'spider\|bot' access.log | sort -u -f >> bots.txt

Still trying to work out how to just print out the spider / bot name and remove the duplicates

 

I think spider and bots is most common one if you can find a list of other ones you could add them into the grep command or even grep the file against a list.

I have a solution its a bit hacky but works:

Code:

grep 'spider\|bot' access.log | sort -u -f >> bots.txt
grep -o -E '\w+' bots.txt | sort -u -f >> bots2.txt
grep 'spider\|bot' bots2.txt

To input the final results into a file to make it easier:

Code:

grep 'spider\|bot' bots2.txt >> botsfinal.txt

There's prob a cleaner way but it works

 

You could even make the above into a quick script

Code:

#!/bin/bash
grep 'spider\|bot' access.log | sort -u -f >> bots.txt
grep -o -E '\w+' bots.txt | sort -u -f >> bots2.txt
grep 'spider\|bot' bots2.txt >> botsfinal.txt
rm -f bots.txt bots2.txt

The results in botsfinal.txt that you want to block, you could add them into:

Code:

/usr/local/nginx/conf/block.conf

With an additional

Code:

if ($http_user_agent ~ "Name of user agent") {
        set $block_user_agents 1;
    }

Also aslong as the below line is uncommented in your Vhost.

Code:

# block common exploits, sql injections etc
include /usr/local/nginx/conf/block.conf;

Once added to block.conf and that line is uncommented restart nginx for the changes to take affect. Sorry if thats a bit long winded