Featured Security CSF Firewall native fail2ban functionality

With inspiration from @ethanpil work and my fail2ban work, I am now doing some testing for CSF Firewall native layer 7 application level protections similar to what fail2ban would do but without needing fail2ban to be installed. The thresholds for banning IPs in CSF Firewall for these native rules will need fine tuning and testing so best you test on test Centmin Mod server. Testing and feedback will help improve the rule sets and eventually an addon would be created so folks can optionally install and/or uninstall this CSF Firewall extended functionality

CSF Firewall Layer 7 Application Level Banning

CSF Firewall allows custom server log level regex match banning of visitor IP addresses via customising the file at /usr/local/csf/bin/regex.custom.pm and defining up to 9 custom logs in /etc/csf/csf.conf for CSF Firewall's lfd daemon to monitor and scan. There is no Cloudflare Firewall API support added yet, will do that later.

To setup and test this, please try on test server first with Centmin Mod 123.09beta01. The SSH commands to run to setup are below:
This will

1. backup csf firewall /etc/csf/csf.conf config to profile suffix labelled as backup-b4-customregex
2. backup /usr/local/csf/bin/regex.custom.pm to /usr/local/csf/bin/regex.custom.pm.bak
3. setup 4 custom log definitions in /etc/csf/csf.conf config file
4. wget download the regex.custom.pm from hosted gist file
5. restart csf firewall and lfd daemon
6. grep for new custom log definitions in lfd log

Code (Text):

csf --profile backup backup-b4-customregex
cp -a /usr/local/csf/bin/regex.custom.pm /usr/local/csf/bin/regex.custom.pm.bak
egrep 'CUSTOM1_LOG|CUSTOM2_LOG|CUSTOM3_LOG|CUSTOM4_LOG' /etc/csf/csf.conf
sed -i "s|CUSTOM1_LOG = .*|CUSTOM1_LOG = \"/home/nginx/domains/\*/log/access.log\"|" /etc/csf/csf.conf
sed -i "s|CUSTOM2_LOG = .*|CUSTOM2_LOG = \"/home/nginx/domains/\*/log/error.log\"|" /etc/csf/csf.conf
sed -i "s|CUSTOM3_LOG = .*|CUSTOM3_LOG = \"/var/log/nginx/localhost.access.log\"|" /etc/csf/csf.conf
sed -i "s|CUSTOM4_LOG = .*|CUSTOM4_LOG = \"/var/log/nginx/localhost.error.log\"|" /etc/csf/csf.conf
egrep 'CUSTOM1_LOG|CUSTOM2_LOG|CUSTOM3_LOG|CUSTOM4_LOG' /etc/csf/csf.conf
wget -O /usr/local/csf/bin/regex.custom.pm https://gist.github.com/centminmod/f5551b92b8aba768c3b4db84c57e756d/raw/regex.custom.pm
csf -ra
fgrep 'nginx' /var/log/lfd.log | tail -100

So 4 custom logs setup

Code (Text):

egrep 'CUSTOM1_LOG|CUSTOM2_LOG|CUSTOM3_LOG|CUSTOM4_LOG' /etc/csf/csf.conf
CUSTOM1_LOG = "/home/nginx/domains/*/log/access.log"
CUSTOM2_LOG = "/home/nginx/domains/*/log/error.log"
CUSTOM3_LOG = "/var/log/nginx/localhost.access.log"
CUSTOM4_LOG = "/var/log/nginx/localhost.error.log"

check for lfd.log log to see if it picked up and is watching the 4 defined custom logs

Code (Text):

fgrep 'nginx' /var/log/lfd.log | tail -100
Apr 12 07:48:26 host lfd[12904]: Watching /home/nginx/domains/demodomain.com/log/access.log...
Apr 12 07:52:50 host lfd[13141]: Watching /home/nginx/domains/demodomain.com/log/error.log...
Apr 12 07:52:50 host lfd[13141]: Watching /var/log/nginx/localhost.access.log...
Apr 12 07:52:50 host lfd[13141]: Watching /var/log/nginx/localhost.error.log...

Custom CSF Firewall Regexs

Always check for most up to date regex rule sets for the custom /usr/local/csf/bin/regex.custom.pm that is used here.

---

You can always download and use latest regex.custom.pm using below 2 commands

Code (Text):

wget -O /usr/local/csf/bin/regex.custom.pm https://gist.github.com/centminmod/f5551b92b8aba768c3b4db84c57e756d/raw/regex.custom.pm
csf -ra

Below is current example regex rule sets for

1. nginx HTTP 444 status errors ban if more than 5 entries in logs - ban is for 24hrs (86400 seconds)
2. access forbidden by rule matches ban if more than 40 entries entries in logs - ban is for 24hrs (86400 seconds). High threshold used to account for new web apps/new nginx users configuring their web apps/scripts so they don't run into issues.
3. 404 not found matches ban if more than 50 entries in logs - ban is for 24hrs (86400 seconds). High threshold used to account for new web apps/new nginx users configuring their web apps/scripts so they don't run into issues.
4. attempts to access .htaccess or .htpasswd ban if more than 1 entries in logs - ban is for 24hrs (86400 seconds)
5. wordpress plugin wp-fail2ban-redux logged entries ban if more than 2 entries in logs - ban is for 24hrs (86400 seconds)
6. connection rate limit rules if more than 30 entries in nginx error logs - ban is for 60 mins (3600 seconds)

Code (Text):

# /home/nginx/domains/*/log/access.log
# Nginx 444  (Default: 5 errors bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /(\S+) -.*[GET|POST|HEAD].*(444)/)) {
    return ("Nginx 444",$1,"nginx_444","5","80,443","86400","0");
}

# /var/log/nginx/localhost.access.log
# Nginx 444  (Default: 5 errors bans for 24 hours)
if (($globlogs{CUSTOM3_LOG}{$lgfile}) and ($line =~ /(\S+) -.*[GET|POST|HEAD].*(444)/)) {
    return ("Nginx 444",$1,"nginx_444","5","80,443","86400","0");
}

# /home/nginx/domains/*/log/error.log
# NginX security rules trigger (Default: 40 errors bans for 24 hours)
if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_security","40","80,443","86400","0");
}

# /var/log/nginx/localhost.error.log
# NginX security rules trigger (Default: 40 errors bans for 24 hours)
if (($globlogs{CUSTOM4_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_security","40","80,443","86400","0");
}

# /home/nginx/domains/*/log/error.log
# NginX 404 errors (Default: 50 errors bans for 24 hours)
if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_404s","50","80,443","86400","0");
}

# /var/log/nginx/localhost.error.log
# NginX 404 errors (Default: 50 errors bans for 24 hours)
if (($globlogs{CUSTOM4_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_404s","50","80,443","86400","0");
}

# /home/nginx/domains/*/log/access.log
#Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
    return ("Trying to download .ht files",$1,"nginx_htfiles","1","80,443","86400","0");
}

# /var/log/nginx/localhost.access.log
#Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
if (($globlogs{CUSTOM3_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
    return ("Trying to download .ht files",$1,"nginx_htfiles","1","80,443","86400","0");
}

# Wordpress fail2ban plugin https://wordpress.org/plugins/wp-fail2ban-redux/
# (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Authentication attempt for unknown user .* from (.*)\n/)) {
  return ("Wordpress unknown user from",$1,"fail2ban_unknownuser","2","80,443","86400","0");
}

# Wordpress fail2ban plugin https://wordpress.org/plugins/wp-fail2ban-redux/
# (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Blocked user enumeration attempt from (.*)\n/)) {
  return ("WordPress user enumeration attempt from",$1,"fail2ban_userenum","2","80,443","86400","0");
}

# Wordpress fail2ban plugin https://wordpress.org/plugins/wp-fail2ban-redux/
# (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Pingback error .* generated from (.*)\n/)) {
  return ("WordPress pingback error",$1,"fail2ban_pingback","2","80,443","86400","0");
}

# Wordpress fail2ban plugin https://wordpress.org/plugins/wp-fail2ban-redux/
# (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Spammed comment from (.*)\n/)) {
  return ("WordPress spam comments from",$1,"fail2ban_spam","2","80,443","86400","0");
}

# Wordpress fail2ban plugin https://wordpress.org/plugins/wp-fail2ban-redux/
# (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*XML-RPC multicall authentication failure (.*)\n/)) {
  return ("WordPress XML-RPC multicall fail from",$1,"fail2ban_xmlrpc","5","80,443","86400","0");
}

# /home/nginx/domains/*/log/error.log
# https://community.centminmod.com/posts/74546/
# Nginx connection limit rule trigger (Default: 30 errors bans for 60mins)
if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*limiting connections by zone .*, client: (\S+),(.*)/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_conn_limit","30","80,443","3600","0");
}

# /var/log/nginx/localhost.error.log
# https://community.centminmod.com/posts/74546/
# Nginx connection limit rule trigger (Default: 30 errors bans for 60mins)
if (($globlogs{CUSTOM4_LOG}{$lgfile}) and ($line =~ /.*limiting connections by zone .*, client: (\S+),(.*)/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_conn_limit_localhost","30","80,443","3600","0");
}

Uninstallation

Now if you want to remove this custom testing configuration from CSF Firewall follow these SSH command steps where:

1. profile list will list csf profile backups look for listing name containing _backup_b4_customregex i.e. 1523525787_backup_b4_customregex
2. then restore /etc/csf/csf.conf config backup
3. restore default /usr/local/csf/bin/regex.custom.pm.bak to /usr/local/csf/bin/regex.custom.pm
4. restart csf firewall and lfd daemon

Code (Text):

csf --profile list
csf --profile restore 1523525787_backup_b4_customregex
\cp -af /usr/local/csf/bin/regex.custom.pm.bak /usr/local/csf/bin/regex.custom.pm
csf -ra

CSF Firewall Layer 7 Application Level Ban Notes

• Centmin Mod by default buffers access log writes so they are not written to access logs in real time which would skew CSF Firewall as well as fail2ban usage unless you disable access log buffering (which will increase nginx usage loads).

Unban IP From CSF Firewall

CSF Firewall does a temp IP ban first and if temp IP ban records more than 4 occurrences, CSF Firewall does a permanent ban. To unblock an IP you need to remove it from CSF Firewall's permanent and temporarily ban lists using below commands:

To very if IP is banned for a grep i.e. ip = xxx.xxx.xxx.xxx

Code (Text):

csf -g xxx.xxx.xxx.xxx

To unban from temp ban list

Code (Text):

csf -tr xxx.xxx.xxx.xxx

To unban form permanent ban list

Code (Text):

csf -dr xxx.xxx.xxx.xxx

To verify grep unbanned ip again

Code (Text):

csf -g xxx.xxx.xxx.xxx

 

Example of blocking at CSF Firewall lfd daemon level in /var/log/lfd.log. All custom regex blocks are tagged with label LF_CUSTOMTRIGGER so easy to use fgrep or grep to filter for

Code (Text):

fgrep 'LF_CUSTOMTRIGGER' /var/log/lfd.log                         
Apr 12 08:44:21 hostname lfd[14003]: (nginx_404s) NGINX Security rule triggered from xxx.xxx.xxx.xxx (US/United States/-): 50 in the last 3600 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]
Apr 12 09:04:42 hostname lfd[15422]: (nginx_444) Nginx 444 xxx.xxx.xxx.xxx (US/United States/-): 5 in the last 3600 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]
Apr 12 09:08:23 hostname lfd[15525]: (PERMBLOCK) xxx.xxx.xxx.xxx (US/United States/-) has had more than 4 temp blocks in the last 86400 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]
Apr 12 09:09:22 hostname lfd[15605]: (nginx_444) Nginx 444 xxx.xxx.xxx.xxx (US/United States/-): 5 in the last 3600 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]
Apr 12 09:32:34 hostname lfd[16537]: (nginx_404s) NGINX Security rule triggered from xxx.xxx.xxx.xxx (US/United States/-): 50 in the last 3600 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]

 

Well lets perfect the CSF Firewall regex rules first before looking at Cloudflare Firewall API support

 

Official CSF Firewall support forum also have a dedicated thread where folks post their regex rules too at Custom REGEX rules for CSF. - ConfigServer Community Forum

 

be careful banning on 404 not found can be tricky especially if you misconfigure your initial nginx vhost for your web app i.e. incorrect nginx rewrite rules may lead to all visitors receiving 404 not found which means all visitors would be banned for 24hrs.

 

makes good sense then

 

not tested but something like below

Code (Text):

# /home/nginx/domains/*/log/error.log
# https://community.centminmod.com/posts/74546/
# https://regexr.com/4f47b
# Nginx connection limit rule trigger (Default: 40 errors bans for 24 hours)
if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*limiting connections by zone (.*)/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_conn_limit","40","80,443","86400","0");
}

regex isn't my strength know how wise so used RegExr: Learn, Build, & Test RegEx

 

maybe threshold too high at 100 errors ?

also note

 

ah yes !

check CSF LFD logs ?

Code (Text):

fgrep 'LF_CUSTOMTRIGGER' /var/log/lfd.log | sed -e "s|$(hostname -s)|hostname|g"