Security Use CSF+LFD as a fail2ban equivalent (including cloudflare support)

eva2000 · Jan 3, 2019

rdan said: ↑
So with this code:
Code:
# /home/nginx/domains/*/log/error.log
# NginX security rules trigger (Default: 40 errors bans for 24 hours)
if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_security","40","80,443","86400","0");
}
If a single IP logs 40 access forbidden by rule it will be block for 24 hours.

It doesn't matter if the 40 errors was done for the last 1 minute, 1 day, or 1 week?
It will still be block?
Click to expand...
no that means 40 hits to port 80/443 within past 24hrs (86400 seconds) will be blocked

rdan · Jan 3, 2019

How/Where can we set how many hours it will be blocked on Cloudflare & CSF?
I don't want permanent block.

eva2000 · Jan 3, 2019

correction 86400 is the number of seconds for temp block

Code (Text):

# Example:
#       if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /^\S+\s+\d+\s+\S+ \S+ pure-ftpd: \(\?\@(\d+\.\d+\.\d+\.\d+)\) \[WARNING\] Authentication failed for user/)) {
#               return ("Failed myftpmatch login from",$1,"myftpmatch","5","20,21","1");
#       }
#
# The return values from this example are as follows:
#
# "Failed myftpmatch login from" = text for custom failure message
# $1 = the offending IP address
# "myftpmatch" = a unique identifier for this custom rule, must be alphanumeric and have no spaces
# "5" = the trigger level for blocking
# "20,21" = the ports to block the IP from in a comma separated list, only used if LF_SELECT enabled. To specify the protocol use 53;udp,53;tcp
# "1" = n/temporary (n = number of seconds to temporarily block) or 1/permanant IP block, only used if LF_TRIGGER is disabled

LF_TRIGGER is by default disabled = 0 so it becomes a permanent block - read /etc/csf/csf.conf config file for full details - it's up to you to properly configure this for your needs - part of the info below but full info in /etc/csf/csf.conf CSF Firewall config file.

Code (Text):

###############################################################################
# SECTION:Login Failure Blocking and Alerts
###############################################################################
# The following[*] triggers are application specific. If you set LF_TRIGGER to
# "0" the value of each trigger is the number of failures against that
# application that will trigger lfd to block the IP address
#
# If you set LF_TRIGGER to a value greater than "0" then the following[*]
# application triggers are simply on or off ("0" or "1") and the value of
# LF_TRIGGER is the total cumulative number of failures that will trigger lfd
# to block the IP address
#
# Setting the application trigger to "0" disables it
LF_TRIGGER = "0"

# If LF_TRIGGER is > "0" then LF_TRIGGER_PERM can be set to "1" to permanently
# block the IP address, or LF_TRIGGER_PERM can be set to a value greater than
# "1" and the IP address will be blocked temporarily for that value in seconds.
# For example:
# LF_TRIGGER_PERM = "1" => the IP is blocked permanently
# LF_TRIGGER_PERM = "3600" => the IP is blocked temporarily for 1 hour
#
# If LF_TRIGGER is "0", then the application LF_[application]_PERM value works
# in the same way as above and LF_TRIGGER_PERM serves no function
LF_TRIGGER_PERM = "1"

rc112 · Feb 2, 2019

ethanpil said: ↑
I have been working for a while on getting LFD to perform IP bans from NginX similar to the way fail2ban does. Lately @eva2000 has been implementing fail2ban into centminmod, but this inspired me even more to finish my project.

IMHO this is much easier to manage and maintain than fail2ban and provides the same security with less hassle to maintain.

Benefits of the system:

Reads standard nginx logs and bans IPs that break too many nginx security rules

CSF/LFD is already installed and running well with Centminmod. No new daemons

Easy to add many more blocks and rules as needed for almost any system log file with simple regex (added to regex.custom.pm)

Wordpress support for fail2ban plugins hard ruleset (just install WP plugin)

LFD will unblock from cloudflare when the block expires

stable and reliable, has been running on my VPS for a while with no issues

How does it work?

CSF/LFD already parses system logs using regex to find IPs that should be banned for port scanning, failed SSH logins, etc. We just need to add the nginx logs and some rules for CSF/LFD to follow. Wordpress fail2ban plugins work by creating log entries in the syslog when bad events happen (which CSF/LFD already scans), so we just need to add awareness of to the CSF ruleset as well.

So, we simply need to point CSF/LFD to scan our NginX logs. Next, CSF/LFD has file called regex.custom.pm that allows us to give CSF/LFD extra rules for logfiles. So I converted the fail2ban wordpress rules, added a few of my own and viola, bad web IPs are now banned in CSF/LFD!

But that wont help when working through cloudflare... we still need to update their firewall. Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!

So, how is it done?

1. Backup existing CSF ruleset
Code:
cp /usr/local/csf/bin/regex.custom.pm /usr/local/csf/bin/regex.custom.pm.bak
2. Edit /usr/local/csf/bin/regex.custom.pm and add new entries

This is where we add our regex rules for CSF to find things in the log files we want to ban IP addresses for.
Code:
# NginX security rules trigger (Default: 4 errors bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_security","4","80,443","86400");
}

# NginX 404 errors (Default: 4 errors bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_404s","4","80,443","86400");
}

#Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
    return ("Trying to download .ht files",$2,"nginx_htfiles","1","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Authentication attempt for unknown user .* from (.*)\n/)) {
  return ("Wordpress unknown user from",$1,"fail2ban_unknownuser","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Blocked user enumeration attempt from (.*)\n/)) {
  return ("WordPress user enumeration attempt from",$1,"fail2ban_userenum","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Pingback error .* generated from (.*)\n/)) {
  return ("WordPress pingback error",$1,"fail2ban_pingback","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Spammed comment from (.*)\n/)) {
  return ("WordPress spam comments from",$1,"fail2ban_spam","2","80,443","86400");
}
# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*XML-RPC multicall authentication failure (.*)\n/)) {
  return ("WordPress XML-RPC multicall fail from",$1,"fail2ban_xmlrpc","5","80,443","86400");
}
3. Install lfd-cloudflare script in /etc/csf/scripts

Here is a bash one liner to install it into /etc/csf/scripts/cloudflare, plus also install jq, a json parser for linux cli :
Code:
mkdir /etc/csf/scripts; mkdir /etc/csf/scripts/cloudflare; wget https://github.com/ducohosting/lfd-cloudflare/archive/master.zip -O /etc/csf/scripts/cloudflare/temp.zip; unzip -j /etc/csf/scripts/cloudflare/temp.zip -d /etc/csf/scripts/cloudflare/; rm -f /etc/csf/scripts/cloudflare/temp.zip; chmod +x /etc/csf/scripts/cloudflare/*.sh; yum install jq.x86_64 -y;
4. Add your api keys to /etc/csf/scripts/cloudflare/secret.sh

lfd-cloudflare needs your cloudflare credentials to make changes to their firewall rules
Code:
#!/bin/bash
key="abcdefg1234"
email="johndoe@example.com"
5. Edit /etc/csf/csf.conf in section "Log File Locations" and add nginx logs to LFD scan

CUSTOM1_LOG is variable which contains the NginX logfile path which CSF/LFD will now scan. Luckily we can wildcard here, so it will scan the logs for all domains on the host. We can add up to 9 custom logs. The NginX will be the first new custom log file.

If you look in the file, you will notice SYSLOG_LOG is already there, which is what is used by the wordpress fail2ban plugins. And our rules above reference it.

Later, perhaps we can add other log files and reference them in additional custom rules.
Code:
CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
6. Edit /etc/csf/csf.conf in section "Reporting Settings" and add lfd-cloudflare scripts

This tells CSF/LFD to run the proper scripts from lfd-cloudflare to ban or unban the IP as needed.
Code:
BLOCK_REPORT = "/etc/csf/scripts/cloudflare/block.sh"
UNBLOCK_REPORT = "/etc/csf/scripts/cloudflare/unblock.sh"
7. Optional, install wp-fail2ban-redux/ plugin into your wordpress sites and this will automatically start logging data to the syslog for CSF to find.

WP Fail2Ban Redux
Click to expand...
Hi @ethanpil Nice work. I also think of not installing additional Fail2Ban when CSF+LFD can do the work. My question is your method will work with Apache server? Can I contact you for the details? Thank you.

rc112 · Feb 2, 2019

ethanpil said: ↑
I have been working for a while on getting LFD to perform IP bans from NginX similar to the way fail2ban does. Lately @eva2000 has been implementing fail2ban into centminmod, but this inspired me even more to finish my project.

IMHO this is much easier to manage and maintain than fail2ban and provides the same security with less hassle to maintain.

Benefits of the system:

Reads standard nginx logs and bans IPs that break too many nginx security rules

CSF/LFD is already installed and running well with Centminmod. No new daemons

Easy to add many more blocks and rules as needed for almost any system log file with simple regex (added to regex.custom.pm)

Wordpress support for fail2ban plugins hard ruleset (just install WP plugin)

LFD will unblock from cloudflare when the block expires

stable and reliable, has been running on my VPS for a while with no issues

How does it work?

CSF/LFD already parses system logs using regex to find IPs that should be banned for port scanning, failed SSH logins, etc. We just need to add the nginx logs and some rules for CSF/LFD to follow. Wordpress fail2ban plugins work by creating log entries in the syslog when bad events happen (which CSF/LFD already scans), so we just need to add awareness of to the CSF ruleset as well.

So, we simply need to point CSF/LFD to scan our NginX logs. Next, CSF/LFD has file called regex.custom.pm that allows us to give CSF/LFD extra rules for logfiles. So I converted the fail2ban wordpress rules, added a few of my own and viola, bad web IPs are now banned in CSF/LFD!

But that wont help when working through cloudflare... we still need to update their firewall. Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!

So, how is it done?

1. Backup existing CSF ruleset
Code:
cp /usr/local/csf/bin/regex.custom.pm /usr/local/csf/bin/regex.custom.pm.bak
2. Edit /usr/local/csf/bin/regex.custom.pm and add new entries

This is where we add our regex rules for CSF to find things in the log files we want to ban IP addresses for.
Code:
# NginX security rules trigger (Default: 4 errors bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_security","4","80,443","86400");
}

# NginX 404 errors (Default: 4 errors bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
    return ("NGINX Security rule triggered from",$1,"nginx_404s","4","80,443","86400");
}

#Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
    return ("Trying to download .ht files",$2,"nginx_htfiles","1","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Authentication attempt for unknown user .* from (.*)\n/)) {
  return ("Wordpress unknown user from",$1,"fail2ban_unknownuser","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Blocked user enumeration attempt from (.*)\n/)) {
  return ("WordPress user enumeration attempt from",$1,"fail2ban_userenum","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Pingback error .* generated from (.*)\n/)) {
  return ("WordPress pingback error",$1,"fail2ban_pingback","2","80,443","86400");
}

# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Spammed comment from (.*)\n/)) {
  return ("WordPress spam comments from",$1,"fail2ban_spam","2","80,443","86400");
}
# Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*XML-RPC multicall authentication failure (.*)\n/)) {
  return ("WordPress XML-RPC multicall fail from",$1,"fail2ban_xmlrpc","5","80,443","86400");
}
3. Install lfd-cloudflare script in /etc/csf/scripts

Here is a bash one liner to install it into /etc/csf/scripts/cloudflare, plus also install jq, a json parser for linux cli :
Code:
mkdir /etc/csf/scripts; mkdir /etc/csf/scripts/cloudflare; wget https://github.com/ducohosting/lfd-cloudflare/archive/master.zip -O /etc/csf/scripts/cloudflare/temp.zip; unzip -j /etc/csf/scripts/cloudflare/temp.zip -d /etc/csf/scripts/cloudflare/; rm -f /etc/csf/scripts/cloudflare/temp.zip; chmod +x /etc/csf/scripts/cloudflare/*.sh; yum install jq.x86_64 -y;
4. Add your api keys to /etc/csf/scripts/cloudflare/secret.sh

lfd-cloudflare needs your cloudflare credentials to make changes to their firewall rules
Code:
#!/bin/bash
key="abcdefg1234"
email="johndoe@example.com"
5. Edit /etc/csf/csf.conf in section "Log File Locations" and add nginx logs to LFD scan

CUSTOM1_LOG is variable which contains the NginX logfile path which CSF/LFD will now scan. Luckily we can wildcard here, so it will scan the logs for all domains on the host. We can add up to 9 custom logs. The NginX will be the first new custom log file.

If you look in the file, you will notice SYSLOG_LOG is already there, which is what is used by the wordpress fail2ban plugins. And our rules above reference it.

Later, perhaps we can add other log files and reference them in additional custom rules.
Code:
CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
6. Edit /etc/csf/csf.conf in section "Reporting Settings" and add lfd-cloudflare scripts

This tells CSF/LFD to run the proper scripts from lfd-cloudflare to ban or unban the IP as needed.
Code:
BLOCK_REPORT = "/etc/csf/scripts/cloudflare/block.sh"
UNBLOCK_REPORT = "/etc/csf/scripts/cloudflare/unblock.sh"
7. Optional, install wp-fail2ban-redux/ plugin into your wordpress sites and this will automatically start logging data to the syslog for CSF to find.

WP Fail2Ban Redux
Click to expand...
Hi @ethanpil Nice work. I also think of not installing additional Fail2Ban when CSF+LFD can do the work. My question is your method will work with Apache server? Can I contact you for the details? Thank you.

ethanpil · Feb 3, 2019

It should work without issues, but not with CentMinMod... We are nginx only!
Just need to change the log file formats.... Basically step 2 would need to be redone for your apache log formats.

rdan · Jun 3, 2019

ethanpil said: ↑

UNBLOCK_REPORT = "/etc/csf/scripts/cloudflare/unblock.sh"
Click to expand...

I manually unblock an IP using csf -tr, but it didn't work.
What else could be the problem?

rdan · Sep 1, 2019

ethanpil said: ↑

But that wont help when working through cloudflare... we still need to update their firewall. Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!
Click to expand...

Not working anymore, maybe because of this:
Cloudflare - Cloudflare API Tokens Announced No Longer Need Global API Key !

eva2000 · Sep 1, 2019

rdan said: ↑

Not working anymore, maybe because of this:
Cloudflare - Cloudflare API Tokens Announced No Longer Need Global API Key !
Click to expand...

Cloudflare introduced API Tokens but Global API Key should still work

rdan · Mar 25, 2020

ethanpil said: ↑

Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!
Click to expand...

For new server install, this works fine and tested it myself the first week.
Now after 1 month, I tested this again and suddenly stops working.
Temp Block IP aren't sent to Cloudflare anymore.

Not sure what's the problem.

Log in / Register

Forums

Join the community today

Security Use CSF+LFD as a fail2ban equivalent (including cloudflare support)

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rc112 Member

rc112 Member

ethanpil Active Member

rdan Well-Known Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

.

Log in / Register

Useful Searches

Join the community today

Security Use CSF+LFD as a fail2ban equivalent (including cloudflare support)

eva2000 Administrator Staff Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rc112 Member

rc112 Member

ethanpil Active Member

rdan Well-Known Member

rdan Well-Known Member

eva2000 Administrator Staff Member

rdan Well-Known Member

.