Welcome to Centmin Mod Community
Become a Member

Security Use CSF+LFD as a fail2ban equivalent (including cloudflare support)

Discussion in 'System Administration' started by ethanpil, May 17, 2017.

  1. ethanpil

    ethanpil Active Member

    167
    46
    28
    Nov 8, 2015
    Ratings:
    +86
    Local Time:
    5:17 AM
    I have been working for a while on getting LFD to perform IP bans from NginX similar to the way fail2ban does. Lately @eva2000 has been implementing fail2ban into centminmod, but this inspired me even more to finish my project.

    IMHO this is much easier to manage and maintain than fail2ban and provides the same security with less hassle to maintain.

    Benefits of the system:
    • Reads standard nginx logs and bans IPs that break too many nginx security rules
    • CSF/LFD is already installed and running well with Centminmod. No new daemons
    • Easy to add many more blocks and rules as needed for almost any system log file with simple regex (added to regex.custom.pm)
    • Wordpress support for fail2ban plugins hard ruleset (just install WP plugin)
    • LFD will unblock from cloudflare when the block expires
    • stable and reliable, has been running on my VPS for a while with no issues
    How does it work?

    CSF/LFD already parses system logs using regex to find IPs that should be banned for port scanning, failed SSH logins, etc. We just need to add the nginx logs and some rules for CSF/LFD to follow. Wordpress fail2ban plugins work by creating log entries in the syslog when bad events happen (which CSF/LFD already scans), so we just need to add awareness of to the CSF ruleset as well.

    So, we simply need to point CSF/LFD to scan our NginX logs. Next, CSF/LFD has file called regex.custom.pm that allows us to give CSF/LFD extra rules for logfiles. So I converted the fail2ban wordpress rules, added a few of my own and viola, bad web IPs are now banned in CSF/LFD!

    But that wont help when working through cloudflare... we still need to update their firewall. Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!

    So, how is it done?

    1. Backup existing CSF ruleset


    Code:
    cp /usr/local/csf/bin/regex.custom.pm /usr/local/csf/bin/regex.custom.pm.bak
    
    2. Edit /usr/local/csf/bin/regex.custom.pm and add new entries

    This is where we add our regex rules for CSF to find things in the log files we want to ban IP addresses for.

    Code:
    
    # NginX security rules trigger (Default: 4 errors bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_security","4","80,443","86400");
    }
    
    # NginX 404 errors (Default: 4 errors bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_404s","4","80,443","86400");
    }
    
    #Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
        return ("Trying to download .ht files",$2,"nginx_htfiles","1","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Authentication attempt for unknown user .* from (.*)\n/)) {
      return ("Wordpress unknown user from",$1,"fail2ban_unknownuser","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Blocked user enumeration attempt from (.*)\n/)) {
      return ("WordPress user enumeration attempt from",$1,"fail2ban_userenum","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Pingback error .* generated from (.*)\n/)) {
      return ("WordPress pingback error",$1,"fail2ban_pingback","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Spammed comment from (.*)\n/)) {
      return ("WordPress spam comments from",$1,"fail2ban_spam","2","80,443","86400");
    }
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*XML-RPC multicall authentication failure (.*)\n/)) {
      return ("WordPress XML-RPC multicall fail from",$1,"fail2ban_xmlrpc","5","80,443","86400");
    }
    
    3. Install lfd-cloudflare script in /etc/csf/scripts

    Here is a bash one liner to install it into /etc/csf/scripts/cloudflare, plus also install jq, a json parser for linux cli :

    Code:
    mkdir /etc/csf/scripts; mkdir /etc/csf/scripts/cloudflare; wget https://github.com/ducohosting/lfd-cloudflare/archive/master.zip -O /etc/csf/scripts/cloudflare/temp.zip; unzip -j /etc/csf/scripts/cloudflare/temp.zip -d /etc/csf/scripts/cloudflare/; rm -f /etc/csf/scripts/cloudflare/temp.zip; chmod +x /etc/csf/scripts/cloudflare/*.sh; yum install jq.x86_64 -y;
    
    4. Add your api keys to /etc/csf/scripts/cloudflare/secret.sh

    lfd-cloudflare needs your cloudflare credentials to make changes to their firewall rules

    Code:
    #!/bin/bash
    key="abcdefg1234"
    email="johndoe@example.com"
    
    5. Edit /etc/csf/csf.conf in section "Log File Locations" and add nginx logs to LFD scan

    CUSTOM1_LOG is variable which contains the NginX logfile path which CSF/LFD will now scan. Luckily we can wildcard here, so it will scan the logs for all domains on the host. We can add up to 9 custom logs. The NginX will be the first new custom log file.

    If you look in the file, you will notice SYSLOG_LOG is already there, which is what is used by the wordpress fail2ban plugins. And our rules above reference it.

    Later, perhaps we can add other log files and reference them in additional custom rules.

    Code:
    CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
    
    6. Edit /etc/csf/csf.conf in section "Reporting Settings" and add lfd-cloudflare scripts

    This tells CSF/LFD to run the proper scripts from lfd-cloudflare to ban or unban the IP as needed.

    Code:
    BLOCK_REPORT = "/etc/csf/scripts/cloudflare/block.sh"
    UNBLOCK_REPORT = "/etc/csf/scripts/cloudflare/unblock.sh"
    
    7. Optional, install wp-fail2ban-redux/ plugin into your wordpress sites and this will automatically start logging data to the syslog for CSF to find.

    WP Fail2Ban Redux
     
    Last edited by a moderator: May 19, 2017
    • Like Like x 2
    • Winner Winner x 1
    • Useful Useful x 1
  2. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
    very nice work @ethanpil - just right for folks who don't want the added install for fail2ban though you'd need to install the wordpress plugins. Now I am curious which performs better for high traffic rates of attacks with very large log files, pure CSF as you have now or fail2ban + CSF
     
  3. ethanpil

    ethanpil Active Member

    167
    46
    28
    Nov 8, 2015
    Ratings:
    +86
    Local Time:
    5:17 AM
    @eva2000 thanks. please feel free to incorporate this work into the codebase.
     
  4. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
    most likely this and my fail2ban would be optional addons and not in the core by default so folks can opt to enable so they don't run into troubles i.e. 4x 404 not found entries and ban might be problematic for someone just setting up their site for the first time on centmin mod nginx and without proper nginx rules are getting 404 not found on their specific web apps.
     
  5. ethanpil

    ethanpil Active Member

    167
    46
    28
    Nov 8, 2015
    Ratings:
    +86
    Local Time:
    5:17 AM
    Agreed that the security settings are better for production vs. dev. I'm glad to hear it will be an addon.

    BTW, dont you also need always to install fail2ban plugins in Wordpress when using it the traditional way (standard fail2ban install)?
     
  6. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
    for fail2ban + CSF implementation I have dedicated fail2ban jail filters for both when fail2ban wp plugins are installed as well as when they are not.
     
  7. ethanpil

    ethanpil Active Member

    167
    46
    28
    Nov 8, 2015
    Ratings:
    +86
    Local Time:
    5:17 AM
    Last edited: May 18, 2017
    • Like Like x 2
  8. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
    Very nice addition (y)
     
  9. ethanpil

    ethanpil Active Member

    167
    46
    28
    Nov 8, 2015
    Ratings:
    +86
    Local Time:
    5:17 AM
    I cannot edit the original post. Need to add a tweak to step 5 above. Should read as follows:

    Code:
    CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
    
    It was scanning ALL the log files, including the archived .gz files the old way, this will prevent it.
     
    Last edited: May 19, 2017
  10. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
  11. eva2000

    eva2000 Administrator Staff Member

    27,179
    6,224
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,154
    Local Time:
    5:17 AM
    Nginx 1.13.x
    MariaDB 5.5
    @ethanpil i edited just this sub forum to allow 14 days post edit limit for promoted users like yourself :)
     
    • Winner Winner x 1