Welcome to Centmin Mod Community
Register Now

Security Use CSF+LFD as a fail2ban equivalent (including cloudflare support)

Discussion in 'System Administration' started by ethanpil, May 17, 2017.

  1. ethanpil

    ethanpil Active Member

    173
    55
    28
    Nov 8, 2015
    Ratings:
    +101
    Local Time:
    4:47 AM
    I have been working for a while on getting LFD to perform IP bans from NginX similar to the way fail2ban does. Lately @eva2000 has been implementing fail2ban into centminmod, but this inspired me even more to finish my project.

    IMHO this is much easier to manage and maintain than fail2ban and provides the same security with less hassle to maintain.

    Benefits of the system:
    • Reads standard nginx logs and bans IPs that break too many nginx security rules
    • CSF/LFD is already installed and running well with Centminmod. No new daemons
    • Easy to add many more blocks and rules as needed for almost any system log file with simple regex (added to regex.custom.pm)
    • Wordpress support for fail2ban plugins hard ruleset (just install WP plugin)
    • LFD will unblock from cloudflare when the block expires
    • stable and reliable, has been running on my VPS for a while with no issues
    How does it work?

    CSF/LFD already parses system logs using regex to find IPs that should be banned for port scanning, failed SSH logins, etc. We just need to add the nginx logs and some rules for CSF/LFD to follow. Wordpress fail2ban plugins work by creating log entries in the syslog when bad events happen (which CSF/LFD already scans), so we just need to add awareness of to the CSF ruleset as well.

    So, we simply need to point CSF/LFD to scan our NginX logs. Next, CSF/LFD has file called regex.custom.pm that allows us to give CSF/LFD extra rules for logfiles. So I converted the fail2ban wordpress rules, added a few of my own and viola, bad web IPs are now banned in CSF/LFD!

    But that wont help when working through cloudflare... we still need to update their firewall. Well, thanks to this wonderful script: GitHub - ducohosting/lfd-cloudflare: Synchronize LFD blocks with cloudflare we can use CSF/LFD's own reporting mechanism to update cloudflare as the status of an IP changes. So easy!

    So, how is it done?

    1. Backup existing CSF ruleset



    Code:
    cp /usr/local/csf/bin/regex.custom.pm /usr/local/csf/bin/regex.custom.pm.bak
    
    2. Edit /usr/local/csf/bin/regex.custom.pm and add new entries

    This is where we add our regex rules for CSF to find things in the log files we want to ban IP addresses for.

    Code:
    
    # NginX security rules trigger (Default: 4 errors bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_security","4","80,443","86400");
    }
    
    # NginX 404 errors (Default: 4 errors bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_404s","4","80,443","86400");
    }
    
    #Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
        return ("Trying to download .ht files",$2,"nginx_htfiles","1","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Authentication attempt for unknown user .* from (.*)\n/)) {
      return ("Wordpress unknown user from",$1,"fail2ban_unknownuser","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Blocked user enumeration attempt from (.*)\n/)) {
      return ("WordPress user enumeration attempt from",$1,"fail2ban_userenum","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Pingback error .* generated from (.*)\n/)) {
      return ("WordPress pingback error",$1,"fail2ban_pingback","2","80,443","86400");
    }
    
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*Spammed comment from (.*)\n/)) {
      return ("WordPress spam comments from",$1,"fail2ban_spam","2","80,443","86400");
    }
    # Wordpress fail2ban plugin (Default: 2 errors bans for 24 hours)
    if (($globlogs{SYSLOG_LOG}{$lgfile}) and ($line =~ /.*XML-RPC multicall authentication failure (.*)\n/)) {
      return ("WordPress XML-RPC multicall fail from",$1,"fail2ban_xmlrpc","5","80,443","86400");
    }
    
    3. Install lfd-cloudflare script in /etc/csf/scripts

    Here is a bash one liner to install it into /etc/csf/scripts/cloudflare, plus also install jq, a json parser for linux cli :

    Code:
    mkdir /etc/csf/scripts; mkdir /etc/csf/scripts/cloudflare; wget https://github.com/ducohosting/lfd-cloudflare/archive/master.zip -O /etc/csf/scripts/cloudflare/temp.zip; unzip -j /etc/csf/scripts/cloudflare/temp.zip -d /etc/csf/scripts/cloudflare/; rm -f /etc/csf/scripts/cloudflare/temp.zip; chmod +x /etc/csf/scripts/cloudflare/*.sh; yum install jq.x86_64 -y;
    
    4. Add your api keys to /etc/csf/scripts/cloudflare/secret.sh

    lfd-cloudflare needs your cloudflare credentials to make changes to their firewall rules

    Code:
    #!/bin/bash
    key="abcdefg1234"
    email="johndoe@example.com"
    
    5. Edit /etc/csf/csf.conf in section "Log File Locations" and add nginx logs to LFD scan

    CUSTOM1_LOG is variable which contains the NginX logfile path which CSF/LFD will now scan. Luckily we can wildcard here, so it will scan the logs for all domains on the host. We can add up to 9 custom logs. The NginX will be the first new custom log file.

    If you look in the file, you will notice SYSLOG_LOG is already there, which is what is used by the wordpress fail2ban plugins. And our rules above reference it.

    Later, perhaps we can add other log files and reference them in additional custom rules.

    Code:
    CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
    
    6. Edit /etc/csf/csf.conf in section "Reporting Settings" and add lfd-cloudflare scripts

    This tells CSF/LFD to run the proper scripts from lfd-cloudflare to ban or unban the IP as needed.

    Code:
    BLOCK_REPORT = "/etc/csf/scripts/cloudflare/block.sh"
    UNBLOCK_REPORT = "/etc/csf/scripts/cloudflare/unblock.sh"
    
    7. Optional, install wp-fail2ban-redux/ plugin into your wordpress sites and this will automatically start logging data to the syslog for CSF to find.

    WP Fail2Ban Redux
     
    Last edited by a moderator: May 19, 2017
  2. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    very nice work @ethanpil - just right for folks who don't want the added install for fail2ban though you'd need to install the wordpress plugins. Now I am curious which performs better for high traffic rates of attacks with very large log files, pure CSF as you have now or fail2ban + CSF
     
  3. ethanpil

    ethanpil Active Member

    173
    55
    28
    Nov 8, 2015
    Ratings:
    +101
    Local Time:
    4:47 AM
    @eva2000 thanks. please feel free to incorporate this work into the codebase.
     
  4. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    most likely this and my fail2ban would be optional addons and not in the core by default so folks can opt to enable so they don't run into troubles i.e. 4x 404 not found entries and ban might be problematic for someone just setting up their site for the first time on centmin mod nginx and without proper nginx rules are getting 404 not found on their specific web apps.
     
  5. ethanpil

    ethanpil Active Member

    173
    55
    28
    Nov 8, 2015
    Ratings:
    +101
    Local Time:
    4:47 AM
    Agreed that the security settings are better for production vs. dev. I'm glad to hear it will be an addon.

    BTW, dont you also need always to install fail2ban plugins in Wordpress when using it the traditional way (standard fail2ban install)?
     
  6. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    for fail2ban + CSF implementation I have dedicated fail2ban jail filters for both when fail2ban wp plugins are installed as well as when they are not.
     
  7. ethanpil

    ethanpil Active Member

    173
    55
    28
    Nov 8, 2015
    Ratings:
    +101
    Local Time:
    4:47 AM
    Last edited: May 18, 2017
  8. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Very nice addition (y)
     
  9. ethanpil

    ethanpil Active Member

    173
    55
    28
    Nov 8, 2015
    Ratings:
    +101
    Local Time:
    4:47 AM
    I cannot edit the original post. Need to add a tweak to step 5 above. Should read as follows:

    Code:
    CUSTOM1_LOG = "/home/nginx/domains/*/log/*.log"
    
    It was scanning ALL the log files, including the archived .gz files the old way, this will prevent it.
     
    Last edited: May 19, 2017
  10. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  11. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @ethanpil i edited just this sub forum to allow 14 days post edit limit for promoted users like yourself :)
     
  12. JJC84

    JJC84 Ad astra per aspera

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +169
    Local Time:
    12:47 PM
    1.15.x
    10.x.x
    This is great so thank you!
     
  13. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    @ethanpil will be revisiting your work with CSF firewall to see what can be done about tightening Centmin Mod out of box security :)

    is this a typo for $2 instead of $1 for ip field - looks like (\S+) is the variable for $1
    Code (Text):
    #Trying to download htaccess or htpasswd  (Default: 1 error bans for 24 hours)
    if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /.*\.(htpasswd|htaccess).*client: (\S+),.*GET/)) {
       return ("Trying to download .ht files",$2,"nginx_htfiles","1","80,443","86400");
    }
    

    latest regex.custom.pm has a slightly different example to include a field for cloudflare support too
    Code (Text):
    #
    # The regex matches in this file will supercede the matches in regex.pm
    #
    # Example:
    #       if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /^\S+\s+\d+\s+\S+ \S+ pure-ftpd: \(\?\@(\d+\.\d+\.\d+\.\d+)\) \[WARNING\] Authentication failed for user/)) {
    #               return ("Failed myftpmatch login from",$1,"myftpmatch","5","20,21","1","0");
    #       }
    #
    # The return values from this example are as follows:
    #
    # "Failed myftpmatch login from" = text for custom failure message
    # $1 = the offending IP address
    # "myftpmatch" = a unique identifier for this custom rule, must be alphanumeric and have no spaces
    # "5" = the trigger level for blocking
    # "20,21" = the ports to block the IP from in a comma separated list, only used if LF_SELECT enabled. To specify the protocol use 53;udp,53;tcp
    # "1" = n/temporary (n = number of seconds to temporarily block) or 1/permanant IP block, only used if LF_TRIGGER is disabled
    # "0" = whether to trigger Cloudflare block if CF_ENABLE is set. "0" = disable, "1" = enable
    

    looks like you need to separately account for access vs error logs due to where ip address is placed in error logs

    Code (Text):
    CUSTOM1_LOG = "/home/nginx/domains/*/log/access.log"
    CUSTOM2_LOG = "/home/nginx/domains/*/log/error.log"
    CUSTOM3_LOG = "/var/log/nginx/localhost.access.log"
    CUSTOM4_LOG = "/var/log/nginx/localhost.error.log"
    


    access logs
    Code (Text):
    xxx.xxx.xxx.xxx - - [12/Apr/2018:08:17:13 +0000] "GET / HTTP/1.1" 200 4074 "-" "Mozilla/5.0 (unknown-x86_64-linux-gnu) Siege/4.0.2"
    

    error log for 404 not found errors where client ip is not on 1st field but 16th field column where client-ip is or is it matching the 2nd field after client: ip but looks like (\S+) is the variable for $1
    Code (Text):
    2018/04/12 08:21:31 [error] 13435#13435: *11 open() "/usr/local/nginx/html/cmlogo1.png" failed (2: No such file or directory), client: client-ip, server: domain.com.com, request: "GET /cmlogo1.png HTTP/1.1", host: "xxx.xxx.xxx.xxx"
    

    also forbidden regex is different in nginx if forbidden is due to directory indexing disallowed on directories without index file so instead of 'access forbidden by rule, client'
    Code (Text):
    2018/04/12 08:43:37 [error] 13995#13995: *1 directory index of "/usr/local/nginx/html/test/" is forbidden,
    

    which means ip address is listed on 11th field column for access forbidden by rule and on 16th field for is forbidden for directory indexing disallow
    Code (Text):
    Apr 12 08:44:21 hostname lfd[14003]: (nginx_404s) NGINX Security rule triggered from client-ip (US/United States/-): 50 in the last 3600 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]
    

    also centmin mod by default buffers access log writes so they are not written to access logs in real time which would skew CSF Firewall as well as fail2ban usage.

    Dedicated thread for my work on native CSF Firewall layer 7 application level banning Security - CSF Firewall native fail2ban functionality
     
    Last edited: Apr 12, 2018
  14. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    2:47 AM
    Mainline
    10.2
    This isn't needed anymore on Latest CSF?
    As CSF has already built-in feature?
     
  15. eva2000

    eva2000 Administrator Staff Member

    53,821
    12,160
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,712
    Local Time:
    4:47 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    CSF cloudflare support isn't the same thing https://download.configserver.com/csf/readme.txt see highlighted bold parts as LF_MODSEC and LF_CXS do not apply to non-cpanel or centmin mod servers which don't use modsecurity and apache or CXS ConfigServer eXploit Scanner (cxs)
     
  16. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    2:47 AM
    Mainline
    10.2
  17. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    2:47 AM
    Mainline
    10.2
    I just tested only this part, and works great CSF/CF.
    Thanks!
     
  18. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    2:47 AM
    Mainline
    10.2
    I disable access log on all my site and also deny access to localhost/server IP.
    So I only need these 2 regex:
    Code:
    
    # /home/nginx/domains/*/log/error.log
    # NginX security rules trigger (Default: 40 errors bans for 24 hours)
    if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_security","40","80,443","86400","0");
    }
    
    # /home/nginx/domains/*/log/error.log
    # NginX 404 errors (Default: 50 errors bans for 24 hours)
    if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*No such file or directory\), client: (\S+),.*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_404s","50","80,443","86400","0");
    }
    
     
  19. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    2:47 AM
    Mainline
    10.2
    So with this code:
    Code:
    # /home/nginx/domains/*/log/error.log
    # NginX security rules trigger (Default: 40 errors bans for 24 hours)
    if (($globlogs{CUSTOM2_LOG}{$lgfile}) and ($line =~ /.*access forbidden by rule, client: (\S+).*/)) {
        return ("NGINX Security rule triggered from",$1,"nginx_security","40","80,443","86400","0");
    }
    If a single IP logs 40 access forbidden by rule it will be block for 24 hours.

    It doesn't matter if the 40 errors was done for the last 1 minute, 1 day, or 1 week?
    It will still be block?
     
  20. Jay Chen

    Jay Chen Active Member

    181
    60
    28
    Sep 10, 2017
    Ratings:
    +116
    Local Time:
    1:47 PM
    Now I have to try this on my box. Can't say no to extra security.