Join the community today
Become a Member

Security fail2ban for Centmin Mod + CSF Firewall / Cloudflare API

Discussion in 'System Administration' started by eva2000, May 12, 2017.

  1. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
  2. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok thanks :)

    Does that setup limit connections on server ip as default or i should set it up ?
     
  3. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    set them up as you require them for your specific usage patterns
     
  4. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    What I want to do is to check if an ip has X connections to server ip to get banned....

    Any ideas how can I do that?

    Thank you
     
  5. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    Configuring that is up to you no support from me. Just nginx connection limits Restricting Access | NGINX combined with fail2ban custom rules as per GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall.
    the fail2ban rule already exists in my fail2ban implementation centminmod-fail2ban/nginx-conn-limit.conf at master · centminmod/centminmod-fail2ban · GitHub so just need nginx configured connection limiting wise.
     
  6. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I just try to follow the instructions to install fail2ban so i can do some tests but i got confused at the end as following the bellow commands:

    Code:
    USERIP=$(last -i | grep "still logged in" | awk '{print $3}' | uniq)
    SERVERIPS=$(ip route get 8.8.8.8 | awk 'NR==1 {print $NF}')
    IGNOREIP=$(echo "ignoreip = 127.0.0.1/8 ::1 $USERIP $SERVERIPS")
    cd /svr-setup/
    git clone -b 0.10 https://github.com/fail2ban/fail2ban
    cd fail2ban
    python setup.py install
    cp /svr-setup/fail2ban/files/fail2ban.service /usr/lib/systemd/system/fail2ban.service
    cp /svr-setup/fail2ban/files/fail2ban-tmpfiles.conf /usr/lib/tmpfiles.d/fail2ban.conf
    cp /svr-setup/fail2ban/files/fail2ban-logrotate /etc/logrotate.d/fail2ban
    echo "[DEFAULT]" > /etc/fail2ban/jail.local
    echo "ignoreip = 127.0.0.1/8 ::1 $USERIP $SERVERIPS" >> /etc/fail2ban/jail.local
    systemctl daemon-reload
    systemctl start fail2ban
    systemctl enable fail2ban
    systemctl status fail2ban
    Then there is no fail2ban.sh file to run commands like:

    Code:
    ./fail2ban.sh status
    Then reading at the bottom i can see that i can use fail2ban.sh with the parameter install to automate installation but there are no instructions on how to......

    George can you please help on this?

    Maybe something like his?

    Code:
    cd /svr-setup/
    git clone https://github.com/centminmod/centminmod-fail2ban.git
    cd centminmod-fail2ban
    ./fail2ban.sh install
    Thank you

    PS: Also do i have to create my own or can i copy any sample rules as when i run fail2ban-client status i can see only 1 rule and you have on your sample 16 rules :)
     
  7. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    updated GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall

    as to rules, if you did git clone automated and fail2ban.sh install, should have all rules

    just tested fresh install on CentOS 7 and works fine
    Code (Text):
    ./fail2ban.sh status
    ---------------------------------------
    nginx-auth parameters: 
    maxretry: 3 findtime: 600 bantime: 3600
    allow rate: 288 hits/day
    filter last modified: Sun Aug 20 08:01:11 UTC 2017
    Status for the jail: nginx-auth
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-auth-main parameters: 
    maxretry: 3 findtime: 600 bantime: 3600
    allow rate: 288 hits/day
    filter last modified: Sun Aug 20 08:01:09 UTC 2017
    Status for the jail: nginx-auth-main
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /usr/local/nginx/logs/error.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-badrequests parameters: 
    maxretry: 1 findtime: 600 bantime: 604800
    allow rate: 144 hits/day
    filter last modified: Sun Aug 20 08:01:17 UTC 2017
    Status for the jail: nginx-badrequests
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-botsearch parameters: 
    maxretry: 2 findtime: 600 bantime: 600
    allow rate: 144 hits/day
    filter last modified: Sun Aug 20 08:01:18 UTC 2017
    Status for the jail: nginx-botsearch
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-common parameters: 
    maxretry: 1 findtime: 43200 bantime: 604800
    allow rate: 2 hits/day
    filter last modified: Sun Aug 20 08:01:12 UTC 2017
    Status for the jail: nginx-common
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/nginx/localhost_ssl.access.log /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-conn-limit parameters: 
    maxretry: 5 findtime: 600 bantime: 7200
    allow rate: 576 hits/day
    filter last modified: Sun Aug 20 08:01:20 UTC 2017
    Status for the jail: nginx-conn-limit
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-get-f5 parameters: 
    maxretry: 15 findtime: 1 bantime: 600
    allow rate: 1209600 hits/day
    filter last modified: Sun Aug 20 08:01:21 UTC 2017
    Status for the jail: nginx-get-f5
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-req-limit parameters: 
    maxretry: 5 findtime: 600 bantime: 7200
    allow rate: 576 hits/day
    filter last modified: Sun Aug 20 08:01:24 UTC 2017
    Status for the jail: nginx-req-limit
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-req-limit-main parameters: 
    maxretry: 5 findtime: 600 bantime: 7200
    allow rate: 576 hits/day
    filter last modified: Sun Aug 20 08:01:22 UTC 2017
    Status for the jail: nginx-req-limit-main
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /usr/local/nginx/logs/error.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-req-limit-repeat parameters: 
    maxretry: 5 findtime: 21600 bantime: 259200
    allow rate: 16 hits/day
    filter last modified: Sun Aug 20 08:01:25 UTC 2017
    Status for the jail: nginx-req-limit-repeat
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/fail2ban.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    nginx-xmlrpc parameters: 
    maxretry: 6 findtime: 60 bantime: 600
    allow rate: 7200 hits/day
    filter last modified: Sun Aug 20 08:01:27 UTC 2017
    Status for the jail: nginx-xmlrpc
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    shells parameters: 
    maxretry: 1 findtime: 86400 bantime: 604800
    allow rate: 1 hits/day
    filter last modified: Sun Aug 20 08:01:43 UTC 2017
    Status for the jail: shells
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/nginx/localhost_ssl.access.log /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    vbulletin parameters: 
    maxretry: 3 findtime: 60 bantime: 28800
    allow rate: 2880 hits/day
    filter last modified: Sun Aug 20 08:01:31 UTC 2017
    Status for the jail: vbulletin
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    wordpress-auth parameters: 
    maxretry: 3 findtime: 60 bantime: 600
    allow rate: 2880 hits/day
    filter last modified: Sun Aug 20 08:01:33 UTC 2017
    Status for the jail: wordpress-auth
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    wordpress-comment parameters: 
    maxretry: 5 findtime: 60 bantime: 3600
    allow rate: 5760 hits/day
    filter last modified: Sun Aug 20 08:01:34 UTC 2017
    Status for the jail: wordpress-comment
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    wordpress-fail2ban-plugin parameters: 
    maxretry: 1 findtime: 7200 bantime: 259200
    allow rate: 12 hits/day
    filter last modified: Sun Aug 20 08:01:42 UTC 2017
    Status for the jail: wordpress-fail2ban-plugin
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/secure /var/log/auth.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    wordpress-pingback parameters: 
    maxretry: 1 findtime: 1 bantime: 86400
    allow rate: 1 hits/day
    filter last modified: Sun Aug 20 08:01:35 UTC 2017
    Status for the jail: wordpress-pingback
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log /usr/local/nginx/logs/access.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    wordpress-pingback-repeat parameters: 
    maxretry: 5 findtime: 21600 bantime: 259200
    allow rate: 16 hits/day
    filter last modified: Sun Aug 20 08:01:36 UTC 2017
    Status for the jail: wordpress-pingback-repeat
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/fail2ban.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    ---------------------------------------
    All Time: Top 10 Banned IP Addresses:
    ---------------------------------------
    All Time: Top 10 Restored Banned IP Addresses:
    ---------------------------------------
    Yesterday: Top 10 Banned IP Addresses:
    ---------------------------------------
    Yesterday: Top 10 Restored Banned IP Addresses:
    ---------------------------------------
    Today: Top 10 Banned IP Addresses:
    ---------------------------------------
    Today: Top 10 Restored Banned IP Addresses:
    ---------------------------------------
    1 hr ago: Top 10 Banned IP Addresses:
    ---------------------------------------
    1 hr ago: Top 10 Restored Banned IP Addresses:
    ---------------------------------------
    
     
    Last edited: Aug 20, 2017
  8. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I did the manual setup with the commands above :(

    Is there an easy way to revert what i did with the manual commands or should i reinstall the server?
     
    Last edited: Aug 20, 2017
  9. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    no reverting, but automated script should still work
     
  10. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Working great :)

    I just got:

    Code:
    fatal: destination path 'fail2ban' already exists and is not an empty directory.
    That's normal as i was try to install it manually before.....don't know if you want to add a check for prior installations and do something there...... :)

    Time for tests :)
     
  11. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    yeah ok to ignore that :)
     
  12. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I found searching around this interesting article as i am trying to stop by banning the ip the most known web scanners like Acunetix.....

    Code:
    https://www.acunetix.com/blog/articles/block-automated-scanners/
    Is it possible on Centminmod to set a separate log file only for this specific access on that file so i can use it to block it ?

    Acunetix doesn't use anymore a specific user agent and is hard to block ......

    If there is no easy way then i hope the rate limit to block it :)

    Let's see !

    Also how can i clear all fail2ban banned ip's at once?

    Thank you
     
  13. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    Usage and configuration of fail2ban is left to end users to do. You need to know how to use fail2ban. I am only providing the script and base rules for install and setup for use with Centmin Mod + CSF Firewall. Rest is up to you :)
     
  14. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Ok i will post only test results in case that something needs a fix or to verify that it works :)

    In front of the test site i use Cloudflare and when i scan it reports this error:

    503 Service Temporarily Unavailable

    I can see at the error log file this:

    Code:
    2017/08/20 09:54:53 [error] 13688#13688: *3500 limiting requests, excess: 180.163 by zone "xwplogin",
    so it seems that limit rate is working.....

    I think is not banning the ip due to the error that cloudflare provide "503" and not the default that rate limit provide i think 444 ....

    What solution do you recommend for this one?

    I mean is it safe to ban 503 status code.... ?

    Thank you


     
  15. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    re-read nginx rate limiting and what it does - it returns 503 when you hit the rate limits nginx set which is as expected

    if you're using my bad bot rate limiting and it matches then 444 errors need to check in logs but cloudflare won't be able to connect to backend with 444 so cloudflare gives 503 - actually Cloudflare will give 520 http status code when it encounters 444 on backend.
     
    Last edited: Aug 20, 2017
  16. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
  17. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Sorry i just post at the same time with you :)

    I am checking the notes now :)
     
  18. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Well i think the reason that is not banning the ip is that i am using the same ip for installing the Centminmod and attacking it now :)

    Csf has whitelist my ip LOL

    I just remember that when i check the logs of fail2ban:

    Code:
    [15005]: INFO    [nginx-req-limit] Ignore 123.456.789.000 by ip
     
    Last edited: Aug 20, 2017
  19. pamamolf

    pamamolf Premium Member Premium Member

    3,983
    412
    83
    May 31, 2014
    Ratings:
    +799
    Local Time:
    12:30 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    It seems that the test is correct as it detects the matches but i don't know now how to re run fail2ban after removing my ip from csf firewall (i restart it also) as i just restart fail2ban but it is not banning the ip ....

    May i need to do something else?

    Code:
    fail2ban-regex error.log /etc/fail2ban/filter.d/nginx-req-limit.conf
    
    Running tests
    =============
    
    Use   failregex filter file : nginx-req-limit, basedir: /etc/fail2ban
    Use      datepattern : Default Detectors
    Use         log file : error.log
    Use         encoding : UTF-8
    
    
    Results
    =======
    
    Failregex: 58911 total
    |-  #) [# of hits] regular expression
    |   1) [58911] ^\s*\[error\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by zone "(?:[^"]+)", client: <HOST>,
    `-
    
    Ignoreregex: 0 total
    
    Date template hits:
    |- [# of hits] date format
    |  [58956] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day[T ]24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
    `-
    
    Lines: 58956 lines, 0 ignored, 58911 matched, 45 missed
    [processed in 5.22 sec]
     
  20. eva2000

    eva2000 Administrator Staff Member

    48,440
    11,102
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,281
    Local Time:
    7:30 PM
    Nginx 1.21.x
    MariaDB 10.x
    Centmin Mod will whitelist the IP it detects that is used to install CSF Firewall so yes IP will be ignored by CSF Firewall.

    grep check IP to see if it's allowed/whitelisted
    Code (Text):
    csf -g IPADDR