/ Register

Want to subscribe to topics you're interested in?
Become a Member

Sysadmin Fail2ban lockout on a broad scale?

Discussion in 'System Administration' started by squibs, Aug 2, 2019.

Previous Thread Next Thread
Loading...
  1. Aug 2, 2019 #1
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Not sure this is a bug, but wasn't sure where else to post. Please move if I'm in the wrong place.

    I've been rocking a new centminmod install for a few months now. Yesterday I couldn't connect to the hosted website, couldn't connect via SSH or VNC, however I could launch a rescue system and ssh into that, mounting the drive and verifying data is fine. This suggested to me that fail2ban or firewall didn't like something I did and locked me out. However I've tried from work network and home network - 2 completely different IP addresses, and SSH or VNC will not connect. The VPS company in another country insist that they can open a connection without issue.

    Could fail2ban block a whole country? Can't figure out what the issue might be, but as the rescue system accepts ssh connections, it must be a firewall thing, right? Any suggestions appreciated.
     
  2. Aug 2, 2019 #2
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    12:58 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    What centmin mod version ? when was it last updated ?

    OS version ? CentOS 6 or 7 ?

    Web host ?

    How was fail2ban installed and configured ? As Centmin Mod doesn't install fail2ban out of the box

    You can verify if it's CSF Firewall by doing a grep filter of your /var/log/messages log to see if there are entries with your IP address listed.

    Where you replace 184.105.xxx.xxx with your ISP IP address or the IP address of user who is having issues uploading via pure-ftpd virtual FTP user
    Code (Text):
    grep '184.105.xxx.xxx' /var/log/messages |

    If you're behind a VPN or proxy, your ISP IP address maybe masked so you can check to see what IP address your server is seeing for your SSH session using command below:
    Code (Text):
    echo $SSH_CLIENT

    example output where your IP detected is the 1st column of ouput i.e. 184.105.xxx.xxx
    Code (Text):
    echo $SSH_CLIENT
184.105.xxx.xxx 54021 22

    If there are entries that are returned from the grep filter, then could be related to CSF Firewall if the entries have Firewall or Blocked labeled entries

    Can also grep CSF firewall logs for your ISP IP too where 184.105.xxx.xxx is your ISP IP
    Code (Text):
    csf -g 184.105.xxx.xxx


    and grep fail2ban logs
    Code (Text):
    grep '184.105.xxx.xxx' /var/log/fail2ban.log
     
  3. Aug 2, 2019 #3
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Apologies for the lack of detail - as I can't access the server, I can't be sure, other than I'm running Centos 7 64bit and centminmod is largely up to date as the server was only spun up a couple of months ago. My bad on failban - thought it was part of the default install. I should be able to check the CSF logs through the rescue system. Great advice - thanks! I promise to post again when I've got more info. Thanks so much for what you do - centminmod such a great stack.
     
  4. Aug 2, 2019 #4
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    12:58 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    If it is a ISP IP block, just use a VPN to try to see if you can SSH login bypassing your ISP IP block too.
     
  5. Aug 2, 2019 #5
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Don't think an ISP block could be the issue as I can SSH into the VPS , if I boot into the provider's rescue system. Something (seems like the CSF firewall is the only possibility?) would appear to be blocking https, ssh and vnc connections from the 3 different networks I've tried, albeit all in the same country. Anyhoo - thanks for the advice - will post an update later.
     
  6. Aug 2, 2019 #6
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Well, that was a bust unfortunately. I can't run any commands as I can't ssh or vnc in. I accessed log files by mounting partition using rescue system. lfd.log is 0 bytes and there's an older gz archive that's 20 bytes. Not seeing much else in /var/log that's useful

    I thought firewalld log might hold the clue, and it might be an itables thing, but those entries are too old:
    2019-05-28 11:37:48 ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.4.21: goto 'IN_public' is not a chain

    Error occurred at line: 2
    Try `iptables-restore -h' or 'iptables-restore --help' for more information.

    2019-06-18 09:31:49 ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.4.21: goto 'IN_public' is not a chain

    Error occurred at line: 2
    Try `iptables-restore -h' or 'iptables-restore --help' for more information.

    2019-06-18 09:31:49 ERROR: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 2 failed

    2019-06-18 09:31:49 ERROR: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 2 failed​
     
  7. Aug 3, 2019 #7
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    12:58 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    check /var/log/messages and /var/log/lfd.log (CSF) for clues
     
  8. Aug 3, 2019 #8
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Thanks. As per my previous post var/log/lfd.log is empty (not sure how that's possible). I scanned the messages log, but nothing jumped out at me. What files might I edit (remember I can only mount the drive, not ssh into it) to disable csf, iptables and anything else on a default centminmod install that might prevent me ssh-ing in? If I can edit, reboot into centos 7, manage to SSH in, I can then start re-enabling services until I find the culprit.
     
  9. Aug 4, 2019 #9
    eva2000

    eva2000 Administrator Staff Member

    58,893
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    12:58 PM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    If you edit CSF Firewall configuration file at /etc/csf/csf.conf and change over to testing mode TESTING = "1" and restart CSF Firewall service (on reboot)
    change from
    Code (Text):
    TESTING = "0"

    to
    Code (Text):
    TESTING = "1"

    restarting CSF
    Code (Text):
    csf -ra

    In testing mode explained
    Code (Text):
    ###############################################################################
# SECTION:Initial Settings
###############################################################################
# Testing flag - enables a CRON job that clears iptables incase of
# configuration problems when you start csf. This should be enabled until you
# are sure that the firewall works - i.e. incase you get locked out of your
# server! Then do remember to set it to 0 and restart csf when you're sure
# everything is OK. Stopping csf will remove the line from /etc/crontab
#
# lfd will not start while this is enabled
TESTING = "0"

# The interval for the crontab in minutes. Since this uses the system clock the
# CRON job will run at the interval past the hour and not from when you issue
# the start command. Therefore an interval of 5 minutes means the firewall
# will be cleared in 0-5 minutes from the firewall start
TESTING_INTERVAL = "5"
     
  10. Aug 6, 2019 #10
    squibs

    squibs New Member

    8
    1
    3
    Aug 2, 2019
    Ratings:
    +1
    Local Time:
    3:58 AM
    Thanks for all the help. I edited the csf config file in rescue mode and then booted into normal mode. Putty still times out trying to connect on the port I had set. I think I'll have to call it a day, and go with a more reputable cloud provider. Luckily I have all my data backed up...
     
Previous Thread Next Thread
Loading...

.