Welcome to Centmin Mod Community
Register Now

Help! I got locked out of my server

Discussion in 'System Administration' started by Carlo, Mar 13, 2018.

  1. Carlo

    Carlo New Member

    15
    4
    3
    Apr 19, 2017
    Ratings:
    +7
    Local Time:
    4:46 PM
    What should I do? I was browsing my site and doing some changes then suddenly I can't access my site. I tried to login the server through SSH but it won't allow me. I was not doing anything that could have triggered the firewall to ban my IP. It happened today about 6PM GMT.

    Even Linode lish is blocked by the firewall. I have CSF+fail2ban installed. I used
    GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall to install fail2ban.

    I checked webpagetest.org and pingdom and confirm that my sites are still running. So its only my machine and lish that's locked out. My ISP changes my IP dynamically so I can't whitelist it. Hopefully ban will expire soon.
     
  2. Jimmy

    Jimmy Well-Known Member

    1,692
    364
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +917
    Local Time:
    2:46 AM
    1.17.x
    MariaDB 10.3.x
    You need to login via Linode.

    Using the Linode Shell (Lish)

    Also this:
    Rescue and Rebuild
     
  3. Carlo

    Carlo New Member

    15
    4
    3
    Apr 19, 2017
    Ratings:
    +7
    Local Time:
    4:46 PM
    I tried the guide. Lish is blocked:
    Code:
    Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:cb:ca:32:84:78:ac:xx:xx:xx
     
  4. Jon Snow

    Jon Snow Active Member

    514
    72
    28
    Jun 30, 2017
    Ratings:
    +114
    Local Time:
    3:46 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Can you access your site on a proxy?
     
  5. Carlo

    Carlo New Member

    15
    4
    3
    Apr 19, 2017
    Ratings:
    +7
    Local Time:
    4:46 PM
    My sites are working from outside because it gets rendered in webpagetest.org and other page speed sites. I have pingdom also monitoring it and no reports of downtime. I can't use proxy here it's blocked by my ISP :(
     
  6. Jimmy

    Jimmy Well-Known Member

    1,692
    364
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +917
    Local Time:
    2:46 AM
    1.17.x
    MariaDB 10.3.x
  7. Carlo

    Carlo New Member

    15
    4
    3
    Apr 19, 2017
    Ratings:
    +7
    Local Time:
    4:46 PM
    Not in this Middle East country. Even VPN apps, Skype, Whatsapp calls are blocked. But I think I should be able to access my sites tomorrow at office, hopefully my VPS as well because IP address would be different.
     
  8. Jimmy

    Jimmy Well-Known Member

    1,692
    364
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +917
    Local Time:
    2:46 AM
    1.17.x
    MariaDB 10.3.x
  9. Meirami

    Meirami Member

    146
    24
    18
    Dec 21, 2017
    Ratings:
    +57
    Local Time:
    9:46 AM
    Get hourly billing vps and try ssh connection from there.
    Around $0.007/hr.
     
  10. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    FYI, CSF Firewall supports dynamic ip whitelisting via custom hostname whitelisting see CSF Firewall - CentminMod.com LEMP Nginx web stack for CentOS

    If you ISP dynamically assigns IP address, you could try to force it to re-assign a new IP that isn't blocked i.e. rebooting modem/router etc.
     
  11. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    have you contacted Linode tech support to verify that Lish is actually blocked by CSF Firewall ? as Lish is meant to be out of bounds and bypasses firewalls.
     
  12. JJC84

    JJC84 Ad astra per aspera Premium Member

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +168
    Local Time:
    1:46 AM
    1.15.x
    10.x.x
    I have done that. Be careful with the firewall.
     
  13. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    If you have more than one server, you can whitelist each server's respective IP for each server, so can SSH into one server from another whitelisted server as well.
     
  14. JJC84

    JJC84 Ad astra per aspera Premium Member

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +168
    Local Time:
    1:46 AM
    1.15.x
    10.x.x
    That's smart. I got lucky that the host had a web console, but I have locked myself out maybe 3-4 times so far. CSF can be unforgiving.
     
  15. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    I can imagine though in 18+ yrs, I have never been locked out of any firewall ! That's probably partially due to luck I guess :)

    Much harder for some folks on ISPs with dynamic IPs that are in countries with alot of bad IPs/spam/blacklisted IP ranges!
     
  16. JJC84

    JJC84 Ad astra per aspera Premium Member

    247
    109
    43
    Jan 31, 2018
    San Antonio, Texas
    Ratings:
    +168
    Local Time:
    1:46 AM
    1.15.x
    10.x.x
    I make a lot of typographical errors when I have been awake working for too long. And there’s that one time that I flushed the iptables.
     
  17. Carlo

    Carlo New Member

    15
    4
    3
    Apr 19, 2017
    Ratings:
    +7
    Local Time:
    4:46 PM
    Thanks for everyone and @eva2000 for the feedback! All good now after the ban expired. I was able to log back in. I'll try implementing the CSF dynamic IP whitelisting. Need to be more careful now.

    I checked the fail2ban and nginx logs and it was apparently caused by my xenforo test installation from an old backup. For some reason, it was making too many requests triggering the [nginx-req-limit] jail. I didn't do anything out of normal. The last action I did was only to edit a post in an old thread and copy its contents to the clipboard!
     
  18. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    interesting you can see nginx-req-limit parameters at centminmod-fail2ban/jail.local at master · centminmod/centminmod-fail2ban · GitHub should allow for 576 hits/day.
     
  19. Jon Snow

    Jon Snow Active Member

    514
    72
    28
    Jun 30, 2017
    Ratings:
    +114
    Local Time:
    3:46 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Checking this today on a server I wasn't blocked from, I see the same firewall notice. I guess I should contact Linode to see what's going on or is this a common thing you experience to?
     
  20. eva2000

    eva2000 Administrator Staff Member

    44,426
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,696
    Local Time:
    4:46 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    yeah you might need to ask Linode regarding Lish being blocked from firewall