Get the most out of your Centmin Mod LEMP stack
Become a Member

Help! I got locked out of my server

Discussion in 'System Administration' started by Carlo, Mar 13, 2018.

  1. Carlo

    Carlo New Member

    14
    4
    3
    Apr 19, 2017
    Ratings:
    +6
    Local Time:
    7:39 AM
    What should I do? I was browsing my site and doing some changes then suddenly I can't access my site. I tried to login the server through SSH but it won't allow me. I was not doing anything that could have triggered the firewall to ban my IP. It happened today about 6PM GMT.

    Even Linode lish is blocked by the firewall. I have CSF+fail2ban installed. I used
    GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall to install fail2ban.

    I checked webpagetest.org and pingdom and confirm that my sites are still running. So its only my machine and lish that's locked out. My ISP changes my IP dynamically so I can't whitelist it. Hopefully ban will expire soon.
     
  2. Jimmy

    Jimmy Premium Member Premium Member

    1,279
    268
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +660
    Local Time:
    5:39 PM
    1.13.x
    MariaDB 10.1.x
    You need to login via Linode.

    Using the Linode Shell (Lish)

    Also this:
    Rescue and Rebuild
     
  3. Carlo

    Carlo New Member

    14
    4
    3
    Apr 19, 2017
    Ratings:
    +6
    Local Time:
    7:39 AM
    I tried the guide. Lish is blocked:
    Code:
    Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:cb:ca:32:84:78:ac:xx:xx:xx
     
  4. Jon Snow

    Jon Snow Active Member

    365
    59
    28
    Jun 30, 2017
    Ratings:
    +87
    Local Time:
    6:39 PM
    Nginx 1.13.9
    MariaDB 10.1.31
    Can you access your site on a proxy?
     
  5. Carlo

    Carlo New Member

    14
    4
    3
    Apr 19, 2017
    Ratings:
    +6
    Local Time:
    7:39 AM
    My sites are working from outside because it gets rendered in webpagetest.org and other page speed sites. I have pingdom also monitoring it and no reports of downtime. I can't use proxy here it's blocked by my ISP :(
     
  6. Jimmy

    Jimmy Premium Member Premium Member

    1,279
    268
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +660
    Local Time:
    5:39 PM
    1.13.x
    MariaDB 10.1.x
  7. Carlo

    Carlo New Member

    14
    4
    3
    Apr 19, 2017
    Ratings:
    +6
    Local Time:
    7:39 AM
    Not in this Middle East country. Even VPN apps, Skype, Whatsapp calls are blocked. But I think I should be able to access my sites tomorrow at office, hopefully my VPS as well because IP address would be different.
     
  8. Jimmy

    Jimmy Premium Member Premium Member

    1,279
    268
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +660
    Local Time:
    5:39 PM
    1.13.x
    MariaDB 10.1.x
  9. Meirami

    Meirami Member

    63
    7
    8
    Dec 21, 2017
    Ratings:
    +20
    Local Time:
    12:39 AM
    Get hourly billing vps and try ssh connection from there.
    Around $0.007/hr.
     
    • Like Like x 1
  10. eva2000

    eva2000 Administrator Staff Member

    34,649
    7,658
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,776
    Local Time:
    7:39 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    FYI, CSF Firewall supports dynamic ip whitelisting via custom hostname whitelisting see CSF Firewall - CentminMod.com LEMP Nginx web stack for CentOS

    If you ISP dynamically assigns IP address, you could try to force it to re-assign a new IP that isn't blocked i.e. rebooting modem/router etc.
     
  11. eva2000

    eva2000 Administrator Staff Member

    34,649
    7,658
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,776
    Local Time:
    7:39 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    have you contacted Linode tech support to verify that Lish is actually blocked by CSF Firewall ? as Lish is meant to be out of bounds and bypasses firewalls.
     
  12. JJC84

    JJC84 Premium Member Premium Member

    139
    56
    28
    Jan 31, 2018
    Ratings:
    +80
    Local Time:
    4:39 PM
    1.13.9
    10.1
    I have done that. Be careful with the firewall.
     
  13. eva2000

    eva2000 Administrator Staff Member

    34,649
    7,658
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,776
    Local Time:
    7:39 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    If you have more than one server, you can whitelist each server's respective IP for each server, so can SSH into one server from another whitelisted server as well.
     
  14. JJC84

    JJC84 Premium Member Premium Member

    139
    56
    28
    Jan 31, 2018
    Ratings:
    +80
    Local Time:
    4:39 PM
    1.13.9
    10.1
    That's smart. I got lucky that the host had a web console, but I have locked myself out maybe 3-4 times so far. CSF can be unforgiving.
     
  15. eva2000

    eva2000 Administrator Staff Member

    34,649
    7,658
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,776
    Local Time:
    7:39 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    I can imagine though in 18+ yrs, I have never been locked out of any firewall ! That's probably partially due to luck I guess :)

    Much harder for some folks on ISPs with dynamic IPs that are in countries with alot of bad IPs/spam/blacklisted IP ranges!
     
  16. JJC84

    JJC84 Premium Member Premium Member

    139
    56
    28
    Jan 31, 2018
    Ratings:
    +80
    Local Time:
    4:39 PM
    1.13.9
    10.1
    I make a lot of typographical errors when I have been awake working for too long. And there’s that one time that I flushed the iptables.
     
    • Winner Winner x 1
  17. Carlo

    Carlo New Member

    14
    4
    3
    Apr 19, 2017
    Ratings:
    +6
    Local Time:
    7:39 AM
    Thanks for everyone and @eva2000 for the feedback! All good now after the ban expired. I was able to log back in. I'll try implementing the CSF dynamic IP whitelisting. Need to be more careful now.

    I checked the fail2ban and nginx logs and it was apparently caused by my xenforo test installation from an old backup. For some reason, it was making too many requests triggering the [nginx-req-limit] jail. I didn't do anything out of normal. The last action I did was only to edit a post in an old thread and copy its contents to the clipboard!
     
    • Like Like x 1
    • Informative Informative x 1
  18. eva2000

    eva2000 Administrator Staff Member

    34,649
    7,658
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +11,776
    Local Time:
    7:39 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    interesting you can see nginx-req-limit parameters at centminmod-fail2ban/jail.local at master · centminmod/centminmod-fail2ban · GitHub should allow for 576 hits/day.
     
..