Get the most out of your Centmin Mod LEMP stack
Become a Member

Both Sites Showing 522 Error / But Can SSH

Discussion in 'Other Centmin Mod Installed software' started by Jimmy, May 31, 2019.

  1. Jimmy

    Jimmy Well-Known Member

    1,655
    362
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +898
    Local Time:
    7:33 AM
    1.17.x
    MariaDB 10.3.x
    Woke up this morning to find both of my servers offline showing a Cloudflare 522 error.

    Both servers (VPS) are:
    cmm 09beta
    nginx 1.17.0
    1 server is 7.3.6 and the other is 7.2.19

    I had no problem logging in via ssh.

    I even rebooted the servers.

    Tried a bunch of stuff, rebuilding php, nginx, checking files.

    Ended up having to restart csf. Once I did that everything was back to normal.

    I looked at my emails and people were still posting as of a few 1 hour ago and the last time CSF was updated was 9 hours ago.

    Thought maybe it was a csf update or something that was bad. Hope I wasn't hacked or something.

    Not sure why it happened on both machines.

    If anyone has any idea of what to check next, I'm all ears. Really curious why this happened.
     
    Last edited: May 31, 2019
  2. eva2000

    eva2000 Administrator Staff Member

    42,833
    9,705
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,970
    Local Time:
    10:33 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    what's output for
    Code (Text):
    grep -i csf /var/log/cron | sed -e "s|$(hostname -s)|hostname|g"

    and
    Code (Text):
    virt-what

    Code (Text):
    grep -i csf /var/log/cron | sed -e "s|$(hostname -s)|hostname|g"
    May 26 05:39:01 hostname CROND[5127]: (root) CMD (/usr/sbin/csf -u)
    May 26 23:00:01 hostname CROND[3276]: (root) CMD (/usr/local/src/centminmod/tools/csfcf.sh auto >/dev/null 2>&1)
    May 27 00:00:01 hostname CROND[7081]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 27 05:39:01 hostname CROND[14004]: (root) CMD (/usr/sbin/csf -u)
    May 27 23:00:01 hostname CROND[10882]: (root) CMD (/usr/local/src/centminmod/tools/csfcf.sh auto >/dev/null 2>&1)
    May 28 00:00:01 hostname CROND[14648]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 28 05:39:01 hostname CROND[21617]: (root) CMD (/usr/sbin/csf -u)
    May 28 23:00:01 hostname CROND[17389]: (root) CMD (/usr/local/src/centminmod/tools/csfcf.sh auto >/dev/null 2>&1)
    May 29 00:00:01 hostname CROND[21132]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 29 05:39:01 hostname CROND[27413]: (root) CMD (/usr/sbin/csf -u)
    May 29 23:00:01 hostname CROND[23239]: (root) CMD (/usr/local/src/centminmod/tools/csfcf.sh auto >/dev/null 2>&1)
    May 30 00:00:01 hostname CROND[26787]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 30 05:39:01 hostname CROND[6115]: (root) CMD (/usr/sbin/csf -u)
    May 30 23:00:01 hostname CROND[29846]: (root) CMD (/usr/local/src/centminmod/tools/csfcf.sh auto >/dev/null 2>&1)
    May 31 00:00:01 hostname CROND[10163]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 31 05:39:01 hostname CROND[12084]: (root) CMD (/usr/sbin/csf -u)
    

    There's other reasons beyond firewall for 522 errors outlined at https://support.cloudflare.com/hc/en-us/articles/200171906-Error-522-Connection-timed-out. So it could be coincidental that CSF firewall restart resolved too.

     
  3. eva2000

    eva2000 Administrator Staff Member

    42,833
    9,705
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,970
    Local Time:
    10:33 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Also any relevant network status events with your web host ?
     
  4. Jimmy

    Jimmy Well-Known Member

    1,655
    362
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +898
    Local Time:
    7:33 AM
    1.17.x
    MariaDB 10.3.x
    No network status events. I checked that and I wasn't having any issue logging in via ssh - that worked fine.

    Code:
    # grep -i csf /var/log/cron | sed -e "s|$(hostname -s)|hostname|g"
    May 26 20:36:01 hostname CROND[22088]: (root) CMD (/usr/sbin/csf -u)
    May 27 00:00:01 hostname CROND[6329]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 27 02:01:01 hostname CROND[16342]: (root) CMD (/root/tools/csfcf_update.sh 2>/dev/null)
    May 27 03:01:02 hostname CROND[21275]: (root) CMD (/root/tools/csf_update.sh 2>/dev/null)
    May 27 20:36:01 hostname CROND[11105]: (root) CMD (/usr/sbin/csf -u)
    May 28 00:00:01 hostname CROND[27787]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 28 02:01:01 hostname CROND[5396]: (root) CMD (/root/tools/csfcf_update.sh 2>/dev/null)
    May 28 03:01:01 hostname CROND[10521]: (root) CMD (/root/tools/csf_update.sh 2>/dev/null)
    May 28 20:36:01 hostname CROND[31824]: (root) CMD (/usr/sbin/csf -u)
    May 29 00:00:01 hostname CROND[16402]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 29 02:01:01 hostname CROND[27675]: (root) CMD (/root/tools/csfcf_update.sh 2>/dev/null)
    May 29 03:01:01 hostname CROND[835]: (root) CMD (/root/tools/csf_update.sh 2>/dev/null)
    May 29 20:36:01 hostname CROND[21826]: (root) CMD (/usr/sbin/csf -u)
    May 30 00:00:01 hostname CROND[5290]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 30 02:01:01 hostname CROND[15067]: (root) CMD (/root/tools/csfcf_update.sh 2>/dev/null)
    May 30 03:01:01 hostname CROND[19834]: (root) CMD (/root/tools/csf_update.sh 2>/dev/null)
    May 30 20:36:01 hostname CROND[4918]: (root) CMD (/usr/sbin/csf -u)
    May 31 00:00:02 hostname CROND[21468]: (root) CMD (/usr/sbin/csf --lfd restart > /dev/null 2>&1)
    May 31 02:01:01 hostname CROND[31138]: (root) CMD (/root/tools/csfcf_update.sh 2>/dev/null)
    May 31 03:01:02 hostname CROND[3582]: (root) CMD (/root/tools/csf_update.sh 2>/dev/null)
    
    Code:
    # virt-what
    kvm
    
     
  5. Jimmy

    Jimmy Well-Known Member

    1,655
    362
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +898
    Local Time:
    7:33 AM
    1.17.x
    MariaDB 10.3.x
    During the process I also put CF into attack mode and restarted nginx and php. When I restarted nginx / php via nprestart sometimes it would load once, after that it will keep loading until I got the error again.

    Ever since I did the csf -r it is working fine (which was the last thing I did).
     
    Last edited: Jun 1, 2019
  6. eva2000

    eva2000 Administrator Staff Member

    42,833
    9,705
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,970
    Local Time:
    10:33 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    load once as in web page or you mean nginx and php-fpm restart process lagged and only sometimes properly restarted ?

    Hard to know why 522 errors occurred though as some many reasons as outlined in that article. Usually for non-cloudflare users you wouldn't even notice such network issues if they're were intermittent - just with cloudflare you'd get a new 522 page notifying you of such.
     
  7. Jimmy

    Jimmy Well-Known Member

    1,655
    362
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +898
    Local Time:
    7:33 AM
    1.17.x
    MariaDB 10.3.x
    Both servers are on ssdnodes, so even if there was a network outage, I highly doubt I'd hear about it unless everything was completely down. Like I mentioned, I had access to ssh the entire time. Not sure how long it was offline before I woke up it's 10:30am here and I noticed the problem at 8:30am.

    Nothing else looks out of place. Hopefully it won't happen again. Since I'm on monthly for 1 server at ssdnodes and yearly is just about up for my other server there, I might be moving back to OVH where they record every little issue that happens.
     
  8. Jimmy

    Jimmy Well-Known Member

    1,655
    362
    83
    Oct 24, 2015
    East Coast USA
    Ratings:
    +898
    Local Time:
    7:33 AM
    1.17.x
    MariaDB 10.3.x
    nprestart always worked 2x [ok] every time. Once or twice when I did the nprestart it would give me the maintenance.html page (I used that page because it would load fast vs the main site page to test).
     
  9. eva2000

    eva2000 Administrator Staff Member

    42,833
    9,705
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,970
    Local Time:
    10:33 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    expected if main site page is php based as static html nginx page would load faster though php shouldn't load that slow - if it does then problem could be php-fpm related - see PHP-FPM - How to troubleshoot & optimize PHP-FPM server?