/ Register

Get the most out of your Centmin Mod LEMP stack
Become a Member

Linode Fremont datacenter outage !

Discussion in 'Forum News' started by eva2000, May 30, 2015.

Tags:
Previous Thread Next Thread
Loading...
  1. May 30, 2015 #1
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Very nasty Linode Fremont datacenter power outage that affected this forum :(

    So much so, I decided to setup a DNS based 503 maintenance failover back up for the forums so in future if such outages occur, it will failover at DNS level to an off host secondary VPS server setup for community.centminmod.com for SSL https which is permanently in 503 status maintenance mode.

    So from end user's perspective hitting forum domain when Linode datacenter is down will instead of broken page unavailable browser message, it will direct to secondary VPS and show a 503 maintenance message for visitors with a more useful message an explanation for visitors :)

    The secondary VPS setup 503 maintenance message

    cm_siteunavailable_htmlindex.png

    from visitor perspective just 503 unavailable maintenance message on secondary VPS

    cm_siteunavailable_ga_realtime.png

    linodestatus2.png

    Looks like my server has been up for a few minutes after power was restored to the datacenter :)

    Code:
    uptime
06:53:47 up  3:57,  1 user,  load average: 0.00, 0.00, 0.00
    Nearly 4hrs downtime
    Code:
    Date  Enabled (hh:mm:ss) Down (hh:mm:ss) Uptime (%)
2015-05-01    1 day 00:00:00    00:00:00    100
2015-05-02    1 day 00:00:00    00:00:00    100
2015-05-03    1 day 00:00:00    00:00:00    100
2015-05-04    1 day 00:00:00    00:00:00    100
2015-05-05    1 day 00:00:00    00:00:00    100
2015-05-06    1 day 00:00:00    00:00:00    100
2015-05-07    1 day 00:00:00    00:00:00    100
2015-05-08    1 day 00:00:00    00:00:00    100
2015-05-09    1 day 00:00:00    00:00:00    100
2015-05-10    1 day 00:00:00    00:00:00    100
2015-05-11    1 day 00:00:00    00:00:00    100
2015-05-12    1 day 00:00:00    00:00:00    100
2015-05-13    1 day 00:00:00    00:00:00    100
2015-05-14    1 day 00:00:00    00:00:00    100
2015-05-15    1 day 00:00:00    00:00:00    100
2015-05-16    1 day 00:00:00    00:00:00    100
2015-05-17    1 day 00:00:00    00:07:55    99.45
2015-05-18    1 day 00:00:00    00:00:00    100
2015-05-19    1 day 00:00:00    00:00:00    100
2015-05-20    1 day 00:00:00    00:00:00    100
2015-05-21    1 day 00:00:00    00:00:00    100
2015-05-22    1 day 00:00:00    00:00:00    100
2015-05-23    1 day 00:00:00    00:00:00    100
2015-05-24    1 day 00:00:00    00:00:00    100
2015-05-25    1 day 00:00:00    00:01:16    99.912
2015-05-26    1 day 00:00:00    00:00:00    100
2015-05-27    1 day 00:00:00    00:00:00    100
2015-05-28    1 day 00:00:00    00:00:00    100
2015-05-29    1 day 00:00:00    00:00:00    100
2015-05-30    07:04:14    03:57:49    43.94
total    29 days 07:04:14    04:07:00    99.414
    muninstats.png
     
    Last edited: May 30, 2015
  2. May 31, 2015 #2
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    wow I am lucky, looks like the Linode Fremont datacenter power outage damaged more >13 servers !

    linodestatus3.png
     
  3. May 31, 2015 #3
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  4. Jun 3, 2015 #4
    SneakyDave

    SneakyDave Member

    84
    14
    8
    Jul 24, 2014
    Ratings:
    +22
    Local Time:
    8:55 PM
    1.0.15
    So what happened, did they say what the real issue was and what steps are being taken to not have this problem in the future.

    I don't know how many times I've seen it, but for whatever reasons, backup generators always seem to not work when they're supposed to work.
     
  5. Jun 3, 2015 #5
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    i think Linode is waiting on the RFO report from the Fremont Datacenter (he.net i believe)
     
  6. Aug 6, 2015 #6
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    As I switched DNS providers from DNSMadeEasy to AWS Route53, had to redo the failover DNS for this forum's domain on Route53. Just a heads up if folks get funky access DNS wise. It should only kick over to failover DNS IP server if main Linode server is down and you'd be greated with that Site Unavailable notice page on the secondary backup server :)

    route53-primary-secondary-failover.png
     
  7. Sep 3, 2015 #7
    BoostN

    BoostN Active Member

    134
    27
    28
    Aug 19, 2014
    Ratings:
    +42
    Local Time:
    8:55 PM
    1.13.6
    10.0.34
    Do you have a post/article on this configuration? I noticed on a different thread how you used this to show a maintenance page as well during the KVM migration.
     
  8. Sep 3, 2015 #8
    eva2000

    eva2000 Administrator Staff Member

    53,488
    12,130
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,672
    Local Time:
    11:55 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    no articles but Route53 docs explains it Configuring DNS Failover - Amazon Route53
     
    Last edited: Sep 3, 2015
Previous Thread Next Thread
Loading...

.