Welcome to Centmin Mod Community
Register Now

Linode Fremont datacenter outage !

Discussion in 'Forum News' started by eva2000, May 30, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    Very nasty Linode Fremont datacenter power outage that affected this forum :(

    So much so, I decided to setup a DNS based 503 maintenance failover back up for the forums so in future if such outages occur, it will failover at DNS level to an off host secondary VPS server setup for community.centminmod.com for SSL https which is permanently in 503 status maintenance mode.

    So from end user's perspective hitting forum domain when Linode datacenter is down will instead of broken page unavailable browser message, it will direct to secondary VPS and show a 503 maintenance message for visitors with a more useful message an explanation for visitors :)

    The secondary VPS setup 503 maintenance message

    cm_siteunavailable_htmlindex.png

    from visitor perspective just 503 unavailable maintenance message on secondary VPS

    cm_siteunavailable_ga_realtime.png

    linodestatus2.png

    Looks like my server has been up for a few minutes after power was restored to the datacenter :)

    Code:
    uptime
    06:53:47 up  3:57,  1 user,  load average: 0.00, 0.00, 0.00
    Nearly 4hrs downtime
    Code:
    Date  Enabled (hh:mm:ss) Down (hh:mm:ss) Uptime (%)
    2015-05-01    1 day 00:00:00    00:00:00    100
    2015-05-02    1 day 00:00:00    00:00:00    100
    2015-05-03    1 day 00:00:00    00:00:00    100
    2015-05-04    1 day 00:00:00    00:00:00    100
    2015-05-05    1 day 00:00:00    00:00:00    100
    2015-05-06    1 day 00:00:00    00:00:00    100
    2015-05-07    1 day 00:00:00    00:00:00    100
    2015-05-08    1 day 00:00:00    00:00:00    100
    2015-05-09    1 day 00:00:00    00:00:00    100
    2015-05-10    1 day 00:00:00    00:00:00    100
    2015-05-11    1 day 00:00:00    00:00:00    100
    2015-05-12    1 day 00:00:00    00:00:00    100
    2015-05-13    1 day 00:00:00    00:00:00    100
    2015-05-14    1 day 00:00:00    00:00:00    100
    2015-05-15    1 day 00:00:00    00:00:00    100
    2015-05-16    1 day 00:00:00    00:00:00    100
    2015-05-17    1 day 00:00:00    00:07:55    99.45
    2015-05-18    1 day 00:00:00    00:00:00    100
    2015-05-19    1 day 00:00:00    00:00:00    100
    2015-05-20    1 day 00:00:00    00:00:00    100
    2015-05-21    1 day 00:00:00    00:00:00    100
    2015-05-22    1 day 00:00:00    00:00:00    100
    2015-05-23    1 day 00:00:00    00:00:00    100
    2015-05-24    1 day 00:00:00    00:00:00    100
    2015-05-25    1 day 00:00:00    00:01:16    99.912
    2015-05-26    1 day 00:00:00    00:00:00    100
    2015-05-27    1 day 00:00:00    00:00:00    100
    2015-05-28    1 day 00:00:00    00:00:00    100
    2015-05-29    1 day 00:00:00    00:00:00    100
    2015-05-30    07:04:14    03:57:49    43.94
    total    29 days 07:04:14    04:07:00    99.414
    muninstats.png
     
    Last edited: May 30, 2015
  2. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    wow I am lucky, looks like the Linode Fremont datacenter power outage damaged more >13 servers !

    linodestatus3.png
     
  3. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    • Informative Informative x 1
  4. SneakyDave

    SneakyDave Member

    68
    11
    8
    Jul 24, 2014
    Ratings:
    +16
    Local Time:
    6:11 AM
    1.0.15
    So what happened, did they say what the real issue was and what steps are being taken to not have this problem in the future.

    I don't know how many times I've seen it, but for whatever reasons, backup generators always seem to not work when they're supposed to work.
     
  5. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    i think Linode is waiting on the RFO report from the Fremont Datacenter (he.net i believe)
     
  6. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    As I switched DNS providers from DNSMadeEasy to AWS Route53, had to redo the failover DNS for this forum's domain on Route53. Just a heads up if folks get funky access DNS wise. It should only kick over to failover DNS IP server if main Linode server is down and you'd be greated with that Site Unavailable notice page on the secondary backup server :)

    route53-primary-secondary-failover.png
     
  7. BoostN

    BoostN Member

    88
    18
    8
    Aug 19, 2014
    Ratings:
    +22
    Local Time:
    6:11 AM
    1.11.1
    10.0.28
    Do you have a post/article on this configuration? I noticed on a different thread how you used this to show a maintenance page as well during the KVM migration.
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,146
    6,782
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,133
    Local Time:
    9:11 PM
    Nginx 1.13.x
    MariaDB 5.5
    no articles but Route53 docs explains it Configuring DNS Failover - Amazon Route53
     
    Last edited: Sep 3, 2015
    • Like Like x 1