Join the community today
Register Now

OVH Massive, catastrophic fire at OVH Strasbourg data center

Discussion in 'Virtual Private Server (VPS) hosting' started by deltahf, Mar 11, 2021.

  1. deltahf

    deltahf Premium Member Premium Member

    461
    199
    43
    Jun 8, 2014
    Ratings:
    +351
    Local Time:
    12:16 AM
    This is one of the biggest incidents in the hosting industry in a while — early indications suggest that every server in OVH’s SGB2 data center has been destroyed, and their other facilities in Strasbourg, France have also been heavily damaged.

    OVH data centre destroyed by fire in Strasbourg – all services unavailable


    https://twitter.com/olesovhcom/status/1369478732247932929?s=21

    https://twitter.com/mothattacks/status/1369582816921587712?s=21

    https://twitter.com/hhackenbecker/status/1369658283804352515?s=21

    Reading through Twitter is pretty amazing right now, as webmasters report that they didn’t have any disaster plan or backups in place, and their sites are completely gone.

    This is a good opportunity to make sure you are fully prepared for something catastrophic like this. The worst can absolutely happen, unfortunately.
     
  2. Tsikura

    Tsikura New Member

    9
    8
    3
    May 18, 2016
    New York
    Ratings:
    +10
    Local Time:
    12:16 AM
    1.15.6
    MariaDB 10.1.137
    Yikes. so many websites gone because of lack of offsite backups. Everyone always assume a datacenter is invincible but this is a good kick in the rear to remind people to always do backups!
     
  3. deltahf

    deltahf Premium Member Premium Member

    461
    199
    43
    Jun 8, 2014
    Ratings:
    +351
    Local Time:
    12:16 AM
    More coverage:

    Faits divers. Strasbourg: important incendie chez OVHcloud, de nombreux sites internet indisponibles partout dans le monde

    Here are some of the affected sites and services, some of which include local French government resources and the airport (!).

     
  4. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Just read about this and dam that is a massive blow to OVH and all the customers who had servers in there!

    At least no lives lost from what I have read so far!

    Indeed it does! Seems I might have to release my Centmin Mod backup scripts a bit earlier to Centmin Mod Premium users first at least (yeah they'll have a cost attached to them due to amount of testing/support and continual development I plan/expect for).

    I mentioned the proposed backup system at Amazon AWS - How many folks use Amazon S3 or S3 compatible Object Storage?
    and Amazon AWS - How many folks use Amazon S3 or S3 compatible Object Storage?
    Tool has been field tested and used. Including for my private paid client work. Example at https://blog.centminmod.com/2021/01...eed-for-linux-backups-using-zstd-compression/
     
    Last edited: Mar 11, 2021
  5. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  6. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Official statement from OVH https://www.ovh.ie/news/press/cpl1786.fire-our-strasbourg-site

     
  7. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Another lesson, is not to self host your inbound email @yourdomain.com. Read reports of folks in SBG2 self hosting email and now can't access their email accounts. Email - Free @yourdomain.com Email Accounts & Alternatives ;)
     
  8. deltahf

    deltahf Premium Member Premium Member

    461
    199
    43
    Jun 8, 2014
    Ratings:
    +351
    Local Time:
    12:16 AM
    This would also be a good time to have Cloudflare.

    Even if all your servers are down, you can quickly re-route to other IP addresses as you bring other servers online, without having to wait for DNS updates, or quickly set up a maintenance page with more information using Workers.
     
  9. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    +1 I already do this with this forum and my Wordpress blog at https://blog.centminmod.com/ at least for Cloudflare usage and Cloudflare Workers for a maintenance page when site goes offline. The blog goes one further to using Cloudflare Loadbalancer and Argo Tunnel failover to CF Worker HTML maintenance page :)

    Won't help for actual data, but to give a downed sites' visitors information or any type of maintenance/news/update page is better than an site that doesn't load :)
     
  10. pamamolf

    pamamolf Premium Member Premium Member

    3,952
    402
    83
    May 31, 2014
    Ratings:
    +782
    Local Time:
    7:16 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    About the backup i am using another storage server like one from Time4VPS and Backblaze or Wasabi....

    I think is faster to restore from the storage server than the cloud services... Didn't test thought....

    Also it is free to pull back the backup but i think the cloud services will charge per GB....
     
  11. radu81

    radu81 New Member

    3
    1
    3
    Apr 22, 2015
    Ratings:
    +4
    Local Time:
    3:16 PM
    at work we have (or maybe is more appropriate to say we had) 3 OVH VPS on SBG1. Hopefully I did not use the OVH backups, but Amazon S3. Today I I moved the most important websites to a new vps Hetzner, tomorrow I'll do the rest.
     
  12. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Slowness might be Blackblaze/Wasabi side as it won't be as fast as AWS S3 at least from my testing

    Good luck!

    Saw interesting info posted at Down - OVH - SBG - Lots and lots of tears. - Page 6
    So high cpu load could also mean your server is being cooked to death from heat!!! Didn't expect that would be a thing!
     
  13. pamamolf

    pamamolf Premium Member Premium Member

    3,952
    402
    83
    May 31, 2014
    Ratings:
    +782
    Local Time:
    7:16 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Yes it is slow from their side. AWS S3 may be faster but is a lot more expensive.
     
  14. deltahf

    deltahf Premium Member Premium Member

    461
    199
    43
    Jun 8, 2014
    Ratings:
    +351
    Local Time:
    12:16 AM
    Wow. :eek:

    I guess the CPU kept thermal throttling itself more and more as ambient temperatures continued to rise?
     
  15. gabel

    gabel Member

    67
    38
    18
    Jan 7, 2016
    Ratings:
    +63
    Local Time:
    5:16 AM
    1.9.9
    MariaDB 10
    seems the perfect time to release that one :D
     
  16. eva2000

    eva2000 Administrator Staff Member

    47,515
    10,783
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,763
    Local Time:
    3:16 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    No idea to be honest. It's not something I've experienced before!
    :D