OVH Massive, catastrophic fire at OVH Strasbourg data center

deltahf · Mar 11, 2021

This is one of the biggest incidents in the hosting industry in a while — early indications suggest that every server in OVH’s SGB2 data center has been destroyed, and their other facilities in Strasbourg, France have also been heavily damaged.

OVH data centre destroyed by fire in Strasbourg – all services unavailable

https://twitter.com/olesovhcom/status/1369478732247932929?s=21

https://twitter.com/mothattacks/status/1369582816921587712?s=21

https://twitter.com/hhackenbecker/status/1369658283804352515?s=21

Reading through Twitter is pretty amazing right now, as webmasters report that they didn’t have any disaster plan or backups in place, and their sites are completely gone.

This is a good opportunity to make sure you are fully prepared for something catastrophic like this. The worst can absolutely happen, unfortunately.

Tsikura · Mar 11, 2021

Yikes. so many websites gone because of lack of offsite backups. Everyone always assume a datacenter is invincible but this is a good kick in the rear to remind people to always do backups!

deltahf · Mar 11, 2021

More coverage:

Faits divers. Strasbourg: important incendie chez OVHcloud, de nombreux sites internet indisponibles partout dans le monde

Here are some of the affected sites and services, some of which include local French government resources and the airport (!).

Due to the fire, many websites were or are still unavailable this Wednesday, such as those of the Colmar or Saverne tourist office, the Meteor brewery, the Lalique museum, the Freihof in Wangen (children's home), that of Recyclivre, the European People's University, Strasbourg airport, the ENT (digital workspace) ONE & NEO in Sarreguemines; the towns of Colmar, Vichy (Allier), Bourg-Saint-Andéol (Ardèche), Cherbourg (Manche), Bruay-la-Buissière (Pas-de-Calais); the websites of the Puy-de-Dôme department, the website of ASM, the emblematic rugby club of Clermont-Ferrand (Puy-de-Dôme); numerous sites specializing in meteorology (Météo Aude 11, Météo Gard-Hérault); La Malle Postale, which provides services to hikers, particularly on the Saint-Jacques-de-Compostelle routes; The Front Populaire media, launched by Michel Onfray and Stéphane Simon in October 2020; Endofrance, combination fight against endometriosis; the web agency TooEasy, based in Valence (Drôme) has seen many of its customers affected; many online radios have seen their flow cut off; the central works council of Peugeot-Citroën Sochaux (Doubs) had to close its doors; Bigmagie online store of magic accessories; the media library of Lannion (Côtes-d'Armor) had to close; the Cityscoot electric scooter reservation center (Paris region); the Republican Popular Union (UPR), the political party of François Asselineau; several online video games (Rust, whose players will have to start their games from scratch); the Square Palace online video game encyclopedia; the stadium of Caen (Calvados); the access “kiosk” of the So Press press group; the OuestGo carpooling service present throughout Brittany; the Créteil handball club (Val-de-Marne); the Generation-NT new technologies portal; the site of youtubeur Julien Chièze; the Stan's website, the public transport network of the agglomeration of Nancy, the Pompidou center in Paris, and even, temporarily, the government site data.gouv.fr

German, Italian, Spanish, Polish and Turkish sites are also affected.

Very concrete practical consequences

Strasbourg airport, whose service provider uses OVH to host the website, was affected by the consequences of the fire.

“The airport continues to operate normally, but anyone who would like to obtain information or even make a reservation on our site is unable to do so,” explains Martine Sahl, head of the aeronautics division, who also reported on the airport. 'Intranet inaccessibility.

The site recovered quickly
"It is not catastrophic", nevertheless noted the head of the aeronautical pole. “We are in permanent contact with our supplier who is hopeful that a solution will be found quickly, a backup is available”. In fact, the Strasbourg airport website was re-established shortly after noon, before being unavailable again around 2 p.m.
Click to expand...

eva2000 · Mar 11, 2021

Just read about this and dam that is a massive blow to OVH and all the customers who had servers in there!

Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.
Click to expand...

At least no lives lost from what I have read so far!

deltahf said: ↑

This is a good opportunity to make sure you are fully prepared for something catastrophic like this. The worst can absolutely happen, unfortunately.
Click to expand...

Indeed it does! Seems I might have to release my Centmin Mod backup scripts a bit earlier to Centmin Mod Premium users first at least (yeah they'll have a cost attached to them due to amount of testing/support and continual development I plan/expect for).

I mentioned the proposed backup system at Amazon AWS - How many folks use Amazon S3 or S3 compatible Object Storage?

I ask, as I am working on the Centmin Mod to Centmin Mod transfer feature's routines and part of that will be the backup and restore aspects. So vote and if you use other S3 Object, compatible providers, please reply to this thread with the details

FYI, also planned to make Centmin Mod a S3 compatible backup server itself. So you can make a Centmin Mod server a backup server to host backup files and have it support using S3 clients to transfer in/out the backup files to Centmin Mod.

If you like what you see or will find such developments useful, make sure you support Centmin Mod via various ways outlined here

Some S3 and S3 compatible Object Storage providers which I intend to support include:

Amazon S3

Backblaze B2 Storage

DigitalOcean Spaces Object Storage

Linode Object Storage

Wasabi Cloud Storage

Click to expand...

and Amazon AWS - How many folks use Amazon S3 or S3 compatible Object Storage?

cmm-remote-backup.sh is a 100% remote backup only tool for Centmin Mod LEMP stack servers running CentOS 7.x 64bit. All Nginx server, site data files and all MySQL databases and data directory /var/lib/mysql are transferred to a remote backup server via netcat + Facebook's zstd compression tool for optimal speed. For MariaDB MySQL data, MariaBackup tool was used to perform physical online snapshot backups of InnoDB, MyRocks, Aria and MyISAM database tables safely. Optionally, you can do test restores of MariaDB backed up data to throw away standalone MariaDB server instances on remote backup server. You first configure via genkey option the remote backup server settings and generate a SSH key pair for the remote backup server.
Click to expand...

Tool has been field tested and used. Including for my private paid client work. Example at https://blog.centminmod.com/2021/01...eed-for-linux-backups-using-zstd-compression/

I recently migrated hundreds of gigabytes of data for a Centmin Mod LEMP stack based client moving servers and web hosts and using Tar 1.32+ and Rsync 3.2.3 and leveraging Facebook’s zstd compression and Rsync’s xxHash checksum support, the data transfer speed was an eye opener. Especially, when the tested network transfer speed between the servers from USA East Coast to Mid USA was ~40-50MB/s over a 1Gbps network connection due to network and geographical distance.

Over an SSH encrypted and zstd compressed netcat connections, I managed to transfer:

144GB of file data (uncompressed size) in ~21.8 minutes with Tar + zstd and

65GB of MariaDB 10 MySQL data (uncompressed size) in ~8.8 minutes with MariaBackup!

That’s an equivalent for files transferred at 112.73MB/s for Tar + zstd and MariaDB MySQL data transferred at 126MB/s !
Click to expand...

eva2000 · Mar 11, 2021

Also WHT discussions at OVH France SBG is in Fire | Web Hosting Talk

eva2000 · Mar 11, 2021

Official statement from OVH https://www.ovh.ie/news/press/cpl1786.fire-our-strasbourg-site

Fire at Our Strasbourg Site
At 00:47 on Wednesday, March 10, 2021, a fire broke out in a room in one of our 4 datacenters in Strasbourg, SBG2. Please note that the site is not classified as a Seveso site. Firefighters immediately intervened to protect our teams and prevent the spread of the fire.

At 2:54 am they isolated the site and closed off its perimeter. By 4:09 am, the fire had destroyed SBG2 and continued to present risks to the nearby datacenters until the fire brigade brought the fire under control. From 5:30 am, the site has been unavailable to our teams for obvious security reasons, under the direction of the prefecture. The fire is now contained. We are relieved that no one was injured, neither among our teams nor among the firefighters and the services of the prefecture, whom we thank for their exemplary mobilization at our side.

With our 15 datacenters in Europe, our technical and commercial teams are fully invested to support our customers, implement solutions and mitigate the impact of the unavailability of our Strasbourg site. Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologize for the difficulties and challenges caused by this fire. We are committed to continue to communicate with the greatest transparency about the cause of the fire and its consequences.

We are currently assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the implementation of solutions. All of our communication channels, starting with our incident tracking platform (travaux.ovh.com), are accessible so that you can stay informed of developments in real-time.
Click to expand...

eva2000 · Mar 11, 2021

Tsikura said: ↑

Yikes. so many websites gone because of lack of offsite backups. Everyone always assume a datacenter is invincible but this is a good kick in the rear to remind people to always do backups!
Click to expand...

Another lesson, is not to self host your inbound email @yourdomain.com. Read reports of folks in SBG2 self hosting email and now can't access their email accounts. Email - Free @yourdomain.com Email Accounts & Alternatives

deltahf · Mar 11, 2021

This would also be a good time to have Cloudflare.

Even if all your servers are down, you can quickly re-route to other IP addresses as you bring other servers online, without having to wait for DNS updates, or quickly set up a maintenance page with more information using Workers.

eva2000 · Mar 11, 2021

deltahf said: ↑

This would also be a good time to have Cloudflare.

Even if all your servers are down, you can quickly re-route to other IP addresses as you bring other servers online, without having to wait for DNS updates, or quickly set up a maintenance page with more information using Workers.
Click to expand...

+1 I already do this with this forum and my Wordpress blog at https://blog.centminmod.com/ at least for Cloudflare usage and Cloudflare Workers for a maintenance page when site goes offline. The blog goes one further to using Cloudflare Loadbalancer and Argo Tunnel failover to CF Worker HTML maintenance page

Won't help for actual data, but to give a downed sites' visitors information or any type of maintenance/news/update page is better than an site that doesn't load

pamamolf · Mar 11, 2021

About the backup i am using another storage server like one from Time4VPS and Backblaze or Wasabi....

I think is faster to restore from the storage server than the cloud services... Didn't test thought....

Also it is free to pull back the backup but i think the cloud services will charge per GB....

radu81 · Mar 11, 2021

at work we have (or maybe is more appropriate to say we had) 3 OVH VPS on SBG1. Hopefully I did not use the OVH backups, but Amazon S3. Today I I moved the most important websites to a new vps Hetzner, tomorrow I'll do the rest.

eva2000 · Mar 11, 2021

pamamolf said: ↑

About the backup i am using another storage server like one from Time4VPS and Backblaze or Wasabi....

I think is faster to restore from the storage server than the cloud services... Didn't test thought....

Also it is free to pull back the backup but i think the cloud services will charge per GB....
Click to expand...

Slowness might be Blackblaze/Wasabi side as it won't be as fast as AWS S3 at least from my testing

radu81 said: ↑

at work we have (or maybe is more appropriate to say we had) 3 OVH VPS on SBG1. Hopefully I did not use the OVH backups, but Amazon S3. Today I I moved the most important websites to a new vps Hetzner, tomorrow I'll do the rest.
Click to expand...

Good luck!

Saw interesting info posted at Down - OVH - SBG - Lots and lots of tears. - Page 6

I only have... had.. one server in SBG, but it is a VPS system. Sad but funny I was watching the load skyrocket and wondering what was wrong, I never would have thought that the server was dying from the heat. Then it stopped pinging.

R.I.P SBG2.
Click to expand...

So high cpu load could also mean your server is being cooked to death from heat!!! Didn't expect that would be a thing!

pamamolf · Mar 12, 2021

Slowness might be Blackblaze/Wasabi side as it won't be as fast as AWS S3 at least from my testing
Click to expand...

Yes it is slow from their side. AWS S3 may be faster but is a lot more expensive.

deltahf · Mar 12, 2021

eva2000 said: ↑

So high cpu load could also mean your server is being cooked to death from heat!!! Didn't expect that would be a thing!
Click to expand...

Wow.

I guess the CPU kept thermal throttling itself more and more as ambient temperatures continued to rise?

gabel · Mar 12, 2021

eva2000 said: ↑

Indeed it does! Seems I might have to release my Centmin Mod backup scripts a bit earlier to Centmin Mod Premium users first at least (yeah they'll have a cost attached to them due to amount of testing/support and continual development I plan/expect for).

Tool has been field tested and used. Including for my private paid client work. Example at https://blog.centminmod.com/2021/01...eed-for-linux-backups-using-zstd-compression/
Click to expand...

seems the perfect time to release that one

eva2000 · Mar 13, 2021

deltahf said: ↑

Wow.

I guess the CPU kept thermal throttling itself more and more as ambient temperatures continued to rise?
Click to expand...

No idea to be honest. It's not something I've experienced before!

gabel said: ↑

seems the perfect time to release that one
Click to expand...

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

OVH Massive, catastrophic fire at OVH Strasbourg data center

deltahf Premium Member Premium Member

Tsikura New Member

deltahf Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

deltahf Premium Member Premium Member

eva2000 Administrator Staff Member

pamamolf Well-Known Member

radu81 New Member

eva2000 Administrator Staff Member

pamamolf Well-Known Member

deltahf Premium Member Premium Member

gabel Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

OVH Massive, catastrophic fire at OVH Strasbourg data center

deltahf Premium Member Premium Member

Tsikura New Member

deltahf Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

deltahf Premium Member Premium Member

eva2000 Administrator Staff Member

pamamolf Well-Known Member

radu81 New Member

eva2000 Administrator Staff Member

pamamolf Well-Known Member

deltahf Premium Member Premium Member

gabel Member

eva2000 Administrator Staff Member

.