DigitalOcean Digitalocean Killed A Company (RaisUP) Story

eva2000 · Jun 1, 2019

Wow saw this on twitter about RaisUP's DigitalOcean experience and reminds us not to put all your eggs in one basket with backups external and remote from your web host Nicolas Beauvais on Twitter. Apparently issue was due to Digitalocean false positives in their internal fraud and abuse systems according to DO CTO As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled.... | Hacker News

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community.

This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happen, and we take every effort to get customers back online as quickly as possible. In this particular scenario, we were slow to respond and had missteps in handling the false positive. This led the user to be locked out for an extended period of time. We apologize for our mistake and will share more details in our public postmortem.
Click to expand...

hanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns, led to the initial flag. We recognize and embrace our customers ability to spin up highly variable workloads, which would normally not lead to any issues. Clearly we messed up in this case.

Additionally, the steps taken in our response to the false positive did not follow our typical process. As part of our investigation, we are looking into our process and how we responded so we can improve upon this moving forward.
Click to expand...

How DigitalOcean just killed our company raisupcom
. A long thread for a very sad story.
Click to expand...

wmtech · Jun 2, 2019

A story that repeats itself very often and needs to be learned.

Do not put all your cloud servers into the same provider.
Do not put all your money into the same bank.
Do not put all your trust in one person.

Otherwise there is some risk to loose everything.

However, I would have thought that DO is able to solve such situations within an hour. There must have been some more serious flags triggered beside the sudden excessive load and droplet creation...

buik · Jun 2, 2019

wmtech said: ↑

A story that repeats itself very often and needs to be learned.

Do not put all your cloud servers into the same provider.
Do not put all your money into the same bank.
Do not put all your trust in one person.

Otherwise there is some risk to loose everything.

However, I would have thought that DO is able to solve such situations within an hour. There must have been some more serious flags triggered beside the sudden excessive load and droplet creation...
Click to expand...

From the Tweet:

1 year of database backups
Click to expand...

What a tragedy. Strength to that guy.

You can choose to put everything at one party servers, money, trust (not my choice, otherwise but ok)

But pffffff That was a choice.
What I just don't understand. where is the chapter back-up??
You serve Fortune 500 customers as Raisup and don't make external backups?

A wise lesson: Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up.

eva2000 · Jun 2, 2019

Indeed backups which are remote/external from web host are a must if you're data is important. I do backups every 3, 4, 6 or 8 hrs locally + Amazon S3 storage remotely and then also have web host providers own backups in play i.e. DigitalOcean or Linode's native backups.

One thing I am planning is more extensive backup options in future Centmin Mod versions - on the to do list at centminmod/centminmod for Extended Backup, Restore, Migration Tools - including for Centmin Mod install to act as a backup server itself with built in useful backup/restore/compression related tool features

Also looking at Wasabi Cloud Storage as an alternative to Amazon S3 for remote storage as they are part of Cloudflare Bandwidth Alliance and also are cheaper.

buik · Jun 2, 2019

As written in his Tweet messages he is the only Developer out of 2.
The only reason I can think of it is that they are far too busy and that the backup plan is always being pushed forward.

They have learned from it and I understand that the backups are back and the site is online again.
A hard learning lesson but I think they have learned from it for sure.

eva2000 · Jun 5, 2019

Indee. DigitalOcean is addressing this case https://twitter.com/digitalocean/status/1136049647871545344

Full blog post at https://blog.digitalocean.com/an-update-on-last-weeks-customer-shutdown-incident/

An update on last week's customer shutdown incident
Written by:Barry Cooks June 4, 2019 • 15 Comments
Update 0120 UTC 5 June – We want to clarify that all customer details shared in this post have been approved by the customer in advance. We would never share such company information without express permission.

Original post
On May 29, DigitalOcean customer Raisup’s account was locked, and their resources were powered down due to a false positive generated by our anti-fraud and abuse automation system. The follow-up in handling the false positive resulted in a subsequent lock, and a communication of permanent denial of access to the account was sent to the customer. The account owner leveraged Twitter as an avenue to call attention to the mistake. Shortly thereafter, DigitalOcean investigated the issue and the Raisup account was unlocked and powered back on. We'd like to apologize and share more details about exactly what happened.

The Incident
The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers. Before any action is taken against accounts, automated safeties are checked to avoid action on a customer that is in good standing without warning.

Unfortunately in this case, the safeties were insufficient to prevent automated action. Additionally, because the customer was running on credit, they did not have a clear payment history, which meant that one of the primary safeties (payment history) was not triggered. The automated service created a support ticket on behalf of the customer to allow for rapid communication regarding the action.
Upon recognizing his resources had been powered off, and the account locked, the customer replied to the ticket created for communication on the action. An Abuse Operations agent re-enabled the account 12 hours after the initial ticket. However, a mistake occurred and the agent did not flag the account as approved for the CPU-intensive activity that was the cause of the initial flag.

On May 30, the same automated service then acted on the account a second time, due to the absence of a safety flag. Upon a second review by a different Abuse Operations agent (nearly 29 hours after the customer responded to the second flag), the agent failed to recognize this was a false positive, and the agent fully denied access back into the account. This action triggered the final “access denied” communication to the customer. At this point, the customer initiated the series of tweets to gain the attention of DigitalOcean.

After further investigation the Droplets were powered back on, access was regranted to the account, and the appropriate safeties were flagged. DigitalOcean leadership initiated communication with the customer to extend apologies, offer credit, and fully explain what happened to resolve the issue.
Click to expand...

buik · Jun 5, 2019

On May 29, DigitalOcean customer Raisup’s account was locked. DigitalOcean investigated the issue and the Raisup account was unlocked and powered back on.
Click to expand...

On May 30, the same automated service then acted on the account a second time
Click to expand...

Wtf and sorry for the wtf but if your account is locked and unlocked thereafter the first available second is starting the process of getting your data into safety and less important but useful, create an emergency migration plan if needed if locked again or am I silly?

This is very strange and I did not know that there was temporary access.

eva2000 · Jun 5, 2019

bassie said: ↑

thereafter the first available second is starting the process of getting your data into safety
Click to expand...

indeed that would of been my plan too first - ensure you safe guard your data off site

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

DigitalOcean Digitalocean Killed A Company (RaisUP) Story

eva2000 Administrator Staff Member

wmtech Active Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

DigitalOcean Digitalocean Killed A Company (RaisUP) Story

eva2000 Administrator Staff Member

wmtech Active Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

.