Want more timely Centmin Mod News Updates?
Become a Member

DigitalOcean Digitalocean Killed A Company (RaisUP) Story

Discussion in 'Virtual Private Server (VPS) hosting' started by eva2000, Jun 1, 2019.

  1. eva2000

    eva2000 Administrator Staff Member

    42,350
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    2:35 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Wow saw this on twitter about RaisUP's DigitalOcean experience and reminds us not to put all your eggs in one basket with backups external and remote from your web host Nicolas Beauvais on Twitter. Apparently issue was due to Digitalocean false positives in their internal fraud and abuse systems according to DO CTO As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled.... | Hacker News

    do-killed-company-01.png do-killed-company-02.png do-killed-company-03.png do-killed-company-04.png
     
    • Informative Informative x 2
  2. wmtech

    wmtech Active Member

    116
    33
    28
    Jul 22, 2017
    Ratings:
    +88
    Local Time:
    5:35 PM
    A story that repeats itself very often and needs to be learned.

    Do not put all your cloud servers into the same provider.
    Do not put all your money into the same bank.
    Do not put all your trust in one person.

    Otherwise there is some risk to loose everything.

    However, I would have thought that DO is able to solve such situations within an hour. There must have been some more serious flags triggered beside the sudden excessive load and droplet creation...
     
    • Agree Agree x 2
  3. buik

    buik Well-Known Member

    1,214
    328
    83
    Apr 29, 2016
    Ratings:
    +962
    Local Time:
    5:35 PM
    From the Tweet:
    What a tragedy. Strength to that guy.

    You can choose to put everything at one party servers, money, trust (not my choice, otherwise but ok)

    But pffffff That was a choice.
    What I just don't understand. where is the chapter back-up??
    You serve Fortune 500 customers as Raisup and don't make external backups?

    A wise lesson: Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up, Back-up.
     
    • Agree Agree x 2
  4. eva2000

    eva2000 Administrator Staff Member

    42,350
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    2:35 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Indeed backups which are remote/external from web host are a must if you're data is important. I do backups every 3, 4, 6 or 8 hrs locally + Amazon S3 storage remotely and then also have web host providers own backups in play i.e. DigitalOcean or Linode's native backups.

    One thing I am planning is more extensive backup options in future Centmin Mod versions - on the to do list at centminmod/centminmod for Extended Backup, Restore, Migration Tools - including for Centmin Mod install to act as a backup server itself with built in useful backup/restore/compression related tool features :)

    Also looking at Wasabi Cloud Storage as an alternative to Amazon S3 for remote storage as they are part of Cloudflare Bandwidth Alliance and also are cheaper.
     
    • Like Like x 1
    • Agree Agree x 1
  5. buik

    buik Well-Known Member

    1,214
    328
    83
    Apr 29, 2016
    Ratings:
    +962
    Local Time:
    5:35 PM
    As written in his Tweet messages he is the only Developer out of 2.
    The only reason I can think of it is that they are far too busy and that the backup plan is always being pushed forward.

    They have learned from it and I understand that the backups are back and the site is online again.
    A hard learning lesson but I think they have learned from it for sure.
     
    • Agree Agree x 1
  6. eva2000

    eva2000 Administrator Staff Member

    42,350
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    2:35 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Indee. DigitalOcean is addressing this case https://twitter.com/digitalocean/status/1136049647871545344

    Full blog post at https://blog.digitalocean.com/an-update-on-last-weeks-customer-shutdown-incident/

     
  7. buik

    buik Well-Known Member

    1,214
    328
    83
    Apr 29, 2016
    Ratings:
    +962
    Local Time:
    5:35 PM
    Wtf and sorry for the wtf but if your account is locked and unlocked thereafter the first available second is starting the process of getting your data into safety and less important but useful, create an emergency migration plan if needed if locked again or am I silly?

    This is very strange and I did not know that there was temporary access.
     
    Last edited: Jun 5, 2019
    • Agree Agree x 2
  8. eva2000

    eva2000 Administrator Staff Member

    42,350
    9,562
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,734
    Local Time:
    2:35 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    indeed that would of been my plan too first - ensure you safe guard your data off site