Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin What auto-backup method do you use or you would you recommend?

Discussion in 'System Administration' started by Jon Snow, Apr 19, 2018.

  1. Jon Snow

    Jon Snow Active Member

    433
    65
    28
    Jun 30, 2017
    Ratings:
    +103
    Local Time:
    3:47 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    What automatic backup method would you guys highly recommend in 2018 for:
    • Backing up onto the same server.
    • Backing up and sending data off-site securely (eg. Dropbox).
    What are you using and what do you recommend the most that doesn't put a hole in your pocket?

    If you have recommendations for small sized sites and large sized sites, please specify when posting.
     
  2. buik

    buik Well-Known Member

    1,163
    308
    83
    Apr 29, 2016
    Ratings:
    +907
    Local Time:
    8:47 AM
    Depends on what kind of site i.e. webshop, blog etc.
    Never store backups on the same server permanently.

    For cheap backuping, create a backup on the local server and rsync it to another server (i.e. lowend).
    Cron can automate this task and if I am not mistaken @eva2000 does have practical examples
     
  3. eva2000

    eva2000 Administrator Staff Member

    41,350
    9,279
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,236
    Local Time:
    4:47 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    I use several tiers of backup solutions which usually backup locally then transfer a copy to Amazon S3 remote buckets.
    1. web host backups i.e. linode/digitalocean snapshot backups
    2. my own custom written backup scripts which run every 3, 4 or 8 hrs depending on server/data. Similar to centminmod's dbbackup.sh database script MariaDB - dbbackup.sh - quick mysql database backups for Centmin Mod stack but extended to backup files etc + db with support for transfering backups to remote ftp + amazon s3 with push notification support on backup completion to mobile/tablet devices :)
    3. remote rsnapshot backend servers pulling the local copy of backups from custom backup scripts
    For very large data set backups, just enable nginx maintenance mode Beta Branch - sitestatus maintenance mode + implement multi-threaded database and compression based backups - so more cpu cores you have, the faster the backup process Sysadmin - Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc so 30-40GB backup can complete in as little as 2-3 minutes :)
     
    • Like Like x 4
  4. Derek

    Derek Member

    45
    9
    8
    Aug 5, 2016
    Ratings:
    +21
    Local Time:
    2:47 AM
    You're dealing with a couple of issues here:
    • Static files that exist on the filesystem
    • Static files that exist on the filesystem but that can't be backed up directly without risk of problems (think database files where transactions occur that affect two files on disk during the backup process, making the database you would restore inconsistent because one DB file includes the new transaction, and another doesn't.)
    You can use something like rsnapshot to make automatic and free and easy backups of files in the first category, and if you do something like dump your database to disk ever hour/day/whatever, then you're in good shape to do a restore - just uncompress your database backup and reimport it and you're golden. The DB will exist as it did at the last DB dump you backed up.

    Modern filesystems support the concept of snapshots though, which makes things easier. I use a program from r1soft that automates this. In my case:
    • The backup software has been given the mysql login credentials.
    • When it's time for a backup the backup system logs into mysql, orders the database to write all pending writes to the tables on disk and pause new writes for a moment. Then it initiates a backup of the filesystem as it existed at that second and releases the database to resume normal function. This takes less than a second but insures that the database files copied from the filesystem represent a consistent state. (There's a better way to describe this, but I've never looked very deep into filesystem snapshotting under Linux so I can't explain it better.)
    • Backups are scheduled for every hour. Only changed data is copied across, so each backup takes something like 4 minutes.
    • An off-site backup server can be configured, so on a replication schedule you set the backups are automatically pushed to your off-site server. I've used this for years without any real difficulty.
    • Restores can be made on an individual file level (to the original machine or another one), or can be a bare-metal operation by booting from a recovery iso image and recreating the system as it last existed.
    • Retention is highly configurable. You can configure to save XX number of hourly backups, YY number of dailies, ZZ number of weeklies, and so on. It's easy with a proper setup to look at data as it existed at some arbitrary point in the past if you need to.
    Using my current system if something terrible happened that required a full restore I would lose on average about 30 minutes worth of data which isn't too bad for a forum.

    This sounds like an ad for r1soft, but it's not. I love the software - it works great and performs well - but the company has been sold many times since I've used it and I generally end up screaming at my monitor whenever I need to do something on their web site like get a license key, but overall I like it. There's no easy way to replicate this level of functionality.
     
    • Informative Informative x 2
  5. robert syputa

    robert syputa Member

    68
    13
    8
    Jan 18, 2018
    Seattle
    Ratings:
    +38
    Local Time:
    2:47 AM
    latest
    10
    I use a command-line backup of the server to the server and copy that to another server. I also backup user/app folders including databases.

    For Wordpress and Joomla CMS I use backup plugins with local and Amazon S3 storage and have used Dropbox. That is cheap for daily backups. If data is critical and you want to save cost, you might back up the databases hourly or more often and the entire CMS daily.
     
  6. Jon Snow

    Jon Snow Active Member

    433
    65
    28
    Jun 30, 2017
    Ratings:
    +103
    Local Time:
    3:47 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Bumping this since it's 2019 now. Anyone else?
     
  7. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    2:47 AM
    1.1.5
    mariadb 10
  8. ArisC

    ArisC Active Member

    119
    29
    28
    Jun 1, 2017
    Ratings:
    +57
    Local Time:
    9:47 AM
    Nginx Latest
    MariaDB Latest
    I'm using UpdraftPlus with DreamObject.
     
    • Informative Informative x 1
  9. upgrade81

    upgrade81 Premium Member Premium Member

    233
    14
    18
    Sep 5, 2016
    Switzerland
    Ratings:
    +21
    Local Time:
    8:47 AM
    1.17
    10.3
    it's interesting
    Have you already done a restore?
     
  10. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    2:47 AM
    1.1.5
    mariadb 10
    @upgrade81 Not yet, lot of backup but no restore yet. I guess I should give it a try.
     
  11. steph40

    steph40 Premium Member Premium Member

    58
    13
    8
    Jan 28, 2019
    Ratings:
    +25
    Local Time:
    2:47 AM
    1.1.5
    mariadb 10
    @upgrade81 Just did a restore, took very little time and was successful. Nice and easy step by step to follow. Site files and database restored with no error.
     
    • Like Like x 1
  12. Benjamin74

    Benjamin74 Premium Member Premium Member

    46
    4
    8
    May 2, 2016
    Ratings:
    +10
    Local Time:
    8:47 AM
    Using All-in-One WP Migration for Wordpress and payed for their Dropbox backup pro extension.

    Didn't like the fact that they use a proprietary compressed format but it's easy as pie to reinstall somewhere temporary if you just need to get back a couple files.

    Now I'm looking for an "all in one" backup script for both Mysql + PHP files for non-wordpress PHP apps, which would support Dropbox + run with cron.

    There was a good app on CodeCanyon called SmartBackup but it's been removed from their store and (as always with codecanyon) no reason was given so I don't feel really safe continuing using it.

    If anyone has a suggestion...
     
    • Informative Informative x 1
  13. Jay Chen

    Jay Chen Member

    95
    30
    18
    Sep 10, 2017
    Ratings:
    +51
    Local Time:
    2:47 AM
    I have tried many php based backup solution for my wordpress sites, including All-in-One WP Migration. They work great for small sites, but performed poorly on sites with many posts/files.

    I am still in the market hunting for a good backup script, not php based, I will share once I find one and tested it to my liking.
     
    • Like Like x 1
  14. Benjamin74

    Benjamin74 Premium Member Premium Member

    46
    4
    8
    May 2, 2016
    Ratings:
    +10
    Local Time:
    8:47 AM
    I am very surprised regarding All-In-One WP migration because it handles backing up some of my sites with 20 to 30K entries in gravityforms (with tons of fields per entry so quite a big database).

    Now I didn't test with a site with tons of images so maybe that's your problem with that plugin but it's unlikely that the DB size is the issue imo.

    How many files and posts are you backing up?

    Cheers,
     
  15. Jay Chen

    Jay Chen Member

    95
    30
    18
    Sep 10, 2017
    Ratings:
    +51
    Local Time:
    2:47 AM
    I never had any problem with DB backup, the problem is the static files.

    The site is an image heavy site, each post has at least one image. and I have over 5K posts on that site, so you can imagine how many images files I have there.
    Last time I checked, it has over 30k image files.