Sysadmin How do you back up your public directory?

I usually do per site vhost backups so /home/nginx/domains/domain1.com/public i just backup the web roots unless there's important stuff in /home/nginx/domains/domain1.com/* so you exclude all the logs in /home/nginx/domains/domain1.com/logs/* . Tar first then multi-threaded compression via pigz, lbzip2 or pxz after tar.

Like dbbackup.sh, I save backup locally + Amazon S3 remote backup every 6-8/hrs depending on site and also have a backup server with rsnapshot to pull backups from servers every 4hrs, 24hrs, weekly, monthly.

Amazon S3 I save to Standard Infrequent Access storage class after 30 days but retain backup depending on importance up to 366 days in Glacier. You never know when you need a working backup and only realise when you need them most i.e. data corruption, data restoration etc

 

I usually write backup scripts to back per site vhost + db pairs. That way I can schedule them at site's specific off peak rather than server off peak as they are not necessarily the same time. For very large data set sites, i use site maintenance mode i.e. https://community.centminmod.com/threads/sitestatus-maintenance-mode.5599/. All my backups are timed and monitored so I have a set target criteria for how fast my backups must be locally and remotely i.e. locally 30GB data in 5 minutes or 60GB data in 10-15 mins and spec the servers accordingly.

Remote time criteria can vary - had clients needing 150-200MB/s backup speed But wanting 600MB/s but priced way out of their budget heh

 

Yeah using Postfix relay for AWS SES smtp as well on servers that need to hide their sending IP address. And same setup email and pushover notifications

yup reading is good

Yeah having that many cpu threads helps. Looking forward to AMD Zen Naples giving Intel some competition in the price to cpu threads ratios

In dbbackup.sh i scripted it so that if it detects 100% InnoDB based mysql database it uses single-transaction flag for mysqldump otherwise if mysql database has other non-InnoDB tables i.e. MyISAM, it doesn't use single-transaction flag.

 

Yeah AWS S3 with Standard-IA storage class is the best balance of AWS S3 pricing to relative speed of access to S3 bucket saved data

 

yeah true after a while in Standard-IA, i set the life cycle in AWS S3 to move to Glacier

Starting to play with my addons/acmetool.sh and acme.sh for Letsencrypt SSL certificates via DNS validation for Cloudflare and AWS Route53 so looking at scripting syncing of Letsencrypt SSL certs issued via DNS validation to multiple servers i.e. clusters serving the same domain/ssl cert

• Add DNS mode to get Letsencrypt SSL cert only (acmetool.sh 0.8.3)
• Cloudflare API Support in DNS Mode (acmetool.sh 0.8.6)

Cloudflare DNS validation already supported, just AWS Route53 DNS takes more manual work for setting up dedicated AWS IAM user and permissions/groups and then syncing across server clusters via AWS S3 dedicated ssl bucket

 

depends on storage class used Cloud Storage Pricing – Amazon Simple Storage Service (S3) – AWS but AWS S3 has greater availability and redundancy then normal dedicated servers or region availability selections to get closer to your server location for better speed

snapshot from one of my AWS accounts at @1.5TB AWS S3 storage where i mainly use Standard-IA and Glacier storage classes

Code (Text):

Amazon Simple Storage Service TimedStorage-ByteHrs
$0.023 per GB - first 50 TB / month of storage used    74.047 GB-Mo    $1.70

Amazon Simple Storage Service TimedStorage-SIA-ByteHrs
$0.0125 per GB-Month of storage used in Standard-Infrequent Access    1,133.646 GB-Mo    $14.17

Amazon Simple Storage Service USW2-TimedStorage-ByteHrs
$0.023 per GB - first 50 TB / month of storage used    36.708 GB-Mo    $0.84

Amazon Simple Storage Service USW2-TimedStorage-GlacierByteHrs
$0.004 per GB / month of storage used - Amazon Glacier    187.502 GB-Mo    $0.75

Amazon Simple Storage Service USW2-TimedStorage-SIA-ByteHrs
$0.0125 per GB-Month of storage used in Standard-Infrequent Access    134.054 GB-Mo    $1.68

I also setup dedicated AWS IAM users/groups per AWS S3 bucket per server for greater security. I like AWS IAM security because you control, grant or revoke privileges from central AWS Console management so if you have 100s of servers it easier to manage user privileges Identity and Access Management (IAM) - Amazon Web Services (AWS)

use AWS S3 Bucket's lifecycle management to automatically transition data to move from one storage class to another

 

Utilise the right S3 bucket life cycle management and storage classes and it is relatively cheap. I have ~3TB of S3 backups now at around US$34/month.

 

I found Backblaze too slow for some geographic servers compared to AWS S3.