Get the most out of your Centmin Mod LEMP stack
Become a Member

Amazon S3 new Standard Infrequent Access storage class

Discussion in 'System Administration' started by eva2000, Sep 24, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    28,935
    6,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,747
    Local Time:
    9:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    Amazon S3 announced lower object storage pricing for their Glacier storage class and a new storage class called Standard IA (Infrequent Access). Details at AWS Storage Update – New Lower Cost S3 Storage Option & Glacier Price Reduction | AWS Official Blog

    S3 Pricing page Amazon S3 Pricing | AWS Cloud Storage Pricing

    And price reduction on Amazon Glacier storage class
    Good news indeed. I already implemented the new S3 Standard IA storage class in my Amazon S3 bucketss Life Cycle management for my backup processes for auto backing up Wordpress and forum sites :D
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    28,935
    6,567
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,747
    Local Time:
    9:30 PM
    Nginx 1.13.x
    MariaDB 5.5
    Example from one of my paid client backup scripts for Xenforo backups with pushover and email notifications after each backup with summary of backups done and Amazon S3 remote backup transfers + local backup retention management :)

    This is the summary email you get after a backup runs for Xenforo forum which backups up forum files, database, database schema only, mysql user grants and auto deletes older than XX threshold day backups (default is older than 14 days) and auto uploads backups to Amazon S3 bucket.

    Script also checks disk free space and whether or not your backup data set size can fit into your available disk free space + uses parallel gzip compression via Pigz multi-threaded gzip compression tool which you can install via Centmin Mod menu system :)
    Code:
    /root/tools/mandbbackup.sh backup xenforodbname
    
    Successfully backed up database: xenforodbname
    Successfully backed up web root directory for xenforodbname
    Successfully backed up database schema: xenforodbname
    pigz -4R -p4 /home/nginx/domains/domain.com/backup/xenforodbname_230915-094757.sql
    pigz -4R -p4 /home/nginx/domains/domain.com/backup/xenforodbname_schema_230915-094757.sql
    Number of backup files in /home/nginx/domains/domain.com/backup before backup or deletion: 20
    Number of backup files in /home/nginx/domains/domain.com/backup after backup or deletion: 23
    
    ----------------------------------------------------------------------------
    /root/tools/mandbbackup.sh 0.2.0
    ----------------------------------------------------------------------------
    /home/nginx/domains/domain.com/backup resides on partition which has
    19884 Megabytes of disk space available
    ----------------------------------------------------------------------------
    MySQL data directory located at: /var/lib/mysql/
    MySQL data directory size: 270 Megabytes
    ----------------------------------------------------------------------------
    [PASS] you have enough free disk space to store MySQL directory size on:
    /home/nginx/domains/domain.com/backup
    ----------------------------------------------------------------------------
    
    ----------------------------------------------------------------------------
    Backup xenforodbname Data: real: 0.00s cpu: 40% maxmem: 3244 KB cswaits: 38
    ----------------------------------------------------------------------------
    Backup xenforodbname associated web root directory: real: 0.00s cpu: 0% maxmem: 996 KB cswaits: 1
    ----------------------------------------------------------------------------
    Backup xenforodbname Schema: real: 0.00s cpu: 20% maxmem: 3244 KB cswaits: 37
    ----------------------------------------------------------------------------
    Pigz xenforodbname database: real: 0.00s cpu: 0% maxmem: 908 KB cswaits: 3
    ----------------------------------------------------------------------------
    Backup MySQL user GRANT permissions for:
    #mysqlusername root @ 127.0.0.1
    
    *************************************************************************
    created MySQL database backups at: /home/nginx/domains/domain.com/backup
    *************************************************************************
    Sep 23  09:27   499    xenforodbname_schema_230915-092707.sql.gz
    Sep 23  09:27   492    xenforodbname_230915-092707.sql.gz
    Sep 23  09:30   498    xenforodbname_schema_230915-093042.sql.gz
    Sep 23  09:30   491    xenforodbname_230915-093042.sql.gz
    Sep 23  09:32   498    xenforodbname_schema_230915-093201.sql.gz
    Sep 23  09:32   491    xenforodbname_230915-093201.sql.gz
    Sep 23  09:42   499    xenforodbname_schema_230915-094217.sql.gz
    Sep 23  09:42   492    xenforodbname_230915-094217.sql.gz
    Sep 23  09:45   499    xenforodbname_schema_230915-094528.sql.gz
    Sep 23  09:45   492    xenforodbname_230915-094528.sql.gz
    Sep 23  09:47   500    xenforodbname_schema_230915-094757.sql.gz
    Sep 23  09:47   493    xenforodbname_230915-094757.sql.gz
    
    *************************************************************************
    created web root directory backups at: /home/nginx/domains/domain.com/backup
    *************************************************************************
    Sep 23  09:27   7.5K   backup_public_230915-092707.zip
    Sep 23  09:30   7.5K   backup_public_230915-093042.zip
    Sep 23  09:32   7.5K   backup_public_230915-093201.zip
    Sep 23  09:42   7.5K   backup_public_230915-094217.zip
    Sep 23  09:45   7.5K   backup_public_230915-094528.zip
    Sep 23  09:47   7.5K   backup_public_230915-094757.zip
    
    *************************************************************************
    created MySQL user GRANT backups at: /home/nginx/domains/domain.com/backup
    *************************************************************************
    Sep 23  09:27   247    backup_mysqlgrants_230915-092707.sql.gz
    Sep 23  09:30   247    backup_mysqlgrants_230915-093042.sql.gz
    Sep 23  09:32   247    backup_mysqlgrants_230915-093201.sql.gz
    Sep 23  09:42   247    backup_mysqlgrants_230915-094217.sql.gz
    Sep 23  09:45   247    backup_mysqlgrants_230915-094528.sql.gz
    Sep 23  09:47   247    backup_mysqlgrants_230915-094757.sql.gz
    ------------------------------------------------------------------------
    Last backup sql / file(s) created at:
    /home/nginx/domains/domain.com/backup/xenforodbname_230915-094757.sql.gz
    /home/nginx/domains/domain.com/backup/backup_mysqlgrants_230915-094757.sql.gz
    /home/nginx/domains/domain.com/backup/backup_public_230915-094757.zip
    ------------------------------------------------------------------------
    Backup Log:
    /home/nginx/domains/domain.com/backuplogs/xenforodbname_backup_230915-094757.log
    ------------------------------------------------------------------------
    
    ----------------------------------------------------------------------------
    upload to Amazon S3
    s3cmd -v --cache-file='/home/s3cmd_cachefile_backups' put /home/nginx/domains/domain.com/backup/backup_mysqlgrants_230915-094757.sql.gz s3://BUCKETNAME/xenforo-backups/backup_mysqlgrants_230915-094757.sql.gz
    s3cmd -v --cache-file='/home/s3cmd_cachefile_backups' put /home/nginx/domains/domain.com/backup/xenforodbname_schema_230915-094757.sql.gz s3://BUCKETNAME/xenforo-backups/xenforodbname_schema_230915-094757.sql.gz
    s3cmd -v --cache-file='/home/s3cmd_cachefile_backups' put /home/nginx/domains/domain.com/backup/xenforodbname_230915-094757.sql.gz s3://BUCKETNAME/xenforo-backups/xenforodbname_230915-094757.sql.gz
    s3cmd -v --cache-file='/home/s3cmd_cachefile_backups' put /home/nginx/domains/domain.com/backup/backup_public_230915-094757.zip s3://BUCKETNAME/xenforo-backups/backup_public_230915-094757.zip
    
    INFO: Compiling list of local files...
    INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
    INFO: Summary: 1 local files to upload
    INFO: Forwarding request to us-west-2
    INFO: Sending file '/home/nginx/domains/domain.com/backup/backup_mysqlgrants_230915-094757.sql.gz', please wait...
    File '/home/nginx/domains/domain.com/backup/backup_mysqlgrants_230915-094757.sql.gz' stored as 's3://BUCKETNAME/xenforo-backups/backup_mysqlgrants_230915-094757.sql.gz' (247 bytes in 0.1 seconds, 1946.44 B/s) [1 of 1]
    
    INFO: Compiling list of local files...
    INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
    INFO: Summary: 1 local files to upload
    INFO: Forwarding request to us-west-2
    INFO: Sending file '/home/nginx/domains/domain.com/backup/xenforodbname_schema_230915-094757.sql.gz', please wait...
    File '/home/nginx/domains/domain.com/backup/xenforodbname_schema_230915-094757.sql.gz' stored as 's3://BUCKETNAME/xenforo-backups/xenforodbname_schema_230915-094757.sql.gz' (500 bytes in 0.2 seconds, 2.50 kB/s) [1 of 1]
    
    INFO: Compiling list of local files...
    INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
    INFO: Summary: 1 local files to upload
    INFO: Forwarding request to us-west-2
    INFO: Sending file '/home/nginx/domains/domain.com/backup/xenforodbname_230915-094757.sql.gz', please wait...
    File '/home/nginx/domains/domain.com/backup/xenforodbname_230915-094757.sql.gz' stored as 's3://BUCKETNAME/xenforo-backups/xenforodbname_230915-094757.sql.gz' (493 bytes in 0.1 seconds, 3.38 kB/s) [1 of 1]
    
    INFO: Compiling list of local files...
    INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
    INFO: Summary: 1 local files to upload
    INFO: Forwarding request to us-west-2
    INFO: Sending file '/home/nginx/domains/domain.com/backup/backup_public_230915-094757.zip', please wait...
    File '/home/nginx/domains/domain.com/backup/backup_public_230915-094757.zip' stored as 's3://BUCKETNAME/xenforo-backups/backup_public_230915-094757.zip' (7633 bytes in 0.1 seconds, 54.84 kB/s) [1 of 1]
    
    ----------------------------------------------------------------------------
    
    
    *************************************************************************
    Amazon S3 backups located in bucket name s3://BUCKETNAME/xenforo-backups
    *************************************************************************
    2015-09-23 09:42       247   s3://BUCKETNAME/xenforo-backups/backup_mysqlgrants_230915-094217.sql.gz
    2015-09-23 09:45       247   s3://BUCKETNAME/xenforo-backups/backup_mysqlgrants_230915-094528.sql.gz
    2015-09-23 09:47       247   s3://BUCKETNAME/xenforo-backups/backup_mysqlgrants_230915-094757.sql.gz
    2015-09-23 09:42      7633   s3://BUCKETNAME/xenforo-backups/backup_public_230915-094217.zip
    2015-09-23 09:45      7633   s3://BUCKETNAME/xenforo-backups/backup_public_230915-094528.zip
    2015-09-23 09:48      7633   s3://BUCKETNAME/xenforo-backups/backup_public_230915-094757.zip
    2015-09-23 09:42       492   s3://BUCKETNAME/xenforo-backups/xenforodbname_230915-094217.sql.gz
    2015-09-23 09:45       492   s3://BUCKETNAME/xenforo-backups/xenforodbname_230915-094528.sql.gz
    2015-09-23 09:48       493   s3://BUCKETNAME/xenforo-backups/xenforodbname_230915-094757.sql.gz
    2015-09-23 09:42       499   s3://BUCKETNAME/xenforo-backups/xenforodbname_schema_230915-094217.sql.gz
    2015-09-23 09:45       499   s3://BUCKETNAME/xenforo-backups/xenforodbname_schema_230915-094528.sql.gz
    2015-09-23 09:47       500   s3://BUCKETNAME/xenforo-backups/xenforodbname_schema_230915-094757.sql.gz
    ----------------------------------------------------------------------------
    With the new Amazon S3 Standard IA storage class, I have now setup my S3 bucket's Life Cycle to transfer backup files from S3 Standard storage to Standard IA storage after 30 days :)
     
    Last edited: Sep 24, 2015
    • Like Like x 1
    • Creative Creative x 1