Learn about Centmin Mod LEMP Stack today
Become a Member

Amazon AWS Script to sync a folder to Amazon S3

Discussion in 'System Administration' started by pamamolf, Feb 13, 2018.

  1. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    9:51 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    Hello

    I am looking for a script to sync my /backup/ folder to Amazon S3.

    Thank you
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    4:51 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  3. BoostN

    BoostN Active Member

    118
    26
    28
    Aug 19, 2014
    Ratings:
    +37
    Local Time:
    1:51 AM
    1.13.6
    10.0.34
    Here is what I wrote:

    Code:
    #!/bin/bash
    DESDIR=/home/nginx/domains/mydomain.com/backup/
    S3DIR="s3://mylocation/mysite/"
    
    # RSYNC to AWS S3
    s3cmd sync --skip-existing $DESDIR $S3DIR
    Not sure if it's right, but it's been working for over a year. I have a 10 day retention policy setup on the S3 folder itself.
     
    • Like Like x 1
    • Informative Informative x 1
  4. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    4:51 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  5. pamamolf

    pamamolf Well-Known Member

    3,113
    295
    83
    May 31, 2014
    Ratings:
    +530
    Local Time:
    9:51 AM
    Nginx-1.13.x
    MariaDB 10.1.x
    I am using at the moment the "put" parameter :)
     
  6. Matt

    Matt Moderator Staff Member

    765
    343
    63
    May 25, 2014
    Sheffield, UK
    Ratings:
    +512
    Local Time:
    7:51 AM
    1.7.1
    MariaDB 10
    Also, be mindful of the costs involved with syncing whole directories with S3, the costs can soon mount up, as they bill for each request type.

    upload_2018-2-15_11-0-20.png
     
    • Agree Agree x 2
    • Old Old x 1
..