Want more timely Centmin Mod News Updates?
Become a Member

Amazon AWS How many folks use Amazon S3 or S3 compatible Object Storage?

Discussion in 'System Administration' started by eva2000, Sep 29, 2020.

  1. Amazon S3

    5 vote(s)
    50.0%
  2. Backblaze B2 Storage

    5 vote(s)
    50.0%
  3. DigitalOcean Spaces Object Storage

    1 vote(s)
    10.0%
  4. Linode Object Storage

    3 vote(s)
    30.0%
  5. Wasabi Cloud Storage

    2 vote(s)
    20.0%
  6. Other S3 Compatible Object Storage Providers

    1 vote(s)
    10.0%
Multiple votes are allowed.
  1. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    How many folks using Centmin Mod or other control panels are using Amazon S3 Object Storage or other cloud provider's S3 compatible Object Storage products?

    I ask, as I am working on the Centmin Mod to Centmin Mod transfer feature's routines and part of that will be the backup and restore aspects. So vote and if you use other S3 Object, compatible providers, please reply to this thread with the details :)

    FYI, also planned to make Centmin Mod a S3 compatible backup server itself. So you can make a Centmin Mod server a backup server to host backup files and have it support using S3 clients to transfer in/out the backup files to Centmin Mod.

    If you like what you see or will find such developments useful, make sure you support Centmin Mod via various ways outlined here ;)

    Some S3 and S3 compatible Object Storage providers which I intend to support include:

    S3 Compatible Object Storage & Clients



    Part of the backup aspect is supporting 3rd party storage providers which are S3 object storage compatible. My dbbackup.sh MySQL database script for Centmin Mod premium members has optional support for backing up to Amazon S3 object storage via s3cmd S3 client.

    However, S3 clients have evolved so looking at adding support for official AWS aws-cli command line client as well as s5cmd S3 client which are both faster eventually for any Centmin Mod backup system I plan to release.

    awscli-get.sh AWS CLI & s5cmd Installer For Centmin Mod



    As part of Centmin Mod's backup/transfer features, I am writing a new AWS aws-cli and s5cmd S3 client install and configuration script to make it easier for Centmin Mod users to install and configure aws-cli command line client and s5cmd S3 client.

    A preview of what it does so far

    By default the script will install Linux 64bit aws-cli from https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip and install it to /usr/local/bin/aws and install s5cmd from latest releases at Releases · peak/s5cmd to /usr/local/bin/s5cmd.
    Code (Text):
    ./awscli-get.sh help
    
    Usage:
    
    ./awscli-get.sh {install|update} {default|profilename|defaultreset} AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION AWS_DEFAULT_OUTPUT
    ./awscli-get.sh regions
    ./awscli-get.sh regions-wasabi

    Both Amazon AWS CLI and s5cmd clients use the official Amazon S3 SDK so you need to export the environment variables for AWS Access Key and Secret Key credentials for the script to pick them up otherwise you get the following message. So in your SSH logged in session type the export commands and fill in the variables with your credentials you generated via AWS Console or 3rd party provider provided details.
    Code (Text):
    ./awscli-get.sh
    
    Error. AWS Key & Secret not detected
    export relevant environment variables
    and re-run script
    
    example:
    
    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=us-west-2
    export AWS_DEFAULT_OUTPUT=text


    Configuration

    First time running awscli-get.sh to configure aws-cli default profile
    Code (Text):
    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=us-west-2
    export AWS_DEFAULT_OUTPUT=text
    
    ./awscli-get.sh
    
    configure default aws-cli profile
    
    aws-cli profile: default set:
    
    aws_access_key_id: AKIAIOSFODNN7EXAMPLE
    aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    default.region: us-west-2
    default output format: text

    If default aws-cli profile already set and detected, you'll get below message about skipping default profile configuration i.e. you already have AWS CLI command line or another S3 client with reads the same AWS CLI config/credential files. You'll also get info on how to use defaulreset command to overwrite your existing default AWS CLI command line profile's AWS Key and Secret if you desire.
    Code (Text):
    ./awscli-get.sh                      
    
    existing config file detected: /root/.aws/config
    existing credential file detected: /root/.aws/credentials
    
    skipping configuration...
    
    detected existing default credentials
    in config file: /root/.aws/config
    in credential file: /root/.aws/credentials
    
    To override the default profile, use defaultreset mode:
    
    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=us-west-2
    export AWS_DEFAULT_OUTPUT=text
    
    ./awscli-get.sh install defaultreset

    Example commands supported

    List AWS S3 region codes
    Code (Text):
    ./awscli-get.sh regions
    Africa (Cape Town)           af-south-1
    Asia Pacific (Hong Kong)     ap-east-1
    Asia Pacific (Mumbai)        ap-south-1
    Asia Pacific (Osaka-Local)   ap-northeast-3
    Asia Pacific (Seoul)         ap-northeast-2
    Asia Pacific (Singapore)     ap-southeast-1
    Asia Pacific (Sydney)        ap-southeast-2
    Asia Pacific (Tokyo)         ap-northeast-1
    AWS GovCloud (US-East)       us-gov-east-1
    AWS GovCloud (US)            us-gov-west-1
    Canada (Central)             ca-central-1
    China (Beijing)              cn-north-1
    China (Ningxia)              cn-northwest-1
    Europe (Frankfurt)           eu-central-1
    Europe (Ireland)             eu-west-1
    Europe (London)              eu-west-2
    Europe (Milan)               eu-south-1
    Europe (Paris)               eu-west-3
    Europe (Stockholm)           eu-north-1
    Middle East (Bahrain)        me-south-1
    South America (São Paulo)    sa-east-1
    US East (N. Virginia)        us-east-1
    US East (Ohio)               us-east-2
    US West (N. California)      us-west-1
    

    List Wasabi S3 region codes
    Code (Text):
    ./awscli-get.sh regions-wasabi
    Wasabi US East 1 (N. Virginia)    s3.wasabisys.com or s3.us-east-1.wasabisys.com
    Wasabi US East 2 (N. Virginia)    s3.us-east-2.wasabisys.com
    Wasabi US West 1 (Oregon)         s3.us-west-1.wasabisys.com
    Wasabi EU Central 1 (Amsterdam)   s3.eu-central-1.wasabisys.com
    

    AWS CLI command line client supports using multiple AWS profiles for AWS IAM user generated AWS Key and Secret Keys. To configure aws-cli custom profile called myprofile, you can use install profile command.
    Code (Text):
    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=us-west-2
    export AWS_DEFAULT_OUTPUT=text
    
    ./awscli-get.sh install myprofile
    
    existing config file detected: /root/.aws/config
    existing credential file detected: /root/.aws/credentials
    
    configure aws-cli profile: myprofile
    
    aws-cli profile: myprofile set:
    
    aws_access_key_id: AKIAIOSFODNN7EXAMPLE
    aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    default.region: us-west-2
    default output format: text
    
    list aws-cli profiles:
    
    default
    myprofile

    Backblaze B2 Cloud Storage Support



    Example of using AWS CLI and s5cmd with Backblaze B2 Cloud Storage which is S3 compatible.

    Create aws-cli profile named b2 by running install profile command. Backblaze when you generate a new API credentials will give you a key id and app key which are equivalent to AWS Access Key ID and AWS Secret Access Key.
    Code (Text):
    keyID=your_b2_keyid
    applicationKey=your_b2_app_key
    export AWS_ACCESS_KEY_ID=$keyID
    export AWS_SECRET_ACCESS_KEY=$applicationKey
    export AWS_DEFAULT_REGION=us-west-001
    export AWS_DEFAULT_OUTPUT=text
    
    ./awscli-get.sh install b2
    
    existing config file detected: /root/.aws/config
    existing credential file detected: /root/.aws/credentials
    
    configure aws-cli profile: b2
    
    aws-cli profile: b2 set:
    
    aws_access_key_id: your_b2_keyid
    aws_secret_access_key: your_b2_app_key
    default.region: us-west-001
    default output format: text
    
    list aws-cli profiles:
    
    default
    myprofile
    b2

    Now when you use AWS CLI client you need to use --profile b2 and --endpoint-url=https://s3.us-west-001.backblazeb2.com passed on command line. Example for listing contents of bucket named your_b2_bucket_name
    Code (Text):
    aws s3 ls --profile b2 --endpoint-url=https://s3.us-west-001.backblazeb2.com s3://your_b2_bucket_name
    2020-09-26 08:27:31          3 test.txt
    

    s5cmd known issues
    • s5cmd doesn't support copy/transfer of files greater than 5GB from one s3 bucket to another as s5cmd doesn't support multipart uploading
    • s5cmd doesn't support multiple AWS s3 user profiles and needs for the profile to be set via environment variable AWS_PROFILE
    example
    Code (Text):
    export AWS_PROFILE=myprofile
    export AWS_PROFILE=b2
    export AWS_PROFILE=default
    

    So using Backblaze B2 Cloud Storage bucket as an example. Comparing speed via time command of listing a single file in your_b2_bucket_name s3 bucket. s5cmd is approximately 56.5% faster than aws-cli.
    Code (Text):
    export AWS_PROFILE=b2
    time s5cmd --endpoint-url=https://s3.us-west-001.backblazeb2.com ls s3://your_b2_bucket_name
    2020/09/26 08:27:31                 3 test.txt
    
    real    0m0.809s
    user    0m0.054s
    sys     0m0.017s

    Code (Text):
    time aws s3 ls --profile b2 --endpoint-url=https://s3.us-west-001.backblazeb2.com s3://your_b2_bucket_name
    2020-09-26 08:27:31          3 test.txt
    
    real    0m1.861s
    user    0m1.039s
    sys     0m0.067s
     
  2. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I am using Wasabi and Backblaze B2 Cloud Storage :)

    Both are S3 compatible Object Storage providers.

    Both of them through Cloudflare... Cost around 5-6 $ per month for 1TB

    Backblaze is a LOT more stable !!!! Wasabi is free if you need direct downloads and Backblaze is $0.01 per GB...

    My vote for Backblaze !

    Isn't better for a solution that will do that directly from server A to server B without to have to move to a storage and then pull from there?

    Let's say that i want to get a better server..... what i would like to have is to connect from the new server to the old server and pull per domain or all (as an option) to the new server.

    It is the faster and will help for big sites...

    Thank you !!!!
     
    Last edited: Sep 29, 2020
  3. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yes transfer system is direct server to server. Just transfer system has backup and restore routines. I am adding S3 support to backup/restore so the routines can also be used for backup and restore and not necessarily for transfers.

    what's unstable about Wasabi?
     
  4. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Perfect !!!

    The infrastructure... At the beginning of the year almost every week there was an issue with the infrastructure and once they had a downtime for more than 10 hours. Most for the East region.

    Today things are better as i am getting an issue around once per month.

    I am not talking for issues related to me and my account or my settings. I am talking about issues on there end that send out notifications about that...

    Using Backblaze from the beginning of the year.... i got only once an issue and they fix it in minutes.

    Rock solid !
     
  5. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    I think it will help a lot for the direct transfer from server A to server B to use Rsync.

    As we will be able to transfer for example 100GB of files and the next day to just sync them with the source and be ready for the migration in seconds :)

    Thank you !!!!!
     
  6. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    The intended purpose is for initial transfers i.e. moving servers and the way I plan it is for fastest initial first time transfers. 100GB of data on decent server should be transferred in ~20-60 minutes for files and MySQL databases.

    Rsync will be magnitudes slower for transfers compared to what I have planned - which will allow initial first time transfers at speeds close to the line rate of your network connection and disk while still being encrypted and compressed :)

    But yes doing a re-sync with rsync would be faster, so will need to account for that as well.

    Quoting myself - the name of the backup script hasn't been finalised yet
     
  7. deltahf

    deltahf Premium Member Premium Member

    381
    176
    43
    Jun 8, 2014
    Ratings:
    +294
    Local Time:
    1:53 PM
    Another Backblaze B2 user here.

    It's not connected with my server but I use it to back up my Synology NAS. :)
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Black Friday is coming in a few weeks and it will be great if we can have that tool before :)

    How is going the development?

    Thank you George !!!
     
  9. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Is there some Black Friday specials going on ? Tool still in development as I have been busy with paid client work (server and Centmin Mod LEMP stack optimization work) :) Also starting a new refresh of my blog.centminmod.com which I have big plans for. Used latest centmin.sh menu option 22 Wordpress installer again to create it :D
     
  10. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Not yet :) But they will start i think in 10-15 days :)

    I am ready to test it when is ready (y)
     
  11. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    At the end of the upcoming week i think they will start the Black Friday offers :)

    It will be good if we can have this tool in the next 7 days :)

    Thank you !!!!!
     
  12. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Haven't had time to do more testing on the awscli-get.sh tool. But if you really want to play with it with no guarantees from me right now, you can download it and install it via dedicated instructions page at https://awscli-get.centminmod.com/ - the page and script is solely powered by Cloudflare Workers for static HTML page and CF Worker proxy to Git backed repo hosted script. Just playing with Cloudflare Workers :D
     
  13. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Hello George :)

    Thanks for the link and the info !!!!

    The most useful part for me it will be the ability to transfer from server A to server B everything needed for a simple domain or for all domains in the server.

    The info and instructions from the link is related to upload files at a Cloud storage provider...

    I am not complaining... I am just saying that the part that it will be more helpful for me is not developed yet :)

    Thank you
     
  14. eva2000

    eva2000 Administrator Staff Member

    45,662
    10,360
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,079
    Local Time:
    4:53 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Oh you mean transfer all data from Centmin Mod to Centmin Mod server tools? Oh that isn't ready for public use yet only the awscli-get.sh tool is - sort of.
     
    Last edited: Nov 9, 2020
  15. pamamolf

    pamamolf Premium Member Premium Member

    3,885
    381
    83
    May 31, 2014
    Ratings:
    +741
    Local Time:
    8:53 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Yes :)