Amazon AWS How many folks use Amazon S3 or S3 compatible Object Storage?

How many folks using Centmin Mod or other control panels are using Amazon S3 Object Storage or other cloud provider's S3 compatible Object Storage products?

I ask, as I am working on the Centmin Mod to Centmin Mod transfer feature's routines and part of that will be the backup and restore aspects. So vote and if you use other S3 Object, compatible providers, please reply to this thread with the details

FYI, also planned to make Centmin Mod a S3 compatible backup server itself. So you can make a Centmin Mod server a backup server to host backup files and have it support using S3 clients to transfer in/out the backup files to Centmin Mod.

If you like what you see or will find such developments useful, make sure you support Centmin Mod via various ways outlined here

Some S3 and S3 compatible Object Storage providers which I intend to support include:

• Amazon S3
• Backblaze B2 Storage
• DigitalOcean Spaces Object Storage
• Linode Object Storage
• Wasabi Cloud Storage

S3 Compatible Object Storage & Clients

Part of the backup aspect is supporting 3rd party storage providers which are S3 object storage compatible. My dbbackup.sh MySQL database script for Centmin Mod premium members has optional support for backing up to Amazon S3 object storage via s3cmd S3 client.

However, S3 clients have evolved so looking at adding support for official AWS aws-cli command line client as well as s5cmd S3 client which are both faster eventually for any Centmin Mod backup system I plan to release.

• AWS aws-cli command line client
• s5cmd S3 client

awscli-get.sh AWS CLI & s5cmd Installer For Centmin Mod

As part of Centmin Mod's backup/transfer features, I am writing a new AWS aws-cli and s5cmd S3 client install and configuration script to make it easier for Centmin Mod users to install and configure aws-cli command line client and s5cmd S3 client.

A preview of what it does so far

By default the script will install Linux 64bit aws-cli from https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip and install it to /usr/local/bin/aws and install s5cmd from latest releases at Releases · peak/s5cmd to /usr/local/bin/s5cmd.

Code (Text):

./awscli-get.sh help

Usage:

./awscli-get.sh {install|update} {default|profilename|defaultreset} AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION AWS_DEFAULT_OUTPUT
./awscli-get.sh regions
./awscli-get.sh regions-wasabi

Both Amazon AWS CLI and s5cmd clients use the official Amazon S3 SDK so you need to export the environment variables for AWS Access Key and Secret Key credentials for the script to pick them up otherwise you get the following message. So in your SSH logged in session type the export commands and fill in the variables with your credentials you generated via AWS Console or 3rd party provider provided details.

Code (Text):

./awscli-get.sh

Error. AWS Key & Secret not detected
export relevant environment variables
and re-run script

example:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
export AWS_DEFAULT_OUTPUT=text

Configuration

---

First time running awscli-get.sh to configure aws-cli default profile

Code (Text):

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
export AWS_DEFAULT_OUTPUT=text

./awscli-get.sh

configure default aws-cli profile

aws-cli profile: default set:

aws_access_key_id: AKIAIOSFODNN7EXAMPLE
aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
default.region: us-west-2
default output format: text

If default aws-cli profile already set and detected, you'll get below message about skipping default profile configuration i.e. you already have AWS CLI command line or another S3 client with reads the same AWS CLI config/credential files. You'll also get info on how to use defaulreset command to overwrite your existing default AWS CLI command line profile's AWS Key and Secret if you desire.

Code (Text):

./awscli-get.sh                      

existing config file detected: /root/.aws/config
existing credential file detected: /root/.aws/credentials

skipping configuration...

detected existing default credentials
in config file: /root/.aws/config
in credential file: /root/.aws/credentials

To override the default profile, use defaultreset mode:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
export AWS_DEFAULT_OUTPUT=text

./awscli-get.sh install defaultreset

Example commands supported

List AWS S3 region codes

Code (Text):

./awscli-get.sh regions
Africa (Cape Town)           af-south-1
Asia Pacific (Hong Kong)     ap-east-1
Asia Pacific (Mumbai)        ap-south-1
Asia Pacific (Osaka-Local)   ap-northeast-3
Asia Pacific (Seoul)         ap-northeast-2
Asia Pacific (Singapore)     ap-southeast-1
Asia Pacific (Sydney)        ap-southeast-2
Asia Pacific (Tokyo)         ap-northeast-1
AWS GovCloud (US-East)       us-gov-east-1
AWS GovCloud (US)            us-gov-west-1
Canada (Central)             ca-central-1
China (Beijing)              cn-north-1
China (Ningxia)              cn-northwest-1
Europe (Frankfurt)           eu-central-1
Europe (Ireland)             eu-west-1
Europe (London)              eu-west-2
Europe (Milan)               eu-south-1
Europe (Paris)               eu-west-3
Europe (Stockholm)           eu-north-1
Middle East (Bahrain)        me-south-1
South America (São Paulo)    sa-east-1
US East (N. Virginia)        us-east-1
US East (Ohio)               us-east-2
US West (N. California)      us-west-1

List Wasabi S3 region codes

Code (Text):

./awscli-get.sh regions-wasabi
Wasabi US East 1 (N. Virginia)    s3.wasabisys.com or s3.us-east-1.wasabisys.com
Wasabi US East 2 (N. Virginia)    s3.us-east-2.wasabisys.com
Wasabi US West 1 (Oregon)         s3.us-west-1.wasabisys.com
Wasabi EU Central 1 (Amsterdam)   s3.eu-central-1.wasabisys.com

AWS CLI command line client supports using multiple AWS profiles for AWS IAM user generated AWS Key and Secret Keys. To configure aws-cli custom profile called myprofile, you can use install profile command.

Code (Text):

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2
export AWS_DEFAULT_OUTPUT=text

./awscli-get.sh install myprofile

existing config file detected: /root/.aws/config
existing credential file detected: /root/.aws/credentials

configure aws-cli profile: myprofile

aws-cli profile: myprofile set:

aws_access_key_id: AKIAIOSFODNN7EXAMPLE
aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
default.region: us-west-2
default output format: text

list aws-cli profiles:

default
myprofile

Backblaze B2 Cloud Storage Support

Example of using AWS CLI and s5cmd with Backblaze B2 Cloud Storage which is S3 compatible.

Create aws-cli profile named b2 by running install profile command. Backblaze when you generate a new API credentials will give you a key id and app key which are equivalent to AWS Access Key ID and AWS Secret Access Key.

Code (Text):

keyID=your_b2_keyid
applicationKey=your_b2_app_key
export AWS_ACCESS_KEY_ID=$keyID
export AWS_SECRET_ACCESS_KEY=$applicationKey
export AWS_DEFAULT_REGION=us-west-001
export AWS_DEFAULT_OUTPUT=text

./awscli-get.sh install b2

existing config file detected: /root/.aws/config
existing credential file detected: /root/.aws/credentials

configure aws-cli profile: b2

aws-cli profile: b2 set:

aws_access_key_id: your_b2_keyid
aws_secret_access_key: your_b2_app_key
default.region: us-west-001
default output format: text

list aws-cli profiles:

default
myprofile
b2

Now when you use AWS CLI client you need to use --profile b2 and --endpoint-url=https://s3.us-west-001.backblazeb2.com passed on command line. Example for listing contents of bucket named your_b2_bucket_name

Code (Text):

aws s3 ls --profile b2 --endpoint-url=https://s3.us-west-001.backblazeb2.com s3://your_b2_bucket_name
2020-09-26 08:27:31          3 test.txt

s5cmd known issues

• s5cmd doesn't support copy/transfer of files greater than 5GB from one s3 bucket to another as s5cmd doesn't support multipart uploading
• s5cmd doesn't support multiple AWS s3 user profiles and needs for the profile to be set via environment variable AWS_PROFILE

example

Code (Text):

export AWS_PROFILE=myprofile
export AWS_PROFILE=b2
export AWS_PROFILE=default

So using Backblaze B2 Cloud Storage bucket as an example. Comparing speed via time command of listing a single file in your_b2_bucket_name s3 bucket. s5cmd is approximately 56.5% faster than aws-cli.

Code (Text):

export AWS_PROFILE=b2
time s5cmd --endpoint-url=https://s3.us-west-001.backblazeb2.com ls s3://your_b2_bucket_name
2020/09/26 08:27:31                 3 test.txt

real    0m0.809s
user    0m0.054s
sys     0m0.017s

Code (Text):

time aws s3 ls --profile b2 --endpoint-url=https://s3.us-west-001.backblazeb2.com s3://your_b2_bucket_name
2020-09-26 08:27:31          3 test.txt

real    0m1.861s
user    0m1.039s
sys     0m0.067s

 

Yes transfer system is direct server to server. Just transfer system has backup and restore routines. I am adding S3 support to backup/restore so the routines can also be used for backup and restore and not necessarily for transfers.

what's unstable about Wasabi?

 

The intended purpose is for initial transfers i.e. moving servers and the way I plan it is for fastest initial first time transfers. 100GB of data on decent server should be transferred in ~20-60 minutes for files and MySQL databases.

Rsync will be magnitudes slower for transfers compared to what I have planned - which will allow initial first time transfers at speeds close to the line rate of your network connection and disk while still being encrypted and compressed

But yes doing a re-sync with rsync would be faster, so will need to account for that as well.

Quoting myself - the name of the backup script hasn't been finalised yet

 

Is there some Black Friday specials going on ? Tool still in development as I have been busy with paid client work (server and Centmin Mod LEMP stack optimization work) Also starting a new refresh of my blog.centminmod.com which I have big plans for. Used latest centmin.sh menu option 22 Wordpress installer again to create it

 

Haven't had time to do more testing on the awscli-get.sh tool. But if you really want to play with it with no guarantees from me right now, you can download it and install it via dedicated instructions page at https://awscli-get.centminmod.com/ - the page and script is solely powered by Cloudflare Workers for static HTML page and CF Worker proxy to Git backed repo hosted script. Just playing with Cloudflare Workers

 

Oh you mean transfer all data from Centmin Mod to Centmin Mod server tools? Oh that isn't ready for public use yet only the awscli-get.sh tool is - sort of.