Beta Branch addons/rclone.sh client for syncing to remote cloud storage providers

eva2000 · Oct 28, 2016

Centmin Mod 123.09beta01 has added a new addons/rclone.sh addon to install and configure rclone rsync client which allows to you to sync and transfer files to several common cloud storage providers. You can discuss and provide beta testing feedback here. If you like and want to support Centmin Mod, consider making a donation or signing up for Centmin Mod Premium Membership. Also if your country supports paypal.me https://www.paypal.me/georgeliu2k

The first providers I will focus on is Dropbox, Google Drive and OneDrive with example further below in this thread. The aim of addons/rclone.sh is to make it noob friendly for setup from both client and server side and Dropbox, Google Drive and OneDrive are as easy as they come. After that will look at other provider examples.

Rclone is a command line program to sync files and directories to and from

Google drive

Amazon S3

Swift / Rackspace Cloudfiles / Memset Memstore

Dropbox

Google Cloud Storage

Local filesystem

Amazon Drive

Backblaze B2

Hubic

Microsoft One Drive

Yandex Disk

addons/rclone.sh Examples

Dropbox config

Google Drive config

OneDrive config

addons/rclone.sh copyssl & syncssl

Rclone Documentation

Documentation

Rclone Support Forums

rclone forum

Rclone.sh Objective

The addon came about from discussions at https://community.centminmod.com/threads/which-cloud-storage-provider-do-you-use-most.9291/ regarding how I can easily sync /root/centminlogs to cloud storage with easy access client side app to view the logs. These logs are created everytime you run centmin.sh menu and are critical to troubleshooting issues with Centmin Mod so end user access to them is important.

eva2000 · Oct 28, 2016

Rclone Dropbox Configuration

You'll need a Dropbox account for this. Best to create a new separate Dropbox account for testing first. You can change/reconfigure the Dropbox account later on. You can discuss and provide beta testing feedback here.

Steps:

Login into your Dropbox account from your web browser
Update Centmin Mod 123.09beta01 to latest code via centmin.sh menu optio 23 so you get access to addons/rclone.sh
Enable addons/rclone.sh via persistent config file setting RCLONE_ENABLE='y' in /etc/centminmod/custom_config.inc. This is required during beta testing addons/rclone.sh
In SSH, go to /usr/local/src/centminmod/addons directory and run rclone.sh
Code (Text):
```
./rclone.sh
```

Running rclone config

At initial install, you will need to configure rclone to add your remote storage providers info (setup remotes).
For Dropbox the config parameters are outlined at Dropbox

Command options available to rclone are below. Note copy and sync options don't do anything right now as I need to figure out how to script it to find the remote names from rclone command line.

Code (Text):

./rclone.sh
./rclone.sh {config|install|update|copy|sync|copyssl|syncssl}

Example for install and configuration for Dropbox

Code (Text):

./rclone install

output

Code (Text):

./rclone.sh install   
------------------------------------------------
Install Rclone 64bit
------------------------------------------------
2016-10-27 21:27:18 URL:http://downloads.rclone.org/rclone-current-linux-amd64.zip [3662290/3662290] -> "rclone-current-linux-amd64.zip" [1]
Archive:  rclone-current-linux-amd64.zip
   creating: rclone-v1.33-linux-amd64/
  inflating: rclone-v1.33-linux-amd64/README.html
  inflating: rclone-v1.33-linux-amd64/rclone
  inflating: rclone-v1.33-linux-amd64/rclone.1
  inflating: rclone-v1.33-linux-amd64/README.txt
-rwxr-xr-x 1 root root 11M Oct 27 21:27 /usr/sbin/rclone

------------------------------------------------
rclone binary at /usr/sbin/rclone
------------------------------------------------

Code (Text):

------------------------------------------------
You need to now manually configure your storage provider
from instructions at http://rclone.org/docs/
------------------------------------------------

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> dropbox1
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 7 / Google Drive
   \ "drive"
 8 / Hubic
   \ "hubic"
 9 / Local Disk
   \ "local"
10 / Microsoft OneDrive
   \ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
12 / Yandex Disk
   \ "yandex"
Storage> 4
Dropbox App Key - leave blank normally.
app_key>
Dropbox App Secret - leave blank normally.
app_secret>

You'll be asked to visit a dropbox url in your web browser where you logged into your Dropbox account to authorize and connect to the Dropbox account

Code (Text):

Remote config
Please visit:
https://www.dropbox.com/1/oauth2/authorize?client_id=5jxadasiasz0rqy&response_type=code
Enter the code: plahsagxUnxkAAAAAAAAACjrz-kAteycMXctAqF55xnw
--------------------
[dropbox1]
app_key =
app_secret =
token = plahsagxUnxkAAAAAAAAAC66j0RyUqrCDWdlCMXWD_pcsasfDlv_z2k3u
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
dropbox1             dropbox

e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q> q

Total Rclone Install Time: 67.209031379 seconds

Rclone Commands

Common commands where remote is name you entered at config time i.e. for example above name = dropbox1

Code (Text):

rclone lsd remote:

list remote directories

Code (Text):

rclone lsd dropbox1:
2016/10/27 21:35:12
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.2s
list remote files
[CODEB]
rclone ls dropbox1:
2016/10/27 21:36:36
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.4s

Both return empty as my Dropbox account remote is empty right now. So lets add and transfer some files form local server to Dropbox account

Transferring /root/centminlogs log directory to remote:centminlogs i.e. dropbox1:centminlogs

Code (Text):

rclone copy /root/centminlogs/ dropbox1:centminlogs

Code (Text):

rclone copy /root/centminlogs/ dropbox1:centminlogs
2016/10/27 21:38:29 Dropbox root 'centminlogs': Waiting for checks to finish
2016/10/27 21:38:29 Dropbox root 'centminlogs': Waiting for transfers to finish
2016/10/27 21:38:32
Transferred:   3.061 MBytes (121.723 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           35
Elapsed time:       25.7s

Now if you list directories should see centminlogs

Code (Text):

rclone lsd dropbox1:
           0 0001-01-01 00:00:00        -1 centminlogs
2016/10/27 21:39:33
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.1s

If you list files

Code (Text):

rclone ls dropbox1:
        0 centminlogs/patch_patchnginx_111016-112321.log
       45 centminlogs/centminmod_libresslinstalltime_111016-112321.log
     1697 centminlogs/nginx-configure-111016-112321.log
     1388 centminlogs/gcc_native.log
   126775 centminlogs/centminmod_downloadtimes_111016-112321.log
   630333 centminlogs/centminmod_ngxinstalltime_111016-112321.log
      783 centminlogs/centminmod_setio_111016-013111.log
  2288914 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
       58 centminlogs/centminmod_phpinstalltime_111016-112321.log
     9568 centminlogs/setio_innodbstatus-after-111016-013111.log
    70711 centminlogs/centminmod_yumtimes_111016-112321.log
    17644 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
     2807 centminlogs/validcheck_downloads_111016-112321.log
       38 centminlogs/centminmod_profiletimes_111016-112321.log
      275 centminlogs/securedtmp.log
     9587 centminlogs/setio_innodbstatus-before-111016-013111.log
      293 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
      705 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
     3005 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
      999 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
      293 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
     8074 centminlogs/rclone_installer_271016-204143.log
      512 centminlogs/rclone_config_271016-204359.log
      251 centminlogs/rclone_config_271016-204416.log
     6651 centminlogs/rclone_installer_271016-212254.log
     6606 centminlogs/rclone_installer_271016-212459.log
      125 centminlogs/install.utc.time.log
     2358 centminlogs/rclone_installer_271016-212713.log
     6898 centminlogs/rclone_installer_271016-212621.log
      293 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
       47 centminlogs/firstyum_installtime_111016-112102.log
      293 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
      387 centminlogs/zendopcache_passfile.txt
     1701 centminlogs/gcc_php_native.log
     9068 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
2016/10/27 21:40:18
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.4s

list files with time and size info

Code (Text):

rclone lsl dropbox1:
        0 2016-10-27 21:38:16.000000000 centminlogs/patch_patchnginx_111016-112321.log
     1697 2016-10-27 21:38:18.000000000 centminlogs/nginx-configure-111016-112321.log
     1388 2016-10-27 21:38:20.000000000 centminlogs/gcc_native.log
     9587 2016-10-27 21:38:21.000000000 centminlogs/setio_innodbstatus-before-111016-013111.log
   630333 2016-10-27 21:38:21.000000000 centminlogs/centminmod_ngxinstalltime_111016-112321.log
      783 2016-10-27 21:38:21.000000000 centminlogs/centminmod_setio_111016-013111.log
  2288914 2016-10-27 21:38:22.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
       58 2016-10-27 21:38:22.000000000 centminlogs/centminmod_phpinstalltime_111016-112321.log
     9568 2016-10-27 21:38:23.000000000 centminlogs/setio_innodbstatus-after-111016-013111.log
     1701 2016-10-27 21:38:23.000000000 centminlogs/gcc_php_native.log
      387 2016-10-27 21:38:24.000000000 centminlogs/zendopcache_passfile.txt
     9068 2016-10-27 21:38:25.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
      125 2016-10-27 21:38:25.000000000 centminlogs/install.utc.time.log
      293 2016-10-27 21:38:25.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
       47 2016-10-27 21:38:26.000000000 centminlogs/firstyum_installtime_111016-112102.log
      293 2016-10-27 21:38:26.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
      293 2016-10-27 21:38:27.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
      705 2016-10-27 21:38:28.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
       45 2016-10-27 21:38:16.000000000 centminlogs/centminmod_libresslinstalltime_111016-112321.log
      999 2016-10-27 21:38:29.000000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
      293 2016-10-27 21:38:29.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
     8074 2016-10-27 21:38:29.000000000 centminlogs/rclone_installer_271016-204143.log
      512 2016-10-27 21:38:30.000000000 centminlogs/rclone_config_271016-204359.log
      251 2016-10-27 21:38:30.000000000 centminlogs/rclone_config_271016-204416.log
     6651 2016-10-27 21:38:30.000000000 centminlogs/rclone_installer_271016-212254.log
     6606 2016-10-27 21:38:31.000000000 centminlogs/rclone_installer_271016-212459.log
     6898 2016-10-27 21:38:31.000000000 centminlogs/rclone_installer_271016-212621.log
     2358 2016-10-27 21:38:32.000000000 centminlogs/rclone_installer_271016-212713.log
    70711 2016-10-27 21:38:12.000000000 centminlogs/centminmod_yumtimes_111016-112321.log
    17644 2016-10-27 21:38:12.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
   126775 2016-10-27 21:38:12.000000000 centminlogs/centminmod_downloadtimes_111016-112321.log
     3005 2016-10-27 21:38:28.000000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
     2807 2016-10-27 21:38:14.000000000 centminlogs/validcheck_downloads_111016-112321.log
       38 2016-10-27 21:38:13.000000000 centminlogs/centminmod_profiletimes_111016-112321.log
      275 2016-10-27 21:38:15.000000000 centminlogs/securedtmp.log
2016/10/27 21:51:07
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                35
Transferred:            0
Elapsed time:        1.2s

Now if you try to copy again, and no files have changed since, then you will see 0 Bytes transferred just like rsync would do

Code (Text):

rclone copy /root/centminlogs/ dropbox1:centminlogs
2016/10/27 21:46:01 Dropbox root 'centminlogs': Waiting for checks to finish
2016/10/27 21:46:01 Dropbox root 'centminlogs': Waiting for transfers to finish
2016/10/27 21:46:01
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                35
Transferred:            0
Elapsed time:        2.8s

Deleting files in remote = dropbox1:centminlogs

Code (Text):

rclone delete dropbox1:
2016/10/27 22:04:08 Waiting for deletions to finish
2016/10/27 22:04:15
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                35
Transferred:            0
Elapsed time:        7.9s

eva2000 · Oct 28, 2016

Rclone Command & Help Options

Rclone commands docs at Commands

The main rclone commands with most used first

rclone config - Enter an interactive configuration session.
rclone copy - Copy files from source to dest, skipping already copied
rclone sync - Make source and dest identical, modifying destination only.
rclone move - Move files from source to dest.
rclone delete - Remove the contents of path.
rclone purge - Remove the path and all of its contents.
rclone mkdir - Make the path if it doesn’t already exist.
rclone rmdir - Remove the path.
rclone check - Checks the files in the source and destination match.
rclone ls - List all the objects in the the path with size and path.
rclone lsd - List all directories/containers/buckets in the the path.
rclone lsl - List all the objects path with modification time, size and path.
rclone md5sum - Produces an md5sum file for all the objects in the path.
rclone sha1sum - Produces an sha1sum file for all the objects in the path.
rclone size - Returns the total size and number of objects in remoteath.
rclone version - Show the version number.
rclone cleanup - Clean up the remote if possible
rclone dedupe - Interactively find duplicate files delete/rename them.

See the commands index for the full list.

Help info

Code (Text):

rclone -h
Rclone is a command line program to sync files and directories to and
from various cloud storage systems, such as:

  * Google Drive
  * Amazon S3
  * Openstack Swift / Rackspace cloud files / Memset Memstore
  * Dropbox
  * Google Cloud Storage
  * Amazon Drive
  * Microsoft One Drive
  * Hubic
  * Backblaze B2
  * Yandex Disk
  * The local filesystem

Features

  * MD5/SHA1 hashes checked at all times for file integrity
  * Timestamps preserved on files
  * Partial syncs supported on a whole file basis
  * Copy mode to just copy new/changed files
  * Sync (one way) mode to make a directory identical
  * Check mode to check for file hash equality
  * Can sync to and from network, eg two different cloud accounts

Code (Text):

See the home page for installation, usage, documentation, changelog
and configuration walkthroughs.

  * http://rclone.org/

Usage:
  rclone [flags]
  rclone [command]

Available Commands:
  authorize       Remote authorization.
  cat             Concatenates any files and sends them to stdout.
  check           Checks the files in the source and destination match.
  cleanup         Clean up the remote if possible
  config          Enter an interactive configuration session.
  copy            Copy files from source to dest, skipping already copied
  dedupe          Interactively find duplicate files delete/rename them.
  delete          Remove the contents of path.
  genautocomplete Output bash completion script for rclone.
  gendocs         Output markdown docs for rclone to the directory supplied.
  ls              List all the objects in the the path with size and path.
  lsd             List all directories/containers/buckets in the the path.
  lsl             List all the objects path with modification time, size and path.
  md5sum          Produces an md5sum file for all the objects in the path.
  mkdir           Make the path if it doesn't already exist.
  mount           Mount the remote as a mountpoint. **EXPERIMENTAL**
  move            Move files from source to dest.
  purge           Remove the path and all of its contents.
  rmdir           Remove the path if empty.
  sha1sum         Produces an sha1sum file for all the objects in the path.
  size            Prints the total size and number of objects in remote:path.
  sync            Make source and dest identical, modifying destination only.
  version         Show the version number.

Code (Text):

Flags:
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
--ask-password Allow prompt for password for encrypted configuration. (default true)
--b2-chunk-size int Upload chunk size. Must fit in memory.
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
--b2-upload-cutoff int Cutoff for switching to chunked upload
--b2-versions Include old versions in directory listings.
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
--checkers int Number of checkers to run in parallel. (default 8)
-c, --checksum Skip based on checksum & size, not mod-time & size
--config string Config file. (default "/root/.rclone.conf")
--contimeout duration Connect timeout (default 1m0s)
--cpuprofile string Write cpu profile to file
--delete-after When synchronizing, delete files on destination after transfering
--delete-before When synchronizing, delete files on destination before transfering
--delete-during When synchronizing, delete files during transfer (default)
--delete-excluded Delete files on dest excluded from sync
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
--drive-upload-cutoff int Cutoff for switching to chunked upload
--drive-use-trash Send files to the trash instead of deleting permanently.
--dropbox-chunk-size int Upload chunk size. Max 150M.
-n, --dry-run Do a trial run with no permanent changes
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-filters Dump the filters to the output
--dump-headers Dump HTTP headers - may contain sensitive info
--exclude string Exclude files matching pattern
--exclude-from string Read exclude patterns from file
--files-from string Read list of source-file names from file
-f, --filter string Add a file-filtering rule
--filter-from string Read filtering patterns from a file
--ignore-existing Skip all files that exist on destination
--ignore-size Ignore size when skipping use mod-time or checksum.
-I, --ignore-times Don't skip files that match size and time - transfer all files
--include string Include files matching pattern
--include-from string Read include patterns from file
--log-file string Log everything to this file
--low-level-retries int Number of low level retries to do. (default 10)
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
--max-depth int If set limits the recursion depth to this. (default -1)
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
--memprofile string Write memory profile to file
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
--modify-window duration Max time diff to be considered the same (default 1ns)
--no-check-certificate Do not verify the server SSL certificate. Insecure.
--no-gzip-encoding Don't set Accept-Encoding: gzip.
--no-traverse Don't traverse destination file system on copy.
--no-update-modtime Don't update destination mod-time if files identical.
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
-q, --quiet Print as little stuff as possible
--retries int Retry operations this many times if they fail (default 3)
--size-only Skip based on size only, not mod-time or checksum
--stats duration Interval to print stats (0 to disable) (default 1m0s)
--swift-chunk-size int Above this size files will be chunked into a _segments container.
--timeout duration IO idle timeout (default 5m0s)
--transfers int Number of file transfers to run in parallel. (default 4)
-u, --update Skip files that are newer on the destination.
-v, --verbose Print lots more stuff
-V, --version Print the version number

Use "rclone [command] --help" for more information about a command.

eva2000 · Oct 28, 2016

Discussion Feedback

You can discuss and provide beta testing feedback here.

eva2000 · Oct 28, 2016

addons/rclone.sh copy & sync options

The copy & sync options allow you to pass the remote name you setup via the command line to copy or sync /root/centminlogs to the remote cloud storage provider. Using above Dropbox remote = dropbox1 as an example. Update: add /usr/local/nginx/conf too.

Copy is one way direction local to remote copy. Sync is making the remote same as local copy.

copy

Code (Text):

./rclone.sh copy dropbox1

sync

Code (Text):

./rclone.sh sync dropbox1

Example output

copy

Code (Text):

./rclone.sh copy dropbox1       
remote = dropbox1

copy /root/centminlogs to cloud storage remote dropbox1
https://community.centminmod.com/posts/39071/

rclone copy /root/centminlogs dropbox1:centminlogs
rclone copy /root/centminlogs dropbox1:centminlogs --exclude rclone_copy_281016-000242.log
2016/10/28 00:02:44 Dropbox root 'centminlogs': Waiting for checks to finish
2016/10/28 00:02:44 Dropbox root 'centminlogs': Waiting for transfers to finish
2016/10/28 00:02:47
Transferred:    908 Bytes (177 Bytes/s)
Errors:                 0
Checks:                53
Transferred:            1
Elapsed time:        5.1s


Total Rclone Copy Time: 5.121360189 seconds

sync

Code (Text):

./rclone.sh sync dropbox1
remote = dropbox1

sync /root/centminlogs to cloud storage remote dropbox1
https://community.centminmod.com/posts/39071/

rclone sync /root/centminlogs dropbox1:centminlogs
rclone sync /root/centminlogs dropbox1:centminlogs --exclude rclone_sync_281016-000453.log
2016/10/28 00:04:55 Dropbox root 'centminlogs': Waiting for checks to finish
2016/10/28 00:04:55 Dropbox root 'centminlogs': Waiting for transfers to finish
2016/10/28 00:04:58 Waiting for deletions to finish
2016/10/28 00:04:58
Transferred:    632 Bytes (124 Bytes/s)
Errors:                 0
Checks:                54
Transferred:            1
Elapsed time:          5s


Total Rclone Sync Time: 5.089820731 seconds

eva2000 · Oct 28, 2016

Rclone Google Drive Config

As per Google Drive setup guide at Google drive you should setup your own client_id for Google Drive for best performance.

Here is how to create your own Google Drive client ID for rclone:

Log into the Google API Console with your Google account. It doesn’t matter what Google account you use. (It need not be the same account as the Google Drive you want to access)
Select a project or create a new project.
Under Overview, Google APIs, Google Apps APIs, click “Drive API”, then “Enable”.
Click “Credentials” in the left-side panel (not “Go to credentials”, which opens the wizard), then “Create credentials”, then “OAuth client ID”. It will prompt you to set the OAuth consent screen product name, if you haven’t set one already.
Choose an application type of “other”, and click “Create”. (the default name is fine)
It will show you a client ID and client secret. Use these values in rclone config to add a new remote or edit an existing remote.

Once you have your clientid and secret, fill them in when prompted for Google Application Client Id and Google Application Client Secret

Config

This will involve you being logged into your Google Drive account on your desktop PC and copying and pasting a url generated from Rclone config

Code (Text):

./rclone.sh config

Run addons/rclone.sh config and setup Google Drive remote called = gdrive1

Code (Text):

./rclone.sh config

------------------------------------------------
You need to now manually configure your storage provider
from instructions at http://rclone.org/docs/
------------------------------------------------


Current remotes:

Name                 Type
====                 ====
dropbox1             dropbox

e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q> n
name> gdrive1
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 7 / Google Drive
   \ "drive"
 8 / Hubic
   \ "hubic"
 9 / Local Disk
   \ "local"
10 / Microsoft OneDrive
   \ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
12 / Yandex Disk
   \ "yandex"
Storage> 7
Google Application Client Id - leave blank normally.
client_id>
Google Application Client Secret - leave blank normally.
client_secret>

Select N for headless machine and copy and paste the google account link into your web browser which you already logged into your Google Drive account for. You will get a verification code you need to enter to continue

Code (Text):

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine or Y didn't work
y) Yes
n) No
y/n> n
If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?client_id=22***644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=ad58XS******4e
Log in and authorize rclone for access
Enter verification code> 4/3_1cAAAANYw4VTYMSQM4J_uw

Enter y if correct

Code (Text):

--------------------
[gdrive1]
client_id =
client_secret =
token = {"access_token":"ya29.Ci-KAyoBd6*****_sssssXZYUy60CjELsqggu9mpklKX_FlXA","token_type":"Bearer","refresh_token":"1/uMDepQ******8nuYM","expiry":"2016-10-28T04:57:20.202908281Z"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Listing of current remotes

Code (Text):

Current remotes:

Name                 Type
====                 ====
dropbox1             dropbox
gdrive1              drive

e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q> q

Total Rclone Config Time: 51.692002451 seconds

check directories in remote = gdrive1

Code (Text):

rclone lsd gdrive1:
2016/10/28 04:03:04
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.1s

test addons/rclone.sh copy command to gdrive1

Code (Text):

./rclone.sh copy gdrive1

Code (Text):

./rclone.sh copy gdrive1
remote = gdrive1

copy /root/centminlogs to cloud storage remote gdrive1
https://community.centminmod.com/posts/39071/

rclone copy /root/centminlogs gdrive1:centminlogs --exclude rclone_copy_281016-040452.log
2016/10/28 04:05:39 Google drive root 'centminlogs': Waiting for checks to finish
2016/10/28 04:05:39 Google drive root 'centminlogs': Waiting for transfers to finish
2016/10/28 04:05:45
Transferred:   3.090 MBytes (60.126 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           65
Elapsed time:       52.6s

copy /usr/local/nginx/conf to cloud storage remote gdrive1

rclone copy /usr/local/nginx/conf gdrive1:nginxconf
2016/10/28 04:06:28 Google drive root 'nginxconf': Waiting for checks to finish
2016/10/28 04:06:28 Google drive root 'nginxconf': Waiting for transfers to finish
2016/10/28 04:06:38
Transferred:   104.285 kBytes (1.963 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           64
Elapsed time:       53.1s


Total Rclone Copy Time: 105.761092037 seconds

then check contents on remote = gdrive1

Code (Text):

rclone lsl gdrive1:

Code (Text):

rclone lsl gdrive1:
      212 2016-10-11 01:37:05.723000000 nginxconf/include_opcache.conf
      292 2016-10-11 01:29:04.963000000 nginxconf/vts_mainserver.conf
       24 2016-10-11 01:29:04.963000000 nginxconf/vts_server.conf
      888 2016-10-11 01:29:04.963000000 nginxconf/wpffpc.conf
       26 2016-10-11 01:29:04.963000000 nginxconf/vts_http.conf
     2236 2016-10-11 01:29:04.963000000 nginxconf/staticfiles.conf
       10 2016-10-11 01:29:04.961000000 nginxconf/sitestatus.conf
      110 2016-10-11 01:29:04.961000000 nginxconf/ssl-include.conf
     2460 2016-10-11 01:29:04.960000000 nginxconf/phpssl.conf
      884 2016-10-11 01:29:04.960000000 nginxconf/proxy.conf
      362 2016-10-11 01:29:04.960000000 nginxconf/phpstatus.conf
     2441 2016-10-11 01:29:04.960000000 nginxconf/pscontrol
     3371 2016-10-11 01:29:04.960000000 nginxconf/php-pool5.conf
     3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool4.conf
     3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool3.conf
     3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool2.conf
     2682 2016-10-11 01:29:04.955000000 nginxconf/phpalt.conf
      668 2016-10-11 01:29:04.955000000 nginxconf/pagespeedhandler.conf
     1740 2016-10-11 01:29:04.954000000 nginxconf/pagespeedadmin.conf
     7152 2016-10-11 01:29:04.954000000 nginxconf/pagespeed.conf
      645 2016-10-11 01:29:04.953000000 nginxconf/maintenance.conf
      283 2016-10-11 01:29:04.952000000 nginxconf/geoip.conf
     1271 2016-10-11 01:29:04.952000000 nginxconf/fastcgi_params
      626 2016-10-11 01:29:04.951000000 nginxconf/drop.conf
      140 2016-10-11 01:29:04.951000000 nginxconf/fastcgi_param_https_map.conf
     1219 2016-10-11 01:29:04.951000000 nginxconf/errorpage.conf
      705 2016-10-11 01:29:04.950000000 nginxconf/cloudflare.conf
     5108 2016-10-11 01:29:04.950000000 nginxconf/botlimit.conf
      564 2016-10-11 01:29:04.948000000 nginxconf/blockbots.conf
     4414 2016-10-11 01:29:04.948000000 nginxconf/block.conf
      361 2016-10-11 01:29:04.947000000 nginxconf/503include-main.conf
     4045 2016-10-11 01:29:19.092000000 nginxconf/htpasswd.py
     2656 2016-10-11 01:29:04.822000000 nginxconf/nginx.conf.default
      636 2016-10-11 01:29:04.818000000 nginxconf/scgi_params.default
      636 2016-10-11 01:29:04.816000000 nginxconf/scgi_params
      664 2016-10-11 01:29:04.812000000 nginxconf/uwsgi_params.default
      664 2016-10-11 01:29:04.810000000 nginxconf/uwsgi_params
     1077 2016-10-11 01:29:04.808000000 nginxconf/fastcgi.conf.default
     1077 2016-10-11 01:29:04.806000000 nginxconf/fastcgi.conf
     1007 2016-10-11 01:29:04.803000000 nginxconf/fastcgi_params.default
     3957 2016-10-11 01:29:04.798000000 nginxconf/mime.types.default
     3957 2016-10-11 01:29:04.797000000 nginxconf/mime.types
     3610 2016-10-11 01:29:04.792000000 nginxconf/win-utf
     2837 2016-10-11 01:29:04.791000000 nginxconf/koi-utf
     2223 2016-10-11 01:29:04.789000000 nginxconf/koi-win
     4601 2016-10-22 12:44:29.288000000 nginxconf/nginx.conf
        0 2016-10-11 01:29:19.088000000 nginxconf/htpasswd
      152 2016-10-11 01:29:04.968000000 nginxconf/dynamic-modules.conf
       46 2016-10-11 01:37:05.709000000 nginxconf/htpasswd_opcache
     1459 2016-10-11 01:37:05.680000000 nginxconf/htpasswd.sh
       37 2016-10-11 01:29:04.947000000 nginxconf/503include-only.conf
     3365 2016-10-11 01:29:19.501000000 nginxconf/php.conf
     1022 2016-10-28 00:25:03.421000000 centminlogs/rclone_copy_281016-002424.log
      684 2016-10-28 00:10:32.031000000 centminlogs/rclone_sync_281016-001026.log
      684 2016-10-28 00:04:58.265000000 centminlogs/rclone_sync_281016-000453.log
      632 2016-10-28 00:04:01.125000000 centminlogs/rclone_copy_281016-000356.log
      632 2016-10-28 00:02:47.437000000 centminlogs/rclone_copy_281016-000242.log
      908 2016-10-27 23:55:46.347000000 centminlogs/rclone_copy_271016-235512.log
     1386 2016-10-27 23:54:25.683000000 centminlogs/rclone_copy_271016-235344.log
      176 2016-10-27 23:52:23.720000000 centminlogs/rclone_copy_271016-235223.log
      176 2016-10-27 23:52:09.040000000 centminlogs/rclone_copy_271016-235209.log
      314 2016-10-27 23:51:02.518000000 centminlogs/rclone_copy_271016-235102.log
      212 2016-10-27 23:49:43.720000000 centminlogs/rclone_copy_271016-234943.log
     1465 2016-10-28 03:30:42.086000000 centminlogs/rclone_config_281016-033020.log
     1613 2016-10-28 03:17:23.476000000 centminlogs/rclone_config_281016-031606.log
      212 2016-10-27 23:48:33.495000000 centminlogs/rclone_copy_271016-234833.log
     1012 2016-10-28 00:46:29.175000000 centminlogs/rclone_copy_281016-004621.log
     2358 2016-10-28 03:57:25.763000000 centminlogs/rclone_config_281016-035634.log
     1423 2016-10-28 03:31:44.448000000 centminlogs/rclone_config_281016-033130.log
      113 2016-10-27 23:45:22.849000000 centminlogs/rclone_copy_271016-234522.log
      113 2016-10-27 23:43:23.547000000 centminlogs/rclone_copy_271016-234323.log
      113 2016-10-27 21:55:54.590000000 centminlogs/rclone_copy_271016-215554.log
     1423 2016-10-28 03:17:40.104000000 centminlogs/rclone_config_281016-031726.log
      774 2016-10-27 21:54:22.852000000 centminlogs/rclone_sync_271016-215417.log
      776 2016-10-27 21:54:11.167000000 centminlogs/rclone_update_271016-215406.log
     2358 2016-10-27 21:28:20.980000000 centminlogs/rclone_installer_271016-212713.log
     6898 2016-10-27 21:26:30.682000000 centminlogs/rclone_installer_271016-212621.log
     6606 2016-10-27 21:25:15.698000000 centminlogs/rclone_installer_271016-212459.log
     6651 2016-10-27 21:23:17.055000000 centminlogs/rclone_installer_271016-212254.log
      251 2016-10-27 20:44:24.195000000 centminlogs/rclone_config_271016-204416.log
      512 2016-10-27 20:44:15.133000000 centminlogs/rclone_config_271016-204359.log
     8074 2016-10-27 20:43:36.860000000 centminlogs/rclone_installer_271016-204143.log
      999 2016-10-24 23:16:24.218000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
      212 2016-10-27 23:45:40.814000000 centminlogs/rclone_copy_271016-234540.log
      212 2016-10-27 23:47:46.601000000 centminlogs/rclone_copy_271016-234746.log
      113 2016-10-27 23:49:40.768000000 centminlogs/rclone_copy_271016-234940.log
     1466 2016-10-28 03:53:27.625000000 centminlogs/rclone_config_281016-035307.log
      113 2016-10-27 21:55:47.576000000 centminlogs/rclone_sync_271016-215547.log
      113 2016-10-27 23:47:43.198000000 centminlogs/rclone_copy_271016-234743.log
      113 2016-10-27 23:48:30.661000000 centminlogs/rclone_copy_271016-234830.log
      293 2016-10-23 14:38:45.978000000 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
  2288914 2016-10-11 01:40:55.584000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
     3005 2016-10-24 23:16:24.218000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
      387 2016-10-11 01:37:05.804000000 centminlogs/zendopcache_passfile.txt
      705 2016-10-24 01:27:50.235000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
      293 2016-10-11 01:40:55.006000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
     9568 2016-10-11 01:31:33.874000000 centminlogs/setio_innodbstatus-after-111016-013111.log
      293 2016-10-24 01:28:13.219000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
       47 2016-10-11 01:40:55.597000000 centminlogs/firstyum_installtime_111016-112102.log
   630333 2016-10-11 01:29:19.522000000 centminlogs/centminmod_ngxinstalltime_111016-112321.log
     1697 2016-10-11 01:27:49.264000000 centminlogs/nginx-configure-111016-112321.log
     1388 2016-10-11 01:27:49.255000000 centminlogs/gcc_native.log
        0 2016-10-11 01:27:33.532000000 centminlogs/patch_patchnginx_111016-112321.log
       45 2016-10-11 01:27:33.516000000 centminlogs/centminmod_libresslinstalltime_111016-112321.log
      275 2016-10-11 01:26:12.476000000 centminlogs/securedtmp.log
   126775 2016-10-11 01:26:11.746000000 centminlogs/centminmod_downloadtimes_111016-112321.log
     2807 2016-10-11 01:26:11.739000000 centminlogs/validcheck_downloads_111016-112321.log
       38 2016-10-11 01:25:28.019000000 centminlogs/centminmod_profiletimes_111016-112321.log
     1701 2016-10-11 01:32:01.379000000 centminlogs/gcc_php_native.log
    17644 2016-10-11 01:24:13.053000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
      783 2016-10-11 01:31:33.841000000 centminlogs/centminmod_setio_111016-013111.log
     9587 2016-10-11 01:31:26.112000000 centminlogs/setio_innodbstatus-before-111016-013111.log
       58 2016-10-11 01:37:05.293000000 centminlogs/centminmod_phpinstalltime_111016-112321.log
     9068 2016-10-11 01:40:50.045000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
      125 2016-10-11 01:40:50.053000000 centminlogs/install.utc.time.log
      293 2016-10-22 12:45:08.063000000 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
    70711 2016-10-11 01:25:28.019000000 centminlogs/centminmod_yumtimes_111016-112321.log
      224 2016-10-11 01:29:04.961000000 nginxconf/ssl/.gitignore
     2052 2016-10-24 23:16:19.809000000 nginxconf/conf.d/newdomain.com.conf
     1701 2016-10-24 23:15:00.781000000 nginxconf/conf.d/virtual.conf
      846 2016-10-11 01:29:04.950000000 nginxconf/conf.d/ssl.conf
     1120 2016-10-11 01:29:04.950000000 nginxconf/conf.d/demodomain.com.conf
     1716 2016-10-11 01:29:04.957000000 nginxconf/phpfpmd/phpfpm_pool4.conf
     1716 2016-10-11 01:29:04.956000000 nginxconf/phpfpmd/phpfpm_pool3.conf
     1716 2016-10-11 01:29:04.956000000 nginxconf/phpfpmd/phpfpm_pool2.conf
     1716 2016-10-11 01:29:04.957000000 nginxconf/phpfpmd/phpfpm_pool5.conf
        0 2016-10-28 00:11:03.267000000 nginxconf/autoprotect/demodomain.com/autoprotect-demodomain.com.conf
      449 2016-10-28 00:11:03.372000000 nginxconf/autoprotect/newdomain.com/autoprotect-newdomain.com.conf
     2054 2016-10-24 23:18:32.570000000 nginxconf/autoprotect/newdomain.com/autoprotect-newdomain.com.conf.orig
2016/10/28 04:07:04
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:               129
Transferred:            0
Elapsed time:        5.4s

eva2000 · Oct 29, 2016

Updated sync and copy options

Updated addons/rclone.sh sync and copy options to better organise remote folders where files and directories are saved to by hostname/copy- or hostname/sync- prefixes.

Example for copy option with Dropbox

Code (Text):

./rclone.sh copy dropbox1
remote = dropbox1

copy /root/centminlogs to cloud storage remote dropbox1
https://community.centminmod.com/posts/39071/

rclone copy /root/centminlogs dropbox1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_281016-230102.log
2016/10/28 23:01:35 Dropbox root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
2016/10/28 23:01:35 Dropbox root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
2016/10/28 23:01:39
Transferred:   3.081 MBytes (84.448 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           67
Elapsed time:       37.3s

copy /usr/local/nginx/conf to cloud storage remote dropbox1

rclone copy /usr/local/nginx/conf dropbox1:centos7.localdomain/copy-nginxconf
2016/10/28 23:02:07 Dropbox root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
2016/10/28 23:02:07 Dropbox root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
2016/10/28 23:02:10
Transferred:   102.562 kBytes (3.329 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           64
Elapsed time:       30.8s


Total Rclone Copy Time: 68.180616339 seconds

Example of copy option for Google Drive

Code (Text):

./rclone.sh copy gdrive1
remote = gdrive1

copy /root/centminlogs to cloud storage remote gdrive1
https://community.centminmod.com/posts/39071/

rclone copy /root/centminlogs gdrive1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_281016-230802.log
2016/10/28 23:08:43 Google drive root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
2016/10/28 23:08:43 Google drive root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
2016/10/28 23:08:47
Transferred:   3.086 MBytes (69.330 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           69
Elapsed time:       45.5s

copy /usr/local/nginx/conf to cloud storage remote gdrive1

rclone copy /usr/local/nginx/conf gdrive1:centos7.localdomain/copy-nginxconf
2016/10/28 23:09:27 Google drive root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
2016/10/28 23:09:27 Google drive root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
2016/10/28 23:09:36
Transferred:   103.986 kBytes (2.163 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           64
Elapsed time:         48s


Total Rclone Copy Time: 93.644793241 seconds

eva2000 · Oct 29, 2016

Rclone OneDrive Config

OneDrive (and Hubic) configuration for Rclone is a bit more complicated as it requires access to web browser on the machine running Rclone as outlined at Remote Setup As CentOS server doesn't have a web browser, you need to download the Windows version of Rclone first from Rclone downloads.
Ensure your local PC has web browser opened and logged into your desired OneDrive account.

Then using Windows command line run the configuration steps outlined at Microsoft One Drive
Then at Windows command line run
Code (Text):
rclone -h
and find the --config line which reports where .rclone.conf config file is located. i.e. C:\Users\USERNAME\.rclone.conf
Copy and paste the contents of C:\Users\USERNAME\.rclone.conf into your existing CentOS server's /root/.rclone.conf file. DO NOT overwrite contents of /root/.rclone.conf file but append to it. Example
Code (Text):
[onedrive1]
type = onedrive
client_id = 
client_secret = 
token = {"access_token":"********Tf!A4v$$","expiry":"2016-10-29T14:08:49.0619662+10:00"}
From windows command prompt. Note name i made a typo ondrive1 should be onedrive1 so can be corrected in copy you append to /root/.rclone.conf instead

listing directories on remote called onedrive1
Code (Text):
rclone lsd onedrive1: 
          -1 2016-10-29 02:48:41         0 Documents
          -1 2016-10-29 02:48:41         0 Pictures
2016/10/29 03:23:23
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.1s
running addons/rclone.sh copy command on remote = onedrive1
Code (Text):
./rclone.sh copy onedrive1
Code (Text):
./rclone.sh copy onedrive1
remote = onedrive1

copy /root/centminlogs to cloud storage remote onedrive1
https://community.centminmod.com/posts/39071/

rclone copy /root/centminlogs onedrive1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_291016-032732.log
2016/10/29 03:28:03 One drive root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
2016/10/29 03:28:03 One drive root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
2016/10/29 03:28:06
Transferred:   3.092 MBytes (93.356 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           76
Elapsed time:       33.9s

copy /usr/local/nginx/conf to cloud storage remote onedrive1

rclone copy /usr/local/nginx/conf onedrive1:centos7.localdomain/copy-nginxconf
2016/10/29 03:28:27 One drive root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
2016/10/29 03:28:27 One drive root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
2016/10/29 03:29:05
Transferred:   102.562 kBytes (1.739 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:           64
Elapsed time:       58.9s


Total Rclone Copy Time: 92.897224058 seconds
listing directories on remote called onedrive1
Code (Text):
rclone lsd onedrive1:
          -1 2016-10-29 03:29:05         2 centos7.localdomain
          -1 2016-10-29 02:48:41         0 Documents
          -1 2016-10-29 02:48:41         0 Pictures
2016/10/29 03:29:30
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:       600ms
Code (Text):
rclone lsd onedrive1:centos7.localdomain
          -1 2016-10-29 03:28:06        76 copy-centminlogs
          -1 2016-10-29 03:29:05        56 copy-nginxconf
2016/10/29 03:30:00
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.5s
Code (Text):
rclone lsd onedrive1:centos7.localdomain/copy-nginxconf
          -1 2016-10-29 03:28:30         2 autoprotect
          -1 2016-10-29 03:28:30         4 conf.d
          -1 2016-10-29 03:28:30         4 phpfpmd
          -1 2016-10-29 03:28:25         1 ssl
2016/10/29 03:35:14
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.7s

eva2000 · Oct 31, 2016

addons/rclone.sh copyssl & syncssl Options Added

Add to addons/rclone.sh copyssl and syncssl options to backup your nginx ssl certificates at /usr/local/nginx/conf/ssl to cloud storage. This could be useful for Centmin Mod's addons/acmetool.sh letsencrypt ssl certificate integration for situations where you run a HTTPS domain across a cluster of multiple web servers and need to distribute the letsencrypt ssl certificate issued to all web servers in the cluster

Example for copyssl

for Dropbox with remote name = dropbox1
Code (Text):
./rclone.sh copyssl dropbox1
output
Code (Text):
./rclone.sh copyssl dropbox1
remote = dropbox1

copy /usr/local/nginx/conf/ssl to cloud storage remote dropbox1
https://community.centminmod.com/posts/39071/

rclone copy /usr/local/nginx/conf/ssl dropbox1:centos7.localdomain/copy-nginxconf-ssl
2016/10/30 20:27:45 Dropbox root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
2016/10/30 20:27:45 Dropbox root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
2016/10/30 20:27:48
Transferred:    224 Bytes (44 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:          5s


Total Rclone Copy SSL Time: 5.051761170 seconds
For Google Drive with remote name gdrive1
Code (Text):
./rclone.sh copyssl gdrive1
output
Code (Text):
./rclone.sh copyssl gdrive1
remote = gdrive1

copy /usr/local/nginx/conf/ssl to cloud storage remote gdrive1
https://community.centminmod.com/posts/39071/

rclone copy /usr/local/nginx/conf/ssl gdrive1:centos7.localdomain/copy-nginxconf-ssl
2016/10/30 20:28:02 Google drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
2016/10/30 20:28:02 Google drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
2016/10/30 20:28:04
Transferred:    224 Bytes (30 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:        7.3s


Total Rclone Copy SSL Time: 7.374108217 seconds
For OneDrive with remote name onedrive1
Code (Text):
./rclone.sh copyssl onedrive1
output
Code (Text):
./rclone.sh copyssl onedrive1
remote = onedrive1

copy /usr/local/nginx/conf/ssl to cloud storage remote onedrive1
https://community.centminmod.com/posts/39071/

rclone copy /usr/local/nginx/conf/ssl onedrive1:centos7.localdomain/copy-nginxconf-ssl
2016/10/30 20:28:14 One drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
2016/10/30 20:28:14 One drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
2016/10/30 20:28:15
Transferred:    224 Bytes (40 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:        5.5s


Total Rclone Copy SSL Time: 5.516254221 seconds
How this can be used is if you want to backup your nginx ssl certificates at /usr/local/nginx/conf/ssl to cloud storage like dropbox, google drive and onedrive you can use addons/rclone.sh outlined at Beta Branch - addons/rclone.sh client for syncing to remote cloud storage providers | Centmin Mod Community and the new copyssl or syncssl options in a cronjob to run

where remote = dropbox1
Code (Text):
13 3 * * * /usr/local/src/centminmod/addons/rclone.sh copyssl dropbox1 >/dev/null 2>&1
Listing remote copy-nginxconf-ssl directory and files in copy-nginxconf-ssl/newdomain2.com on dropbox remote = dropbox1
Code (Text):
rclone lsd dropbox1:centos7.localdomain/copy-nginxconf-ssl
output
Code (Text):
rclone lsd dropbox1:centos7.localdomain/copy-nginxconf-ssl
           0 0001-01-01 00:00:00        -1 newdomain2.com
2016/10/30 22:44:37
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1.1s
Code (Text):
rclone lsl dropbox1:centos7.localdomain/copy-nginxconf-ssl/newdomain2.com
output
Code (Text):
rclone lsl dropbox1:centos7.localdomain/copy-nginxconf-ssl/newdomain2.com
      424 2016-10-30 22:44:05.000000000 dhparam.pem
     1704 2016-10-30 22:44:03.000000000 newdomain2.com.key
       45 2016-10-30 22:44:06.000000000 hpkp-info-primary-pin.txt
     1127 2016-10-30 22:44:03.000000000 newdomain2.com.crt
      964 2016-10-30 22:44:03.000000000 newdomain2.com.csr
     1704 2016-10-30 22:44:03.000000000 newdomain2.com-backup.key
      964 2016-10-30 22:44:04.000000000 newdomain2.com-backup.csr
       45 2016-10-30 22:44:05.000000000 hpkp-info-secondary-pin.txt
2016/10/30 22:46:11
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 8
Transferred:            0
Elapsed time:        1.2s
Creating A Local Sync Copy

Now if you have multiple servers serving the same domain name's SSL certificate i.e. Letsencrypt SSL, you could create a local sync copy and edit your domain's nginx vhost ssl config file paths to point to the local sync copy path and write up a cronjob script to sync from remote to local sync copy directory i.e. /usr/local/nginx/conf/ssldropbox

Create the local sync directory at /usr/local/nginx/conf/ssldropbox
Code (Text):
mkdir -p /usr/local/nginx/conf/ssldropbox
Manual or script rclone sync from remote to local /usr/local/nginx/conf/ssldropbox. Example is syncing from dropbox1 remote and is a dry run first --dry-run
Code (Text):
rclone --dry-run sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
output
Code (Text):
rclone --dry-run sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Not making directory as dry run is set
2016/10/30 22:56:54 newdomain2.com/dhparam.pem: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/hpkp-info-secondary-pin.txt: Not copying as --dry-run
2016/10/30 22:56:54 .gitignore: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/newdomain2.com.csr: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/newdomain2.com.crt: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/newdomain2.com.key: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/newdomain2.com-backup.key: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/newdomain2.com-backup.csr: Not copying as --dry-run
2016/10/30 22:56:54 newdomain2.com/hpkp-info-primary-pin.txt: Not copying as --dry-run
2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for checks to finish
2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for transfers to finish
2016/10/30 22:56:54 Waiting for deletions to finish
2016/10/30 22:56:54
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            9
Elapsed time:        1.1s
Then live run
Code (Text):
rclone sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
output
Code (Text):
rclone sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
2016/10/30 22:57:55 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for checks to finish
2016/10/30 22:57:55 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for transfers to finish
2016/10/30 22:57:56 Waiting for deletions to finish
2016/10/30 22:57:56
Transferred:   7.032 kBytes (1.906 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:            9
Elapsed time:        3.6s
check the local sync copy directory contents
Code (Text):
ls -lah /usr/local/nginx/conf/ssldropbox
total 12K
drwxr-xr-x 3 root root   44 Oct 30 22:57 .
drwxr-xr-x 7 root root 4.0K Oct 30 22:53 ..
-rw-r--r-- 1 root root  224 Oct 30 20:27 .gitignore
drwxr-xr-x 2 root root 4.0K Oct 30 22:57 newdomain2.com
Code (Text):
ls -lah /usr/local/nginx/conf/ssldropbox/newdomain2.com/
total 36K
drwxr-xr-x 2 root root 4.0K Oct 30 22:57 .
drwxr-xr-x 3 root root   44 Oct 30 22:57 ..
-rw-r--r-- 1 root root  424 Oct 30 22:44 dhparam.pem
-rw-r--r-- 1 root root   45 Oct 30 22:44 hpkp-info-primary-pin.txt
-rw-r--r-- 1 root root   45 Oct 30 22:44 hpkp-info-secondary-pin.txt
-rw-r--r-- 1 root root  964 Oct 30 22:44 newdomain2.com-backup.csr
-rw-r--r-- 1 root root 1.7K Oct 30 22:44 newdomain2.com-backup.key
-rw-r--r-- 1 root root 1.2K Oct 30 22:44 newdomain2.com.crt
-rw-r--r-- 1 root root  964 Oct 30 22:44 newdomain2.com.csr
-rw-r--r-- 1 root root 1.7K Oct 30 22:44 newdomain2.com.key
Then modify your domain's ssl vhost paths from /usr/local/nginx/conf/ssl/ to /usr/local/nginx/conf/ssldropbox

Relevant paths displayed via egrep filter
Code (Text):
egrep 'ssl_dh|ssl_cert|ssl_trusted' /usr/local/nginx/conf/conf.d/newdomain2.com.ssl.conf
results in the defaults at
Code (Text):
egrep 'ssl_dh|ssl_cert|ssl_trusted' /usr/local/nginx/conf/conf.d/newdomain2.com.ssl.conf
  ssl_dhparam /usr/local/nginx/conf/ssl/newdomain2.com/dhparam.pem;
  ssl_certificate      /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com.crt;
  ssl_certificate_key  /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com.key;
  #ssl_trusted_certificate /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com-trusted.crt;  
would change to
Code (Text):
  ssl_dhparam /usr/local/nginx/conf/ssldropbox/newdomain2.com/dhparam.pem;
  ssl_certificate      /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com.crt;
  ssl_certificate_key  /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com.key;
  #ssl_trusted_certificate /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com-trusted.crt;
restart nginx
Code (Text):
ngxrestart
Of course only do this for other servers in the domain's cluster and leave at least one of the domain's cluster to use the letsencrypt clients normal verification and issuance i.e. via addons/acmetool.sh underlying acme.sh web root verification when issue/reissue/renewing ssl certificates and on that server run addons/rclone.sh copyssl or syncssl to push to cloudstorage via cronjob. So addons/acmetool.sh will use acme.sh to auto renew letsencrypt ssl certificate via webroot authentication and addons/rclone.sh copyssl or syncssl would push the new letsencrypt ssl certificate to cloud storage provider of your choice - Dropbox, Google Drive, Amazon S3 or OneDrive etc.

Or if you rather all Nginx servers in cluster use letsencrypt DNS verification mode via addons/acmetool.sh you could do that instead. More recent example of addons/acmetool.sh DNS mode posted at Oct 26, 2016

Note, Dropbox and Google Drive have versioning support or if you use OneDrive Business edition there is versioning support so if you needed to revert to a previous copy of ssl certificate files, it's available.

Log in / Register

Forums

Want more timely Centmin Mod News Updates?

Beta Branch addons/rclone.sh client for syncing to remote cloud storage providers

eva2000 Administrator Staff Member

addons/rclone.sh Examples

Rclone Documentation

Rclone Support Forums

Rclone.sh Objective

eva2000 Administrator Staff Member

Rclone Dropbox Configuration

Rclone Commands

eva2000 Administrator Staff Member

Rclone Command & Help Options

eva2000 Administrator Staff Member

Discussion Feedback

eva2000 Administrator Staff Member

addons/rclone.sh copy & sync options

eva2000 Administrator Staff Member

Rclone Google Drive Config

Config

eva2000 Administrator Staff Member

Updated sync and copy options

eva2000 Administrator Staff Member

Rclone OneDrive Config

eva2000 Administrator Staff Member

addons/rclone.sh copyssl & syncssl Options Added

Creating A Local Sync Copy

.

Log in / Register

Useful Searches

Want more timely Centmin Mod News Updates?

Beta Branch addons/rclone.sh client for syncing to remote cloud storage providers

eva2000 Administrator Staff Member

addons/rclone.sh Examples

Rclone Documentation

Rclone Support Forums

Rclone.sh Objective

eva2000 Administrator Staff Member

Rclone Dropbox Configuration

Rclone Commands

eva2000 Administrator Staff Member

Rclone Command & Help Options

eva2000 Administrator Staff Member

Discussion Feedback

eva2000 Administrator Staff Member

addons/rclone.sh copy & sync options

eva2000 Administrator Staff Member

Rclone Google Drive Config

Config

eva2000 Administrator Staff Member

Updated sync and copy options

eva2000 Administrator Staff Member

Rclone OneDrive Config

eva2000 Administrator Staff Member

addons/rclone.sh copyssl & syncssl Options Added

Creating A Local Sync Copy

.