Want more timely Centmin Mod News Updates?
Become a Member

Beta Branch addons/rclone.sh client for syncing to remote cloud storage providers

Discussion in 'Beta release code' started by eva2000, Oct 28, 2016.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod 123.09beta01 has added a new addons/rclone.sh addon to install and configure rclone rsync client which allows to you to sync and transfer files to several common cloud storage providers. You can discuss and provide beta testing feedback here. If you like and want to support Centmin Mod, consider making a donation or signing up for Centmin Mod Premium Membership. Also if your country supports paypal.me https://www.paypal.me/georgeliu2k ;)

    The first providers I will focus on is Dropbox, Google Drive and OneDrive with example further below in this thread. The aim of addons/rclone.sh is to make it noob friendly for setup from both client and server side and Dropbox, Google Drive and OneDrive are as easy as they come. After that will look at other provider examples.

    Rclone is a command line program to sync files and directories to and from

    addons/rclone.sh Examples


    Rclone Documentation


    Rclone Support Forums


    Rclone.sh Objective



    The addon came about from discussions at https://community.centminmod.com/threads/which-cloud-storage-provider-do-you-use-most.9291/ regarding how I can easily sync /root/centminlogs to cloud storage with easy access client side app to view the logs. These logs are created everytime you run centmin.sh menu and are critical to troubleshooting issues with Centmin Mod so end user access to them is important.
     
    Last edited: Oct 31, 2016
    • Like Like x 2
    • Winner Winner x 1
  2. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Rclone Dropbox Configuration



    You'll need a Dropbox account for this. Best to create a new separate Dropbox account for testing first. You can change/reconfigure the Dropbox account later on. You can discuss and provide beta testing feedback here.

    Steps:
    1. Login into your Dropbox account from your web browser
    2. Update Centmin Mod 123.09beta01 to latest code via centmin.sh menu optio 23 so you get access to addons/rclone.sh
    3. Enable addons/rclone.sh via persistent config file setting RCLONE_ENABLE='y' in /etc/centminmod/custom_config.inc. This is required during beta testing addons/rclone.sh
    4. In SSH, go to /usr/local/src/centminmod/addons directory and run rclone.sh
      Code (Text):
      ./rclone.sh
    Running rclone config
    1. At initial install, you will need to configure rclone to add your remote storage providers info (setup remotes).
    2. For Dropbox the config parameters are outlined at Dropbox
    Command options available to rclone are below. Note copy and sync options don't do anything right now as I need to figure out how to script it to find the remote names from rclone command line.
    Code (Text):
    ./rclone.sh
    ./rclone.sh {config|install|update|copy|sync|copyssl|syncssl}
    

    Example for install and configuration for Dropbox
    Code (Text):
    ./rclone install

    output
    Code (Text):
    ./rclone.sh install   
    ------------------------------------------------
    Install Rclone 64bit
    ------------------------------------------------
    2016-10-27 21:27:18 URL:http://downloads.rclone.org/rclone-current-linux-amd64.zip [3662290/3662290] -> "rclone-current-linux-amd64.zip" [1]
    Archive:  rclone-current-linux-amd64.zip
       creating: rclone-v1.33-linux-amd64/
      inflating: rclone-v1.33-linux-amd64/README.html
      inflating: rclone-v1.33-linux-amd64/rclone
      inflating: rclone-v1.33-linux-amd64/rclone.1
      inflating: rclone-v1.33-linux-amd64/README.txt
    -rwxr-xr-x 1 root root 11M Oct 27 21:27 /usr/sbin/rclone
    
    ------------------------------------------------
    rclone binary at /usr/sbin/rclone
    ------------------------------------------------
    

    Code (Text):
    ------------------------------------------------
    You need to now manually configure your storage provider
    from instructions at http://rclone.org/docs/
    ------------------------------------------------
    
    No remotes found - make a new one
    n) New remote
    s) Set configuration password
    q) Quit config
    n/s/q> n
    name> dropbox1
    Type of storage to configure.
    Choose a number from below, or type in your own value
     1 / Amazon Drive
       \ "amazon cloud drive"
     2 / Amazon S3 (also Dreamhost, Ceph, Minio)
       \ "s3"
     3 / Backblaze B2
       \ "b2"
     4 / Dropbox
       \ "dropbox"
     5 / Encrypt/Decrypt a remote
       \ "crypt"
     6 / Google Cloud Storage (this is not Google Drive)
       \ "google cloud storage"
     7 / Google Drive
       \ "drive"
     8 / Hubic
       \ "hubic"
     9 / Local Disk
       \ "local"
    10 / Microsoft OneDrive
       \ "onedrive"
    11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
       \ "swift"
    12 / Yandex Disk
       \ "yandex"
    Storage> 4
    Dropbox App Key - leave blank normally.
    app_key>
    Dropbox App Secret - leave blank normally.
    app_secret>
    

    You'll be asked to visit a dropbox url in your web browser where you logged into your Dropbox account to authorize and connect to the Dropbox account
    Code (Text):
    Remote config
    Please visit:
    https://www.dropbox.com/1/oauth2/authorize?client_id=5jxadasiasz0rqy&response_type=code
    Enter the code: plahsagxUnxkAAAAAAAAACjrz-kAteycMXctAqF55xnw
    --------------------
    [dropbox1]
    app_key =
    app_secret =
    token = plahsagxUnxkAAAAAAAAAC66j0RyUqrCDWdlCMXWD_pcsasfDlv_z2k3u
    --------------------
    y) Yes this is OK
    e) Edit this remote
    d) Delete this remote
    y/e/d> y
    Current remotes:
    
    Name                 Type
    ====                 ====
    dropbox1             dropbox
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    s) Set configuration password
    q) Quit config
    e/n/d/s/q> q
    
    Total Rclone Install Time: 67.209031379 seconds


    Rclone Commands



    Common commands where remote is name you entered at config time i.e. for example above name = dropbox1
    Code (Text):
    rclone lsd remote:

    list remote directories
    Code (Text):
    rclone lsd dropbox1:
    2016/10/27 21:35:12
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.2s
    list remote files
    [CODEB]
    rclone ls dropbox1:
    2016/10/27 21:36:36
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.4s
    

    Both return empty as my Dropbox account remote is empty right now. So lets add and transfer some files form local server to Dropbox account :)

    Transferring /root/centminlogs log directory to remote:centminlogs i.e. dropbox1:centminlogs
    Code (Text):
    rclone copy /root/centminlogs/ dropbox1:centminlogs
    

    Code (Text):
    rclone copy /root/centminlogs/ dropbox1:centminlogs
    2016/10/27 21:38:29 Dropbox root 'centminlogs': Waiting for checks to finish
    2016/10/27 21:38:29 Dropbox root 'centminlogs': Waiting for transfers to finish
    2016/10/27 21:38:32
    Transferred:   3.061 MBytes (121.723 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           35
    Elapsed time:       25.7s
    

    Now if you list directories should see centminlogs
    Code (Text):
    rclone lsd dropbox1:
               0 0001-01-01 00:00:00        -1 centminlogs
    2016/10/27 21:39:33
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.1s
    

    If you list files
    Code (Text):
    rclone ls dropbox1:
            0 centminlogs/patch_patchnginx_111016-112321.log
           45 centminlogs/centminmod_libresslinstalltime_111016-112321.log
         1697 centminlogs/nginx-configure-111016-112321.log
         1388 centminlogs/gcc_native.log
       126775 centminlogs/centminmod_downloadtimes_111016-112321.log
       630333 centminlogs/centminmod_ngxinstalltime_111016-112321.log
          783 centminlogs/centminmod_setio_111016-013111.log
      2288914 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
           58 centminlogs/centminmod_phpinstalltime_111016-112321.log
         9568 centminlogs/setio_innodbstatus-after-111016-013111.log
        70711 centminlogs/centminmod_yumtimes_111016-112321.log
        17644 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
         2807 centminlogs/validcheck_downloads_111016-112321.log
           38 centminlogs/centminmod_profiletimes_111016-112321.log
          275 centminlogs/securedtmp.log
         9587 centminlogs/setio_innodbstatus-before-111016-013111.log
          293 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
          705 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
         3005 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
          999 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
          293 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
         8074 centminlogs/rclone_installer_271016-204143.log
          512 centminlogs/rclone_config_271016-204359.log
          251 centminlogs/rclone_config_271016-204416.log
         6651 centminlogs/rclone_installer_271016-212254.log
         6606 centminlogs/rclone_installer_271016-212459.log
          125 centminlogs/install.utc.time.log
         2358 centminlogs/rclone_installer_271016-212713.log
         6898 centminlogs/rclone_installer_271016-212621.log
          293 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
           47 centminlogs/firstyum_installtime_111016-112102.log
          293 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
          387 centminlogs/zendopcache_passfile.txt
         1701 centminlogs/gcc_php_native.log
         9068 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
    2016/10/27 21:40:18
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.4s
    

    list files with time and size info
    Code (Text):
    rclone lsl dropbox1:
            0 2016-10-27 21:38:16.000000000 centminlogs/patch_patchnginx_111016-112321.log
         1697 2016-10-27 21:38:18.000000000 centminlogs/nginx-configure-111016-112321.log
         1388 2016-10-27 21:38:20.000000000 centminlogs/gcc_native.log
         9587 2016-10-27 21:38:21.000000000 centminlogs/setio_innodbstatus-before-111016-013111.log
       630333 2016-10-27 21:38:21.000000000 centminlogs/centminmod_ngxinstalltime_111016-112321.log
          783 2016-10-27 21:38:21.000000000 centminlogs/centminmod_setio_111016-013111.log
      2288914 2016-10-27 21:38:22.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
           58 2016-10-27 21:38:22.000000000 centminlogs/centminmod_phpinstalltime_111016-112321.log
         9568 2016-10-27 21:38:23.000000000 centminlogs/setio_innodbstatus-after-111016-013111.log
         1701 2016-10-27 21:38:23.000000000 centminlogs/gcc_php_native.log
          387 2016-10-27 21:38:24.000000000 centminlogs/zendopcache_passfile.txt
         9068 2016-10-27 21:38:25.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
          125 2016-10-27 21:38:25.000000000 centminlogs/install.utc.time.log
          293 2016-10-27 21:38:25.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
           47 2016-10-27 21:38:26.000000000 centminlogs/firstyum_installtime_111016-112102.log
          293 2016-10-27 21:38:26.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
          293 2016-10-27 21:38:27.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
          705 2016-10-27 21:38:28.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
           45 2016-10-27 21:38:16.000000000 centminlogs/centminmod_libresslinstalltime_111016-112321.log
          999 2016-10-27 21:38:29.000000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
          293 2016-10-27 21:38:29.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
         8074 2016-10-27 21:38:29.000000000 centminlogs/rclone_installer_271016-204143.log
          512 2016-10-27 21:38:30.000000000 centminlogs/rclone_config_271016-204359.log
          251 2016-10-27 21:38:30.000000000 centminlogs/rclone_config_271016-204416.log
         6651 2016-10-27 21:38:30.000000000 centminlogs/rclone_installer_271016-212254.log
         6606 2016-10-27 21:38:31.000000000 centminlogs/rclone_installer_271016-212459.log
         6898 2016-10-27 21:38:31.000000000 centminlogs/rclone_installer_271016-212621.log
         2358 2016-10-27 21:38:32.000000000 centminlogs/rclone_installer_271016-212713.log
        70711 2016-10-27 21:38:12.000000000 centminlogs/centminmod_yumtimes_111016-112321.log
        17644 2016-10-27 21:38:12.000000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
       126775 2016-10-27 21:38:12.000000000 centminlogs/centminmod_downloadtimes_111016-112321.log
         3005 2016-10-27 21:38:28.000000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
         2807 2016-10-27 21:38:14.000000000 centminlogs/validcheck_downloads_111016-112321.log
           38 2016-10-27 21:38:13.000000000 centminlogs/centminmod_profiletimes_111016-112321.log
          275 2016-10-27 21:38:15.000000000 centminlogs/securedtmp.log
    2016/10/27 21:51:07
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                35
    Transferred:            0
    Elapsed time:        1.2s
    


    upload_2016-10-28_7-40-54.png

    upload_2016-10-28_7-41-42.png

    Now if you try to copy again, and no files have changed since, then you will see 0 Bytes transferred just like rsync would do :)
    Code (Text):
    rclone copy /root/centminlogs/ dropbox1:centminlogs
    2016/10/27 21:46:01 Dropbox root 'centminlogs': Waiting for checks to finish
    2016/10/27 21:46:01 Dropbox root 'centminlogs': Waiting for transfers to finish
    2016/10/27 21:46:01
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                35
    Transferred:            0
    Elapsed time:        2.8s
    

    Deleting files in remote = dropbox1:centminlogs
    Code (Text):
    rclone delete dropbox1:
    2016/10/27 22:04:08 Waiting for deletions to finish
    2016/10/27 22:04:15
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                35
    Transferred:            0
    Elapsed time:        7.9s
     
    Last edited: Oct 31, 2016
    • Winner Winner x 2
  3. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Rclone Command & Help Options



    Rclone commands docs at Commands

    The main rclone commands with most used first
    • rclone config - Enter an interactive configuration session.
    • rclone copy - Copy files from source to dest, skipping already copied
    • rclone sync - Make source and dest identical, modifying destination only.
    • rclone move - Move files from source to dest.
    • rclone delete - Remove the contents of path.
    • rclone purge - Remove the path and all of its contents.
    • rclone mkdir - Make the path if it doesn’t already exist.
    • rclone rmdir - Remove the path.
    • rclone check - Checks the files in the source and destination match.
    • rclone ls - List all the objects in the the path with size and path.
    • rclone lsd - List all directories/containers/buckets in the the path.
    • rclone lsl - List all the objects path with modification time, size and path.
    • rclone md5sum - Produces an md5sum file for all the objects in the path.
    • rclone sha1sum - Produces an sha1sum file for all the objects in the path.
    • rclone size - Returns the total size and number of objects in remote:path.
    • rclone version - Show the version number.
    • rclone cleanup - Clean up the remote if possible
    • rclone dedupe - Interactively find duplicate files delete/rename them.
    See the commands index for the full list.

    Help info
    Code (Text):
    rclone -h
    Rclone is a command line program to sync files and directories to and
    from various cloud storage systems, such as:
    
      * Google Drive
      * Amazon S3
      * Openstack Swift / Rackspace cloud files / Memset Memstore
      * Dropbox
      * Google Cloud Storage
      * Amazon Drive
      * Microsoft One Drive
      * Hubic
      * Backblaze B2
      * Yandex Disk
      * The local filesystem
    
    Features
    
      * MD5/SHA1 hashes checked at all times for file integrity
      * Timestamps preserved on files
      * Partial syncs supported on a whole file basis
      * Copy mode to just copy new/changed files
      * Sync (one way) mode to make a directory identical
      * Check mode to check for file hash equality
      * Can sync to and from network, eg two different cloud accounts
    

    Code (Text):
    See the home page for installation, usage, documentation, changelog
    and configuration walkthroughs.
    
      * http://rclone.org/
    
    Usage:
      rclone [flags]
      rclone [command]
    
    Available Commands:
      authorize       Remote authorization.
      cat             Concatenates any files and sends them to stdout.
      check           Checks the files in the source and destination match.
      cleanup         Clean up the remote if possible
      config          Enter an interactive configuration session.
      copy            Copy files from source to dest, skipping already copied
      dedupe          Interactively find duplicate files delete/rename them.
      delete          Remove the contents of path.
      genautocomplete Output bash completion script for rclone.
      gendocs         Output markdown docs for rclone to the directory supplied.
      ls              List all the objects in the the path with size and path.
      lsd             List all directories/containers/buckets in the the path.
      lsl             List all the objects path with modification time, size and path.
      md5sum          Produces an md5sum file for all the objects in the path.
      mkdir           Make the path if it doesn't already exist.
      mount           Mount the remote as a mountpoint. **EXPERIMENTAL**
      move            Move files from source to dest.
      purge           Remove the path and all of its contents.
      rmdir           Remove the path if empty.
      sha1sum         Produces an sha1sum file for all the objects in the path.
      size            Prints the total size and number of objects in remote:path.
      sync            Make source and dest identical, modifying destination only.
      version         Show the version number.
    

    Code (Text):
    Flags:
          --acd-templink-threshold int      Files >= this size will be downloaded via their tempLink.
          --acd-upload-wait-time duration   Time to wait after a failed complete upload to see if it appears. (default 2m0s)
          --ask-password                    Allow prompt for password for encrypted configuration. (default true)
          --b2-chunk-size int               Upload chunk size. Must fit in memory.
          --b2-test-mode string             A flag string for X-Bz-Test-Mode header.
          --b2-upload-cutoff int            Cutoff for switching to chunked upload
          --b2-versions                     Include old versions in directory listings.
          --bwlimit int                     Bandwidth limit in kBytes/s, or use suffix b|k|M|G
          --checkers int                    Number of checkers to run in parallel. (default 8)
      -c, --checksum                        Skip based on checksum & size, not mod-time & size
          --config string                   Config file. (default "/root/.rclone.conf")
          --contimeout duration             Connect timeout (default 1m0s)
          --cpuprofile string               Write cpu profile to file
          --delete-after                    When synchronizing, delete files on destination after transfering
          --delete-before                   When synchronizing, delete files on destination before transfering
          --delete-during                   When synchronizing, delete files during transfer (default)
          --delete-excluded                 Delete files on dest excluded from sync
          --drive-auth-owner-only           Only consider files owned by the authenticated user. Requires drive-full-list.
          --drive-chunk-size int            Upload chunk size. Must a power of 2 >= 256k.
          --drive-formats string            Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
          --drive-full-list                 Use a full listing for directory list. More data but usually quicker. (obsolete)
          --drive-upload-cutoff int         Cutoff for switching to chunked upload
          --drive-use-trash                 Send files to the trash instead of deleting permanently.
          --dropbox-chunk-size int          Upload chunk size. Max 150M.
      -n, --dry-run                         Do a trial run with no permanent changes
          --dump-bodies                     Dump HTTP headers and bodies - may contain sensitive info
          --dump-filters                    Dump the filters to the output
          --dump-headers                    Dump HTTP headers - may contain sensitive info
          --exclude string                  Exclude files matching pattern
          --exclude-from string             Read exclude patterns from file
          --files-from string               Read list of source-file names from file
      -f, --filter string                   Add a file-filtering rule
          --filter-from string              Read filtering patterns from a file
          --ignore-existing                 Skip all files that exist on destination
          --ignore-size                     Ignore size when skipping use mod-time or checksum.
      -I, --ignore-times                    Don't skip files that match size and time - transfer all files
          --include string                  Include files matching pattern
          --include-from string             Read include patterns from file
          --log-file string                 Log everything to this file
          --low-level-retries int           Number of low level retries to do. (default 10)
          --max-age string                  Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
          --max-depth int                   If set limits the recursion depth to this. (default -1)
          --max-size int                    Don't transfer any file larger than this in k or suffix b|k|M|G
          --memprofile string               Write memory profile to file
          --min-age string                  Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
          --min-size int                    Don't transfer any file smaller than this in k or suffix b|k|M|G
          --modify-window duration          Max time diff to be considered the same (default 1ns)
          --no-check-certificate            Do not verify the server SSL certificate. Insecure.
          --no-gzip-encoding                Don't set Accept-Encoding: gzip.
          --no-traverse                     Don't traverse destination file system on copy.
          --no-update-modtime               Don't update destination mod-time if files identical.
          --onedrive-chunk-size int         Above this size files will be chunked - must be multiple of 320k.
          --onedrive-upload-cutoff int      Cutoff for switching to chunked upload - must be <= 100MB
      -q, --quiet                           Print as little stuff as possible
          --retries int                     Retry operations this many times if they fail (default 3)
          --size-only                       Skip based on size only, not mod-time or checksum
          --stats duration                  Interval to print stats (0 to disable) (default 1m0s)
          --swift-chunk-size int            Above this size files will be chunked into a _segments container.
          --timeout duration                IO idle timeout (default 5m0s)
          --transfers int                   Number of file transfers to run in parallel. (default 4)
      -u, --update                          Skip files that are newer on the destination.
      -v, --verbose                         Print lots more stuff
      -V, --version                         Print the version number
    
    Use "rclone [command] --help" for more information about a command.
     
    Last edited: Oct 28, 2016
    • Informative Informative x 1
  4. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Discussion Feedback



    You can discuss and provide beta testing feedback here.
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    addons/rclone.sh copy & sync options



    The copy & sync options allow you to pass the remote name you setup via the command line to copy or sync /root/centminlogs to the remote cloud storage provider. Using above Dropbox remote = dropbox1 as an example. Update: add /usr/local/nginx/conf too.

    Copy is one way direction local to remote copy. Sync is making the remote same as local copy.

    copy
    Code (Text):
    ./rclone.sh copy dropbox1
    

    sync
    Code (Text):
    ./rclone.sh sync dropbox1
    

    Example output

    copy
    Code (Text):
    ./rclone.sh copy dropbox1       
    remote = dropbox1
    
    copy /root/centminlogs to cloud storage remote dropbox1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /root/centminlogs dropbox1:centminlogs
    rclone copy /root/centminlogs dropbox1:centminlogs --exclude rclone_copy_281016-000242.log
    2016/10/28 00:02:44 Dropbox root 'centminlogs': Waiting for checks to finish
    2016/10/28 00:02:44 Dropbox root 'centminlogs': Waiting for transfers to finish
    2016/10/28 00:02:47
    Transferred:    908 Bytes (177 Bytes/s)
    Errors:                 0
    Checks:                53
    Transferred:            1
    Elapsed time:        5.1s
    
    
    Total Rclone Copy Time: 5.121360189 seconds

    sync
    Code (Text):
    ./rclone.sh sync dropbox1
    remote = dropbox1
    
    sync /root/centminlogs to cloud storage remote dropbox1
    https://community.centminmod.com/posts/39071/
    
    rclone sync /root/centminlogs dropbox1:centminlogs
    rclone sync /root/centminlogs dropbox1:centminlogs --exclude rclone_sync_281016-000453.log
    2016/10/28 00:04:55 Dropbox root 'centminlogs': Waiting for checks to finish
    2016/10/28 00:04:55 Dropbox root 'centminlogs': Waiting for transfers to finish
    2016/10/28 00:04:58 Waiting for deletions to finish
    2016/10/28 00:04:58
    Transferred:    632 Bytes (124 Bytes/s)
    Errors:                 0
    Checks:                54
    Transferred:            1
    Elapsed time:          5s
    
    
    Total Rclone Sync Time: 5.089820731 seconds
     
    Last edited: Oct 28, 2016
    • Informative Informative x 1
  6. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Rclone Google Drive Config



    As per Google Drive setup guide at Google drive you should setup your own client_id for Google Drive for best performance.

    Here is how to create your own Google Drive client ID for rclone:
    1. Log into the Google API Console with your Google account. It doesn’t matter what Google account you use. (It need not be the same account as the Google Drive you want to access)
    2. Select a project or create a new project.
    3. Under Overview, Google APIs, Google Apps APIs, click “Drive API”, then “Enable”.
    4. Click “Credentials” in the left-side panel (not “Go to credentials”, which opens the wizard), then “Create credentials”, then “OAuth client ID”. It will prompt you to set the OAuth consent screen product name, if you haven’t set one already.
    5. Choose an application type of “other”, and click “Create”. (the default name is fine)
    6. It will show you a client ID and client secret. Use these values in rclone config to add a new remote or edit an existing remote.
    Once you have your clientid and secret, fill them in when prompted for Google Application Client Id and Google Application Client Secret

    Config



    This will involve you being logged into your Google Drive account on your desktop PC and copying and pasting a url generated from Rclone config
    Code (Text):
    ./rclone.sh config
    

    Run addons/rclone.sh config and setup Google Drive remote called = gdrive1
    Code (Text):
    ./rclone.sh config
    
    ------------------------------------------------
    You need to now manually configure your storage provider
    from instructions at http://rclone.org/docs/
    ------------------------------------------------
    
    
    Current remotes:
    
    Name                 Type
    ====                 ====
    dropbox1             dropbox
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    s) Set configuration password
    q) Quit config
    e/n/d/s/q> n
    name> gdrive1
    Type of storage to configure.
    Choose a number from below, or type in your own value
     1 / Amazon Drive
       \ "amazon cloud drive"
     2 / Amazon S3 (also Dreamhost, Ceph, Minio)
       \ "s3"
     3 / Backblaze B2
       \ "b2"
     4 / Dropbox
       \ "dropbox"
     5 / Encrypt/Decrypt a remote
       \ "crypt"
     6 / Google Cloud Storage (this is not Google Drive)
       \ "google cloud storage"
     7 / Google Drive
       \ "drive"
     8 / Hubic
       \ "hubic"
     9 / Local Disk
       \ "local"
    10 / Microsoft OneDrive
       \ "onedrive"
    11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
       \ "swift"
    12 / Yandex Disk
       \ "yandex"
    Storage> 7
    Google Application Client Id - leave blank normally.
    client_id>
    Google Application Client Secret - leave blank normally.
    client_secret>
    

    Select N for headless machine and copy and paste the google account link into your web browser which you already logged into your Google Drive account for. You will get a verification code you need to enter to continue
    Code (Text):
    Remote config
    Use auto config?
     * Say Y if not sure
     * Say N if you are working on a remote or headless machine or Y didn't work
    y) Yes
    n) No
    y/n> n
    If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?client_id=22***644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=ad58XS******4e
    Log in and authorize rclone for access
    Enter verification code> 4/3_1cAAAANYw4VTYMSQM4J_uw
    

    Enter y if correct
    Code (Text):
    --------------------
    [gdrive1]
    client_id =
    client_secret =
    token = {"access_token":"ya29.Ci-KAyoBd6*****_sssssXZYUy60CjELsqggu9mpklKX_FlXA","token_type":"Bearer","refresh_token":"1/uMDepQ******8nuYM","expiry":"2016-10-28T04:57:20.202908281Z"}
    --------------------
    y) Yes this is OK
    e) Edit this remote
    d) Delete this remote
    y/e/d> y
    

    Listing of current remotes
    Code (Text):
    Current remotes:
    
    Name                 Type
    ====                 ====
    dropbox1             dropbox
    gdrive1              drive
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    s) Set configuration password
    q) Quit config
    e/n/d/s/q> q
    
    Total Rclone Config Time: 51.692002451 seconds
    

    check directories in remote = gdrive1
    Code (Text):
    rclone lsd gdrive1:
    2016/10/28 04:03:04
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.1s
    

    test addons/rclone.sh copy command to gdrive1
    Code (Text):
    ./rclone.sh copy gdrive1
    

    Code (Text):
    ./rclone.sh copy gdrive1
    remote = gdrive1
    
    copy /root/centminlogs to cloud storage remote gdrive1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /root/centminlogs gdrive1:centminlogs --exclude rclone_copy_281016-040452.log
    2016/10/28 04:05:39 Google drive root 'centminlogs': Waiting for checks to finish
    2016/10/28 04:05:39 Google drive root 'centminlogs': Waiting for transfers to finish
    2016/10/28 04:05:45
    Transferred:   3.090 MBytes (60.126 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           65
    Elapsed time:       52.6s
    
    copy /usr/local/nginx/conf to cloud storage remote gdrive1
    
    rclone copy /usr/local/nginx/conf gdrive1:nginxconf
    2016/10/28 04:06:28 Google drive root 'nginxconf': Waiting for checks to finish
    2016/10/28 04:06:28 Google drive root 'nginxconf': Waiting for transfers to finish
    2016/10/28 04:06:38
    Transferred:   104.285 kBytes (1.963 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           64
    Elapsed time:       53.1s
    
    
    Total Rclone Copy Time: 105.761092037 seconds
    

    then check contents on remote = gdrive1
    Code (Text):
    rclone lsl gdrive1:
    

    Code (Text):
    rclone lsl gdrive1:
          212 2016-10-11 01:37:05.723000000 nginxconf/include_opcache.conf
          292 2016-10-11 01:29:04.963000000 nginxconf/vts_mainserver.conf
           24 2016-10-11 01:29:04.963000000 nginxconf/vts_server.conf
          888 2016-10-11 01:29:04.963000000 nginxconf/wpffpc.conf
           26 2016-10-11 01:29:04.963000000 nginxconf/vts_http.conf
         2236 2016-10-11 01:29:04.963000000 nginxconf/staticfiles.conf
           10 2016-10-11 01:29:04.961000000 nginxconf/sitestatus.conf
          110 2016-10-11 01:29:04.961000000 nginxconf/ssl-include.conf
         2460 2016-10-11 01:29:04.960000000 nginxconf/phpssl.conf
          884 2016-10-11 01:29:04.960000000 nginxconf/proxy.conf
          362 2016-10-11 01:29:04.960000000 nginxconf/phpstatus.conf
         2441 2016-10-11 01:29:04.960000000 nginxconf/pscontrol
         3371 2016-10-11 01:29:04.960000000 nginxconf/php-pool5.conf
         3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool4.conf
         3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool3.conf
         3371 2016-10-11 01:29:04.958000000 nginxconf/php-pool2.conf
         2682 2016-10-11 01:29:04.955000000 nginxconf/phpalt.conf
          668 2016-10-11 01:29:04.955000000 nginxconf/pagespeedhandler.conf
         1740 2016-10-11 01:29:04.954000000 nginxconf/pagespeedadmin.conf
         7152 2016-10-11 01:29:04.954000000 nginxconf/pagespeed.conf
          645 2016-10-11 01:29:04.953000000 nginxconf/maintenance.conf
          283 2016-10-11 01:29:04.952000000 nginxconf/geoip.conf
         1271 2016-10-11 01:29:04.952000000 nginxconf/fastcgi_params
          626 2016-10-11 01:29:04.951000000 nginxconf/drop.conf
          140 2016-10-11 01:29:04.951000000 nginxconf/fastcgi_param_https_map.conf
         1219 2016-10-11 01:29:04.951000000 nginxconf/errorpage.conf
          705 2016-10-11 01:29:04.950000000 nginxconf/cloudflare.conf
         5108 2016-10-11 01:29:04.950000000 nginxconf/botlimit.conf
          564 2016-10-11 01:29:04.948000000 nginxconf/blockbots.conf
         4414 2016-10-11 01:29:04.948000000 nginxconf/block.conf
          361 2016-10-11 01:29:04.947000000 nginxconf/503include-main.conf
         4045 2016-10-11 01:29:19.092000000 nginxconf/htpasswd.py
         2656 2016-10-11 01:29:04.822000000 nginxconf/nginx.conf.default
          636 2016-10-11 01:29:04.818000000 nginxconf/scgi_params.default
          636 2016-10-11 01:29:04.816000000 nginxconf/scgi_params
          664 2016-10-11 01:29:04.812000000 nginxconf/uwsgi_params.default
          664 2016-10-11 01:29:04.810000000 nginxconf/uwsgi_params
         1077 2016-10-11 01:29:04.808000000 nginxconf/fastcgi.conf.default
         1077 2016-10-11 01:29:04.806000000 nginxconf/fastcgi.conf
         1007 2016-10-11 01:29:04.803000000 nginxconf/fastcgi_params.default
         3957 2016-10-11 01:29:04.798000000 nginxconf/mime.types.default
         3957 2016-10-11 01:29:04.797000000 nginxconf/mime.types
         3610 2016-10-11 01:29:04.792000000 nginxconf/win-utf
         2837 2016-10-11 01:29:04.791000000 nginxconf/koi-utf
         2223 2016-10-11 01:29:04.789000000 nginxconf/koi-win
         4601 2016-10-22 12:44:29.288000000 nginxconf/nginx.conf
            0 2016-10-11 01:29:19.088000000 nginxconf/htpasswd
          152 2016-10-11 01:29:04.968000000 nginxconf/dynamic-modules.conf
           46 2016-10-11 01:37:05.709000000 nginxconf/htpasswd_opcache
         1459 2016-10-11 01:37:05.680000000 nginxconf/htpasswd.sh
           37 2016-10-11 01:29:04.947000000 nginxconf/503include-only.conf
         3365 2016-10-11 01:29:19.501000000 nginxconf/php.conf
         1022 2016-10-28 00:25:03.421000000 centminlogs/rclone_copy_281016-002424.log
          684 2016-10-28 00:10:32.031000000 centminlogs/rclone_sync_281016-001026.log
          684 2016-10-28 00:04:58.265000000 centminlogs/rclone_sync_281016-000453.log
          632 2016-10-28 00:04:01.125000000 centminlogs/rclone_copy_281016-000356.log
          632 2016-10-28 00:02:47.437000000 centminlogs/rclone_copy_281016-000242.log
          908 2016-10-27 23:55:46.347000000 centminlogs/rclone_copy_271016-235512.log
         1386 2016-10-27 23:54:25.683000000 centminlogs/rclone_copy_271016-235344.log
          176 2016-10-27 23:52:23.720000000 centminlogs/rclone_copy_271016-235223.log
          176 2016-10-27 23:52:09.040000000 centminlogs/rclone_copy_271016-235209.log
          314 2016-10-27 23:51:02.518000000 centminlogs/rclone_copy_271016-235102.log
          212 2016-10-27 23:49:43.720000000 centminlogs/rclone_copy_271016-234943.log
         1465 2016-10-28 03:30:42.086000000 centminlogs/rclone_config_281016-033020.log
         1613 2016-10-28 03:17:23.476000000 centminlogs/rclone_config_281016-031606.log
          212 2016-10-27 23:48:33.495000000 centminlogs/rclone_copy_271016-234833.log
         1012 2016-10-28 00:46:29.175000000 centminlogs/rclone_copy_281016-004621.log
         2358 2016-10-28 03:57:25.763000000 centminlogs/rclone_config_281016-035634.log
         1423 2016-10-28 03:31:44.448000000 centminlogs/rclone_config_281016-033130.log
          113 2016-10-27 23:45:22.849000000 centminlogs/rclone_copy_271016-234522.log
          113 2016-10-27 23:43:23.547000000 centminlogs/rclone_copy_271016-234323.log
          113 2016-10-27 21:55:54.590000000 centminlogs/rclone_copy_271016-215554.log
         1423 2016-10-28 03:17:40.104000000 centminlogs/rclone_config_281016-031726.log
          774 2016-10-27 21:54:22.852000000 centminlogs/rclone_sync_271016-215417.log
          776 2016-10-27 21:54:11.167000000 centminlogs/rclone_update_271016-215406.log
         2358 2016-10-27 21:28:20.980000000 centminlogs/rclone_installer_271016-212713.log
         6898 2016-10-27 21:26:30.682000000 centminlogs/rclone_installer_271016-212621.log
         6606 2016-10-27 21:25:15.698000000 centminlogs/rclone_installer_271016-212459.log
         6651 2016-10-27 21:23:17.055000000 centminlogs/rclone_installer_271016-212254.log
          251 2016-10-27 20:44:24.195000000 centminlogs/rclone_config_271016-204416.log
          512 2016-10-27 20:44:15.133000000 centminlogs/rclone_config_271016-204359.log
         8074 2016-10-27 20:43:36.860000000 centminlogs/rclone_installer_271016-204143.log
          999 2016-10-24 23:16:24.218000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv-remove-cmds-newdomain.com.log
          212 2016-10-27 23:45:40.814000000 centminlogs/rclone_copy_271016-234540.log
          212 2016-10-27 23:47:46.601000000 centminlogs/rclone_copy_271016-234746.log
          113 2016-10-27 23:49:40.768000000 centminlogs/rclone_copy_271016-234940.log
         1466 2016-10-28 03:53:27.625000000 centminlogs/rclone_config_281016-035307.log
          113 2016-10-27 21:55:47.576000000 centminlogs/rclone_sync_271016-215547.log
          113 2016-10-27 23:47:43.198000000 centminlogs/rclone_copy_271016-234743.log
          113 2016-10-27 23:48:30.661000000 centminlogs/rclone_copy_271016-234830.log
          293 2016-10-23 14:38:45.978000000 centminlogs/centminmod_1.2.3-eva2000.09.001_231016-143844_git_updatecur_branch.log
      2288914 2016-10-11 01:40:55.584000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_install.log
         3005 2016-10-24 23:16:24.218000000 centminlogs/centminmod_241016-231614_nginx_addvhost_nv.log
          387 2016-10-11 01:37:05.804000000 centminlogs/zendopcache_passfile.txt
          705 2016-10-24 01:27:50.235000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012745_nginx_addvhost.log
          293 2016-10-11 01:40:55.006000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-014052_git_updatecur_branch.log
         9568 2016-10-11 01:31:33.874000000 centminlogs/setio_innodbstatus-after-111016-013111.log
          293 2016-10-24 01:28:13.219000000 centminlogs/centminmod_1.2.3-eva2000.09.001_241016-012811_git_updatecur_branch.log
           47 2016-10-11 01:40:55.597000000 centminlogs/firstyum_installtime_111016-112102.log
       630333 2016-10-11 01:29:19.522000000 centminlogs/centminmod_ngxinstalltime_111016-112321.log
         1697 2016-10-11 01:27:49.264000000 centminlogs/nginx-configure-111016-112321.log
         1388 2016-10-11 01:27:49.255000000 centminlogs/gcc_native.log
            0 2016-10-11 01:27:33.532000000 centminlogs/patch_patchnginx_111016-112321.log
           45 2016-10-11 01:27:33.516000000 centminlogs/centminmod_libresslinstalltime_111016-112321.log
          275 2016-10-11 01:26:12.476000000 centminlogs/securedtmp.log
       126775 2016-10-11 01:26:11.746000000 centminlogs/centminmod_downloadtimes_111016-112321.log
         2807 2016-10-11 01:26:11.739000000 centminlogs/validcheck_downloads_111016-112321.log
           38 2016-10-11 01:25:28.019000000 centminlogs/centminmod_profiletimes_111016-112321.log
         1701 2016-10-11 01:32:01.379000000 centminlogs/gcc_php_native.log
        17644 2016-10-11 01:24:13.053000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yuminstall_centos7.log
          783 2016-10-11 01:31:33.841000000 centminlogs/centminmod_setio_111016-013111.log
         9587 2016-10-11 01:31:26.112000000 centminlogs/setio_innodbstatus-before-111016-013111.log
           58 2016-10-11 01:37:05.293000000 centminlogs/centminmod_phpinstalltime_111016-112321.log
         9068 2016-10-11 01:40:50.045000000 centminlogs/centminmod_1.2.3-eva2000.09.001_111016-112321_yum-log.log
          125 2016-10-11 01:40:50.053000000 centminlogs/install.utc.time.log
          293 2016-10-22 12:45:08.063000000 centminlogs/centminmod_1.2.3-eva2000.09.001_221016-124505_git_updatecur_branch.log
        70711 2016-10-11 01:25:28.019000000 centminlogs/centminmod_yumtimes_111016-112321.log
          224 2016-10-11 01:29:04.961000000 nginxconf/ssl/.gitignore
         2052 2016-10-24 23:16:19.809000000 nginxconf/conf.d/newdomain.com.conf
         1701 2016-10-24 23:15:00.781000000 nginxconf/conf.d/virtual.conf
          846 2016-10-11 01:29:04.950000000 nginxconf/conf.d/ssl.conf
         1120 2016-10-11 01:29:04.950000000 nginxconf/conf.d/demodomain.com.conf
         1716 2016-10-11 01:29:04.957000000 nginxconf/phpfpmd/phpfpm_pool4.conf
         1716 2016-10-11 01:29:04.956000000 nginxconf/phpfpmd/phpfpm_pool3.conf
         1716 2016-10-11 01:29:04.956000000 nginxconf/phpfpmd/phpfpm_pool2.conf
         1716 2016-10-11 01:29:04.957000000 nginxconf/phpfpmd/phpfpm_pool5.conf
            0 2016-10-28 00:11:03.267000000 nginxconf/autoprotect/demodomain.com/autoprotect-demodomain.com.conf
          449 2016-10-28 00:11:03.372000000 nginxconf/autoprotect/newdomain.com/autoprotect-newdomain.com.conf
         2054 2016-10-24 23:18:32.570000000 nginxconf/autoprotect/newdomain.com/autoprotect-newdomain.com.conf.orig
    2016/10/28 04:07:04
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:               129
    Transferred:            0
    Elapsed time:        5.4s
    


    upload_2016-10-28_14-7-45.png
     
    Last edited: Oct 29, 2016
    • Winner Winner x 1
  7. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Updated sync and copy options



    Updated addons/rclone.sh sync and copy options to better organise remote folders where files and directories are saved to by hostname/copy- or hostname/sync- prefixes.

    Example for copy option with Dropbox
    Code (Text):
    ./rclone.sh copy dropbox1
    remote = dropbox1
    
    copy /root/centminlogs to cloud storage remote dropbox1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /root/centminlogs dropbox1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_281016-230102.log
    2016/10/28 23:01:35 Dropbox root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
    2016/10/28 23:01:35 Dropbox root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
    2016/10/28 23:01:39
    Transferred:   3.081 MBytes (84.448 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           67
    Elapsed time:       37.3s
    
    copy /usr/local/nginx/conf to cloud storage remote dropbox1
    
    rclone copy /usr/local/nginx/conf dropbox1:centos7.localdomain/copy-nginxconf
    2016/10/28 23:02:07 Dropbox root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
    2016/10/28 23:02:07 Dropbox root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
    2016/10/28 23:02:10
    Transferred:   102.562 kBytes (3.329 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           64
    Elapsed time:       30.8s
    
    
    Total Rclone Copy Time: 68.180616339 seconds


    upload_2016-10-29_9-7-25.png

    upload_2016-10-29_9-7-47.png

    Example of copy option for Google Drive

    Code (Text):
    ./rclone.sh copy gdrive1
    remote = gdrive1
    
    copy /root/centminlogs to cloud storage remote gdrive1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /root/centminlogs gdrive1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_281016-230802.log
    2016/10/28 23:08:43 Google drive root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
    2016/10/28 23:08:43 Google drive root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
    2016/10/28 23:08:47
    Transferred:   3.086 MBytes (69.330 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           69
    Elapsed time:       45.5s
    
    copy /usr/local/nginx/conf to cloud storage remote gdrive1
    
    rclone copy /usr/local/nginx/conf gdrive1:centos7.localdomain/copy-nginxconf
    2016/10/28 23:09:27 Google drive root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
    2016/10/28 23:09:27 Google drive root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
    2016/10/28 23:09:36
    Transferred:   103.986 kBytes (2.163 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           64
    Elapsed time:         48s
    
    
    Total Rclone Copy Time: 93.644793241 seconds
    


    upload_2016-10-29_9-9-48.png

    upload_2016-10-29_9-12-6.png
     
    Last edited: Oct 29, 2016
  8. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    Rclone OneDrive Config



    OneDrive (and Hubic) configuration for Rclone is a bit more complicated as it requires access to web browser on the machine running Rclone as outlined at Remote Setup As CentOS server doesn't have a web browser, you need to download the Windows version of Rclone first from Rclone downloads.
    • Ensure your local PC has web browser opened and logged into your desired OneDrive account.
    • Then using Windows command line run the configuration steps outlined at Microsoft One Drive
    • Then at Windows command line run
      Code (Text):
      rclone -h
      and find the --config line which reports where .rclone.conf config file is located. i.e. C:\Users\USERNAME\.rclone.conf
    • Copy and paste the contents of C:\Users\USERNAME\.rclone.conf into your existing CentOS server's /root/.rclone.conf file. DO NOT overwrite contents of /root/.rclone.conf file but append to it. Example
      Code (Text):
      [onedrive1]
      type = onedrive
      client_id = 
      client_secret = 
      token = {"access_token":"********Tf!A4v$$","expiry":"2016-10-29T14:08:49.0619662+10:00"}
      
    From windows command prompt. Note name i made a typo ondrive1 should be onedrive1 so can be corrected in copy you append to /root/.rclone.conf instead

    rclone-onedrive1.png

    listing directories on remote called onedrive1
    Code (Text):
    rclone lsd onedrive1: 
              -1 2016-10-29 02:48:41         0 Documents
              -1 2016-10-29 02:48:41         0 Pictures
    2016/10/29 03:23:23
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.1s

    running addons/rclone.sh copy command on remote = onedrive1
    Code (Text):
    ./rclone.sh copy onedrive1
    

    Code (Text):
    ./rclone.sh copy onedrive1
    remote = onedrive1
    
    copy /root/centminlogs to cloud storage remote onedrive1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /root/centminlogs onedrive1:centos7.localdomain/copy-centminlogs --exclude rclone_copy_291016-032732.log
    2016/10/29 03:28:03 One drive root 'centos7.localdomain/copy-centminlogs': Waiting for checks to finish
    2016/10/29 03:28:03 One drive root 'centos7.localdomain/copy-centminlogs': Waiting for transfers to finish
    2016/10/29 03:28:06
    Transferred:   3.092 MBytes (93.356 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           76
    Elapsed time:       33.9s
    
    copy /usr/local/nginx/conf to cloud storage remote onedrive1
    
    rclone copy /usr/local/nginx/conf onedrive1:centos7.localdomain/copy-nginxconf
    2016/10/29 03:28:27 One drive root 'centos7.localdomain/copy-nginxconf': Waiting for checks to finish
    2016/10/29 03:28:27 One drive root 'centos7.localdomain/copy-nginxconf': Waiting for transfers to finish
    2016/10/29 03:29:05
    Transferred:   102.562 kBytes (1.739 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:           64
    Elapsed time:       58.9s
    
    
    Total Rclone Copy Time: 92.897224058 seconds
    

    listing directories on remote called onedrive1
    Code (Text):
    rclone lsd onedrive1:
              -1 2016-10-29 03:29:05         2 centos7.localdomain
              -1 2016-10-29 02:48:41         0 Documents
              -1 2016-10-29 02:48:41         0 Pictures
    2016/10/29 03:29:30
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:       600ms
    

    Code (Text):
    rclone lsd onedrive1:centos7.localdomain
              -1 2016-10-29 03:28:06        76 copy-centminlogs
              -1 2016-10-29 03:29:05        56 copy-nginxconf
    2016/10/29 03:30:00
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.5s
    

    Code (Text):
    rclone lsd onedrive1:centos7.localdomain/copy-nginxconf
              -1 2016-10-29 03:28:30         2 autoprotect
              -1 2016-10-29 03:28:30         4 conf.d
              -1 2016-10-29 03:28:30         4 phpfpmd
              -1 2016-10-29 03:28:25         1 ssl
    2016/10/29 03:35:14
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.7s
    


    onedrive1.png
     
    Last edited: Oct 31, 2016
    • Informative Informative x 1
  9. eva2000

    eva2000 Administrator Staff Member

    29,720
    6,711
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,015
    Local Time:
    8:00 PM
    Nginx 1.13.x
    MariaDB 5.5

    addons/rclone.sh copyssl & syncssl Options Added



    Add to addons/rclone.sh copyssl and syncssl options to backup your nginx ssl certificates at /usr/local/nginx/conf/ssl to cloud storage. This could be useful for Centmin Mod's addons/acmetool.sh letsencrypt ssl certificate integration for situations where you run a HTTPS domain across a cluster of multiple web servers and need to distribute the letsencrypt ssl certificate issued to all web servers in the cluster :)

    Example for copyssl

    for Dropbox with remote name = dropbox1
    Code (Text):
    ./rclone.sh copyssl dropbox1
    

    output
    Code (Text):
    ./rclone.sh copyssl dropbox1
    remote = dropbox1
    
    copy /usr/local/nginx/conf/ssl to cloud storage remote dropbox1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /usr/local/nginx/conf/ssl dropbox1:centos7.localdomain/copy-nginxconf-ssl
    2016/10/30 20:27:45 Dropbox root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
    2016/10/30 20:27:45 Dropbox root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
    2016/10/30 20:27:48
    Transferred:    224 Bytes (44 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            1
    Elapsed time:          5s
    
    
    Total Rclone Copy SSL Time: 5.051761170 seconds
    

    For Google Drive with remote name gdrive1
    Code (Text):
    ./rclone.sh copyssl gdrive1
    

    output
    Code (Text):
    ./rclone.sh copyssl gdrive1
    remote = gdrive1
    
    copy /usr/local/nginx/conf/ssl to cloud storage remote gdrive1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /usr/local/nginx/conf/ssl gdrive1:centos7.localdomain/copy-nginxconf-ssl
    2016/10/30 20:28:02 Google drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
    2016/10/30 20:28:02 Google drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
    2016/10/30 20:28:04
    Transferred:    224 Bytes (30 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            1
    Elapsed time:        7.3s
    
    
    Total Rclone Copy SSL Time: 7.374108217 seconds
    

    For OneDrive with remote name onedrive1
    Code (Text):
    ./rclone.sh copyssl onedrive1
    

    output
    Code (Text):
    ./rclone.sh copyssl onedrive1
    remote = onedrive1
    
    copy /usr/local/nginx/conf/ssl to cloud storage remote onedrive1
    https://community.centminmod.com/posts/39071/
    
    rclone copy /usr/local/nginx/conf/ssl onedrive1:centos7.localdomain/copy-nginxconf-ssl
    2016/10/30 20:28:14 One drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for checks to finish
    2016/10/30 20:28:14 One drive root 'centos7.localdomain/copy-nginxconf-ssl': Waiting for transfers to finish
    2016/10/30 20:28:15
    Transferred:    224 Bytes (40 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            1
    Elapsed time:        5.5s
    
    
    Total Rclone Copy SSL Time: 5.516254221 seconds


    upload_2016-10-31_6-34-11.png

    upload_2016-10-31_6-37-27.png

    upload_2016-10-31_6-36-10.png

    How this can be used is if you want to backup your nginx ssl certificates at /usr/local/nginx/conf/ssl to cloud storage like dropbox, google drive and onedrive you can use addons/rclone.sh outlined at Beta Branch - addons/rclone.sh client for syncing to remote cloud storage providers | Centmin Mod Community and the new copyssl or syncssl options in a cronjob to run

    where remote = dropbox1
    Code (Text):
    13 3 * * * /usr/local/src/centminmod/addons/rclone.sh copyssl dropbox1 >/dev/null 2>&1


    Listing remote copy-nginxconf-ssl directory and files in copy-nginxconf-ssl/newdomain2.com on dropbox remote = dropbox1
    Code (Text):
    rclone lsd dropbox1:centos7.localdomain/copy-nginxconf-ssl

    output
    Code (Text):
    rclone lsd dropbox1:centos7.localdomain/copy-nginxconf-ssl
               0 0001-01-01 00:00:00        -1 newdomain2.com
    2016/10/30 22:44:37
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            0
    Elapsed time:        1.1s

    Code (Text):
    rclone lsl dropbox1:centos7.localdomain/copy-nginxconf-ssl/newdomain2.com
    

    output
    Code (Text):
    rclone lsl dropbox1:centos7.localdomain/copy-nginxconf-ssl/newdomain2.com
          424 2016-10-30 22:44:05.000000000 dhparam.pem
         1704 2016-10-30 22:44:03.000000000 newdomain2.com.key
           45 2016-10-30 22:44:06.000000000 hpkp-info-primary-pin.txt
         1127 2016-10-30 22:44:03.000000000 newdomain2.com.crt
          964 2016-10-30 22:44:03.000000000 newdomain2.com.csr
         1704 2016-10-30 22:44:03.000000000 newdomain2.com-backup.key
          964 2016-10-30 22:44:04.000000000 newdomain2.com-backup.csr
           45 2016-10-30 22:44:05.000000000 hpkp-info-secondary-pin.txt
    2016/10/30 22:46:11
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 8
    Transferred:            0
    Elapsed time:        1.2s
    


    upload_2016-10-31_8-48-0.png

    upload_2016-10-31_8-48-23.png

    Creating A Local Sync Copy



    Now if you have multiple servers serving the same domain name's SSL certificate i.e. Letsencrypt SSL, you could create a local sync copy and edit your domain's nginx vhost ssl config file paths to point to the local sync copy path and write up a cronjob script to sync from remote to local sync copy directory i.e. /usr/local/nginx/conf/ssldropbox

    Create the local sync directory at /usr/local/nginx/conf/ssldropbox
    Code (Text):
    mkdir -p /usr/local/nginx/conf/ssldropbox

    Manual or script rclone sync from remote to local /usr/local/nginx/conf/ssldropbox. Example is syncing from dropbox1 remote and is a dry run first --dry-run
    Code (Text):
    rclone --dry-run sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
    

    output
    Code (Text):
    rclone --dry-run sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
    2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Not making directory as dry run is set
    2016/10/30 22:56:54 newdomain2.com/dhparam.pem: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/hpkp-info-secondary-pin.txt: Not copying as --dry-run
    2016/10/30 22:56:54 .gitignore: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/newdomain2.com.csr: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/newdomain2.com.crt: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/newdomain2.com.key: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/newdomain2.com-backup.key: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/newdomain2.com-backup.csr: Not copying as --dry-run
    2016/10/30 22:56:54 newdomain2.com/hpkp-info-primary-pin.txt: Not copying as --dry-run
    2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for checks to finish
    2016/10/30 22:56:54 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for transfers to finish
    2016/10/30 22:56:54 Waiting for deletions to finish
    2016/10/30 22:56:54
    Transferred:      0 Bytes (0 Bytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            9
    Elapsed time:        1.1s
    

    Then live run
    Code (Text):
    rclone sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
    

    output
    Code (Text):
    rclone sync dropbox1:centos7.localdomain/copy-nginxconf-ssl /usr/local/nginx/conf/ssldropbox
    2016/10/30 22:57:55 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for checks to finish
    2016/10/30 22:57:55 Local file system at /usr/local/nginx/conf/ssldropbox: Waiting for transfers to finish
    2016/10/30 22:57:56 Waiting for deletions to finish
    2016/10/30 22:57:56
    Transferred:   7.032 kBytes (1.906 kBytes/s)
    Errors:                 0
    Checks:                 0
    Transferred:            9
    Elapsed time:        3.6s
    

    check the local sync copy directory contents
    Code (Text):
    ls -lah /usr/local/nginx/conf/ssldropbox
    total 12K
    drwxr-xr-x 3 root root   44 Oct 30 22:57 .
    drwxr-xr-x 7 root root 4.0K Oct 30 22:53 ..
    -rw-r--r-- 1 root root  224 Oct 30 20:27 .gitignore
    drwxr-xr-x 2 root root 4.0K Oct 30 22:57 newdomain2.com
    

    Code (Text):
    ls -lah /usr/local/nginx/conf/ssldropbox/newdomain2.com/
    total 36K
    drwxr-xr-x 2 root root 4.0K Oct 30 22:57 .
    drwxr-xr-x 3 root root   44 Oct 30 22:57 ..
    -rw-r--r-- 1 root root  424 Oct 30 22:44 dhparam.pem
    -rw-r--r-- 1 root root   45 Oct 30 22:44 hpkp-info-primary-pin.txt
    -rw-r--r-- 1 root root   45 Oct 30 22:44 hpkp-info-secondary-pin.txt
    -rw-r--r-- 1 root root  964 Oct 30 22:44 newdomain2.com-backup.csr
    -rw-r--r-- 1 root root 1.7K Oct 30 22:44 newdomain2.com-backup.key
    -rw-r--r-- 1 root root 1.2K Oct 30 22:44 newdomain2.com.crt
    -rw-r--r-- 1 root root  964 Oct 30 22:44 newdomain2.com.csr
    -rw-r--r-- 1 root root 1.7K Oct 30 22:44 newdomain2.com.key
    

    Then modify your domain's ssl vhost paths from /usr/local/nginx/conf/ssl/ to /usr/local/nginx/conf/ssldropbox

    Relevant paths displayed via egrep filter
    Code (Text):
    egrep 'ssl_dh|ssl_cert|ssl_trusted' /usr/local/nginx/conf/conf.d/newdomain2.com.ssl.conf
    

    results in the defaults at
    Code (Text):
    egrep 'ssl_dh|ssl_cert|ssl_trusted' /usr/local/nginx/conf/conf.d/newdomain2.com.ssl.conf
      ssl_dhparam /usr/local/nginx/conf/ssl/newdomain2.com/dhparam.pem;
      ssl_certificate      /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com.crt;
      ssl_certificate_key  /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com.key;
      #ssl_trusted_certificate /usr/local/nginx/conf/ssl/newdomain2.com/newdomain2.com-trusted.crt;  

    would change to
    Code (Text):
      ssl_dhparam /usr/local/nginx/conf/ssldropbox/newdomain2.com/dhparam.pem;
      ssl_certificate      /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com.crt;
      ssl_certificate_key  /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com.key;
      #ssl_trusted_certificate /usr/local/nginx/conf/ssldropbox/newdomain2.com/newdomain2.com-trusted.crt;
    

    restart nginx
    Code (Text):
    ngxrestart
    

    Of course only do this for other servers in the domain's cluster and leave at least one of the domain's cluster to use the letsencrypt clients normal verification and issuance i.e. via addons/acmetool.sh underlying acme.sh web root verification when issue/reissue/renewing ssl certificates and on that server run addons/rclone.sh copyssl or syncssl to push to cloudstorage via cronjob. So addons/acmetool.sh will use acme.sh to auto renew letsencrypt ssl certificate via webroot authentication and addons/rclone.sh copyssl or syncssl would push the new letsencrypt ssl certificate to cloud storage provider of your choice - Dropbox, Google Drive, Amazon S3 or OneDrive etc.

    Or if you rather all Nginx servers in cluster use letsencrypt DNS verification mode via addons/acmetool.sh you could do that instead. More recent example of addons/acmetool.sh DNS mode posted at Oct 26, 2016

    Note, Dropbox and Google Drive have versioning support or if you use OneDrive Business edition there is versioning support so if you needed to revert to a previous copy of ssl certificate files, it's available.
     
    Last edited: Oct 31, 2016
Thread Status:
Not open for further replies.