Sysadmin hubiC

So like OneDrive, Google Drive and Dropbox ?

 

k signed up with your referral code and here's mine https://hubic.com/home/new/?referral=RCGJQW

 

Nice S3 sync is good

 

rlone can clone to local filesystem too cool

 

FAQ so only one way syncing not bi-directional

 

yeah might be good for one way syncs basically

 

indeed

Still you'd get duplicates if files have same names

 

playtime Documentation

Code:

rclone help
Sync files and directories to and from local and remote object stores - v1.27.

Syntax: [options] subcommand <parameters> <parameters...>

Subcommands:

    copy source:path dest:path

        Copy the source to the destination.  Doesn't transfer
        unchanged files, testing by size and modification time or
        MD5SUM.  Doesn't delete files from the destination.

    sync source:path dest:path

        Sync the source to the destination, changing the destination
        only.  Doesn't transfer unchanged files, testing by size and
        modification time or MD5SUM.  Destination is updated to match
        source, including deleting files if necessary.  Since this can
        cause data loss, test first with the --dry-run flag.

    move source:path dest:path

        Moves the source to the destination.  This is equivalent to a
        copy followed by a purge, but may use server side operations
        to speed it up. Since this can cause data loss, test first
        with the --dry-run flag.

    ls remote:path

        List all the objects in the the path with size and path.

    lsd remote:path

        List all directories/containers/buckets in the the path.

    lsl remote:path

        List all the objects in the the path with modification time,
        size and path.

    md5sum remote:path

        Produces an md5sum file for all the objects in the path.  This
        is in the same format as the standard md5sum tool produces.

    sha1sum remote:path

        Produces an sha1sum file for all the objects in the path.  This
        is in the same format as the standard sha1sum tool produces.

    size remote:path

        Returns the total size of objects in remote:path and the number
        of objects.

    mkdir remote:path

        Make the path if it doesn't already exist

    rmdir remote:path

        Remove the path.  Note that you can't remove a path with
        objects in it, use purge for that.

    purge remote:path

        Remove the path and all of its contents.  Does not obey
        filters - use remove for that.

    delete remote:path

        Remove the contents of path.  Obeys include/exclude filters.

    check source:path dest:path

        Checks the files in the source and destination match.  It
        compares sizes and MD5SUMs and prints a report of files which
        don't match.  It doesn't alter the source or destination.

    dedupe remote:path

        Interactively find duplicate files and offer to delete all
        but one or rename them to be different. Only useful with
        Google Drive which can have duplicate file names.

    config

        Enter an interactive configuration session.

    authorize

        Remote authorization. Used to authorize a remote or headless
        rclone from a machine with a browser - use as instructed by
        rclone config.

    help

        This help.

Options:
      --acd-templink-threshold value   Files >= this size will be downloaded via their tempLink. (default 9G)
      --bwlimit value                  Bandwidth limit in kBytes/s, or use suffix k|M|G
      --checkers int                   Number of checkers to run in parallel. (default 8)
  -c, --checksum                       Skip based on checksum & size, not mod-time & size
      --config string                  Config file. (default "/root/.rclone.conf")
      --contimeout duration            Connect timeout (default 1m0s)
      --cpuprofile string              Write cpu profile to file
      --delete-after                   When synchronizing, delete files on destination after transfering
      --delete-before                  When synchronizing, delete files on destination before transfering
      --delete-during                  When synchronizing, delete files during transfer (default)
      --delete-excluded                Delete files on dest excluded from sync
      --drive-auth-owner-only          Only consider files owned by the authenticated user. Requires drive-full-list.
      --drive-chunk-size value         Upload chunk size. Must a power of 2 >= 256k. (default 256k)
      --drive-formats string           Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
      --drive-full-list                Use a full listing for directory list. More data but usually quicker. (default true)
      --drive-upload-cutoff value      Cutoff for switching to chunked upload (default 256k)
      --drive-use-trash                Send files to the trash instead of deleting permanently.
      --dropbox-chunk-size value       Upload chunk size. Max 150M. (default 128M)
  -n, --dry-run                        Do a trial run with no permanent changes
      --dump-bodies                    Dump HTTP headers and bodies - may contain sensitive info
      --dump-filters                   Dump the filters to the output
      --dump-headers                   Dump HTTP headers - may contain sensitive info
      --exclude string                 Exclude files matching pattern
      --exclude-from string            Read exclude patterns from file
      --files-from string              Read list of source-file names from file
  -f, --filter string                  Add a file-filtering rule
      --filter-from string             Read filtering patterns from a file
      --ignore-existing                Skip all files that exist on destination
      --include string                 Include files matching pattern
      --include-from string            Read include patterns from file
      --log-file string                Log everything to this file
      --max-age string                 Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
      --max-size value                 Don't transfer any file larger than this in k or suffix k|M|G
      --memprofile string              Write memory profile to file
      --min-age string                 Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
      --min-size value                 Don't transfer any file smaller than this in k or suffix k|M|G
      --modify-window duration         Max time diff to be considered the same (default 1ns)
      --no-check-certificate           Do not verify the server SSL certificate. Insecure.
      --onedrive-chunk-size value      Above this size files will be chunked - must be multiple of 320k. (default 10M)
      --onedrive-upload-cutoff value   Cutoff for switching to chunked upload - must be <= 100MB (default 10M)
  -q, --quiet                          Print as little stuff as possible
      --retries int                    Retry operations this many times if they fail (default 3)
      --size-only                      Skip based on size only, not mod-time or checksum
      --stats duration                 Interval to print stats (0 to disable) (default 1m0s)
      --swift-chunk-size value         Above this size files will be chunked into a _segments container. (default 5G)
      --timeout duration               IO idle timeout (default 5m0s)
      --transfers int                  Number of file transfers to run in parallel. (default 4)
  -v, --verbose                        Print lots more stuff
  -V, --version                        Print the version number

It is only necessary to use a unique prefix of the subcommand, eg 'mo'
for 'move'.

dummy sync dry run

Code:

rclone sync --dry-run /home/rsource/ /home/rdest/
2016/02/04 06:08:22 Local file system at /home/rdest: Building file list
2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for checks to finish
2016/02/04 06:08:22 Waiting for deletions to finish
2016/02/04 06:08:22 1.txt: Not copying as --dry-run
2016/02/04 06:08:22 2.txt: Not copying as --dry-run
2016/02/04 06:08:22 3.txt: Not copying as --dry-run
2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for transfers to finish
2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for deletes to finish (during+after)

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            3
Elapsed time:           0

live sync

Code:

rclone sync /home/rsource/ /home/rdest/ 
2016/02/04 06:09:35 Local file system at /home/rdest: Building file list
2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for checks to finish
2016/02/04 06:09:35 Waiting for deletions to finish
2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for transfers to finish
2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for deletes to finish (during+after)

Transferred:            6 Bytes (   1.76 kByte/s)
Errors:                 0
Checks:                 0
Transferred:            3
Elapsed time:           0

verbose live sync

Code:

rclone sync -v /home/rsource/ /home/rdest/
2016/02/04 06:10:12 Local file system at /home/rdest: Modify window is 1ns
2016/02/04 06:10:12 Local file system at /home/rdest: Building file list
2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for checks to finish
2016/02/04 06:10:12 1.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:10:12 Local file system at /home/rdest: Starting deletion
2016/02/04 06:10:12 1.txt: Unchanged skipping
2016/02/04 06:10:12 2.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:10:12 2.txt: Unchanged skipping
2016/02/04 06:10:12 Waiting for deletions to finish
2016/02/04 06:10:12 3.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:10:12 3.txt: Unchanged skipping
2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for transfers to finish
2016/02/04 06:10:12 Local file system at /home/rdest: Deletion finished
2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for deletes to finish (during+after)

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 3
Transferred:            0
Elapsed time:           0

2016/02/04 06:10:12 Go routines at exit 7

Code:

ls -lah /home/rsource/
total 20K
drwxr-xr-x  2 root root 4.0K Feb  4 06:07 .
drwxr-xr-x. 8 root root 4.0K Feb  4 06:07 ..
-rw-r--r--  1 root root    2 Feb  4 06:07 1.txt
-rw-r--r--  1 root root    2 Feb  4 06:07 2.txt
-rw-r--r--  1 root root    2 Feb  4 06:07 3.txt

ls -lah /home/rdest/
total 20K
drwxr-xr-x  2 root root 4.0K Feb  4 06:09 .
drwxr-xr-x. 8 root root 4.0K Feb  4 06:07 ..
-rw-r--r--  1 root root    2 Feb  4 06:07 1.txt
-rw-r--r--  1 root root    2 Feb  4 06:07 2.txt
-rw-r--r--  1 root root    2 Feb  4 06:07 3.txt

verbose check

Code:

rclone check  -v /home/rsource/ /home/rdest/         
2016/02/04 06:15:41 Local file system at /home/rdest: Modify window is 1ns
2016/02/04 06:15:41 Local file system at /home/rsource: Building file list
2016/02/04 06:15:41 Local file system at /home/rdest: Building file list
2016/02/04 06:15:41 Local file system at /home/rdest: Done building file list
2016/02/04 06:15:41 Local file system at /home/rdest: Done building file list
2016/02/04 06:15:41 Local file system at /home/rdest: 0 files not in Local file system at /home/rsource
2016/02/04 06:15:41 Local file system at /home/rsource: 0 files not in Local file system at /home/rdest
2016/02/04 06:15:41 Local file system at /home/rdest: Waiting for checks to finish
2016/02/04 06:15:41 1.txt: OK
2016/02/04 06:15:41 2.txt: OK
2016/02/04 06:15:41 3.txt: OK
2016/02/04 06:15:41 Local file system at /home/rdest: 0 differences found

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 3
Transferred:            0
Elapsed time:           0

2016/02/04 06:15:41 Go routines at exit 7

liking this migt have found something useful for premium member's cmbackup.sh file + db big brother script to dbbackup.sh MariaDB - dbbackup.sh - quick mysql database backups for Centmin Mod stack | Centmin Mod Community

multi-threaded parallel transfers support too

Code:

      --transfers int                  Number of file transfers to run in parallel. (default 4)

Code:

rclone sync --dry-run -v /home/rsource/ /home/rdest/                                              
2016/02/04 06:19:16 Local file system at /home/rdest: Modify window is 1ns
2016/02/04 06:19:16 Local file system at /home/rdest: Building file list
2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for checks to finish
2016/02/04 06:19:16 Local file system at /home/rdest: Starting deletion
2016/02/04 06:19:16 Waiting for deletions to finish
2016/02/04 06:19:16 4.txt: Not copying as --dry-run
2016/02/04 06:19:16 5.txt: Not copying as --dry-run
2016/02/04 06:19:16 6.txt: Not copying as --dry-run
2016/02/04 06:19:16 7.txt: Not copying as --dry-run
2016/02/04 06:19:16 8.txt: Not copying as --dry-run
2016/02/04 06:19:16 1.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:19:16 1.txt: Unchanged skipping
2016/02/04 06:19:16 2.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:19:16 Local file system at /home/rdest: Deletion finished
2016/02/04 06:19:16 2.txt: Unchanged skipping
2016/02/04 06:19:16 3.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
2016/02/04 06:19:16 3.txt: Unchanged skipping
2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for transfers to finish
2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for deletes to finish (during+after)

Transferred:            0 Bytes (   0.00 kByte/s)
Errors:                 0
Checks:                 3
Transferred:            5
Elapsed time:           0

2016/02/04 06:19:16 Go routines at exit 7

 

hubicfuse GitHub - TurboGit/hubicfuse: Support for mounting HubiC drive in GNU/Linux

 

Ah they have a forum for hubic hubiC Forums !