Want to subscribe to topics you're interested in?
Become a Member

Sysadmin hubiC

Discussion in 'System Administration' started by dorobo, Feb 4, 2016.

Tags:
  1. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    Anyone else using hubiC from ovh?

    They offer FREE 25GB space. 30GB if you sign up using my referral link :D

    Too bad I only have 25GB because I didn't know there was a referral system and I just kinda jumped into it.

    I'm using it to store backups.

    I use borg to backup files into another VPS and then on that VPS I use rclone to sync that backup to hubiC and also rclone it to Google Drive.

    I also use borg check to double check the integrity of the backups.

    borg --> VPS backup --> rclone --> hubiC
     
    Last edited: Feb 4, 2016
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    So like OneDrive, Google Drive and Dropbox ?
     
  3. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    Yep but 30GB.

    You can also upgrade if you want

    The hubiC cloud storage offers – hubiC.com

    100GB is just 1 euro per month or 10 euros per year

    10TB is just 5 euros per month or 50 euros per year

    dang that's cheap as far as storage is concerned isn't it?
     
    • Informative Informative x 1
  4. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    k signed up with your referral code and here's mine https://hubic.com/home/new/?referral=RCGJQW :D

     
  5. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    thanks. also check out rclone - ncw/rclone · GitHub

    if you need to sync files into hubic. It also supports other cloud storages:

    • Google Drive
    • Amazon S3
    • Openstack Swift / Rackspace cloud files / Memset Memstore
    • Dropbox
    • Google Cloud Storage
    • Amazon Cloud Drive
    • Microsoft One Drive
    • Hubic
    • Backblaze B2
    • Yandex Disk
    • The local filesystem
     
    • Informative Informative x 1
  6. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    Nice S3 sync is good :)

    upload_2016-2-4_5-20-40.png
     
  7. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    I've tried rcloning to Google Drive, Dropbox, Google Cloud Platform, Hubic.

    rclone is awesome.

    And my hubic is now 30GB too :D

    Thanks George.
     
    • Like Like x 1
  8. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    rlone can clone to local filesystem too cool :)
     
  9. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    FAQ so only one way syncing not bi-directional
     
  10. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    I've tried it but it can A > B then B > A

    rclone sync /backup remote:borg

    and then

    rclone sync remote:borg /test

    and then I will do borg check /test to check the integrity
     
    • Informative Informative x 1
  11. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    Oh I see. Maybe it can't do

    rclone sync remote:borg /backup (sync back to the source)

    but you can just sync back to a new folder.
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    yeah might be good for one way syncs basically :)
     
  13. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    But maybe it can do copy bi-directionally

    rclone copy

    I will try it.
     
  14. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    ahh right. if you want bi-directional use copy.

    From FAQ
     
    • Informative Informative x 1
  15. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    indeed
    Still you'd get duplicates if files have same names
     
    • Informative Informative x 1
  16. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    playtime Documentation :)

    Code:
    rclone help
    Sync files and directories to and from local and remote object stores - v1.27.
    
    Syntax: [options] subcommand <parameters> <parameters...>
    
    Subcommands:
    
        copy source:path dest:path
    
            Copy the source to the destination.  Doesn't transfer
            unchanged files, testing by size and modification time or
            MD5SUM.  Doesn't delete files from the destination.
    
        sync source:path dest:path
    
            Sync the source to the destination, changing the destination
            only.  Doesn't transfer unchanged files, testing by size and
            modification time or MD5SUM.  Destination is updated to match
            source, including deleting files if necessary.  Since this can
            cause data loss, test first with the --dry-run flag.
    
        move source:path dest:path
    
            Moves the source to the destination.  This is equivalent to a
            copy followed by a purge, but may use server side operations
            to speed it up. Since this can cause data loss, test first
            with the --dry-run flag.
    
        ls remote:path
    
            List all the objects in the the path with size and path.
    
        lsd remote:path
    
            List all directories/containers/buckets in the the path.
    
        lsl remote:path
    
            List all the objects in the the path with modification time,
            size and path.
    
        md5sum remote:path
    
            Produces an md5sum file for all the objects in the path.  This
            is in the same format as the standard md5sum tool produces.
    
        sha1sum remote:path
    
            Produces an sha1sum file for all the objects in the path.  This
            is in the same format as the standard sha1sum tool produces.
    
        size remote:path
    
            Returns the total size of objects in remote:path and the number
            of objects.
    
        mkdir remote:path
    
            Make the path if it doesn't already exist
    
        rmdir remote:path
    
            Remove the path.  Note that you can't remove a path with
            objects in it, use purge for that.
    
        purge remote:path
    
            Remove the path and all of its contents.  Does not obey
            filters - use remove for that.
    
        delete remote:path
    
            Remove the contents of path.  Obeys include/exclude filters.
    
        check source:path dest:path
    
            Checks the files in the source and destination match.  It
            compares sizes and MD5SUMs and prints a report of files which
            don't match.  It doesn't alter the source or destination.
    
        dedupe remote:path
    
            Interactively find duplicate files and offer to delete all
            but one or rename them to be different. Only useful with
            Google Drive which can have duplicate file names.
    
        config
    
            Enter an interactive configuration session.
    
        authorize
    
            Remote authorization. Used to authorize a remote or headless
            rclone from a machine with a browser - use as instructed by
            rclone config.
    
        help
    
            This help.
    
    Options:
          --acd-templink-threshold value   Files >= this size will be downloaded via their tempLink. (default 9G)
          --bwlimit value                  Bandwidth limit in kBytes/s, or use suffix k|M|G
          --checkers int                   Number of checkers to run in parallel. (default 8)
      -c, --checksum                       Skip based on checksum & size, not mod-time & size
          --config string                  Config file. (default "/root/.rclone.conf")
          --contimeout duration            Connect timeout (default 1m0s)
          --cpuprofile string              Write cpu profile to file
          --delete-after                   When synchronizing, delete files on destination after transfering
          --delete-before                  When synchronizing, delete files on destination before transfering
          --delete-during                  When synchronizing, delete files during transfer (default)
          --delete-excluded                Delete files on dest excluded from sync
          --drive-auth-owner-only          Only consider files owned by the authenticated user. Requires drive-full-list.
          --drive-chunk-size value         Upload chunk size. Must a power of 2 >= 256k. (default 256k)
          --drive-formats string           Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
          --drive-full-list                Use a full listing for directory list. More data but usually quicker. (default true)
          --drive-upload-cutoff value      Cutoff for switching to chunked upload (default 256k)
          --drive-use-trash                Send files to the trash instead of deleting permanently.
          --dropbox-chunk-size value       Upload chunk size. Max 150M. (default 128M)
      -n, --dry-run                        Do a trial run with no permanent changes
          --dump-bodies                    Dump HTTP headers and bodies - may contain sensitive info
          --dump-filters                   Dump the filters to the output
          --dump-headers                   Dump HTTP headers - may contain sensitive info
          --exclude string                 Exclude files matching pattern
          --exclude-from string            Read exclude patterns from file
          --files-from string              Read list of source-file names from file
      -f, --filter string                  Add a file-filtering rule
          --filter-from string             Read filtering patterns from a file
          --ignore-existing                Skip all files that exist on destination
          --include string                 Include files matching pattern
          --include-from string            Read include patterns from file
          --log-file string                Log everything to this file
          --max-age string                 Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
          --max-size value                 Don't transfer any file larger than this in k or suffix k|M|G
          --memprofile string              Write memory profile to file
          --min-age string                 Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
          --min-size value                 Don't transfer any file smaller than this in k or suffix k|M|G
          --modify-window duration         Max time diff to be considered the same (default 1ns)
          --no-check-certificate           Do not verify the server SSL certificate. Insecure.
          --onedrive-chunk-size value      Above this size files will be chunked - must be multiple of 320k. (default 10M)
          --onedrive-upload-cutoff value   Cutoff for switching to chunked upload - must be <= 100MB (default 10M)
      -q, --quiet                          Print as little stuff as possible
          --retries int                    Retry operations this many times if they fail (default 3)
          --size-only                      Skip based on size only, not mod-time or checksum
          --stats duration                 Interval to print stats (0 to disable) (default 1m0s)
          --swift-chunk-size value         Above this size files will be chunked into a _segments container. (default 5G)
          --timeout duration               IO idle timeout (default 5m0s)
          --transfers int                  Number of file transfers to run in parallel. (default 4)
      -v, --verbose                        Print lots more stuff
      -V, --version                        Print the version number
    
    It is only necessary to use a unique prefix of the subcommand, eg 'mo'
    for 'move'.
    dummy sync dry run
    Code:
    rclone sync --dry-run /home/rsource/ /home/rdest/
    2016/02/04 06:08:22 Local file system at /home/rdest: Building file list
    2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for checks to finish
    2016/02/04 06:08:22 Waiting for deletions to finish
    2016/02/04 06:08:22 1.txt: Not copying as --dry-run
    2016/02/04 06:08:22 2.txt: Not copying as --dry-run
    2016/02/04 06:08:22 3.txt: Not copying as --dry-run
    2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for transfers to finish
    2016/02/04 06:08:22 Local file system at /home/rdest: Waiting for deletes to finish (during+after)
    
    Transferred:            0 Bytes (   0.00 kByte/s)
    Errors:                 0
    Checks:                 0
    Transferred:            3
    Elapsed time:           0
    live sync
    Code:
    rclone sync /home/rsource/ /home/rdest/ 
    2016/02/04 06:09:35 Local file system at /home/rdest: Building file list
    2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for checks to finish
    2016/02/04 06:09:35 Waiting for deletions to finish
    2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for transfers to finish
    2016/02/04 06:09:35 Local file system at /home/rdest: Waiting for deletes to finish (during+after)
    
    Transferred:            6 Bytes (   1.76 kByte/s)
    Errors:                 0
    Checks:                 0
    Transferred:            3
    Elapsed time:           0
    verbose live sync
    Code:
    rclone sync -v /home/rsource/ /home/rdest/
    2016/02/04 06:10:12 Local file system at /home/rdest: Modify window is 1ns
    2016/02/04 06:10:12 Local file system at /home/rdest: Building file list
    2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for checks to finish
    2016/02/04 06:10:12 1.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:10:12 Local file system at /home/rdest: Starting deletion
    2016/02/04 06:10:12 1.txt: Unchanged skipping
    2016/02/04 06:10:12 2.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:10:12 2.txt: Unchanged skipping
    2016/02/04 06:10:12 Waiting for deletions to finish
    2016/02/04 06:10:12 3.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:10:12 3.txt: Unchanged skipping
    2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for transfers to finish
    2016/02/04 06:10:12 Local file system at /home/rdest: Deletion finished
    2016/02/04 06:10:12 Local file system at /home/rdest: Waiting for deletes to finish (during+after)
    
    Transferred:            0 Bytes (   0.00 kByte/s)
    Errors:                 0
    Checks:                 3
    Transferred:            0
    Elapsed time:           0
    
    2016/02/04 06:10:12 Go routines at exit 7
    Code:
    ls -lah /home/rsource/
    total 20K
    drwxr-xr-x  2 root root 4.0K Feb  4 06:07 .
    drwxr-xr-x. 8 root root 4.0K Feb  4 06:07 ..
    -rw-r--r--  1 root root    2 Feb  4 06:07 1.txt
    -rw-r--r--  1 root root    2 Feb  4 06:07 2.txt
    -rw-r--r--  1 root root    2 Feb  4 06:07 3.txt
    
    ls -lah /home/rdest/
    total 20K
    drwxr-xr-x  2 root root 4.0K Feb  4 06:09 .
    drwxr-xr-x. 8 root root 4.0K Feb  4 06:07 ..
    -rw-r--r--  1 root root    2 Feb  4 06:07 1.txt
    -rw-r--r--  1 root root    2 Feb  4 06:07 2.txt
    -rw-r--r--  1 root root    2 Feb  4 06:07 3.txt
    verbose check
    Code:
    rclone check  -v /home/rsource/ /home/rdest/         
    2016/02/04 06:15:41 Local file system at /home/rdest: Modify window is 1ns
    2016/02/04 06:15:41 Local file system at /home/rsource: Building file list
    2016/02/04 06:15:41 Local file system at /home/rdest: Building file list
    2016/02/04 06:15:41 Local file system at /home/rdest: Done building file list
    2016/02/04 06:15:41 Local file system at /home/rdest: Done building file list
    2016/02/04 06:15:41 Local file system at /home/rdest: 0 files not in Local file system at /home/rsource
    2016/02/04 06:15:41 Local file system at /home/rsource: 0 files not in Local file system at /home/rdest
    2016/02/04 06:15:41 Local file system at /home/rdest: Waiting for checks to finish
    2016/02/04 06:15:41 1.txt: OK
    2016/02/04 06:15:41 2.txt: OK
    2016/02/04 06:15:41 3.txt: OK
    2016/02/04 06:15:41 Local file system at /home/rdest: 0 differences found
    
    Transferred:            0 Bytes (   0.00 kByte/s)
    Errors:                 0
    Checks:                 3
    Transferred:            0
    Elapsed time:           0
    
    2016/02/04 06:15:41 Go routines at exit 7
    liking this migt have found something useful for premium member's cmbackup.sh file + db big brother script to dbbackup.sh MariaDB - dbbackup.sh - quick mysql database backups for Centmin Mod stack | Centmin Mod Community :)

    multi-threaded parallel transfers support too :cool:
    Code:
          --transfers int                  Number of file transfers to run in parallel. (default 4)
    Code:
    rclone sync --dry-run -v /home/rsource/ /home/rdest/                                              
    2016/02/04 06:19:16 Local file system at /home/rdest: Modify window is 1ns
    2016/02/04 06:19:16 Local file system at /home/rdest: Building file list
    2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for checks to finish
    2016/02/04 06:19:16 Local file system at /home/rdest: Starting deletion
    2016/02/04 06:19:16 Waiting for deletions to finish
    2016/02/04 06:19:16 4.txt: Not copying as --dry-run
    2016/02/04 06:19:16 5.txt: Not copying as --dry-run
    2016/02/04 06:19:16 6.txt: Not copying as --dry-run
    2016/02/04 06:19:16 7.txt: Not copying as --dry-run
    2016/02/04 06:19:16 8.txt: Not copying as --dry-run
    2016/02/04 06:19:16 1.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:19:16 1.txt: Unchanged skipping
    2016/02/04 06:19:16 2.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:19:16 Local file system at /home/rdest: Deletion finished
    2016/02/04 06:19:16 2.txt: Unchanged skipping
    2016/02/04 06:19:16 3.txt: Size and modification time the same (differ by 0, within tolerance 1ns)
    2016/02/04 06:19:16 3.txt: Unchanged skipping
    2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for transfers to finish
    2016/02/04 06:19:16 Local file system at /home/rdest: Waiting for deletes to finish (during+after)
    
    Transferred:            0 Bytes (   0.00 kByte/s)
    Errors:                 0
    Checks:                 3
    Transferred:            5
    Elapsed time:           0
    
    2016/02/04 06:19:16 Go routines at exit 7
    
     
    Last edited: Feb 4, 2016
    • Informative Informative x 1
  17. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Useful Useful x 1
  18. eva2000

    eva2000 Administrator Staff Member

    30,168
    6,785
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,137
    Local Time:
    7:24 AM
    Nginx 1.13.x
    MariaDB 5.5
  19. Cr0w

    Cr0w Member

    165
    9
    18
    Mar 17, 2015
    Iran
    Ratings:
    +13
    Local Time:
    12:54 AM
    1.9.x
    5.5
    and rclone ist work on centos . what is centos version of this program ?

    and what is this :|

    If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
    Log in and authorize rclone for access
    Waiting for code...

    i can't run on centos .
    i tryed by lynx but nothing happend
     
  20. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    5:24 AM
    latest
    latest
    it works on CentOS. Here's how to install on CentOS 7 64bit

    Code:
    wget http://downloads.rclone.org/rclone-v1.27-linux-amd64.zip
    unzip rclone-v1.27-linux-amd64.zip
    cd rclone-v1.27-linux-amd64/
    cp rclone /usr/sbin/
    chown root:root /usr/sbin/rclone
    chmod 755 /usr/sbin/rclone
    as for the second part, open another ssh connection and in there do

    Code:
    lynx http://127.0.0.1:53682/auth
     
    • Informative Informative x 3
    • Like Like x 1