Welcome to Centmin Mod Community
Become a Member

Beta Branch Centmin Mod Backups - Attic Deduplication Backups

Discussion in 'Beta release code' started by eva2000, Aug 18, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    41,039
    9,169
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,069
    Local Time:
    7:30 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Last edited: Sep 27, 2015
    • Informative Informative x 1
  2. pamamolf

    pamamolf Premium Member Premium Member

    3,393
    320
    83
    May 31, 2014
    Ratings:
    +604
    Local Time:
    12:30 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Looking for some help to install it and test it as it seems the debian instructions for dependencies are not the same as on Centos 64Bit :

    for example:

    Code:
    [[email protected] ~]# yum install libacl1 liblz4-1
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    * base: ftp.plusline.de
    * extras: ftp.plusline.de
    * updates: mirror.imt-systems.com
    No package libacl1 available.
    No package liblz4-1 available.
    Error: Nothing to do
     
  3. eva2000

    eva2000 Administrator Staff Member

    41,039
    9,169
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,069
    Local Time:
    7:30 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Seems for Borgbackup it runs in a python virtual environment
    Code:
    cd /usr/local/src/centminmod/addons
    ./python34_install.sh
    yum -y install libacl-devel libacl lz4-devel fuse-devel fuse pkgconfig
    cd /root/tools
    mkdir -p /home/borgtmp
    chmod 1777 /home/borgtmp
    export TMPDIR=/home/borgtmp
    easy_install-3.4 pip
    pip install virtualenv
    virtualenv --python=python3 borg-env
    source borg-env/bin/activate
    pip install 'llfuse<0.41'
    pip install borgbackup
    Code:
    /root/tools/borg-env/bin/borg -h
    usage: borg [-h]
             
                {serve,init,check,change-passphrase,create,extract,rename,delete,list,mount,info,prune,help}
                ...
    
    Borg 0.26.0 - Deduplicated Backups
    
    optional arguments:
      -h, --help            show this help message and exit
    
    Available commands:
      {serve,init,check,change-passphrase,create,extract,rename,delete,list,mount,info,prune,help}
    
    Quick Start — Borg - Deduplicating Archiver 0.26.0 documentation
     
    • Informative Informative x 1
  4. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:30 AM
    latest
    latest
    • Informative Informative x 1
  5. eva2000

    eva2000 Administrator Staff Member

    41,039
    9,169
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,069
    Local Time:
    7:30 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    interesting info thanks @dorobo

    interesting both his test servers would of been desktop grade with non-ECC memory FYI
     
  6. ModeltogTossen

    ModeltogTossen I wish I could??

    313
    97
    28
    Dec 20, 2015
    Denmark
    Ratings:
    +143
    Local Time:
    11:30 PM
    1.9.12
    10.0.23
    For the borg install you give this your post:

    After running the pip install virtualenv I got this:

    Code:
    You are using pip version 7.1.2, however version 8.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    
    I'm not a programmer - so can I just executed that statement before I go on with the rest of those pip install commands?
     
  7. eva2000

    eva2000 Administrator Staff Member

    41,039
    9,169
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,069
    Local Time:
    7:30 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    should be okay
    Code:
    pip install --upgrade pip
     
  8. ModeltogTossen

    ModeltogTossen I wish I could??

    313
    97
    28
    Dec 20, 2015
    Denmark
    Ratings:
    +143
    Local Time:
    11:30 PM
    1.9.12
    10.0.23
    It was - thanks:

    Code:
    [[email protected] tools]# pip install --upgrade pip
    Collecting pip
      Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB)
        100% |████████████████████████████████| 1.2MB 515kB/s
    Installing collected packages: pip
      Found existing installation: pip 7.1.2
        Uninstalling pip-7.1.2:
          Successfully uninstalled pip-7.1.2
    Successfully installed pip-8.0.2
    
     
    • Like Like x 1
  9. ModeltogTossen

    ModeltogTossen I wish I could??

    313
    97
    28
    Dec 20, 2015
    Denmark
    Ratings:
    +143
    Local Time:
    11:30 PM
    1.9.12
    10.0.23
    So - after install of newer python on 09beta1 - I use the @eva2000 borg guide for install.. After that I created a new virtual disk for storing of local backup while testing it. The goal is for sure - remote storage like nfs or perhaps something like sshfs. Got the disk mounted - execute borg backup on my XF install.. Added -v --stats as parameters and got this for you:

    Code:
    [[email protected] tools]# borg-env/bin/borg create -v --stats --compression lz4 /backupstore/borgstorage::mt-backup-`date +%Y-%m-%d` /home/nginx/domains/yourdomin/public
    Using a pure-python msgpack! This will result in lower performance.
    reading files cache
    processing files
    ------------------------------------------------------------------------------
    Archive name: mt-backup-2016-01-25
    Archive fingerprint: 80e9a4ae743057277901e21465a0c756df7c446594f4102603ff66a07dcf12bc
    Start time: Mon Jan 25 21:35:16 2016
    End time: Mon Jan 25 21:36:14 2016
    Duration: 57.93 seconds
    Number of files: 8622
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    This archive:                1.87 GB              1.83 GB              1.82 GB
    All archives:                1.87 GB              1.83 GB              1.82 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   34640                35879
    ------------------------------------------------------------------------------
    
    So all is fine for now.. So @eva2000 - again, thanks for your guide, that got me started..
     
    • Like Like x 1
  10. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:30 AM
    latest
    latest
    I'm playing with Attic mount and I got this

    fuse: device not found, try 'modprobe fuse' first
    attic: Exiting with failure status due to previous errors

    I tried modprobe fuse but still gets the same error when mounting
     
  11. eva2000

    eva2000 Administrator Staff Member

    41,039
    9,169
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,069
    Local Time:
    7:30 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    you on OpenVZ ? Your web host has to support fuse from OpenVZ host node level
     
  12. ModeltogTossen

    ModeltogTossen I wish I could??

    313
    97
    28
    Dec 20, 2015
    Denmark
    Ratings:
    +143
    Local Time:
    11:30 PM
    1.9.12
    10.0.23
    The last update from me about this borg backup. I got fuse-sshfs installed - make a share on another server - did sshfs to that and mounted it 'here' on this server.. Same files, same server but remote storages by sshfs:

    Code:
    [[email protected] tools]# borg-env/bin/borg create -v --stats --compression lz4 /backupstore/borgremote/borgstorage::mt-backup-`date +%Y-%m-%d` /home/nginx/domains/yourdomain/public
    Using a pure-python msgpack! This will result in lower performance.
    reading files cache
    processing files
    ------------------------------------------------------------------------------
    Archive name: mt-backup-2016-01-25
    Archive fingerprint: 394d878ceeeb20556c329e8f690622a679801f6edd854eed31a6d885e81c584d
    Start time: Mon Jan 25 21:59:56 2016
    End time: Mon Jan 25 22:01:05 2016
    Duration: 1 minutes 9.14 seconds
    Number of files: 8622
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    This archive:                1.87 GB              1.83 GB              1.82 GB
    All archives:                1.87 GB              1.83 GB              1.82 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   34640                35879
    ------------------------------------------------------------------------------
    
    So the local storages duration against remote storage duration is:

    Local -Duration: 57.93 seconds
    Remote - Duration: 1 minutes 9.14 seconds

    Not much lost in time - allright - its only 2 GB and without DB. So now comes a cron with mysqldump first and then backup of files.
     
    • Informative Informative x 1
  13. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:30 AM
    latest
    latest
    Yes OpenVZ. Oh snap.
     
  14. ethanpil

    ethanpil Active Member

    170
    53
    28
    Nov 8, 2015
    Ratings:
    +98
    Local Time:
    7:30 AM
    • Informative Informative x 1
  15. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +162
    Local Time:
    5:30 AM
    latest
    latest
    • Informative Informative x 1
  16. pamamolf

    pamamolf Premium Member Premium Member

    3,393
    320
    83
    May 31, 2014
    Ratings:
    +604
    Local Time:
    12:30 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    So does Attic create corrupted backups on huge files or not?
     
  17. ModeltogTossen

    ModeltogTossen I wish I could??

    313
    97
    28
    Dec 20, 2015
    Denmark
    Ratings:
    +143
    Local Time:
    11:30 PM
    1.9.12
    10.0.23
    I don't think you can get the right answer to that in here .. Maybe take a look on the maillist archive - [attic] archives