Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin Faster & smaller compressed file backups with tar + zstd compression

Discussion in 'System Administration' started by eva2000, Dec 20, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    8:10 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Heads up Centmin Mod 123.09beta01+ and newer users, I am testing a custom tar archiver RPM build with native Facebook zstd compression support for CentOS 6/7 64bit systems which allows faster backups and smaller compressed sized backups than the standard tar + gzip/pigz method.

    You can read up on how to install the custom tar RPM and zstd itself in 1st post in thread at https://community.centminmod.com/th...with-facebook-zstd-compression-support.16243/. The 5th post in this thread also shows a manual example of how you can backup and restore Centmin Mod Nginx file/directories in /home/nginx/domain/yourdomain.com using tar + zstd as well :)

    I'm excited to eventually add this tar custom RPM to Centmin Mod 123.09beta01 out of the box as one of my concerns with automated file/directory backups is the additional server resources for memory and disk space required. For a 50GB sized site, one backup would need at least 40-50GB of free disk space and if you keep 7 daily backups, you'd need up to 7x50 = 350GB of free disk space to go with the backup process. With tar + zstd compression, you can potentially 1/2 the size of your compressed backups compared to tar + gzip/pigz standard method and still do it faster as well :)

    Feedback from testing is much appreciated.

    Charted benchmarks comparing tar zstd with tar gzip/pigz and tar xz/pxz.

    tar-zstd-table-03.png tar-zstd-chart-03b.png
     
    • Like Like x 3
  2. digij

    digij New Member

    18
    6
    3
    Jan 25, 2018
    Ratings:
    +9
    Local Time:
    10:10 AM
    Interesting, thanks @eva2000!

    Right now I use tar.gz, I keep a local copy and scp the file to two remote locations.

    Before I scp it, I have the files AES256 encrypted via openssl. Do you encrypt archives? I assume this might at least kill the "faster-advantage" of zstd.

    I will try a test setup later :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    8:10 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    rsync is faster than scp usually :) but no don't really encrypt backups though probably should look into it or see what speedy ways there are for encrypted backups :)
     
  4. bassie

    bassie Well-Known Member

    1,045
    255
    83
    Apr 29, 2016
    Ratings:
    +748
    Local Time:
    11:10 AM
    I made a simple Tar backport-patch based on the git code.
    This allows you to use Red Hat Enterprise Linux 7's TAR with zstd.
     
    • Like Like x 2
  5. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    8:10 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    cheers thanks for sharing :)
     
..