Learn about Centmin Mod LEMP Stack today
Register Now

Sysadmin Faster & smaller compressed file backups with tar + zstd compression

Discussion in 'System Administration' started by eva2000, Dec 20, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Heads up Centmin Mod 123.09beta01+ and newer users, I am testing a custom tar archiver RPM build with native Facebook zstd compression support for CentOS 6/7 64bit systems which allows faster backups and smaller compressed sized backups than the standard tar + gzip/pigz method.

    You can read up on how to install the custom tar RPM and zstd itself in 1st post in thread at https://community.centminmod.com/th...with-facebook-zstd-compression-support.16243/. The 5th post in this thread also shows a manual example of how you can backup and restore Centmin Mod Nginx file/directories in /home/nginx/domain/yourdomain.com using tar + zstd as well :)

    I'm excited to eventually add this tar custom RPM to Centmin Mod 123.09beta01 out of the box as one of my concerns with automated file/directory backups is the additional server resources for memory and disk space required. For a 50GB sized site, one backup would need at least 40-50GB of free disk space and if you keep 7 daily backups, you'd need up to 7x50 = 350GB of free disk space to go with the backup process. With tar + zstd compression, you can potentially 1/2 the size of your compressed backups compared to tar + gzip/pigz standard method and still do it faster as well :)

    Feedback from testing is much appreciated.

    Charted benchmarks comparing tar zstd with tar gzip/pigz and tar xz/pxz.

    tar-zstd-table-03.png tar-zstd-chart-03b.png
     
    • Like Like x 3
  2. digij

    digij New Member

    18
    7
    3
    Jan 25, 2018
    Ratings:
    +10
    Local Time:
    5:05 PM
    Interesting, thanks @eva2000!

    Right now I use tar.gz, I keep a local copy and scp the file to two remote locations.

    Before I scp it, I have the files AES256 encrypted via openssl. Do you encrypt archives? I assume this might at least kill the "faster-advantage" of zstd.

    I will try a test setup later :)
     
  3. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    rsync is faster than scp usually :) but no don't really encrypt backups though probably should look into it or see what speedy ways there are for encrypted backups :)
     
  4. buik

    buik Well-Known Member

    1,162
    308
    83
    Apr 29, 2016
    Ratings:
    +906
    Local Time:
    6:05 PM
    I made a simple Tar backport-patch based on the git code.
    This allows you to use Red Hat Enterprise Linux 7's TAR with zstd.
     
    • Like Like x 2
  5. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    cheers thanks for sharing :)
     
  6. pamamolf

    pamamolf Premium Member Premium Member

    3,440
    329
    83
    May 31, 2014
    Ratings:
    +627
    Local Time:
    7:05 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Got confused on that as i would like to try Zstd and i already run from menu:

    17). Multi-thread compression: zstd,pigz,pbzip2,lbzip2

    But is that alone ok?

    Should i use the custom RPM to get the new tar with native zstd support?

    Thank you
     
  7. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    centmin.sh menu option 17 is all you need for zstd and other multi-threaded compression tools to play with compression

    but if you want to try tar native zstd compression, you need to install a version of tar v1.31+ that supports it so need this custom tar RPM so tar can make use of zstd installed
     
  8. pamamolf

    pamamolf Premium Member Premium Member

    3,440
    329
    83
    May 31, 2014
    Ratings:
    +627
    Local Time:
    7:05 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    So if i use only the menu option 17 how should i compress and decompress files?

    For example this will not work :

    Code:
    tar -caf
     
  9. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    if not using tar v1.31+ and above then for tar + zstd compression you can either do in 2 steps, tar first and then zstd compress tar or do tar stream piping it into zstd.
     
  10. pamamolf

    pamamolf Premium Member Premium Member

    3,440
    329
    83
    May 31, 2014
    Ratings:
    +627
    Local Time:
    7:05 PM
    Nginx-1.17.x
    MariaDB 10.3.x
  11. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    you don't need custom tar v1.31+ for nginx and php-fpm log rotation, zstd is all you need for that

    only downside right now is once custom tar 1.31+ is that updating will be manually done same way it was installed via yum localinstall command

    test and see, i am already using custom tar v1.32 latest version for some of my servers backup routines paired with zstd.
     
    • Informative Informative x 1
  12. pamamolf

    pamamolf Premium Member Premium Member

    3,440
    329
    83
    May 31, 2014
    Ratings:
    +627
    Local Time:
    7:05 PM
    Nginx-1.17.x
    MariaDB 10.3.x
    Do you recommend me to use it for all profiles at:

    Code:
    /etc/logrotate.d/
    ?
     
  13. eva2000

    eva2000 Administrator Staff Member

    41,321
    9,274
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,229
    Local Time:
    2:05 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    You can decide which logroate profiles to use zstd compression with. Nginx and php-fpm logrotate profiles is just an example of what can be done and chosen by me as they are one of the most common log files to have variable sizes that can grow quite big depending on site's traffic and number of php errors and entries logged.