Sysadmin Faster & smaller compressed file backups with tar + zstd compression

Heads up Centmin Mod 123.09beta01+ and newer users, I am testing a custom tar archiver RPM build with native Facebook zstd compression support for CentOS 6/7 64bit systems which allows faster backups and smaller compressed sized backups than the standard tar + gzip/pigz method.

You can read up on how to install the custom tar RPM and zstd itself in 1st post in thread at https://community.centminmod.com/th...with-facebook-zstd-compression-support.16243/. The 5th post in this thread also shows a manual example of how you can backup and restore Centmin Mod Nginx file/directories in /home/nginx/domain/yourdomain.com using tar + zstd as well

---

I'm excited to eventually add this tar custom RPM to Centmin Mod 123.09beta01 out of the box as one of my concerns with automated file/directory backups is the additional server resources for memory and disk space required. For a 50GB sized site, one backup would need at least 40-50GB of free disk space and if you keep 7 daily backups, you'd need up to 7x50 = 350GB of free disk space to go with the backup process. With tar + zstd compression, you can potentially 1/2 the size of your compressed backups compared to tar + gzip/pigz standard method and still do it faster as well

Feedback from testing is much appreciated.

Charted benchmarks comparing tar zstd with tar gzip/pigz and tar xz/pxz.

 

rsync is faster than scp usually but no don't really encrypt backups though probably should look into it or see what speedy ways there are for encrypted backups

 

cheers thanks for sharing

 

centmin.sh menu option 17 is all you need for zstd and other multi-threaded compression tools to play with compression

but if you want to try tar native zstd compression, you need to install a version of tar v1.31+ that supports it so need this custom tar RPM so tar can make use of zstd installed

 

if not using tar v1.31+ and above then for tar + zstd compression you can either do in 2 steps, tar first and then zstd compress tar or do tar stream piping it into zstd.

 

you don't need custom tar v1.31+ for nginx and php-fpm log rotation, zstd is all you need for that

only downside right now is once custom tar 1.31+ is that updating will be manually done same way it was installed via yum localinstall command

test and see, i am already using custom tar v1.32 latest version for some of my servers backup routines paired with zstd.

 

You can decide which logroate profiles to use zstd compression with. Nginx and php-fpm logrotate profiles is just an example of what can be done and chosen by me as they are one of the most common log files to have variable sizes that can grow quite big depending on site's traffic and number of php errors and entries logged.

 

I'm built updated tar v1.33 and v1.34 custom RPMs for Centmin Mod on CentOS 7 that folks can test if they want https://community.centminmod.com/th...ok-zstd-compression-support.16243/#post-90901