Featured Beta Branch CentOS 7.x Custom Tar Archiver RPM Build With Facebook Zstd Compression Support

eva2000 · Dec 16, 2018

Folks may have read my multi-threaded compression tool benchmarks here, here, here & here and may wonder how we can take advantage of faster compression tools such as Facebook's zstd ? Centmin Mod 123.09beta01 and higher allow you to optionally install all the multi-threaded compression tools via centmin.sh menu option 17 (see below). You can also test and provide feedback here. Or jump straight to the final zstd benchmark comparison results including zstd long range mode and automatic level determination modes for higher compression ratios here. Also rsync 3.2.3 with zstd support is a nice companion to tar 1.32+ with zstd support

Code (Text):

--------------------------------------------------------
     Centmin Mod Menu 123.09beta01 centminmod.com
--------------------------------------------------------
1).  Centmin Install
2).  Add Nginx vhost domain
3).  NSD setup domain name DNS
4).  Nginx Upgrade / Downgrade
5).  PHP Upgrade / Downgrade
6).  XCache Re-install
7).  APC Cache Re-install
8).  XCache Install
9).  APC Cache Install
10). Memcached Server Re-install
11). MariaDB MySQL Upgrade & Management
12). Zend OpCache Install/Re-install
13). Install/Reinstall Redis PHP Extension
14). SELinux disable
15). Install/Reinstall ImagicK PHP Extension
16). Change SSHD Port Number
17). Multi-thread compression: pigz,pbzip2,lbzip2...
18). Suhosin PHP Extension install
19). Install FFMPEG and FFMPEG PHP Extension
20). NSD Install/Re-Install
21). Update - Nginx + PHP-FPM + Siege
22). Add Wordpress Nginx vhost + Cache Plugin
23). Update Centmin Mod Code Base
24). Exit
--------------------------------------------------------
Enter option [ 1 - 24 ] 17

zstd v1.3.8

Code (Text):

zstd --version
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

For backup purposes with tar archiving tool, you would currently need to tar first and then use zstd. However, latest development version of tar 1.30.90 has native support for zstd now and now tar 1.31 stable release is out.

Code (Text):

Compression options:

  -a, --auto-compress        use archive suffix to determine the compression
                             program
  -I, --use-compress-program=PROG
                             filter through PROG (must accept -d)
  -j, --bzip2                filter the archive through bzip2
  -J, --xz                   filter the archive through xz
      --lzip                 filter the archive through lzip
      --lzma                 filter the archive through lzma
      --lzop                 filter the archive through lzop
      --no-auto-compress     do not use archive suffix to determine the
                             compression program
  -z, --gzip, --gunzip, --ungzip   filter the archive through gzip
      --zstd                 filter the archive through zstd
  -Z, --compress, --uncompress   filter the archive through compress

So I built a custom tar 1.31+ RPM for CentOS 6.x & 7.x 64bit based Centmin Mod 123.09beta01 or higher installations.

After running centmin.sh menu option 17 to install zstd etc, install custom tar zstd supported rpm. December 6, 2021 updated with tar v1.33 and v1.34 RPMs

CentOS 7 64bit tar v1.34

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc9-1.34-1.el7.x86_64.rpm -O tar-zstd-gcc9-1.34-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc9-1.34-1.el7.x86_64.rpm

Code (Text):

rpm -qa --changelog tar-zstd-gcc9
* Sun Dec 05 2021 George Liu <centminmod.com> 1.34
- tar 1.34 zstd supported for centminmod.com LEMP stack
- Fix extraction over pipe (savannah bug #60002)
- Fix memory leak in read_header (savannah bug #59897)
- Fix extraction when . and .. are unreadable
  See https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00012.html
- Gracefully handle duplicate symlinks when extracting
  See https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00026.html
- Re-initialize supplementary groups when switching to user privileges

CentOS 7 64bit tar v1.33

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc9-1.33-1.el7.x86_64.rpm -O tar-zstd-gcc9-1.33-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc9-1.33-1.el7.x86_64.rpm

Code (Text):

rpm -qa --changelog tar-zstd-gcc9
* Mon Jan 25 2021 George Liu <centminmod.com> 1.33
- tar 1.33 zstd supported for centminmod.com LEMP stack
- Fix memory leak in read_header CVE-2021-20193 patch fixed

CentOS 7 64bit tar v1.32

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.32-1.el7.x86_64.rpm -O tar-zstd-gcc7-1.32-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc7-1.32-1.el7.x86_64.rpm

Updating with January 3, 2019 tar 1.32 build

Code (Text):

rpm -e tar-zstd-gcc7
cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.32-1.el7.x86_64.rpm -O tar-zstd-gcc7-1.32-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc7-1.32-1.el7.x86_64.rpm

Then exit existing SSH session and re-login via SSH to pick up the updated tar version

CentOS 6 64bit

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.32-1.el6.x86_64.rpm -O tar-zstd-gcc7-1.32-1.el6.x86_64.rpm
yum localinstall tar-zstd-gcc7-1.32-1.el6.x86_64.rpm

Updating with January 3, 2019 tar 1.32 build

Code (Text):

rpm -e tar-zstd-gcc7
cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.32-1.el6.x86_64.rpm -O tar-zstd-gcc7-1.32-1.el6.x86_64.rpm
yum localinstall tar-zstd-gcc7-1.32-1.el6.x86_64.rpm

Then exit existing SSH session and re-login via SSH to pick up the updated tar version

Example for CentOS 7 64bit install for tar 1.30.90

Code (Text):

yum localinstall tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm
Loaded plugins: fastestmirror, priorities, versionlock
Examining tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm: tar-zstd-gcc7-1.30.90-1.el7.x86_64
Marking tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package tar-zstd-gcc7.x86_64 0:1.30.90-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================
 Package                                                  Arch                                              Version                                                  Repository                                                                      Size
==========================================================================================================================================================================================================================================================
Installing:
 tar-zstd-gcc7                                            x86_64                                            1.30.90-1.el7                                            /tar-zstd-gcc7-1.30.90-1.el7.x86_64                                            3.4 M

Transaction Summary
==========================================================================================================================================================================================================================================================
Install  1 Package

Total size: 3.4 M
Installed size: 3.4 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tar-zstd-gcc7-1.30.90-1.el7.x86_64                                                                                                                                                                                                     1/1
  Verifying  : tar-zstd-gcc7-1.30.90-1.el7.x86_64                                                                                                                                                                                                     1/1

Installed:
  tar-zstd-gcc7.x86_64 0:1.30.90-1.el7                                                                                                                                                                                           
Complete!

Usage

Create a tar archive compressed with zstd

Code (Text):

tar -caf directory.tar.zst directory/

Decompress zstd compressed archive

Code (Text):

tar -xaf directory.tar.zst

RPM Info

Code (Text):

tar --version
tar (GNU tar) 1.30.90
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Code (Text):

yum -q info tar-zstd-gcc7
Installed Packages
Name        : tar-zstd-gcc7
Arch        : x86_64
Version     : 1.30.90
Release     : 1.el7
Size        : 3.4 M
Repo        : installed
Summary     : tar 1.30.90 zstd supported for centminmod.com LEMP stack
URL         : https://centminmod.com
License     : unknown
Description : tar 1.30.90 zstd supported for centminmod.com LEMP stack

Code (Text):

rpm -qa --changelog tar-zstd-gcc7
* Fri Dec 28 2018 George Liu <centminmod.com> 1.30.90
- tar 1.30.90 zstd supported for centminmod.com LEMP stack

Code (Text):

rpm -ql tar-zstd-gcc7
/usr/local/bin/tar
/usr/local/libexec/rmt
/usr/local/share/info/dir
/usr/local/share/info/tar.info
/usr/local/share/info/tar.info-1
/usr/local/share/info/tar.info-2
/usr/local/share/locale/bg/LC_MESSAGES/tar.mo
/usr/local/share/locale/ca/LC_MESSAGES/tar.mo
/usr/local/share/locale/cs/LC_MESSAGES/tar.mo
/usr/local/share/locale/da/LC_MESSAGES/tar.mo
/usr/local/share/locale/de/LC_MESSAGES/tar.mo
/usr/local/share/locale/el/LC_MESSAGES/tar.mo
/usr/local/share/locale/eo/LC_MESSAGES/tar.mo
/usr/local/share/locale/es/LC_MESSAGES/tar.mo
/usr/local/share/locale/et/LC_MESSAGES/tar.mo
/usr/local/share/locale/eu/LC_MESSAGES/tar.mo
/usr/local/share/locale/fi/LC_MESSAGES/tar.mo
/usr/local/share/locale/fr/LC_MESSAGES/tar.mo
/usr/local/share/locale/ga/LC_MESSAGES/tar.mo
/usr/local/share/locale/gl/LC_MESSAGES/tar.mo
/usr/local/share/locale/hr/LC_MESSAGES/tar.mo
/usr/local/share/locale/hu/LC_MESSAGES/tar.mo
/usr/local/share/locale/id/LC_MESSAGES/tar.mo
/usr/local/share/locale/it/LC_MESSAGES/tar.mo
/usr/local/share/locale/ja/LC_MESSAGES/tar.mo
/usr/local/share/locale/ko/LC_MESSAGES/tar.mo
/usr/local/share/locale/ky/LC_MESSAGES/tar.mo
/usr/local/share/locale/ms/LC_MESSAGES/tar.mo
/usr/local/share/locale/nb/LC_MESSAGES/tar.mo
/usr/local/share/locale/nl/LC_MESSAGES/tar.mo
/usr/local/share/locale/pl/LC_MESSAGES/tar.mo
/usr/local/share/locale/pt/LC_MESSAGES/tar.mo
/usr/local/share/locale/pt_BR/LC_MESSAGES/tar.mo
/usr/local/share/locale/ro/LC_MESSAGES/tar.mo
/usr/local/share/locale/ru/LC_MESSAGES/tar.mo
/usr/local/share/locale/sk/LC_MESSAGES/tar.mo
/usr/local/share/locale/sl/LC_MESSAGES/tar.mo
/usr/local/share/locale/sr/LC_MESSAGES/tar.mo
/usr/local/share/locale/sv/LC_MESSAGES/tar.mo
/usr/local/share/locale/tr/LC_MESSAGES/tar.mo
/usr/local/share/locale/uk/LC_MESSAGES/tar.mo
/usr/local/share/locale/vi/LC_MESSAGES/tar.mo
/usr/local/share/locale/zh_CN/LC_MESSAGES/tar.mo
/usr/local/share/locale/zh_TW/LC_MESSAGES/tar.mo
/usr/local/share/man/man1/tar.1
/usr/local/share/man/man8/rmt.8

Code (Text):

tar --help
Usage: tar [OPTION...] [FILE]...
GNU 'tar' saves many files together into a single tape or disk archive, and can
restore individual files from the archive.

Examples:
  tar -cf archive.tar foo bar  # Create archive.tar from files foo and bar.
  tar -tvf archive.tar         # List all files in archive.tar verbosely.
  tar -xf archive.tar          # Extract all files from archive.tar.

 Local file name selection:

      --add-file=FILE        add given FILE to the archive (useful if its name
                             starts with a dash)
  -C, --directory=DIR        change to directory DIR
      --exclude=PATTERN      exclude files, given as a PATTERN
      --exclude-backups      exclude backup and lock files
      --exclude-caches       exclude contents of directories containing
                             CACHEDIR.TAG, except for the tag file itself
      --exclude-caches-all   exclude directories containing CACHEDIR.TAG
      --exclude-caches-under exclude everything under directories containing
                             CACHEDIR.TAG
      --exclude-ignore=FILE  read exclude patterns for each directory from
                             FILE, if it exists
      --exclude-ignore-recursive=FILE
                             read exclude patterns for each directory and its
                             subdirectories from FILE, if it exists
      --exclude-tag=FILE     exclude contents of directories containing FILE,
                             except for FILE itself
      --exclude-tag-all=FILE exclude directories containing FILE
      --exclude-tag-under=FILE   exclude everything under directories
                             containing FILE
      --exclude-vcs          exclude version control system directories
      --exclude-vcs-ignores  read exclude patterns from the VCS ignore files
      --no-null              disable the effect of the previous --null option
      --no-recursion         avoid descending automatically in directories
      --no-unquote           do not unquote input file or member names
      --no-verbatim-files-from   -T treats file names starting with dash as
                             options (default)
      --null                 -T reads null-terminated names; implies
                             --verbatim-files-from
      --recursion            recurse into directories (default)
  -T, --files-from=FILE      get names to extract or create from FILE
      --unquote              unquote input file or member names (default)
      --verbatim-files-from  -T reads file names verbatim (no escape or option
                             handling)
  -X, --exclude-from=FILE    exclude patterns listed in FILE

 File name matching options (affect both exclude and include patterns):

      --anchored             patterns match file name start
      --ignore-case          ignore case
      --no-anchored          patterns match after any '/' (default for
                             exclusion)
      --no-ignore-case       case sensitive matching (default)
      --no-wildcards         verbatim string matching
      --no-wildcards-match-slash   wildcards do not match '/'
      --wildcards            use wildcards (default for exclusion)
      --wildcards-match-slash   wildcards match '/' (default for exclusion)

 Main operation mode:

  -A, --catenate, --concatenate   append tar files to an archive
  -c, --create               create a new archive
  -d, --diff, --compare      find differences between archive and file system
      --delete               delete from the archive (not on mag tapes!)
  -r, --append               append files to the end of an archive
  -t, --list                 list the contents of an archive
      --test-label           test the archive volume label and exit
  -u, --update               only append files newer than copy in archive
  -x, --extract, --get       extract files from an archive

 Operation modifiers:

      --check-device         check device numbers when creating incremental
                             archives (default)
  -g, --listed-incremental=FILE   handle new GNU-format incremental backup
  -G, --incremental          handle old GNU-format incremental backup
      --hole-detection=TYPE  technique to detect holes
      --ignore-failed-read   do not exit with nonzero on unreadable files
      --level=NUMBER         dump level for created listed-incremental archive
  -n, --seek                 archive is seekable
      --no-check-device      do not check device numbers when creating
                             incremental archives
      --no-seek              archive is not seekable
      --occurrence[=NUMBER]  process only the NUMBERth occurrence of each file
                             in the archive; this option is valid only in
                             conjunction with one of the subcommands --delete,
                             --diff, --extract or --list and when a list of
                             files is given either on the command line or via
                             the -T option; NUMBER defaults to 1
      --sparse-version=MAJOR[.MINOR]
                             set version of the sparse format to use (implies
                             --sparse)
  -S, --sparse               handle sparse files efficiently

 Overwrite control:

  -k, --keep-old-files       don't replace existing files when extracting,
                             treat them as errors
      --keep-directory-symlink   preserve existing symlinks to directories when
                             extracting
      --keep-newer-files     don't replace existing files that are newer than
                             their archive copies
      --no-overwrite-dir     preserve metadata of existing directories
      --one-top-level[=DIR]  create a subdirectory to avoid having loose files
                             extracted
      --overwrite            overwrite existing files when extracting
      --overwrite-dir        overwrite metadata of existing directories when
                             extracting (default)
      --recursive-unlink     empty hierarchies prior to extracting directory
      --remove-files         remove files after adding them to the archive
      --skip-old-files       don't replace existing files when extracting,
                             silently skip over them
  -U, --unlink-first         remove each file prior to extracting over it
  -W, --verify               attempt to verify the archive after writing it

 Select output stream:

      --ignore-command-error ignore exit codes of children
      --no-ignore-command-error   treat non-zero exit codes of children as
                             error
  -O, --to-stdout            extract files to standard output
      --to-command=COMMAND   pipe extracted files to another program

 Handling of file attributes:

      --atime-preserve[=METHOD]   preserve access times on dumped files, either
                             by restoring the times after reading
                             (METHOD='replace'; default) or by not setting the
                             times in the first place (METHOD='system')
      --clamp-mtime          only set time when the file is more recent than
                             what was given with --mtime
      --delay-directory-restore   delay setting modification times and
                             permissions of extracted directories until the end
                             of extraction
      --group=NAME           force NAME as group for added files
      --group-map=FILE       use FILE to map file owner GIDs and names
      --mode=CHANGES         force (symbolic) mode CHANGES for added files
      --mtime=DATE-OR-FILE   set mtime for added files from DATE-OR-FILE
  -m, --touch                don't extract file modified time
      --no-delay-directory-restore
                             cancel the effect of --delay-directory-restore
                             option
      --no-same-owner        extract files as yourself (default for ordinary
                             users)
      --no-same-permissions  apply the user's umask when extracting permissions
                             from the archive (default for ordinary users)
      --numeric-owner        always use numbers for user/group names
      --owner=NAME           force NAME as owner for added files
      --owner-map=FILE       use FILE to map file owner UIDs and names
  -p, --preserve-permissions, --same-permissions
                             extract information about file permissions
                             (default for superuser)
      --same-owner           try extracting files with the same ownership as
                             exists in the archive (default for superuser)
  -s, --preserve-order, --same-order
                             member arguments are listed in the same order as
                             the files in the archive
      --sort=ORDER           directory sorting order: none (default), name or
                             inode

 Handling of extended file attributes:

      --acls                 Enable the POSIX ACLs support
      --no-acls              Disable the POSIX ACLs support
      --no-selinux           Disable the SELinux context support
      --no-xattrs            Disable extended attributes support
      --selinux              Enable the SELinux context support
      --xattrs               Enable extended attributes support
      --xattrs-exclude=MASK  specify the exclude pattern for xattr keys
      --xattrs-include=MASK  specify the include pattern for xattr keys

 Device selection and switching:

  -f, --file=ARCHIVE         use archive file or device ARCHIVE
      --force-local          archive file is local even if it has a colon
  -F, --info-script=NAME, --new-volume-script=NAME
                             run script at end of each tape (implies -M)
  -L, --tape-length=NUMBER   change tape after writing NUMBER x 1024 bytes
  -M, --multi-volume         create/list/extract multi-volume archive
      --rmt-command=COMMAND  use given rmt COMMAND instead of rmt
      --rsh-command=COMMAND  use remote COMMAND instead of rsh
      --volno-file=FILE      use/update the volume number in FILE

 Device blocking:

  -b, --blocking-factor=BLOCKS   BLOCKS x 512 bytes per record
  -B, --read-full-records    reblock as we read (for 4.2BSD pipes)
  -i, --ignore-zeros         ignore zeroed blocks in archive (means EOF)
      --record-size=NUMBER   NUMBER of bytes per record, multiple of 512

 Archive format selection:

  -H, --format=FORMAT        create archive of the given format

 FORMAT is one of the following:

    gnu                      GNU tar 1.13.x format
    oldgnu                   GNU format as per tar <= 1.12
    pax                      POSIX 1003.1-2001 (pax) format
    posix                    same as pax
    ustar                    POSIX 1003.1-1988 (ustar) format
    v7                       old V7 tar format

      --old-archive, --portability
                             same as --format=v7
      --pax-option=keyword[[:]=value][,keyword[[:]=value]]...
                             control pax keywords
      --posix                same as --format=posix
  -V, --label=TEXT           create archive with volume name TEXT; at
                             list/extract time, use TEXT as a globbing pattern
                             for volume name

 Compression options:

  -a, --auto-compress        use archive suffix to determine the compression
                             program
  -I, --use-compress-program=PROG
                             filter through PROG (must accept -d)
  -j, --bzip2                filter the archive through bzip2
  -J, --xz                   filter the archive through xz
      --lzip                 filter the archive through lzip
      --lzma                 filter the archive through lzma
      --lzop                 filter the archive through lzop
      --no-auto-compress     do not use archive suffix to determine the
                             compression program
  -z, --gzip, --gunzip, --ungzip   filter the archive through gzip
      --zstd                 filter the archive through zstd
  -Z, --compress, --uncompress   filter the archive through compress

 Local file selection:

      --backup[=CONTROL]     backup before removal, choose version CONTROL
  -h, --dereference          follow symlinks; archive and dump the files they
                             point to
      --hard-dereference     follow hard links; archive and dump the files they
                             refer to
  -K, --starting-file=MEMBER-NAME
                             begin at member MEMBER-NAME when reading the
                             archive
      --newer-mtime=DATE     compare date and time when data changed only
  -N, --newer=DATE-OR-FILE, --after-date=DATE-OR-FILE
                             only store files newer than DATE-OR-FILE
      --one-file-system      stay in local file system when creating archive
  -P, --absolute-names       don't strip leading '/'s from file names
      --suffix=STRING        backup before removal, override usual suffix ('~'
                             unless overridden by environment variable
                             SIMPLE_BACKUP_SUFFIX)

 File name transformations:

      --strip-components=NUMBER   strip NUMBER leading components from file
                             names on extraction
      --transform=EXPRESSION, --xform=EXPRESSION
                             use sed replace EXPRESSION to transform file
                             names

 Informative output:

      --checkpoint[=NUMBER]  display progress messages every NUMBERth record
                             (default 10)
      --checkpoint-action=ACTION   execute ACTION on each checkpoint
      --full-time            print file time to its full resolution
      --index-file=FILE      send verbose output to FILE
  -l, --check-links          print a message if not all links are dumped
      --no-quote-chars=STRING   disable quoting for characters from STRING
      --quote-chars=STRING   additionally quote characters from STRING
      --quoting-style=STYLE  set name quoting style; see below for valid STYLE
                             values
  -R, --block-number         show block number within archive with each message
        
      --show-defaults        show tar defaults
      --show-omitted-dirs    when listing or extracting, list each directory
                             that does not match search criteria
      --show-snapshot-field-ranges
                             show valid ranges for snapshot-file fields
      --show-transformed-names, --show-stored-names
                             show file or archive names after transformation
      --totals[=SIGNAL]      print total bytes after processing the archive;
                             with an argument - print total bytes when this
                             SIGNAL is delivered; Allowed signals are: SIGHUP,
                             SIGQUIT, SIGINT, SIGUSR1 and SIGUSR2; the names
                             without SIG prefix are also accepted
      --utc                  print file modification times in UTC
  -v, --verbose              verbosely list files processed
      --warning=KEYWORD      warning control
  -w, --interactive, --confirmation
                             ask for confirmation for every action

 Compatibility options:

  -o                         when creating, same as --old-archive; when
                             extracting, same as --no-same-owner

 Other options:

  -?, --help                 give this help list
      --restrict             disable use of some potentially harmful options
      --usage                give a short usage message
      --version              print program version

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.
The version control may be set with --backup or VERSION_CONTROL, values are:

  none, off       never make backups
  t, numbered     make numbered backups
  nil, existing   numbered if numbered backups exist, simple otherwise
  never, simple   always make simple backups

Valid arguments for the --quoting-style option are:

  literal
  shell
  shell-always
  shell-escape
  shell-escape-always
  c
  c-maybe
  escape
  locale
  clocale

*This* tar defaults to:
--format=gnu -f- -b20 --quoting-style=escape
--rmt-command=/usr/local/libexec/rmt

eva2000 · Dec 16, 2018

Example usage on 2 cpu thread server for zstd run backing up /svr-setup directory
Code (Text):
du -sh /svr-setup
1.6G    /svr-setup
tar + zstd compression took 16.74s
Code (Text):
time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup

real    0m16.740s
user    0m15.451s
sys     0m5.694s
Code (Text):
ls -lah /home/speedtest/svr-setup.tar.zst
-rw-r--r-- 1 root root 621M Dec 15 17:24 /home/speedtest/svr-setup.tar.zst
Compared with standard gzip run which took 1m 47.782s or 107.782 seconds
Code (Text):
time tar -caf /home/speedtest/svr-setup.tar.gz svr-setup

real    1m47.782s
user    1m44.892s
sys     0m6.766s
Code (Text):
ls -lah /home/speedtest
total 1.3G
drwxr-xr-x   2 root root 4.0K Dec 15 17:26 .
drwxr-xr-x. 13 root root 4.0K Dec 15 17:22 ..
-rw-r--r--   1 root root 633M Dec 15 17:28 svr-setup.tar.gz
-rw-r--r--   1 root root 621M Dec 15 17:24 svr-setup.tar.
Compared with pigz multi-threaded gzip run took 54.774s
Code (Text):
time tar -I pigz -cf /home/speedtest/svr-setup-pigz.tar.gz svr-setup

real    0m54.774s
user    1m46.527s
sys     0m2.268s
Final compressed sizes show zstd compression which defaults to level 3 compress the svr-setup tar archive better and was much faster in doing it in 16.74s versus gzip at 107.782s versus pigz gzip at 54.774s.
Code (Text):
ls -lah /home/speedtest
total 1.9G
drwxr-xr-x   2 root root 4.0K Dec 15 17:30 .
drwxr-xr-x. 13 root root 4.0K Dec 15 17:22 ..
-rw-r--r--   1 root root 632M Dec 15 17:31 svr-setup-pigz.tar.gz
-rw-r--r--   1 root root 633M Dec 15 17:28 svr-setup.tar.gz
-rw-r--r--   1 root root 621M Dec 15 17:24 svr-setup.tar.zst

eva2000 · Dec 16, 2018

Above tests are on 2 cpu thread VPSes. This time test on Intel Core i7 4790K with 4 cpu core/8 cpu threaded dedicated server to see how zstd handles more multi-threaded compression runs.

Code (Text):

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:              3
CPU MHz:               4199.707
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              7981.85
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1d

test against /svr-setup directory of 3.7GB in size

Code (Text):

du -sh /svr-setup/
3.7G    /svr-setup/

tar uncompressed

Code (Text):

time tar -cf /home/speedtest/svr-setup.tar svr-setup

real    0m10.270s
user    0m0.347s
sys     0m2.804s

First is tar + standard gzip which has default level 6 compression which took 1mi 57.043s or 117.043 seconds

Code (Text):

time tar -caf /home/speedtest/svr-setup.tar.gz svr-setup

real    1m57.043s
user    1m45.699s
sys     0m3.690s

Then tar + pigz multi-threaded gzip run with default level 6 compression took 22.374 seconds

Code (Text):

time tar -I pigz -cf /home/speedtest/svr-setup-pigz.tar.gz svr-setup

real    0m22.374s
user    2m34.483s
sys     0m3.911s

Then tar + zstd single threaded run with default level 3 compression took 13.87 seconds

Code (Text):

time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup

real    0m13.870s
user    0m14.148s
sys     0m2.830s

Resulting compressed and uncompressed sizes

Code (Text):

ls -lAhrt /home/speedtest/                   
total 11G
-rw-r--r-- 1 root root 1.4G Dec 16 04:10 svr-setup.tar.gz
-rw-r--r-- 1 root root 1.4G Dec 16 04:11 svr-setup-pigz.tar.gz
-rw-r--r-- 1 root root 1.3G Dec 16 04:12 svr-setup.tar.zst
-rw-r--r-- 1 root root 3.5G Dec 16 04:16 svr-setup.tar

Then try higher levels of zstd compression against svr-setup.tar

zstd level 6

Code (Text):

time zstd -6f svr-setup.tar -o svr-setup-zstd-lvl6.tar.zst
svr-setup.tar        : 35.89%   (3690383360 => 1324652195 bytes, svr-setup-zstd-lvl6.tar.zst)

real    0m32.766s
user    0m32.016s
sys     0m0.966s

zstd level 9

Code (Text):

time zstd -9f svr-setup.tar -o svr-setup-zstd-lvl9.tar.zst
svr-setup.tar        : 35.22%   (3690383360 => 1299591955 bytes, svr-setup-zstd-lvl9.tar.zst)

real    1m5.178s
user    1m3.793s
sys     0m1.043s

zstd level 12

Code (Text):

time zstd -12f svr-setup.tar -o svr-setup-zstd-lvl12.tar.zst
svr-setup.tar        : 29.75%   (3690383360 => 1098039642 bytes, svr-setup-zstd-lvl12.tar.zst)

real    1m32.450s
user    1m32.256s
sys     0m1.039s

Resulting compressed and uncompressed sizes

Code (Text):

ls -lAhrt /home/speedtest/
total 11G
-rw-r--r-- 1 root root 1.4G Dec 16 04:10 svr-setup.tar.gz
-rw-r--r-- 1 root root 1.4G Dec 16 04:11 svr-setup-pigz.tar.gz
-rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup-zstd-lvl9.tar.zst
-rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup-zstd-lvl6.tar.zst
-rw-r--r-- 1 root root 1.1G Dec 16 04:16 svr-setup-zstd-lvl12.tar.zst
-rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup.tar.zst
-rw-r--r-- 1 root root 3.5G Dec 16 04:16 svr-setup.tar

Code (Text):

ls -lArt /home/speedtest/
total 11359368
-rw-r--r-- 1 root root 1448823495 Dec 16 04:10 svr-setup.tar.gz
-rw-r--r-- 1 root root 1445803664 Dec 16 04:11 svr-setup-pigz.tar.gz
-rw-r--r-- 1 root root 1299591955 Dec 16 04:16 svr-setup-zstd-lvl9.tar.zst
-rw-r--r-- 1 root root 1324652195 Dec 16 04:16 svr-setup-zstd-lvl6.tar.zst
-rw-r--r-- 1 root root 1098039642 Dec 16 04:16 svr-setup-zstd-lvl12.tar.zst
-rw-r--r-- 1 root root 1324652195 Dec 16 04:16 svr-setup.tar.zst
-rw-r--r-- 1 root root 3690383360 Dec 16 04:16 svr-setup.tar

tar + native zstd multi-threaded tests

By default zstd used natively by tar uses only a single thread from above tests. That is already fast.

tar + zstd single threaded run with default level 3 compression took 13.87 seconds

Code (Text):

time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup

real    0m13.870s
user    0m14.148s
sys     0m2.830s

tar + zstd single threaded run with level 12 compression took 1m 34.634s or 94.634 seconds

Code (Text):

time tar "-I zstd -12 -T1" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup

real    1m34.634s
user    1m34.634s
sys     0m2.933s

tar + zstd single threaded run with level 19 compression took 14m 17.188s or 94.634 seconds

Code (Text):

time tar "-I zstd -19 -T1" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup

real    14m17.188s
user    14m18.003s
sys     0m2.763s

However it can be sped up dramatically by using more than one cpu thread via zstd -T flag passed through tar's -I flag.

4 cpu thread zstd test at level 3 compression

Code (Text):

time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup

real    0m7.625s
user    0m19.528s
sys     0m3.413s

4 cpu thread zstd test at level 12 compression

Code (Text):

time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup

real    0m37.832s
user    2m23.620s
sys     0m3.383s

4 cpu thread zstd test at level 19 compression

Code (Text):

time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup

real    4m13.144s
user    16m36.715s
sys     0m3.370s

bump to 8 cpu thread zstd test at level 3 compression

Code (Text):

time tar "-I zstd -3 -T8" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup

real    0m7.918s
user    0m36.137s
sys     0m3.482s

bump to 8 cpu thread zstd test at level 12 compression

Code (Text):

time tar "-I zstd -12 -T8" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup

real    0m31.399s
user    3m35.345s
sys     0m3.627s

bump to 8 cpu thread zstd test at level 19 compression

Code (Text):

time tar "-I zstd -19 -T8" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup

real    3m14.418s
user    24m25.890s
sys     0m4.215s

Speed up on 8 zstd threads wasn't as much compared to 4 zstd threads seeing as Intel Core i7 4790K only really has 4 physical cpu cores and the additional hyperthreaded virtual 4 threads isn't as fast as physical cpu cores.

Compared to popular xz single threaded and pxz multi-threaded compression tools as well

tar + xz took 987.607 seconds

Code (Text):

time tar "-I xz -6" -cf /home/speedtest/svr-setup-xz-lvl6.tar.xz svr-setup                    

real    16m27.607s
user    16m25.924s
sys     0m4.541s

tar + pxz took 221.896 seconds

Code (Text):

time tar "-I pxz -6" -cf /home/speedtest/svr-setup-pxz-lvl6.tar.xz svr-setup

real    3m41.896s
user    25m9.745s
sys     0m10.244s

Comparison chart and table

eva2000 · Dec 20, 2018

Facebook also posted an update on 5 ways Facebook improved compression at scale with Zstandard

Two years ago, Facebook open-sourced Zstandard v1.0, a landmark data compression solution that offers best-in-kind performance. Since then, we’ve released a variety of enhancements and advanced capabilities. While some of the improvements, such as faster decompression speed and stronger compression ratios, are included with each upgrade, others need to be explicitly requested. This post will walk through these new features and how they’ve delivered a second round of benefits at Facebook.

The initial promise of Zstandard (zstd) was that it would allow users to replace their existing data compression implementation (e.g., zlib) for one with significant improvements on all metrics (compression speed, compression ratio, and decompression speed). Once it delivered on that promise, we quickly began to replace zlib with zstd. This was no small undertaking, and there is a long tail of projects to get to. However, through creative refactoring of accretion points, notably folly, we have converted a significant portion in record time (as reported by our internal monitoring system). The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed.

In addition to replacing zlib, zstd has taken over many of the jobs that traditionally relied on fast compression alternatives. Save the fastest ones, which are still served by LZ4, zstd provides a much better ratio (+50 percent). It has also begun to replace the strong compression scenarios previously served by XZ, for the benefit of >10x faster decompression speed. With all these use cases combined, zstd now processes a significant amount of data every day at Facebook.
Click to expand...

Interesting better compression with zstd long range mode

Long range mode is a serial preprocessor which finds large matches and passes the rest to a parallel zstd backend.

Development server backups use the full 2 GB window (--long=31) because it is fine to require decompressors to allocate 2 GB during server recovery. Adding long range mode reduced the full backup size by 16 percent and the diff backup size by 27 percent. Switching to long range mode also reduced the P90 backup time by 16 percent, while the median backup time remained approximately the same. Fbpkg uses only a 128 MB window size (--long) because we can’t allocate too much memory to decompression on production hosts. However, large packages became about 10 percent smaller, and in extreme cases the size was reduced 5x. You can learn more about multithreading and long range mode in the zstd-1.3.4 release notes.
Click to expand...

from zstd 1.3.4 release notes https://github.com/facebook/zstd/releases/tag/v1.3.4
Long Range Mode with Multi-threading
zstd-1.3.2 introduced the long range mode, capable to deduplicate long distance redundancies in a large data stream, a situation typical in backup scenarios for example. But its usage in association with multi-threading was discouraged, due to inefficient use of memory.

zstd-1.3.4 solves this issue, by making long range match finder run in serial mode, like a pre-processor, before passing its result to backend compressors (regular zstd). Memory usage is now bounded to the maximum of the long range window size, and the memory that zstdmt would require without long range matching. As the long range mode runs at about 200 MB/s, depending on the number of cores available, it's possible to tune compression level to match the LRM speed, which becomes the upper limit.
Code:
zstd -T0  -5  --long    file # autodetect threads, level 5, 128 MB window
zstd -T16 -10 --long=31 file # 16 threads, level 10, 2 GB window
Click to expand...
zstd lvl 3 compression 4 threads without long range mode
Code (Text):
time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup

real    0m7.625s
user    0m19.528s
sys     0m3.413s
zstd lvl 3 compression 4 threads long=27
Code (Text):
time tar "-I zstd -3 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl3-long27.tar.zst svr-setup

real    0m13.171s
user    0m25.689s
sys     0m3.338s
zstd lvl 3 compression 4 threads long=31
Code (Text):
time tar "-I zstd -3 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl3-long31.tar.zst svr-setup

real    0m13.317s
user    0m24.447s
sys     0m3.647s
zstd lvl 12 compression 4 threads without long range mode
Code (Text):
time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup

real    0m37.832s
user    2m23.620s
sys     0m3.383s
zstd lvl 12 compression 4 threads long=27
Code (Text):
time tar "-I zstd -12 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl12-long27.tar.zst svr-setup

real    0m30.850s
user    1m59.543s
sys     0m3.604s
zstd lvl 12 compression 4 threads long=31
Code (Text):
time tar "-I zstd -12 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl12-long31.tar.zst svr-setup

real    0m29.371s
user    1m48.642s
sys     0m4.176s
zstd lvl 19 compression 4 threads without long range mode
Code (Text):
time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup

real    4m13.144s
user    16m36.715s
sys     0m3.370s
zstd lvl 19 compression 4 threads long=27
Code (Text):
time tar "-I zstd -19 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl19-long27.tar.zst svr-setup

real    4m13.295s
user    15m33.865s
sys     0m3.877s
zstd lvl 19 compression 4 threads long=31
Code (Text):
time tar "-I zstd -19 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl19-long31.tar.zst svr-setup

real    3m32.687s
user    13m9.800s
sys     0m4.400s
Code (Text):
ls -lah /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
-rw-r--r--   1 root root  911M Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
-rw-r--r--   1 root root  844M Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
-rw-r--r--   1 root root  1.1G Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
-rw-r--r--   1 root root  813M Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
-rw-r--r--   1 root root  760M Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
-rw-r--r--   1 root root  937M Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
-rw-r--r--   1 root root 1023M Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
-rw-r--r--   1 root root  943M Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
-rw-r--r--   1 root root  1.3G Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
Code (Text):
ls -la /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
-rw-r--r--   1 root root  954719263 Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
-rw-r--r--   1 root root  884625200 Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
-rw-r--r--   1 root root 1098039638 Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
-rw-r--r--   1 root root  852163352 Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
-rw-r--r--   1 root root  795986668 Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
-rw-r--r--   1 root root  981577668 Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
-rw-r--r--   1 root root 1072611166 Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
-rw-r--r--   1 root root  988555457 Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
-rw-r--r--   1 root root 1395589259 Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
zstd long range mode improves compression ratio and results in smaller compressed sizes

Then there's Automatic level determination mode

Automatic Level Determination
The last feature introduced for large data streams is automatic level determination (--adapt). Adaptive mode measures the speed of the input and output files, or pipes, and adjusts the compression level to match the bottleneck. This mode can be combined with multithreading and long range mode and finely controlled through lower and upper bounds. Combining all three is perfect for server backups because the algorithm can opportunistically convert wait time due to network congestion into improved compression ratio. In initial experiments, we measured an improvement of approximately 10 percent in compression ratio for equivalent or better transmission time.
Click to expand...

zstd lvl 3 compression 4 threads without long range mode without automatic level determination
Code (Text):
time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup

real    0m7.625s
user    0m19.528s
sys     0m3.413s
zstd lvl 3 compression 4 threads long=27 with automatic level determination
Code (Text):
time tar "-I zstd -3 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl3-long27-adapt.tar.zst svr-setup

real    0m13.063s
user    0m29.756s
sys     0m3.363s
zstd lvl 3 compression 4 threads long=31 with automatic level determination
Code (Text):
time tar "-I zstd -3 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl3-long31-adapt.tar.zst svr-setup

real    0m13.782s
user    0m26.946s
sys     0m3.654s
zstd lvl 12 compression 4 threads without long range mode without automatic level determination
Code (Text):
time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup

real    0m37.832s
user    2m23.620s
sys     0m3.383s
zstd lvl 12 compression 4 threads long=27 with automatic level determination
Code (Text):
time tar "-I zstd -12 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl12-long27-adapt.tar.zst svr-setup

real    0m30.198s
user    1m55.895s
sys     0m3.487s
zstd lvl 12 compression 4 threads long=31 with automatic level determination
Code (Text):
time tar "-I zstd -12 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl12-long31-adapt.tar.zst svr-setup

real    0m27.908s
user    1m42.629s
sys     0m4.037s
zstd lvl 19 compression 4 threads without long range mode without automatic level determination
Code (Text):
time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup

real    4m13.144s
user    16m36.715s
sys     0m3.370s
zstd lvl 19 compression 4 threads long=27 with automatic level determination
Code (Text):
time tar "-I zstd -19 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl19-long27-adapt.tar.zst svr-setup

real    4m16.817s
user    15m54.161s
sys     0m3.906s
zstd lvl 19 compression 4 threads long=31 with automatic level determination
Code (Text):
time tar "-I zstd -19 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl19-long31-adapt.tar.zst svr-setup

real    3m30.615s
user    13m1.289s
sys     0m4.204s
Code (Text):
ls -lah /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
-rw-r--r--   1 root root  911M Dec 19 23:12 svr-setup-zstd-lvl12-long27-adapt.tar.zst
-rw-r--r--   1 root root  911M Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
-rw-r--r--   1 root root  844M Dec 19 23:12 svr-setup-zstd-lvl12-long31-adapt.tar.zst
-rw-r--r--   1 root root  844M Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
-rw-r--r--   1 root root  1.1G Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
-rw-r--r--   1 root root  813M Dec 19 23:17 svr-setup-zstd-lvl19-long27-adapt.tar.zst
-rw-r--r--   1 root root  813M Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
-rw-r--r--   1 root root  760M Dec 19 23:23 svr-setup-zstd-lvl19-long31-adapt.tar.zst
-rw-r--r--   1 root root  760M Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
-rw-r--r--   1 root root  937M Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
-rw-r--r--   1 root root 1020M Dec 19 23:10 svr-setup-zstd-lvl3-long27-adapt.tar.zst
-rw-r--r--   1 root root 1023M Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
-rw-r--r--   1 root root  941M Dec 19 23:11 svr-setup-zstd-lvl3-long31-adapt.tar.zst
-rw-r--r--   1 root root  943M Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
-rw-r--r--   1 root root  1.3G Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
Code (Text):
ls -la /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
-rw-r--r--   1 root root  954949017 Dec 19 23:12 svr-setup-zstd-lvl12-long27-adapt.tar.zst
-rw-r--r--   1 root root  954719263 Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
-rw-r--r--   1 root root  884729008 Dec 19 23:12 svr-setup-zstd-lvl12-long31-adapt.tar.zst
-rw-r--r--   1 root root  884625200 Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
-rw-r--r--   1 root root 1098039638 Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
-rw-r--r--   1 root root  852163352 Dec 19 23:17 svr-setup-zstd-lvl19-long27-adapt.tar.zst
-rw-r--r--   1 root root  852163352 Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
-rw-r--r--   1 root root  795894243 Dec 19 23:23 svr-setup-zstd-lvl19-long31-adapt.tar.zst
-rw-r--r--   1 root root  795986668 Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
-rw-r--r--   1 root root  981577668 Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
-rw-r--r--   1 root root 1068830132 Dec 19 23:10 svr-setup-zstd-lvl3-long27-adapt.tar.zst
-rw-r--r--   1 root root 1072611166 Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
-rw-r--r--   1 root root  986262705 Dec 19 23:11 svr-setup-zstd-lvl3-long31-adapt.tar.zst
-rw-r--r--   1 root root  988555457 Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
-rw-r--r--   1 root root 1395589259 Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
table and charts with --adapt added for zstd 4 cpu thread level 3, 12 and 19 tests

eva2000 · Dec 20, 2018

tar + zstd backup for Centmin Mod Nginx Site Files

Once the custom tar zstd supported RPM and zstd are installed, Centmin Mod users can use this version of tar to backup their Centmin Mod Nginx site's vhost directory and files efficiently with zstd compression - speeding up their backup process time + reducing the size of your compressed backups.

Below is example manual SSH command line commands for both backup and restore of Centmin Mod site Nginx vhost directory and files using tar + zstd.

backup

Backup process involves you running the below SSH command line commands. Only variable you need to edit and adjust is vhostname variable change from domain.com to your site domain name that is located on your server. Full outline of Centmin Mod config file structure is outlined here.
Code (Text):
DT=$(date +"%d%m%y-%H%M%S")
vhostname='domain.com'
backupdir='/home/nginx-backup-sites'
if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
pushd /home/nginx/domains/ && mkdir -p "${backupdir}"
time tar "-I zstd -3 -T0 --long=${long} --adapt" --exclude=log/* --exclude=backup/* -cpf "${backupdir}/${vhostname}-${DT}.tar.zst" ${vhostname}
fs=$(ls -lAh "${backupdir}/${vhostname}-${DT}.tar.zst" | awk '{print $5,$9}') && echo "$fs"
popd && echo "backup ${vhostname} files only complete at ${backupdir}/${vhostname}-${DT}.tar.zst"
1st line assigns to DT variable a date timestamp which is appended to your tar.zst backup file

2nd line for vhostname variable assigns variable for intended domain name you want to backup

3rd line assigns backupdir variable the directory where you want to save your tar.zst backup file

4th line determines which zstd long range method is used depending on how much free memory you have available. So if you have more than 2,500,000 KB free memory, zstd will use long range method with --long=31 for 2GB window. If you have less than 2,500,00 KB free memory, zstd will use long range method with --long=27 for 128MB window.

5th line uses pushd to change into /home/nginx/domains directory and create the backup directory set on 3rd line's backupdir variable

6th line, timed tar + zstd backup and compression which also uses zstd automatic level determination --adapt flag and -T0 meaning use all cpu cores. Tar also excludes contents of /home/nginx/domains/domain.com/log and /home/nginx/domains/domain.com/backup directories

7th line assigns to fs variable size and full path name of resulting tar.zst backup file and prints that out via echo command

8th line uses popd to return you to whatever directory you were in prior to pushd command on line 5 above and prints out completion message and full path to tar.zst backup file

Listing of tar.zst backup compressed files
Code (Text):
ls -lah /home/nginx-backup-sites           
total 32K
drwxr-xr-x   2 root root 4.0K Dec 20 02:03 .
drwxr-xr-x. 31 root root 4.0K Dec 20 01:01 ..
-rw-r--r--   1 root root  12K Dec 20 01:08 domain.com-201218-010827.tar.zst
-rw-r--r--   1 root root  12K Dec 20 01:35 domain.com-201218-013419.tar.zst
restore

Edit and assign to backupfile variable the full path to the backup file you want to restore from the backup commands you ran above.
Code (Text):
DT=$(date +"%d%m%y-%H%M%S")
backupfile='/home/nginx-backup-sites/domain.com-201218-010827.tar.zst'
vhostname='domain.com'
restoredir="/home/nginx/domains/${vhostname}-restored-${DT}"
mkdir -p ${restoredir}
if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
time tar "-I zstd -d -T0 --long=${long}" -xvf ${backupfile} -C ${restoredir}
echo "diff compare restore versus existing directories"
diff -r /home/nginx/domains/${vhostname} ${restoredir}/${vhostname}
echo "restored ${vhostname} files to ${restoredir}"
1st line assigns to backupfile variable the full path to the backup file you want to restore from the backup commands you ran above.

2nd line for vhostname variable assigns variable for intended domain name you intend to restore to

3rd line assigns restoredir variable the directory where the backup tar.zst files' contents will be restored to. You can then compare or sync directory/files with the existing version of the nginx vhost or move the restored files to replace existing version of the nginx vhost

4th line creates the restore directory assigned to restoredir variable if it doesn't exist

5th line determines which zstd long range method is used depending on how much free memory you have available. So if you have more than 2,500,000 KB free memory, zstd will use long range method with --long=31 for 2GB window. If you have less than 2,500,00 KB free memory, zstd will use long range method with --long=27 for 128MB window.

6th line run timed tar decompress and extraction with zstd with all cpu cores and extract to restore directory

7th & 8th lines print out the differences only between restored directory from tar.zst backup and the existing nginx vhost site

9th line, restore completion message

example of 5th line's tar decompress and extraction run
Code (Text):
time tar "-I zstd -d -T0" -xvf ${backupfile} -C ${restoredir}
domain.com/
domain.com/private/
domain.com/backup/
domain.com/public/
domain.com/public/404.html
domain.com/public/401.html
domain.com/public/maintenance.html
domain.com/public/50x.html
domain.com/public/index.html
domain.com/public/503.html
domain.com/public/503.jpg
domain.com/public/403.html
domain.com/public/504.html
domain.com/public/502.html
domain.com/public/500.html
domain.com/log/

real    0m0.003s
user    0m0.000s
sys     0m0.003s
Code (Text):
tree /home/nginx/domains/domain.com-restored-201218-013419/domain.com/
/home/nginx/domains/domain.com-restored-201218-013419/domain.com/
├── backup
├── log
├── private
└── public
   ├── 401.html
   ├── 403.html
   ├── 404.html
   ├── 500.html
   ├── 502.html
   ├── 503.html
   ├── 503.jpg
   ├── 504.html
   ├── 50x.html
   ├── index.html
   └── maintenance.html

4 directories, 11 files
listing in /home/nginx/domains/ with existing nginx vhost directory and restored version
Code (Text):
ls -lah /home/nginx/domains/ | grep ${vhostname}
drwxr-s---   6 nginx nginx 4.0K Dec 20 01:04 domain.com
drwxr-sr-x   3 root  nginx 4.0K Dec 20 01:36 domain.com-restored-201218-013419
differences reported between existing ngixn vhost directory and restored version
Code (Text):
diff -r /home/nginx/domains/${vhostname} ${restoredir}/${vhostname}
Only in /home/nginx/domains/domain.com/log: access.log
Only in /home/nginx/domains/domain.com/log: error.log
Backup All Nginx Domain Files on Centmin Mod

Want to backup all files (excluding mysql databases) for all Nginx sites on Centmin Mod located in /home/nginx/domains ?
Code (Text):
DT=$(date +"%d%m%y-%H%M%S")
backupdir='/home/nginx-backup-sites'
if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
pushd /home/nginx/ && mkdir -p "${backupdir}"
time tar "-I zstd -3 -T0 --long=${long} --adapt" --exclude=log/* --exclude=backup/* --exclude=*-all-restored-*/* -cpvvf "${backupdir}/domains-all-${DT}.tar.zst" domains
fs=$(ls -lAh "${backupdir}/domains-all-${DT}.tar.zst" | awk '{print $5,$9}') && echo "$fs"
popd && echo "backup all Nginx vhost files at /home/nginx/domains complete at ${backupdir}/domains-all-${DT}.tar.zst"
Code (Text):
ls -lah /home/nginx-backup-sites/
total 24K
drwxr-xr-x   2 root root 4.0K Dec 21 06:30 .
drwxr-xr-x. 17 root root 4.0K Dec 21 06:29 ..
-rw-r--r--   1 root root  13K Dec 21 06:30 domains-all-211218-062709.tar.zst
restore
Code (Text):
DT=$(date +"%d%m%y-%H%M%S")
backupfile='/home/nginx-backup-sites/domains-all-211218-062709.tar.zst'
restoredir="/home/nginx/domains/domains-all-restored-${DT}"
mkdir -p ${restoredir}
if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
time tar "-I zstd -d -T0 --long=${long}" -xvf ${backupfile} -C ${restoredir}
echo "diff compare restore versus existing directories"
diff -r /home/nginx/domains/ ${restoredir}/domains
echo "restored /home/nginx/domains files to ${restoredir}"
Code (Text):
tree /home/nginx/domains/domains-all-restored-211218-070634/domains/
/home/nginx/domains/domains-all-restored-211218-070634/domains/
├── demodomain.com
│   ├── backup
│   ├── log
│   ├── private
│   └── public
│       ├── 401.html
│       ├── 403.html
│       ├── 404.html
│       ├── 500.html
│       ├── 502.html
│       ├── 503.html
│       ├── 503.jpg
│       ├── 504.html
│       ├── 50x.html
│       ├── index.html
│       └── maintenance.html
├── http2.domain.com
│   ├── backup
│   ├── log
│   ├── private
│   └── public
│       ├── 401.html
│       ├── 403.html
│       ├── 404.html
│       ├── 500.html
│       ├── 502.html
│       ├── 503.html
│       ├── 503.jpg
│       ├── 504.html
│       ├── 50x.html
│       ├── index.html
│       └── maintenance.html
├── newdomain.com
│   └── backup
└── testdb.com
    ├── backup
    ├── log
    ├── private
    └── public
        ├── 401.html
        ├── 403.html
        ├── 404.html
        ├── 500.html
        ├── 502.html
        ├── 503.html
        ├── 503.jpg
        ├── 504.html
        ├── 50x.html
        ├── index.html
        └── maintenance.html

17 directories, 33 files
Code (Text):
diff -r /home/nginx/domains/ ${restoredir}/domains
Only in /home/nginx/domains/demodomain.com/log: access.log
Only in /home/nginx/domains/demodomain.com/log: error.log
Only in /home/nginx/domains/: domains-all-restored-211218-070634
Only in /home/nginx/domains/http2.domain.com/log: access.log
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181212.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181213.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181214.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181215.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181216.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181217.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181218.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181219.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181220.gz
Only in /home/nginx/domains/http2.domain.com/log: access.log-20181221
Only in /home/nginx/domains/http2.domain.com/log: error.log
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181210.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181211.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181212.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181213.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181214.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181217.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181218.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181219.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181220.gz
Only in /home/nginx/domains/http2.domain.com/log: error.log-20181221
Only in /home/nginx/domains/newdomain.com/backup: mysql
Only in /home/nginx/domains/newdomain.com/backup: mysql-logs
Only in /home/nginx/domains/testdb.com/backup: mysql
Only in /home/nginx/domains/testdb.com/backup: mysql-logs
Only in /home/nginx/domains/testdb.com/log: access.log
Only in /home/nginx/domains/testdb.com/log: error.log

eva2000 · Jan 3, 2019

Official Tar 1.31 stable release was announced with native zstd support so updated my above listed custom Tar RPM for CentOS 6 and 7 64 bit OSes.

tar-1.31 released [stable]

GNU Tar 1.31 Released With Zstd Support - Phoronix

List of changes in this release:

* Fix heap-buffer-overrun with --one-top-level.

Bug introduced with the addition of that option in 1.28.

* Support for zstd compression

New option '--zstd' instructs tar to use zstd as compression program.
When listing, extractng and comparing, zstd compressed archives are
recognized automatically.
When '-a' option is in effect, zstd compression is selected if the
destination archive name ends in '.zst' or '.tzst'.

* The -K option interacts properly with member names given in the command line

Names of members to extract can be specified along with the "-K NAME"
option. In this case, tar will extract NAME and those of named members
that appear in the archive after it, which is consistent with the
semantics of the option.

* Fix CVE-2018-20482

When creating archives with the --sparse option, previous versions of
tar would loop endlessly if a sparse file had been truncated while
being archived.
Click to expand...
Code (Text):
tar --version
tar (GNU tar) 1.31
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Code (Text):
yum -q info tar-zstd-gcc7
Installed Packages
Name        : tar-zstd-gcc7
Arch        : x86_64
Version     : 1.31
Release     : 1.el7
Size        : 3.5 M
Repo        : installed
Summary     : tar 1.31 zstd supported for centminmod.com LEMP stack
URL         : https://centminmod.com
License     : unknown
Description : tar 1.31 zstd supported for centminmod.com LEMP stack
Code (Text):
yum -q info tar-zstd-gcc7
Installed Packages
Name        : tar-zstd-gcc7
Arch        : x86_64
Version     : 1.31
Release     : 1.el6
Size        : 3.5 M
Repo        : installed
From repo   : /tar-zstd-gcc7-1.31-1.el6.x86_64
Summary     : tar 1.31 zstd supported for centminmod.com LEMP stack
URL         : https://centminmod.com
License     : unknown
Description : tar 1.31 zstd supported for centminmod.com LEMP stack

eva2000 · Jan 5, 2019

tar 1.31 + zstd compression & decompression benchmarks

I wrote a quick tar-zstd-test.sh script to test out my tar 1.31 custom RPM which has native zstd compression support and ran the script against uncompressed silesia corpus to compare tar 1.31 pairing with gzip, pigz, zstd, xz and pxz compression algorithms. Test on Intel Core i7 4790K 4C/8T 32GB 2x 240GB Samsung PM863 SSD raid 1 server running CentOS 7.6 64bit with Centmin Mod 123.09beta01 LEMP stack with tar 1.31 installed.

tar-zstd-test.sh usage

Code (Text):

./tar-zstd-test.sh

./tar-zstd-test.sh {prep|all|gzip|pigz|zstd|xz|pxz}

Testing tar with gzip vs pigz vs zstd vs xz vs pxz compression algorithms

Code (Text):

gzip -V
gzip 1.5

pigz -V
pigz 2.4

zstd -V
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

xz -V
xz (XZ Utils) 5.2.2
liblzma 5.2.2

pxz -V
Parallel PXZ 4.999.9beta (build 20160426)
xz (XZ Utils) 5.2.2
liblzma 5.2.2

gzip level 1 to 9

Code (Text):

./tar-zstd-test.sh gzip
### gzip test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I gzip -1 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 3.51s cpu: 101% maxmem: 1388 KB cswaits: 6470
 compression ratio: 2.6976 211942676 / 78565167 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.33s cpu: 110% maxmem: 1320 KB cswaits: 6472
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I gzip -2 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 3.67s cpu: 101% maxmem: 1388 KB cswaits: 6469
 compression ratio: 2.7780 211942676 / 76290537 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.30s cpu: 110% maxmem: 1320 KB cswaits: 6473
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I gzip -3 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 4.28s cpu: 101% maxmem: 1388 KB cswaits: 6470
 compression ratio: 2.8546 211942676 / 74243600 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.40s cpu: 100% maxmem: 1320 KB cswaits: 6475
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I gzip -4 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 4.60s cpu: 101% maxmem: 1384 KB cswaits: 6470
 compression ratio: 2.9325 211942676 / 72271408 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.36s cpu: 104% maxmem: 1312 KB cswaits: 6475
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I gzip -5 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 5.78s cpu: 101% maxmem: 1388 KB cswaits: 6470
 compression ratio: 3.0099 211942676 / 70413987 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.26s cpu: 110% maxmem: 1316 KB cswaits: 6474
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I gzip -6 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 7.74s cpu: 100% maxmem: 1388 KB cswaits: 6469
 compression ratio: 3.0508 211942676 / 69470849 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.25s cpu: 111% maxmem: 1320 KB cswaits: 6471
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I gzip -7 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 9.08s cpu: 100% maxmem: 1388 KB cswaits: 6470
 compression ratio: 3.0634 211942676 / 69183567 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.26s cpu: 110% maxmem: 1316 KB cswaits: 6474
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I gzip -8 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 13.43s cpu: 100% maxmem: 1388 KB cswaits: 6470
 compression ratio: 3.0745 211942676 / 68934484 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.33s cpu: 103% maxmem: 1320 KB cswaits: 6478
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I gzip -9 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 16.71s cpu: 100% maxmem: 1388 KB cswaits: 6470
 compression ratio: 3.0765 211942676 / 68890462 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.24s cpu: 111% maxmem: 1316 KB cswaits: 6473

pigz level 1 to 9

notice cpu utilisation for pigz almost used all 8 cpu threads on Intel Core i7 4790K cpu as such it ended up faster than zstd compression but at a lower compression ratio

Code (Text):

./tar-zstd-test.sh pigz
### pigz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I pigz -1 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.63s cpu: 725% maxmem: 7656 KB cswaits: 9902
 compression ratio: 2.7085 211942676 / 78248308 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.66s cpu: 155% maxmem: 1320 KB cswaits: 21809
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I pigz -2 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.67s cpu: 732% maxmem: 7604 KB cswaits: 11371
 compression ratio: 2.7903 211942676 / 75956751 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.65s cpu: 157% maxmem: 1316 KB cswaits: 21719
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I pigz -3 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.78s cpu: 742% maxmem: 7764 KB cswaits: 10880
 compression ratio: 2.8702 211942676 / 73840861 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.63s cpu: 157% maxmem: 1316 KB cswaits: 21648
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I pigz -4 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.84s cpu: 748% maxmem: 7688 KB cswaits: 10766
 compression ratio: 2.9413 211942676 / 72057108 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.75s cpu: 136% maxmem: 1316 KB cswaits: 21622
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I pigz -5 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 1.06s cpu: 771% maxmem: 7656 KB cswaits: 11742
 compression ratio: 3.0187 211942676 / 70208692 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.64s cpu: 158% maxmem: 1320 KB cswaits: 21613
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I pigz -6 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 1.36s cpu: 794% maxmem: 7512 KB cswaits: 11518
 compression ratio: 3.0598 211942676 / 69266472 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.61s cpu: 158% maxmem: 1316 KB cswaits: 21603
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I pigz -7 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 1.67s cpu: 761% maxmem: 7808 KB cswaits: 13123
 compression ratio: 3.0724 211942676 / 68982378 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.61s cpu: 159% maxmem: 1316 KB cswaits: 21588
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I pigz -8 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.24s cpu: 791% maxmem: 7500 KB cswaits: 11178
 compression ratio: 3.0835 211942676 / 68733463 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.61s cpu: 160% maxmem: 1316 KB cswaits: 21490
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I pigz -9 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.80s cpu: 769% maxmem: 7656 KB cswaits: 11361
 compression ratio: 3.0855 211942676 / 68688684 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.61s cpu: 159% maxmem: 1316 KB cswaits: 21563

zstd level 1 to 19

notice cpu utilisation for zstd does not seem to be using all 8 cpu threads on Intel Core i7 4790K cpu and after zstd level 13 compression, cpu utilisation falls.
zstd level 11 compression times at 7.37s were equivalent to gzip level 6 default 7.74s but had higher compression ratio 3.6035 vs 3.0508
parallel xz, pxz level 2 compression times at 6.56s were faster than zstd level 11 at 7.37s and gzip level 6 at 7.74s and had a higher compression ratio at 3.7188 than zstd level 11 at 3.6035 and gzip level 6 at 3.0508
however pxz level 2 decompression times were slower at 2.93s compared to zstd level 11 at 1.93s

Code (Text):

./tar-zstd-test.sh zstd                      
### zstd test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I zstd -1 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 2.92s cpu: 308% maxmem: 162036 KB cswaits: 8492
 compression ratio: 2.9241 211942676 / 72479417 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.97s cpu: 100% maxmem: 132464 KB cswaits: 12855
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I zstd -2 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 2.83s cpu: 321% maxmem: 166780 KB cswaits: 8485
 compression ratio: 3.0652 211942676 / 69143240 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.03s cpu: 104% maxmem: 132460 KB cswaits: 12854
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I zstd -3 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 3.18s cpu: 427% maxmem: 172560 KB cswaits: 8327
 compression ratio: 3.1925 211942676 / 66387130 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.22s cpu: 101% maxmem: 132436 KB cswaits: 12858
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I zstd -4 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 3.48s cpu: 453% maxmem: 196756 KB cswaits: 7915
 compression ratio: 3.2549 211942676 / 65114223 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.15s cpu: 103% maxmem: 132448 KB cswaits: 12887
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I zstd -5 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 3.76s cpu: 517% maxmem: 204400 KB cswaits: 7353
 compression ratio: 3.3322 211942676 / 63603510 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.16s cpu: 104% maxmem: 132444 KB cswaits: 12895
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I zstd -6 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 4.41s cpu: 490% maxmem: 235220 KB cswaits: 8031
 compression ratio: 3.3960 211942676 / 62409187 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.12s cpu: 104% maxmem: 132452 KB cswaits: 12913
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I zstd -7 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 4.68s cpu: 551% maxmem: 237472 KB cswaits: 7654
 compression ratio: 3.4796 211942676 / 60909269 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.01s cpu: 104% maxmem: 132448 KB cswaits: 12918
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I zstd -8 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 5.14s cpu: 571% maxmem: 238724 KB cswaits: 7566
 compression ratio: 3.5162 211942676 / 60275135 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.97s cpu: 104% maxmem: 132440 KB cswaits: 12911
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I zstd -9 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 5.58s cpu: 614% maxmem: 251772 KB cswaits: 7241
 compression ratio: 3.5482 211942676 / 59732045 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.94s cpu: 104% maxmem: 132440 KB cswaits: 12911
------------------------------------------------------------------------------
 [tar compress lvl: 10 ] tar "-I zstd -10 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 6.30s cpu: 593% maxmem: 328888 KB cswaits: 7091
 compression ratio: 3.5797 211942676 / 59206641 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.95s cpu: 105% maxmem: 132532 KB cswaits: 12918
------------------------------------------------------------------------------
 [tar compress lvl: 11 ] tar "-I zstd -11 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 7.37s cpu: 536% maxmem: 473992 KB cswaits: 8304
 compression ratio: 3.6035 211942676 / 58815270 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 104% maxmem: 132496 KB cswaits: 12931
------------------------------------------------------------------------------
 [tar compress lvl: 12 ] tar "-I zstd -12 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 9.07s cpu: 566% maxmem: 473572 KB cswaits: 8539
 compression ratio: 3.6289 211942676 / 58403470 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.92s cpu: 105% maxmem: 132500 KB cswaits: 12940
------------------------------------------------------------------------------
 [tar compress lvl: 13 ] tar "-I zstd -13 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 11.80s cpu: 575% maxmem: 473004 KB cswaits: 7580
 compression ratio: 3.6661 211942676 / 57810870 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.94s cpu: 101% maxmem: 132500 KB cswaits: 12933
------------------------------------------------------------------------------
 [tar compress lvl: 14 ] tar "-I zstd -14 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 18.68s cpu: 360% maxmem: 520832 KB cswaits: 8783
 compression ratio: 3.6990 211942676 / 57297060 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.84s cpu: 105% maxmem: 132504 KB cswaits: 12935
------------------------------------------------------------------------------
 [tar compress lvl: 15 ] tar "-I zstd -15 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 42.39s cpu: 151% maxmem: 470792 KB cswaits: 8790
 compression ratio: 3.7293 211942676 / 56831267 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.84s cpu: 105% maxmem: 132564 KB cswaits: 12934
------------------------------------------------------------------------------
 [tar compress lvl: 16 ] tar "-I zstd -16 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 32.66s cpu: 346% maxmem: 437868 KB cswaits: 8521
 compression ratio: 3.8056 211942676 / 55690893 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.86s cpu: 105% maxmem: 132528 KB cswaits: 12935
------------------------------------------------------------------------------
 [tar compress lvl: 17 ] tar "-I zstd -17 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 68.33s cpu: 163% maxmem: 420208 KB cswaits: 9269
 compression ratio: 3.8555 211942676 / 54971383 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.88s cpu: 105% maxmem: 132564 KB cswaits: 12940
------------------------------------------------------------------------------
 [tar compress lvl: 18 ] tar "-I zstd -18 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 106.88s cpu: 156% maxmem: 420952 KB cswaits: 9103
 compression ratio: 3.9234 211942676 / 54019608 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 104% maxmem: 132556 KB cswaits: 12938
------------------------------------------------------------------------------
 [tar compress lvl: 19 ] tar "-I zstd -19 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 158.96s cpu: 147% maxmem: 434764 KB cswaits: 8389
 compression ratio: 3.9783 211942676 / 53273587 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.95s cpu: 104% maxmem: 132572 KB cswaits: 12938

xz level 1 to 9

Code (Text):

./tar-zstd-test.sh xz
### xz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I xz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 11.94s cpu: 100% maxmem: 9616 KB cswaits: 25870
 compression ratio: 3.6282 211942676 / 58414444 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.06s cpu: 105% maxmem: 2036 KB cswaits: 25873
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I xz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 18.73s cpu: 100% maxmem: 17296 KB cswaits: 25866
 compression ratio: 3.7371 211942676 / 56712980 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.00s cpu: 103% maxmem: 3060 KB cswaits: 25834
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I xz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 30.85s cpu: 100% maxmem: 32652 KB cswaits: 25870
 compression ratio: 3.8016 211942676 / 55749668 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.84s cpu: 106% maxmem: 5104 KB cswaits: 25869
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I xz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 48.43s cpu: 100% maxmem: 49228 KB cswaits: 25872
 compression ratio: 4.0672 211942676 / 52109368 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.82s cpu: 106% maxmem: 5104 KB cswaits: 25861
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I xz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 64.46s cpu: 100% maxmem: 96328 KB cswaits: 25875
 compression ratio: 4.2401 211942676 / 49984712 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.74s cpu: 106% maxmem: 9204 KB cswaits: 25859
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I xz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 75.01s cpu: 100% maxmem: 96332 KB cswaits: 25874
 compression ratio: 4.3044 211942676 / 49237796 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.70s cpu: 106% maxmem: 9200 KB cswaits: 25861
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I xz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 79.00s cpu: 100% maxmem: 190540 KB cswaits: 25876
 compression ratio: 4.3297 211942676 / 48950732 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.73s cpu: 107% maxmem: 17396 KB cswaits: 25867
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I xz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 82.92s cpu: 100% maxmem: 378952 KB cswaits: 25877
 compression ratio: 4.3472 211942676 / 48752848 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.82s cpu: 106% maxmem: 33780 KB cswaits: 25836
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I xz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 84.48s cpu: 100% maxmem: 690248 KB cswaits: 25874
 compression ratio: 4.3500 211942676 / 48721676 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.74s cpu: 106% maxmem: 66548 KB cswaits: 25847

pxz level 1 to 9

Code (Text):

./tar-zstd-test.sh pxz
### pxz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I pxz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 4.29s cpu: 526% maxmem: 94300 KB cswaits: 4968
 compression ratio: 3.6086 211942676 / 58731740 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.17s cpu: 103% maxmem: 2032 KB cswaits: 25846
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I pxz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 6.56s cpu: 536% maxmem: 180152 KB cswaits: 4071
 compression ratio: 3.7188 211942676 / 56992000 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.93s cpu: 106% maxmem: 3056 KB cswaits: 25862
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I pxz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 10.49s cpu: 538% maxmem: 351176 KB cswaits: 2253
 compression ratio: 3.7820 211942676 / 56039192 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.86s cpu: 106% maxmem: 5108 KB cswaits: 25858
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I pxz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 14.48s cpu: 514% maxmem: 481120 KB cswaits: 3044
 compression ratio: 4.0367 211942676 / 52503612 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.96s cpu: 103% maxmem: 5108 KB cswaits: 25853
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I pxz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 18.11s cpu: 539% maxmem: 924368 KB cswaits: 341
 compression ratio: 4.2126 211942676 / 50310532 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.84s cpu: 103% maxmem: 9204 KB cswaits: 25837
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I pxz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 20.61s cpu: 555% maxmem: 952936 KB cswaits: 434
 compression ratio: 4.2753 211942676 / 49573244 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.98s cpu: 98% maxmem: 9200 KB cswaits: 25841
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I pxz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 27.15s cpu: 354% maxmem: 965744 KB cswaits: 127
 compression ratio: 4.3057 211942676 / 49223036 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.73s cpu: 106% maxmem: 17392 KB cswaits: 25866
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I pxz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 47.40s cpu: 181% maxmem: 964016 KB cswaits: 88
 compression ratio: 4.3316 211942676 / 48929028 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.74s cpu: 106% maxmem: 33780 KB cswaits: 25861
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I pxz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 80.17s cpu: 104% maxmem: 897328 KB cswaits: 62
 compression ratio: 4.3415 211942676 / 48816968 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.77s cpu: 107% maxmem: 66544 KB cswaits: 25861

Updated benchmarks

I noticed zstd with -T0 --long --rsyncable benchmarks above were slower than pigz multi-threaded gzip with less cpu utilisation as well. So decided to find which zstd options were causing the slowest and re-run tests with --rsyncable disabled as well as with zstd -8 with and without --long. It seems for silesia corpus tar compression tests, zstd with -T0 was using less cpu tilisation than -T8 and --long was the culprit slowing zstd compression. The --rsyncable option for gzip, pigz slightly slowed down compression times but for zstd --rsyncable had a more dramatic slow down for compression times/speed.

eva2000 · Jan 11, 2019

tar + zstd versus tar alone

Discussion came up about how disk space is cheap these days so backups can be done uncompressed so zstd isn't as useful. So decided to test that theory with tar v1.31 + zstd 1.3.8 versus tar only without compression and check speeds and resulting tar.zst compressed and tar uncompressed file sizes.

Testing on low end ImpactVPS 2 cpu KVM VPS with Intel Xeon Silver 4114 Skylake cpu
Code (Text):
lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
Stepping:              4
CPU MHz:               2199.992
BogoMIPS:              4399.98
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 arat spec_ctrl
All tests to be done using tar v1.31 custom RPM with native zstd support
Code (Text):
tar --version    
tar (GNU tar) 1.31
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
First the /svr-setup directory to be tar'd up is 1.70GB in size
Code (Text):
du -sh /svr-setup/
1.7G    /svr-setup/
Timed tar without compression speed at 27.582 seconds
Code (Text):
time tar cf svr-setup.tar svr-setup/

real    0m27.582s
user    0m1.190s
sys     0m10.934s
resulting in /svr-setup.tar file 1.60GB in size
Code (Text):
ls -lah /svr-setup.tar
-rw-r--r-- 1 root root 1.6G Jan 10 18:18 /svr-setup.tar
tar with zstd level 3 default compression single threaded default
Code (Text):
time tar cfa svr-setup.tar.zst svr-setup

real    1m58.883s
user    1m58.203s
sys     0m7.051s
resulting in 644MB compressed svr-setup.tar.zst file
Code (Text):
ls -lah /svr-setup.tar.zst          
-rw-r--r-- 1 root root 664M Jan 10 18:22 /svr-setup.tar.zst
now test tar v1.31 with zstd level 1 compression 2 cores = 44.765s
Code (Text):
time tar "-I zstd -1 -T2" -cf svr-setup.tar.zst svr-setup      

real    0m44.765s
user    1m9.498s
sys     0m6.019s
724MB compressed svr-setup.tar.zst file
Code (Text):
ls -lah /svr-setup.tar.zst                                    
-rw-r--r-- 1 root root 724M Jan 10 18:32 /svr-setup.tar.zst
zstd v1.3.8 also has negative compression levels which sacrifice compression ratio size for speed.

tar v1.31 with zstd level -1 compression 2 cores = 38.634s getting closer to tar no-compress speeds
Code (Text):
time tar "-I zstd --fast=1 -T2" -cf svr-setup.tar.zst svr-setup

real    0m38.634s
user    0m57.896s
sys     0m6.269s
but file size for compressed svr-setup.tar.zst is 781MB much smaller than tar no-compress 1.60GB size
Code (Text):
ls -lah /svr-setup.tar.zst                                    
-rw-r--r-- 1 root root 781M Jan 10 18:34 /svr-setup.tar.zst
tar v1.31 with zstd level -2 compression 2 cores at 32.454s
Code (Text):
time tar "-I zstd --fast=2 -T2" -cf svr-setup.tar.zst svr-setup

real    0m32.454s
user    0m48.458s
sys     0m5.587s
compressed svr-setup.tar.zst is 834MB
Code (Text):
ls -lah /svr-setup.tar.zst                                    
-rw-r--r-- 1 root root 834M Jan 10 18:35 /svr-setup.tar.zst
tar v1.31 with zstd level -3 compression 2 cores at 29.190s
Code (Text):
time tar "-I zstd --fast=3 -T2" -cf svr-setup.tar.zst svr-setup

real    0m29.190s
user    0m42.845s
sys     0m4.946s
compressed svr-setup.tar.zst is 868MB
Code (Text):
ls -lah /svr-setup.tar.zst                                    
-rw-r--r-- 1 root root 868M Jan 10 18:36 /svr-setup.tar.zst
tar v1.31 with zstd level -4 compression 2 cores at 27.461s which is faster than tar no-compress time of 27.582s
Code (Text):
time tar "-I zstd --fast=4 -T2" -cf svr-setup.tar.zst svr-setup

real    0m27.461s
user    0m39.677s
sys     0m4.809s
compressed svr-setup.tar.zst is 908MB smaller than tar no-compress size of 1.60GB !
Code (Text):
ls -lah /svr-setup.tar.zst                                    
-rw-r--r-- 1 root root 908M Jan 10 18:37 /svr-setup.tar.zst
So yes it's possible for tar + zstd to compress files at speed close to tar uncompressed sizes when using zstd 1.3.8+ with negative compression levels. In this case it took zstd -4 compression level to achieve tar + zstd compression speeds faster than tar uncompressed

Also it would be even faster if you had more cpu cores to work with and set -TX where X is number of cpu cores. This server only had 2 cpu threads.

Edit: now compare that tar + gzip level 1 & 6 and tar + pigz level 1 & 6 tests

pigz multi-threaded gzip default level 6 tar
Code (Text):
time tar "-I pigz" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz                   

real    1m8.257s
user    1m56.274s
sys     0m7.501s
-rw-r--r-- 1 root root 692M Jan 11 00:47 /svr-setup.tar.gz
pigz multi-threaded gzip default level 1 tar
Code (Text):
time tar "-I pigz -1" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz

real    0m34.844s
user    0m53.258s
sys     0m4.954s
-rw-r--r-- 1 root root 742M Jan 11 00:48 /svr-setup.tar.gz
gzip single threaded default level 6 tar
Code (Text):
time tar "-I gzip" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz       

real    1m58.683s
user    1m55.101s
sys     0m7.383s
-rw-r--r-- 1 root root 692M Jan 11 00:51 /svr-setup.tar.gz
gzip single threaded level 1 tar
Code (Text):
time tar "-I gzip -1" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz

real    0m57.323s
user    0m53.631s
sys     0m6.508s
-rw-r--r-- 1 root root 744M Jan 11 00:52 /svr-setup.tar.gz

eva2000 · Jan 12, 2019

tar 1.31 + zstd compression & decompression benchmarks part 2

I wrote a quick tar-zstd-test.sh script to test out my tar 1.31 custom RPM which has native zstd compression support and ran the script against uncompressed silesia corpus to compare tar 1.31 pairing with gzip, pigz, zstd, xz, pxz & lz4 compression algorithms. I have since added pigz level 11 for zopfli compression and added all 32 levels of compression for zstd - from negative levels (-10 to -1) which focus on compression speed, normal levels 1 to 19 and then the 3 ultra levels 20 to 22 which focus on better compression ratios.

Test System

OVH MC-32 Intel Core i7 4790K
32GB Memory
2x240GB SSD Samsung PM863 SSD raid 1
250Mbit Network Bandwidth
CentOS 7.6 64bit
Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.8, MariaDB 10.1.37 MySQL, + CSF Firewall
BHS, Canada
tar 1.31 installed

tar-zstd-test.sh usage

Code (Text):

./tar-zstd-test.sh

./tar-zstd-test.sh {prep|all|gzip|pigz|zstd|xz|pxz|lz4}

Testing tar with gzip vs pigz vs zstd vs xz vs pxz vs lz4 compression algorithms

Code (Text):

gzip -V
gzip 1.5

pigz -V
pigz 2.4

zstd -V
*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

xz -V
xz (XZ Utils) 5.2.2
liblzma 5.2.2

pxz -V
Parallel PXZ 4.999.9beta (build 20160426)
xz (XZ Utils) 5.2.2
liblzma 5.2.2

lz4 -V
*** LZ4 command line interface 64-bits v1.8.3, by Yann Collet ***

I will be running with all flag to test all compression algorithms in same session.

Code (Text):

./tar-zstd-test.sh all

Also to compare with tar uncompressed times, here's a tar uncompressed run

Code (Text):

tar -cf /home/tar-zstd/silesia.tar silesia
real: 0.38s user: 0.01s sys: 0.11s cpu: 31% maxmem: 1308 KB cswaits: 43

ls -lah /home/tar-zstd/silesia.tar
-rw-r--r-- 1 root root 203M Jan 11 20:44 /home/tar-zstd/silesia.tar

tar -xf /home/tar-zstd/silesia.tar -C /home/tar-zstd-extract
real: 0.39s user: 0.00s sys: 0.15s cpu: 39% maxmem: 1312 KB cswaits: 22

tar compression results

Code (Text):

./tar-zstd-test.sh all                            
### gzip test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I gzip -1" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.55s cpu: 102% maxmem: 1392 KB cswaits: 6470
 [compress speed:] 79.26 MB/s
 compression ratio: 2.7394 211942676 / 77365465 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.13s cpu: 112% maxmem: 1320 KB cswaits: 6472
 [decompress speed:] 178.87 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I gzip -2" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.76s cpu: 102% maxmem: 1392 KB cswaits: 6469
 [compress speed:] 73.23 MB/s
 compression ratio: 2.8216 211942676 / 75113751 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.10s cpu: 112% maxmem: 1320 KB cswaits: 6471
 [decompress speed:] 183.74 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I gzip -3" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 3.45s cpu: 101% maxmem: 1392 KB cswaits: 6470
 [compress speed:] 58.58 MB/s
 compression ratio: 2.9013 211942676 / 73050277 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.09s cpu: 114% maxmem: 1316 KB cswaits: 6473
 [decompress speed:] 185.43 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I gzip -4" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 3.57s cpu: 101% maxmem: 1392 KB cswaits: 6468
 [compress speed:] 56.61 MB/s
 compression ratio: 2.9826 211942676 / 71057874 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.08s cpu: 113% maxmem: 1320 KB cswaits: 6472
 [decompress speed:] 187.15 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I gzip -5" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 4.77s cpu: 101% maxmem: 1392 KB cswaits: 6469
 [compress speed:] 42.37 MB/s
 compression ratio: 3.0648 211942676 / 69153440 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.08s cpu: 112% maxmem: 1320 KB cswaits: 6472
 [decompress speed:] 187.15 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I gzip -6" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 6.66s cpu: 100% maxmem: 1392 KB cswaits: 6469
 [compress speed:] 30.34 MB/s
 compression ratio: 3.1066 211942676 / 68222450 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.04s cpu: 113% maxmem: 1320 KB cswaits: 6473
 [decompress speed:] 194.35 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I gzip -7" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 8.04s cpu: 100% maxmem: 1392 KB cswaits: 6469
 [compress speed:] 25.13 MB/s
 compression ratio: 3.1195 211942676 / 67941131 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.07s cpu: 115% maxmem: 1316 KB cswaits: 6473
 [decompress speed:] 188.90 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I gzip -8" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 12.36s cpu: 100% maxmem: 1388 KB cswaits: 6471
 [compress speed:] 16.35 MB/s
 compression ratio: 3.1308 211942676 / 67693882 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.10s cpu: 113% maxmem: 1316 KB cswaits: 6473
 [decompress speed:] 183.74 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I gzip -9" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 15.72s cpu: 100% maxmem: 1388 KB cswaits: 6470
 [compress speed:] 12.85 MB/s
 compression ratio: 3.1329 211942676 / 67649708 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 1.03s cpu: 113% maxmem: 1316 KB cswaits: 6473
 [decompress speed:] 196.23 MB/s
### pigz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I pigz -1" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.50s cpu: 770% maxmem: 7676 KB cswaits: 11309
 [compress speed:] 404.24 MB/s
 compression ratio: 2.7463 211942676 / 77171874 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.59s cpu: 163% maxmem: 1320 KB cswaits: 21748
 [decompress speed:] 342.58 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I pigz -2" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.54s cpu: 782% maxmem: 7760 KB cswaits: 11187
 [compress speed:] 374.30 MB/s
 compression ratio: 2.8298 211942676 / 74894231 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.58s cpu: 163% maxmem: 1320 KB cswaits: 21686
 [decompress speed:] 348.49 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I pigz -3" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.66s cpu: 779% maxmem: 7652 KB cswaits: 13136
 [compress speed:] 306.24 MB/s
 compression ratio: 2.9111 211942676 / 72802779 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.60s cpu: 165% maxmem: 1320 KB cswaits: 21608
 [decompress speed:] 336.87 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I pigz -4" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.72s cpu: 786% maxmem: 7640 KB cswaits: 11574
 [compress speed:] 280.72 MB/s
 compression ratio: 2.9848 211942676 / 71005642 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.56s cpu: 165% maxmem: 1316 KB cswaits: 21569
 [decompress speed:] 360.93 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I pigz -5" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 0.93s cpu: 793% maxmem: 7468 KB cswaits: 11783
 [compress speed:] 217.33 MB/s
 compression ratio: 3.0642 211942676 / 69167137 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.58s cpu: 164% maxmem: 1320 KB cswaits: 21566
 [decompress speed:] 348.49 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I pigz -6" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 1.30s cpu: 764% maxmem: 7640 KB cswaits: 15437
 [compress speed:] 155.48 MB/s
 compression ratio: 3.1061 211942676 / 68234196 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.55s cpu: 167% maxmem: 1320 KB cswaits: 21511
 [decompress speed:] 367.49 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I pigz -7" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 1.49s cpu: 792% maxmem: 7480 KB cswaits: 12014
 [compress speed:] 135.65 MB/s
 compression ratio: 3.1192 211942676 / 67946718 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.55s cpu: 166% maxmem: 1320 KB cswaits: 21540
 [decompress speed:] 367.49 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I pigz -8" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.16s cpu: 788% maxmem: 7684 KB cswaits: 12500
 [compress speed:] 93.57 MB/s
 compression ratio: 3.1306 211942676 / 67699404 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.54s cpu: 170% maxmem: 1316 KB cswaits: 21425
 [decompress speed:] 374.30 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I pigz -9" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 2.69s cpu: 784% maxmem: 7736 KB cswaits: 11938
 [compress speed:] 75.13 MB/s
 compression ratio: 3.1327 211942676 / 67653907 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.54s cpu: 167% maxmem: 1320 KB cswaits: 21536
 [decompress speed:] 374.30 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 11 ] tar "-I pigz -11" -cf /home/tar-zstd/silesia.tar.gz silesia
 [compress stats:] real: 237.85s cpu: 789% maxmem: 55756 KB cswaits: 12005
 [compress speed:] .84 MB/s
 compression ratio: 3.2845 211942676 / 64527580 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
 [decompress stats:] real: 0.57s cpu: 168% maxmem: 1320 KB cswaits: 21216
 [decompress speed:] 354.60 MB/s
### zstd test
------------------------------------------------------------------------------
 [tar compress lvl: -10 ] tar "-I zstd --fast=10 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.49s cpu: 506% maxmem: 44516 KB cswaits: 8121
 [compress speed:] 412.49 MB/s
 compression ratio: 1.7824 211942676 / 118907221 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.21s cpu: 100% maxmem: 1856 KB cswaits: 12944
 [decompress speed:] 167.04 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -9 ] tar "-I zstd --fast=9 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.42s cpu: 618% maxmem: 43712 KB cswaits: 7743
 [compress speed:] 481.24 MB/s
 compression ratio: 1.8016 211942676 / 117639597 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.12s cpu: 103% maxmem: 1856 KB cswaits: 12943
 [decompress speed:] 180.46 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -8 ] tar "-I zstd --fast=8 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.43s cpu: 624% maxmem: 43912 KB cswaits: 7750
 [compress speed:] 470.05 MB/s
 compression ratio: 1.8477 211942676 / 114706049 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.06s cpu: 109% maxmem: 1848 KB cswaits: 12938
 [decompress speed:] 190.68 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -7 ] tar "-I zstd --fast=7 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.45s cpu: 622% maxmem: 44420 KB cswaits: 8220
 [compress speed:] 449.16 MB/s
 compression ratio: 1.8675 211942676 / 113488957 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.11s cpu: 109% maxmem: 1848 KB cswaits: 12925
 [decompress speed:] 182.09 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -6 ] tar "-I zstd --fast=6 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.54s cpu: 533% maxmem: 43236 KB cswaits: 7497
 [compress speed:] 374.30 MB/s
 compression ratio: 1.9370 211942676 / 109417899 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.14s cpu: 108% maxmem: 1848 KB cswaits: 12939
 [decompress speed:] 177.30 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -5 ] tar "-I zstd --fast=5 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.48s cpu: 632% maxmem: 44048 KB cswaits: 7772
 [compress speed:] 421.09 MB/s
 compression ratio: 1.9947 211942676 / 106252061 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.17s cpu: 108% maxmem: 1844 KB cswaits: 12939
 [decompress speed:] 172.75 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -4 ] tar "-I zstd --fast=4 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.51s cpu: 632% maxmem: 42444 KB cswaits: 7522
 [compress speed:] 396.32 MB/s
 compression ratio: 2.0672 211942676 / 102525133 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.21s cpu: 107% maxmem: 1840 KB cswaits: 12936
 [decompress speed:] 167.04 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -3 ] tar "-I zstd --fast=3 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.54s cpu: 669% maxmem: 41696 KB cswaits: 7356
 [compress speed:] 374.30 MB/s
 compression ratio: 2.1507 211942676 / 98541355 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.28s cpu: 107% maxmem: 1836 KB cswaits: 12938
 [decompress speed:] 157.90 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -2 ] tar "-I zstd --fast=2 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.60s cpu: 677% maxmem: 43516 KB cswaits: 8361
 [compress speed:] 336.87 MB/s
 compression ratio: 2.2641 211942676 / 93609955 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.38s cpu: 107% maxmem: 1832 KB cswaits: 12937
 [decompress speed:] 146.46 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: -1 ] tar "-I zstd --fast=1 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.76s cpu: 645% maxmem: 42996 KB cswaits: 7773
 [compress speed:] 265.95 MB/s
 compression ratio: 2.4296 211942676 / 87232776 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.43s cpu: 106% maxmem: 1824 KB cswaits: 12936
 [decompress speed:] 141.34 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I zstd -1 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.82s cpu: 691% maxmem: 40776 KB cswaits: 7889
 [compress speed:] 246.49 MB/s
 compression ratio: 2.8766 211942676 / 73678095 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.89s cpu: 105% maxmem: 1996 KB cswaits: 12931
 [decompress speed:] 106.94 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I zstd -2 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 0.91s cpu: 666% maxmem: 73532 KB cswaits: 8217
 [compress speed:] 222.11 MB/s
 compression ratio: 3.0314 211942676 / 69914803 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.16s cpu: 101% maxmem: 2508 KB cswaits: 12941
 [decompress speed:] 93.57 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I zstd -3 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 1.69s cpu: 626% maxmem: 137708 KB cswaits: 8586
 [compress speed:] 119.60 MB/s
 compression ratio: 3.1722 211942676 / 66811326 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.11s cpu: 105% maxmem: 3504 KB cswaits: 12935
 [decompress speed:] 95.79 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I zstd -4 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 2.05s cpu: 591% maxmem: 147360 KB cswaits: 7922
 [compress speed:] 98.59 MB/s
 compression ratio: 3.2259 211942676 / 65699482 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.10s cpu: 104% maxmem: 3500 KB cswaits: 12940
 [decompress speed:] 96.24 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I zstd -5 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 2.77s cpu: 600% maxmem: 155204 KB cswaits: 8109
 [compress speed:] 72.96 MB/s
 compression ratio: 3.3088 211942676 / 64052950 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.16s cpu: 104% maxmem: 3492 KB cswaits: 12939
 [decompress speed:] 93.57 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I zstd -6 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 3.14s cpu: 592% maxmem: 162916 KB cswaits: 8451
 [compress speed:] 64.37 MB/s
 compression ratio: 3.3660 211942676 / 62964179 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.10s cpu: 104% maxmem: 3496 KB cswaits: 12938
 [decompress speed:] 96.24 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I zstd -7 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 3.66s cpu: 625% maxmem: 161644 KB cswaits: 8228
 [compress speed:] 55.22 MB/s
 compression ratio: 3.4503 211942676 / 61426311 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.09s cpu: 101% maxmem: 3492 KB cswaits: 12940
 [decompress speed:] 96.71 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I zstd -8 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 4.06s cpu: 633% maxmem: 159200 KB cswaits: 7846
 [compress speed:] 49.78 MB/s
 compression ratio: 3.4856 211942676 / 60804851 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.06s cpu: 99% maxmem: 3488 KB cswaits: 12939
 [decompress speed:] 98.11 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I zstd -9 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 4.91s cpu: 604% maxmem: 174712 KB cswaits: 8002
 [compress speed:] 41.16 MB/s
 compression ratio: 3.5182 211942676 / 60241424 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 105% maxmem: 3488 KB cswaits: 12941
 [decompress speed:] 104.72 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 10 ] tar "-I zstd -10 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 5.72s cpu: 586% maxmem: 331796 KB cswaits: 8624
 [compress speed:] 35.33 MB/s
 compression ratio: 3.5579 211942676 / 59569376 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.91s cpu: 105% maxmem: 5536 KB cswaits: 12939
 [decompress speed:] 105.82 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 11 ] tar "-I zstd -11 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 6.21s cpu: 587% maxmem: 429872 KB cswaits: 8508
 [compress speed:] 32.54 MB/s
 compression ratio: 3.5748 211942676 / 59286517 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 2.04s cpu: 104% maxmem: 5536 KB cswaits: 12937
 [decompress speed:] 99.08 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 12 ] tar "-I zstd -12 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 7.97s cpu: 615% maxmem: 429420 KB cswaits: 8521
 [compress speed:] 25.36 MB/s
 compression ratio: 3.6030 211942676 / 58823794 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.95s cpu: 103% maxmem: 5536 KB cswaits: 12941
 [decompress speed:] 103.65 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 13 ] tar "-I zstd -13 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 10.25s cpu: 671% maxmem: 424356 KB cswaits: 8800
 [compress speed:] 19.71 MB/s
 compression ratio: 3.6464 211942676 / 58122395 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.88s cpu: 102% maxmem: 5540 KB cswaits: 12941
 [decompress speed:] 107.51 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 14 ] tar "-I zstd -14 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 11.70s cpu: 655% maxmem: 620516 KB cswaits: 8928
 [compress speed:] 17.27 MB/s
 compression ratio: 3.6775 211942676 / 57632128 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.81s cpu: 105% maxmem: 5540 KB cswaits: 12936
 [decompress speed:] 111.67 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 15 ] tar "-I zstd -15 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 13.67s cpu: 685% maxmem: 751272 KB cswaits: 8219
 [compress speed:] 14.78 MB/s
 compression ratio: 3.7008 211942676 / 57269219 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.80s cpu: 105% maxmem: 5540 KB cswaits: 12937
 [decompress speed:] 112.29 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 16 ] tar "-I zstd -16 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 20.88s cpu: 652% maxmem: 493176 KB cswaits: 8092
 [compress speed:] 9.68 MB/s
 compression ratio: 3.8044 211942676 / 55709560 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.83s cpu: 105% maxmem: 5540 KB cswaits: 12939
 [decompress speed:] 110.45 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 17 ] tar "-I zstd -17 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 28.45s cpu: 540% maxmem: 608544 KB cswaits: 9523
 [compress speed:] 7.10 MB/s
 compression ratio: 3.8802 211942676 / 54620629 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.86s cpu: 105% maxmem: 9652 KB cswaits: 12940
 [decompress speed:] 108.66 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 18 ] tar "-I zstd -18 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 42.73s cpu: 554% maxmem: 611368 KB cswaits: 8969
 [compress speed:] 4.73 MB/s
 compression ratio: 3.9480 211942676 / 53683348 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 105% maxmem: 9628 KB cswaits: 12911
 [decompress speed:] 104.72 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 19 ] tar "-I zstd -19 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 61.23s cpu: 513% maxmem: 840380 KB cswaits: 9803
 [compress speed:] 3.30 MB/s
 compression ratio: 3.9796 211942676 / 53256215 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 105% maxmem: 9628 KB cswaits: 12938
 [decompress speed:] 104.72 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 20 ] tar "-I zstd --ultra -20 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 140.72s cpu: 163% maxmem: 589608 KB cswaits: 8424
 [compress speed:] 1.43 MB/s
 compression ratio: 4.0115 211942676 / 52832599 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.93s cpu: 105% maxmem: 34204 KB cswaits: 12940
 [decompress speed:] 104.72 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 21 ] tar "-I zstd --ultra -21 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 243.44s cpu: 100% maxmem: 588488 KB cswaits: 8578
 [compress speed:] .83 MB/s
 compression ratio: 4.0210 211942676 / 52708823 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.95s cpu: 105% maxmem: 66976 KB cswaits: 12938
 [decompress speed:] 103.65 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 22 ] tar "-I zstd --ultra -22 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
 [compress stats:] real: 280.86s cpu: 100% maxmem: 916256 KB cswaits: 8895
 [compress speed:] .71 MB/s
 compression ratio: 4.0228 211942676 / 52684381 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
 [decompress stats:] real: 1.95s cpu: 105% maxmem: 132508 KB cswaits: 12939
 [decompress speed:] 103.65 MB/s
### pxz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I pxz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 4.21s cpu: 529% maxmem: 94300 KB cswaits: 5869
 [compress speed:] 48.01 MB/s
 compression ratio: 3.6086 211942676 / 58731740 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.10s cpu: 106% maxmem: 2036 KB cswaits: 25850
 [decompress speed:] 65.20 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I pxz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 6.56s cpu: 533% maxmem: 180152 KB cswaits: 3513
 [compress speed:] 30.81 MB/s
 compression ratio: 3.7188 211942676 / 56992000 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.93s cpu: 106% maxmem: 3056 KB cswaits: 25852
 [decompress speed:] 68.98 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I pxz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 10.70s cpu: 527% maxmem: 351180 KB cswaits: 2571
 [compress speed:] 18.89 MB/s
 compression ratio: 3.7820 211942676 / 56039192 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.87s cpu: 106% maxmem: 5108 KB cswaits: 25849
 [decompress speed:] 70.42 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I pxz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 14.36s cpu: 504% maxmem: 481288 KB cswaits: 5207
 [compress speed:] 14.07 MB/s
 compression ratio: 4.0367 211942676 / 52503612 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.89s cpu: 107% maxmem: 5108 KB cswaits: 25857
 [decompress speed:] 69.93 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I pxz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 18.08s cpu: 538% maxmem: 907716 KB cswaits: 357
 [compress speed:] 11.17 MB/s
 compression ratio: 4.2126 211942676 / 50310532 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.84s cpu: 103% maxmem: 9200 KB cswaits: 25848
 [decompress speed:] 71.17 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I pxz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 21.04s cpu: 545% maxmem: 952864 KB cswaits: 369
 [compress speed:] 9.60 MB/s
 compression ratio: 4.2753 211942676 / 49573244 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.95s cpu: 98% maxmem: 9204 KB cswaits: 25837
 [decompress speed:] 68.51 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I pxz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 28.05s cpu: 348% maxmem: 965896 KB cswaits: 146
 [compress speed:] 7.20 MB/s
 compression ratio: 4.3057 211942676 / 49223036 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.80s cpu: 104% maxmem: 17392 KB cswaits: 25843
 [decompress speed:] 72.18 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I pxz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 47.66s cpu: 181% maxmem: 963984 KB cswaits: 94
 [compress speed:] 4.24 MB/s
 compression ratio: 4.3316 211942676 / 48929028 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.80s cpu: 103% maxmem: 33780 KB cswaits: 25852
 [decompress speed:] 72.18 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I pxz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 80.69s cpu: 104% maxmem: 897324 KB cswaits: 70
 [compress speed:] 2.50 MB/s
 compression ratio: 4.3415 211942676 / 48816968 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.73s cpu: 106% maxmem: 66548 KB cswaits: 25861
 [decompress speed:] 74.03 MB/s
### xz test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I xz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 12.13s cpu: 100% maxmem: 9616 KB cswaits: 25870
 [compress speed:] 16.66 MB/s
 compression ratio: 3.6282 211942676 / 58414444 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.08s cpu: 105% maxmem: 2036 KB cswaits: 25864
 [decompress speed:] 65.62 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I xz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 18.77s cpu: 100% maxmem: 17300 KB cswaits: 25869
 [compress speed:] 10.76 MB/s
 compression ratio: 3.7371 211942676 / 56712980 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 3.05s cpu: 105% maxmem: 3056 KB cswaits: 25852
 [decompress speed:] 66.27 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I xz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 30.73s cpu: 100% maxmem: 32656 KB cswaits: 25870
 [compress speed:] 6.57 MB/s
 compression ratio: 3.8016 211942676 / 55749668 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.89s cpu: 105% maxmem: 5108 KB cswaits: 25840
 [decompress speed:] 69.93 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I xz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 48.74s cpu: 100% maxmem: 49224 KB cswaits: 25873
 [compress speed:] 4.14 MB/s
 compression ratio: 4.0672 211942676 / 52109368 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.83s cpu: 106% maxmem: 5108 KB cswaits: 25853
 [decompress speed:] 71.42 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I xz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 64.54s cpu: 100% maxmem: 96328 KB cswaits: 25875
 [compress speed:] 3.13 MB/s
 compression ratio: 4.2401 211942676 / 49984712 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.74s cpu: 106% maxmem: 9204 KB cswaits: 25847
 [decompress speed:] 73.76 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I xz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 75.45s cpu: 100% maxmem: 96332 KB cswaits: 25876
 [compress speed:] 2.67 MB/s
 compression ratio: 4.3044 211942676 / 49237796 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.88s cpu: 103% maxmem: 9204 KB cswaits: 25856
 [decompress speed:] 70.18 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I xz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 79.00s cpu: 100% maxmem: 190540 KB cswaits: 25871
 [compress speed:] 2.55 MB/s
 compression ratio: 4.3297 211942676 / 48950732 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.71s cpu: 106% maxmem: 17392 KB cswaits: 25840
 [decompress speed:] 74.58 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I xz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 83.42s cpu: 100% maxmem: 378956 KB cswaits: 25878
 [compress speed:] 2.42 MB/s
 compression ratio: 4.3472 211942676 / 48752848 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.81s cpu: 106% maxmem: 33776 KB cswaits: 25852
 [decompress speed:] 71.93 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I xz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
 [compress stats:] real: 85.14s cpu: 100% maxmem: 690248 KB cswaits: 25881
 [compress speed:] 2.37 MB/s
 compression ratio: 4.3500 211942676 / 48721676 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
 [decompress stats:] real: 2.82s cpu: 103% maxmem: 66548 KB cswaits: 25845
 [decompress speed:] 71.67 MB/s
### lz4 test
------------------------------------------------------------------------------
 [tar compress lvl: 1 ] tar "-I lz4 -1" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 2.44s cpu: 97% maxmem: 8844 KB cswaits: 9043
 [compress speed:] 82.83 MB/s
 compression ratio: 2.0999 211942676 / 100925719 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.86s cpu: 100% maxmem: 8940 KB cswaits: 18036
 [decompress speed:] 235.02 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 2 ] tar "-I lz4 -2" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 2.35s cpu: 102% maxmem: 8844 KB cswaits: 9037
 [compress speed:] 86.01 MB/s
 compression ratio: 2.0999 211942676 / 100925719 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.79s cpu: 107% maxmem: 8940 KB cswaits: 18102
 [decompress speed:] 255.85 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 3 ] tar "-I lz4 -3" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 7.02s cpu: 100% maxmem: 8600 KB cswaits: 8160
 [compress speed:] 28.79 MB/s
 compression ratio: 2.6023 211942676 / 81444275 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.87s cpu: 102% maxmem: 8440 KB cswaits: 18304
 [decompress speed:] 232.32 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 4 ] tar "-I lz4 -4" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 8.21s cpu: 100% maxmem: 8588 KB cswaits: 8430
 [compress speed:] 24.61 MB/s
 compression ratio: 2.6520 211942676 / 79916565 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.81s cpu: 107% maxmem: 8428 KB cswaits: 18075
 [decompress speed:] 249.53 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 5 ] tar "-I lz4 -5" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 9.68s cpu: 100% maxmem: 8588 KB cswaits: 9421
 [compress speed:] 20.88 MB/s
 compression ratio: 2.6825 211942676 / 79008581 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.81s cpu: 106% maxmem: 8428 KB cswaits: 17928
 [decompress speed:] 249.53 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 6 ] tar "-I lz4 -6" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 11.34s cpu: 100% maxmem: 8588 KB cswaits: 9298
 [compress speed:] 17.82 MB/s
 compression ratio: 2.6998 211942676 / 78501649 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.81s cpu: 107% maxmem: 8428 KB cswaits: 17804
 [decompress speed:] 249.53 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 7 ] tar "-I lz4 -7" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 13.41s cpu: 100% maxmem: 8588 KB cswaits: 9107
 [compress speed:] 15.07 MB/s
 compression ratio: 2.7095 211942676 / 78219928 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.87s cpu: 101% maxmem: 8428 KB cswaits: 18173
 [decompress speed:] 232.32 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 8 ] tar "-I lz4 -8" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 15.82s cpu: 100% maxmem: 8584 KB cswaits: 8001
 [compress speed:] 12.77 MB/s
 compression ratio: 2.7145 211942676 / 78075840 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.80s cpu: 107% maxmem: 8428 KB cswaits: 18134
 [decompress speed:] 252.65 MB/s
------------------------------------------------------------------------------
 [tar compress lvl: 9 ] tar "-I lz4 -9" -cf /home/tar-zstd/silesia.tar.lz4 silesia
 [compress stats:] real: 18.11s cpu: 100% maxmem: 8588 KB cswaits: 7943
 [compress speed:] 11.16 MB/s
 compression ratio: 2.7171 211942676 / 78003178 KB
------------------------------------------------------------------------------
 [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
 [decompress stats:] real: 0.87s cpu: 99% maxmem: 8428 KB cswaits: 17757
 [decompress speed:] 232.32 MB/s

For pure compression speed, seems with 4 core/8 cpu thread server, zstd negative level --9 came closest to pure tar uncompressed speeds of 532MB/s with 481MB/s followed by multi-threaded gzip via pigz level 1 at 404MB/s. Pigz level 1 had much better compression ratio of 2.7463 vs zstd negative level -9 at 1.8016 compressio ratio.

eva2000 · Dec 6, 2021

Update December 6, 2021 - added to 1st post instructions for installing tar v1.33 and v1.34 RPMs for your own testing. Best of both worlds, you can use system tar at /usr/bin/tar or custom tar RPM at /usr/local/bin/tar which takes priority if you just use tar command without a path

Code (Text):

/usr/bin/tar --version
tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Code (Text):

/usr/local/bin/tar --version
tar (GNU tar) 1.34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

CentOS 7 64bit tar v1.34

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc9-1.34-1.el7.x86_64.rpm -O tar-zstd-gcc9-1.34-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc9-1.34-1.el7.x86_64.rpm

Code (Text):

rpm -qa --changelog tar-zstd-gcc9
* Sun Dec 05 2021 George Liu <centminmod.com> 1.34
- tar 1.34 zstd supported for centminmod.com LEMP stack
- Fix extraction over pipe (savannah bug #60002)
- Fix memory leak in read_header (savannah bug #59897)
- Fix extraction when . and .. are unreadable
  See https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00012.html
- Gracefully handle duplicate symlinks when extracting
  See https://lists.gnu.org/archive/html/bug-tar/2021-01/msg00026.html
- Re-initialize supplementary groups when switching to user privileges

CentOS 7 64bit tar v1.33

Code (Text):

cd /svr-setup
wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc9-1.33-1.el7.x86_64.rpm -O tar-zstd-gcc9-1.33-1.el7.x86_64.rpm
yum localinstall tar-zstd-gcc9-1.33-1.el7.x86_64.rpm

Code (Text):

rpm -qa --changelog tar-zstd-gcc9
* Mon Jan 25 2021 George Liu <centminmod.com> 1.33
- tar 1.33 zstd supported for centminmod.com LEMP stack
- Fix memory leak in read_header CVE-2021-20193 patch fixed

tar v1.33 RPM has a backported patch from tar v1.34 for memory leak security bug that I added at the time around Jan 25, 2021

Log in / Register

Forums

Join the community today

Featured Beta Branch CentOS 7.x Custom Tar Archiver RPM Build With Facebook Zstd Compression Support

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar + native zstd multi-threaded tests

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar + zstd backup for Centmin Mod Nginx Site Files

backup

restore

Backup All Nginx Domain Files on Centmin Mod

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar 1.31 + zstd compression & decompression benchmarks

tar-zstd-test.sh usage

gzip level 1 to 9

pigz level 1 to 9

zstd level 1 to 19

xz level 1 to 9

pxz level 1 to 9

Updated benchmarks

eva2000 Administrator Staff Member

tar + zstd versus tar alone

eva2000 Administrator Staff Member

tar 1.31 + zstd compression & decompression benchmarks part 2

tar-zstd-test.sh usage

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Join the community today

Featured Beta Branch CentOS 7.x Custom Tar Archiver RPM Build With Facebook Zstd Compression Support

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar + native zstd multi-threaded tests

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar + zstd backup for Centmin Mod Nginx Site Files

backup

restore

Backup All Nginx Domain Files on Centmin Mod

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

tar 1.31 + zstd compression & decompression benchmarks

tar-zstd-test.sh usage

gzip level 1 to 9

pigz level 1 to 9

zstd level 1 to 19

xz level 1 to 9

pxz level 1 to 9

Updated benchmarks

eva2000 Administrator Staff Member

tar + zstd versus tar alone

eva2000 Administrator Staff Member

tar 1.31 + zstd compression & decompression benchmarks part 2

tar-zstd-test.sh usage

eva2000 Administrator Staff Member

.