Want to subscribe to topics you're interested in?
Become a Member

Featured Beta Branch CentOS 7.x Custom Tar Archiver RPM Build With Facebook Zstd Compression Support

Discussion in 'Beta release code' started by eva2000, Dec 16, 2018.

Tags:
Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Folks may have read my multi-threaded compression tool benchmarks here & here and may wonder how we can take advantage of faster compression tools such as Facebook's zstd ? Centmin Mod 123.09beta01 and higher allow you to optionally install all the multi-threaded compression tools via centmin.sh menu option 17 (see below). You can also test and provide feedback here. Or jump straight to the final zstd benchmark comparison results including zstd long range mode and automatic level determination modes for higher compression ratios here.
    Code (Text):
    --------------------------------------------------------
         Centmin Mod Menu 123.09beta01 centminmod.com
    --------------------------------------------------------
    1).  Centmin Install
    2).  Add Nginx vhost domain
    3).  NSD setup domain name DNS
    4).  Nginx Upgrade / Downgrade
    5).  PHP Upgrade / Downgrade
    6).  XCache Re-install
    7).  APC Cache Re-install
    8).  XCache Install
    9).  APC Cache Install
    10). Memcached Server Re-install
    11). MariaDB MySQL Upgrade & Management
    12). Zend OpCache Install/Re-install
    13). Install/Reinstall Redis PHP Extension
    14). SELinux disable
    15). Install/Reinstall ImagicK PHP Extension
    16). Change SSHD Port Number
    17). Multi-thread compression: pigz,pbzip2,lbzip2...
    18). Suhosin PHP Extension install
    19). Install FFMPEG and FFMPEG PHP Extension
    20). NSD Install/Re-Install
    21). Update - Nginx + PHP-FPM + Siege
    22). Add Wordpress Nginx vhost + Cache Plugin
    23). Update Centmin Mod Code Base
    24). Exit
    --------------------------------------------------------
    Enter option [ 1 - 24 ] 17
    

    zstd v1.3.8
    Code (Text):
    zstd --version
    *** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
    

    For backup purposes with tar archiving tool, you would currently need to tar first and then use zstd. However, latest development version of tar 1.30.90 has native support for zstd now and now tar 1.31 stable release is out.
    Code (Text):
    Compression options:
    
      -a, --auto-compress        use archive suffix to determine the compression
                                 program
      -I, --use-compress-program=PROG
                                 filter through PROG (must accept -d)
      -j, --bzip2                filter the archive through bzip2
      -J, --xz                   filter the archive through xz
          --lzip                 filter the archive through lzip
          --lzma                 filter the archive through lzma
          --lzop                 filter the archive through lzop
          --no-auto-compress     do not use archive suffix to determine the
                                 compression program
      -z, --gzip, --gunzip, --ungzip   filter the archive through gzip
          --zstd                 filter the archive through zstd
      -Z, --compress, --uncompress   filter the archive through compress
    

    So I built a custom tar 1.31 RPM for CentOS 6.x & 7.x 64bit based Centmin Mod 123.09beta01 or higher installations.

    After running centmin.sh menu option 17 to install zstd etc, install custom tar zstd supported rpm.

    CentOS 7 64bit
    Code (Text):
    cd /svr-setup
    wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.31-1.el7.x86_64.rpm -O tar-zstd-gcc7-1.31-1.el7.x86_64.rpm
    yum localinstall tar-zstd-gcc7-1.31-1.el7.x86_64.rpm
    

    Updating with January 3, 2019 tar 1.31 build
    Code (Text):
    rpm -e tar-zstd-gcc7
    cd /svr-setup
    wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.31-1.el7.x86_64.rpm -O tar-zstd-gcc7-1.31-1.el7.x86_64.rpm
    yum localinstall tar-zstd-gcc7-1.31-1.el7.x86_64.rpm
    


    CentOS 6 64bit
    Code (Text):
    cd /svr-setup
    wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.31-1.el6.x86_64.rpm -O tar-zstd-gcc7-1.31-1.el6.x86_64.rpm
    yum localinstall tar-zstd-gcc7-1.31-1.el6.x86_64.rpm
    

    Updating with January 3, 2019 tar 1.31 build
    Code (Text):
    rpm -e tar-zstd-gcc7
    cd /svr-setup
    wget https://centminmod.com/centminmodparts/tar/tar-zstd-gcc7-1.31-1.el6.x86_64.rpm -O tar-zstd-gcc7-1.31-1.el6.x86_64.rpm
    yum localinstall tar-zstd-gcc7-1.31-1.el6.x86_64.rpm
    


    Example for CentOS 7 64bit install for tar 1.30.90
    Code (Text):
    yum localinstall tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm
    Loaded plugins: fastestmirror, priorities, versionlock
    Examining tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm: tar-zstd-gcc7-1.30.90-1.el7.x86_64
    Marking tar-zstd-gcc7-1.30.90-1.el7.x86_64.rpm to be installed
    Resolving Dependencies
    --> Running transaction check
    ---> Package tar-zstd-gcc7.x86_64 0:1.30.90-1.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==========================================================================================================================================================================================================================================================
     Package                                                  Arch                                              Version                                                  Repository                                                                      Size
    ==========================================================================================================================================================================================================================================================
    Installing:
     tar-zstd-gcc7                                            x86_64                                            1.30.90-1.el7                                            /tar-zstd-gcc7-1.30.90-1.el7.x86_64                                            3.4 M
    
    Transaction Summary
    ==========================================================================================================================================================================================================================================================
    Install  1 Package
    
    Total size: 3.4 M
    Installed size: 3.4 M
    Is this ok [y/d/N]: y
    Downloading packages:
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : tar-zstd-gcc7-1.30.90-1.el7.x86_64                                                                                                                                                                                                     1/1
      Verifying  : tar-zstd-gcc7-1.30.90-1.el7.x86_64                                                                                                                                                                                                     1/1
    
    Installed:
      tar-zstd-gcc7.x86_64 0:1.30.90-1.el7                                                                                                                                                                                                  
    Complete!
    


    Usage

    Create a tar archive compressed with zstd
    Code (Text):
    tar -caf directory.tar.zst directory/
    

    Decompress zstd compressed archive
    Code (Text):
    tar -xaf directory.tar.zst
    

    RPM Info
    Code (Text):
    tar --version
    tar (GNU tar) 1.30.90
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Written by John Gilmore and Jay Fenlason.
    

    Code (Text):
    yum -q info tar-zstd-gcc7
    Installed Packages
    Name        : tar-zstd-gcc7
    Arch        : x86_64
    Version     : 1.30.90
    Release     : 1.el7
    Size        : 3.4 M
    Repo        : installed
    Summary     : tar 1.30.90 zstd supported for centminmod.com LEMP stack
    URL         : https://centminmod.com
    License     : unknown
    Description : tar 1.30.90 zstd supported for centminmod.com LEMP stack
    

    Code (Text):
    rpm -qa --changelog tar-zstd-gcc7
    * Fri Dec 28 2018 George Liu <centminmod.com> 1.30.90
    - tar 1.30.90 zstd supported for centminmod.com LEMP stack
    

    Code (Text):
    rpm -ql tar-zstd-gcc7
    /usr/local/bin/tar
    /usr/local/libexec/rmt
    /usr/local/share/info/dir
    /usr/local/share/info/tar.info
    /usr/local/share/info/tar.info-1
    /usr/local/share/info/tar.info-2
    /usr/local/share/locale/bg/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ca/LC_MESSAGES/tar.mo
    /usr/local/share/locale/cs/LC_MESSAGES/tar.mo
    /usr/local/share/locale/da/LC_MESSAGES/tar.mo
    /usr/local/share/locale/de/LC_MESSAGES/tar.mo
    /usr/local/share/locale/el/LC_MESSAGES/tar.mo
    /usr/local/share/locale/eo/LC_MESSAGES/tar.mo
    /usr/local/share/locale/es/LC_MESSAGES/tar.mo
    /usr/local/share/locale/et/LC_MESSAGES/tar.mo
    /usr/local/share/locale/eu/LC_MESSAGES/tar.mo
    /usr/local/share/locale/fi/LC_MESSAGES/tar.mo
    /usr/local/share/locale/fr/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ga/LC_MESSAGES/tar.mo
    /usr/local/share/locale/gl/LC_MESSAGES/tar.mo
    /usr/local/share/locale/hr/LC_MESSAGES/tar.mo
    /usr/local/share/locale/hu/LC_MESSAGES/tar.mo
    /usr/local/share/locale/id/LC_MESSAGES/tar.mo
    /usr/local/share/locale/it/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ja/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ko/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ky/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ms/LC_MESSAGES/tar.mo
    /usr/local/share/locale/nb/LC_MESSAGES/tar.mo
    /usr/local/share/locale/nl/LC_MESSAGES/tar.mo
    /usr/local/share/locale/pl/LC_MESSAGES/tar.mo
    /usr/local/share/locale/pt/LC_MESSAGES/tar.mo
    /usr/local/share/locale/pt_BR/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ro/LC_MESSAGES/tar.mo
    /usr/local/share/locale/ru/LC_MESSAGES/tar.mo
    /usr/local/share/locale/sk/LC_MESSAGES/tar.mo
    /usr/local/share/locale/sl/LC_MESSAGES/tar.mo
    /usr/local/share/locale/sr/LC_MESSAGES/tar.mo
    /usr/local/share/locale/sv/LC_MESSAGES/tar.mo
    /usr/local/share/locale/tr/LC_MESSAGES/tar.mo
    /usr/local/share/locale/uk/LC_MESSAGES/tar.mo
    /usr/local/share/locale/vi/LC_MESSAGES/tar.mo
    /usr/local/share/locale/zh_CN/LC_MESSAGES/tar.mo
    /usr/local/share/locale/zh_TW/LC_MESSAGES/tar.mo
    /usr/local/share/man/man1/tar.1
    /usr/local/share/man/man8/rmt.8
    

    Code (Text):
    tar --help
    Usage: tar [OPTION...] [FILE]...
    GNU 'tar' saves many files together into a single tape or disk archive, and can
    restore individual files from the archive.
    
    Examples:
      tar -cf archive.tar foo bar  # Create archive.tar from files foo and bar.
      tar -tvf archive.tar         # List all files in archive.tar verbosely.
      tar -xf archive.tar          # Extract all files from archive.tar.
    
     Local file name selection:
    
          --add-file=FILE        add given FILE to the archive (useful if its name
                                 starts with a dash)
      -C, --directory=DIR        change to directory DIR
          --exclude=PATTERN      exclude files, given as a PATTERN
          --exclude-backups      exclude backup and lock files
          --exclude-caches       exclude contents of directories containing
                                 CACHEDIR.TAG, except for the tag file itself
          --exclude-caches-all   exclude directories containing CACHEDIR.TAG
          --exclude-caches-under exclude everything under directories containing
                                 CACHEDIR.TAG
          --exclude-ignore=FILE  read exclude patterns for each directory from
                                 FILE, if it exists
          --exclude-ignore-recursive=FILE
                                 read exclude patterns for each directory and its
                                 subdirectories from FILE, if it exists
          --exclude-tag=FILE     exclude contents of directories containing FILE,
                                 except for FILE itself
          --exclude-tag-all=FILE exclude directories containing FILE
          --exclude-tag-under=FILE   exclude everything under directories
                                 containing FILE
          --exclude-vcs          exclude version control system directories
          --exclude-vcs-ignores  read exclude patterns from the VCS ignore files
          --no-null              disable the effect of the previous --null option
          --no-recursion         avoid descending automatically in directories
          --no-unquote           do not unquote input file or member names
          --no-verbatim-files-from   -T treats file names starting with dash as
                                 options (default)
          --null                 -T reads null-terminated names; implies
                                 --verbatim-files-from
          --recursion            recurse into directories (default)
      -T, --files-from=FILE      get names to extract or create from FILE
          --unquote              unquote input file or member names (default)
          --verbatim-files-from  -T reads file names verbatim (no escape or option
                                 handling)
      -X, --exclude-from=FILE    exclude patterns listed in FILE
    
     File name matching options (affect both exclude and include patterns):
    
          --anchored             patterns match file name start
          --ignore-case          ignore case
          --no-anchored          patterns match after any '/' (default for
                                 exclusion)
          --no-ignore-case       case sensitive matching (default)
          --no-wildcards         verbatim string matching
          --no-wildcards-match-slash   wildcards do not match '/'
          --wildcards            use wildcards (default for exclusion)
          --wildcards-match-slash   wildcards match '/' (default for exclusion)
    
     Main operation mode:
    
      -A, --catenate, --concatenate   append tar files to an archive
      -c, --create               create a new archive
      -d, --diff, --compare      find differences between archive and file system
          --delete               delete from the archive (not on mag tapes!)
      -r, --append               append files to the end of an archive
      -t, --list                 list the contents of an archive
          --test-label           test the archive volume label and exit
      -u, --update               only append files newer than copy in archive
      -x, --extract, --get       extract files from an archive
    
     Operation modifiers:
    
          --check-device         check device numbers when creating incremental
                                 archives (default)
      -g, --listed-incremental=FILE   handle new GNU-format incremental backup
      -G, --incremental          handle old GNU-format incremental backup
          --hole-detection=TYPE  technique to detect holes
          --ignore-failed-read   do not exit with nonzero on unreadable files
          --level=NUMBER         dump level for created listed-incremental archive
      -n, --seek                 archive is seekable
          --no-check-device      do not check device numbers when creating
                                 incremental archives
          --no-seek              archive is not seekable
          --occurrence[=NUMBER]  process only the NUMBERth occurrence of each file
                                 in the archive; this option is valid only in
                                 conjunction with one of the subcommands --delete,
                                 --diff, --extract or --list and when a list of
                                 files is given either on the command line or via
                                 the -T option; NUMBER defaults to 1
          --sparse-version=MAJOR[.MINOR]
                                 set version of the sparse format to use (implies
                                 --sparse)
      -S, --sparse               handle sparse files efficiently
    
     Overwrite control:
    
      -k, --keep-old-files       don't replace existing files when extracting,
                                 treat them as errors
          --keep-directory-symlink   preserve existing symlinks to directories when
                                 extracting
          --keep-newer-files     don't replace existing files that are newer than
                                 their archive copies
          --no-overwrite-dir     preserve metadata of existing directories
          --one-top-level[=DIR]  create a subdirectory to avoid having loose files
                                 extracted
          --overwrite            overwrite existing files when extracting
          --overwrite-dir        overwrite metadata of existing directories when
                                 extracting (default)
          --recursive-unlink     empty hierarchies prior to extracting directory
          --remove-files         remove files after adding them to the archive
          --skip-old-files       don't replace existing files when extracting,
                                 silently skip over them
      -U, --unlink-first         remove each file prior to extracting over it
      -W, --verify               attempt to verify the archive after writing it
    
     Select output stream:
    
          --ignore-command-error ignore exit codes of children
          --no-ignore-command-error   treat non-zero exit codes of children as
                                 error
      -O, --to-stdout            extract files to standard output
          --to-command=COMMAND   pipe extracted files to another program
    
     Handling of file attributes:
    
          --atime-preserve[=METHOD]   preserve access times on dumped files, either
                                 by restoring the times after reading
                                 (METHOD='replace'; default) or by not setting the
                                 times in the first place (METHOD='system')
          --clamp-mtime          only set time when the file is more recent than
                                 what was given with --mtime
          --delay-directory-restore   delay setting modification times and
                                 permissions of extracted directories until the end
                                 of extraction
          --group=NAME           force NAME as group for added files
          --group-map=FILE       use FILE to map file owner GIDs and names
          --mode=CHANGES         force (symbolic) mode CHANGES for added files
          --mtime=DATE-OR-FILE   set mtime for added files from DATE-OR-FILE
      -m, --touch                don't extract file modified time
          --no-delay-directory-restore
                                 cancel the effect of --delay-directory-restore
                                 option
          --no-same-owner        extract files as yourself (default for ordinary
                                 users)
          --no-same-permissions  apply the user's umask when extracting permissions
                                 from the archive (default for ordinary users)
          --numeric-owner        always use numbers for user/group names
          --owner=NAME           force NAME as owner for added files
          --owner-map=FILE       use FILE to map file owner UIDs and names
      -p, --preserve-permissions, --same-permissions
                                 extract information about file permissions
                                 (default for superuser)
          --same-owner           try extracting files with the same ownership as
                                 exists in the archive (default for superuser)
      -s, --preserve-order, --same-order
                                 member arguments are listed in the same order as
                                 the files in the archive
          --sort=ORDER           directory sorting order: none (default), name or
                                 inode
    
     Handling of extended file attributes:
    
          --acls                 Enable the POSIX ACLs support
          --no-acls              Disable the POSIX ACLs support
          --no-selinux           Disable the SELinux context support
          --no-xattrs            Disable extended attributes support
          --selinux              Enable the SELinux context support
          --xattrs               Enable extended attributes support
          --xattrs-exclude=MASK  specify the exclude pattern for xattr keys
          --xattrs-include=MASK  specify the include pattern for xattr keys
    
     Device selection and switching:
    
      -f, --file=ARCHIVE         use archive file or device ARCHIVE
          --force-local          archive file is local even if it has a colon
      -F, --info-script=NAME, --new-volume-script=NAME
                                 run script at end of each tape (implies -M)
      -L, --tape-length=NUMBER   change tape after writing NUMBER x 1024 bytes
      -M, --multi-volume         create/list/extract multi-volume archive
          --rmt-command=COMMAND  use given rmt COMMAND instead of rmt
          --rsh-command=COMMAND  use remote COMMAND instead of rsh
          --volno-file=FILE      use/update the volume number in FILE
    
     Device blocking:
    
      -b, --blocking-factor=BLOCKS   BLOCKS x 512 bytes per record
      -B, --read-full-records    reblock as we read (for 4.2BSD pipes)
      -i, --ignore-zeros         ignore zeroed blocks in archive (means EOF)
          --record-size=NUMBER   NUMBER of bytes per record, multiple of 512
    
     Archive format selection:
    
      -H, --format=FORMAT        create archive of the given format
    
     FORMAT is one of the following:
    
        gnu                      GNU tar 1.13.x format
        oldgnu                   GNU format as per tar <= 1.12
        pax                      POSIX 1003.1-2001 (pax) format
        posix                    same as pax
        ustar                    POSIX 1003.1-1988 (ustar) format
        v7                       old V7 tar format
    
          --old-archive, --portability
                                 same as --format=v7
          --pax-option=keyword[[:]=value][,keyword[[:]=value]]...
                                 control pax keywords
          --posix                same as --format=posix
      -V, --label=TEXT           create archive with volume name TEXT; at
                                 list/extract time, use TEXT as a globbing pattern
                                 for volume name
    
     Compression options:
    
      -a, --auto-compress        use archive suffix to determine the compression
                                 program
      -I, --use-compress-program=PROG
                                 filter through PROG (must accept -d)
      -j, --bzip2                filter the archive through bzip2
      -J, --xz                   filter the archive through xz
          --lzip                 filter the archive through lzip
          --lzma                 filter the archive through lzma
          --lzop                 filter the archive through lzop
          --no-auto-compress     do not use archive suffix to determine the
                                 compression program
      -z, --gzip, --gunzip, --ungzip   filter the archive through gzip
          --zstd                 filter the archive through zstd
      -Z, --compress, --uncompress   filter the archive through compress
    
     Local file selection:
    
          --backup[=CONTROL]     backup before removal, choose version CONTROL
      -h, --dereference          follow symlinks; archive and dump the files they
                                 point to
          --hard-dereference     follow hard links; archive and dump the files they
                                 refer to
      -K, --starting-file=MEMBER-NAME
                                 begin at member MEMBER-NAME when reading the
                                 archive
          --newer-mtime=DATE     compare date and time when data changed only
      -N, --newer=DATE-OR-FILE, --after-date=DATE-OR-FILE
                                 only store files newer than DATE-OR-FILE
          --one-file-system      stay in local file system when creating archive
      -P, --absolute-names       don't strip leading '/'s from file names
          --suffix=STRING        backup before removal, override usual suffix ('~'
                                 unless overridden by environment variable
                                 SIMPLE_BACKUP_SUFFIX)
    
     File name transformations:
    
          --strip-components=NUMBER   strip NUMBER leading components from file
                                 names on extraction
          --transform=EXPRESSION, --xform=EXPRESSION
                                 use sed replace EXPRESSION to transform file
                                 names
    
     Informative output:
    
          --checkpoint[=NUMBER]  display progress messages every NUMBERth record
                                 (default 10)
          --checkpoint-action=ACTION   execute ACTION on each checkpoint
          --full-time            print file time to its full resolution
          --index-file=FILE      send verbose output to FILE
      -l, --check-links          print a message if not all links are dumped
          --no-quote-chars=STRING   disable quoting for characters from STRING
          --quote-chars=STRING   additionally quote characters from STRING
          --quoting-style=STYLE  set name quoting style; see below for valid STYLE
                                 values
      -R, --block-number         show block number within archive with each message
                   
          --show-defaults        show tar defaults
          --show-omitted-dirs    when listing or extracting, list each directory
                                 that does not match search criteria
          --show-snapshot-field-ranges
                                 show valid ranges for snapshot-file fields
          --show-transformed-names, --show-stored-names
                                 show file or archive names after transformation
          --totals[=SIGNAL]      print total bytes after processing the archive;
                                 with an argument - print total bytes when this
                                 SIGNAL is delivered; Allowed signals are: SIGHUP,
                                 SIGQUIT, SIGINT, SIGUSR1 and SIGUSR2; the names
                                 without SIG prefix are also accepted
          --utc                  print file modification times in UTC
      -v, --verbose              verbosely list files processed
          --warning=KEYWORD      warning control
      -w, --interactive, --confirmation
                                 ask for confirmation for every action
    
     Compatibility options:
    
      -o                         when creating, same as --old-archive; when
                                 extracting, same as --no-same-owner
    
     Other options:
    
      -?, --help                 give this help list
          --restrict             disable use of some potentially harmful options
          --usage                give a short usage message
          --version              print program version
    
    Mandatory or optional arguments to long options are also mandatory or optional
    for any corresponding short options.
    
    The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.
    The version control may be set with --backup or VERSION_CONTROL, values are:
    
      none, off       never make backups
      t, numbered     make numbered backups
      nil, existing   numbered if numbered backups exist, simple otherwise
      never, simple   always make simple backups
    
    Valid arguments for the --quoting-style option are:
    
      literal
      shell
      shell-always
      shell-escape
      shell-escape-always
      c
      c-maybe
      escape
      locale
      clocale
    
    *This* tar defaults to:
    --format=gnu -f- -b20 --quoting-style=escape
    --rmt-command=/usr/local/libexec/rmt
    
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Example usage on 2 cpu thread server for zstd run backing up /svr-setup directory
    Code (Text):
    du -sh /svr-setup
    1.6G    /svr-setup
    

    tar + zstd compression took 16.74s
    Code (Text):
    time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup
    
    real    0m16.740s
    user    0m15.451s
    sys     0m5.694s
    

    Code (Text):
    ls -lah /home/speedtest/svr-setup.tar.zst
    -rw-r--r-- 1 root root 621M Dec 15 17:24 /home/speedtest/svr-setup.tar.zst
    

    Compared with standard gzip run which took 1m 47.782s or 107.782 seconds
    Code (Text):
    time tar -caf /home/speedtest/svr-setup.tar.gz svr-setup
    
    real    1m47.782s
    user    1m44.892s
    sys     0m6.766s
    

    Code (Text):
    ls -lah /home/speedtest
    total 1.3G
    drwxr-xr-x   2 root root 4.0K Dec 15 17:26 .
    drwxr-xr-x. 13 root root 4.0K Dec 15 17:22 ..
    -rw-r--r--   1 root root 633M Dec 15 17:28 svr-setup.tar.gz
    -rw-r--r--   1 root root 621M Dec 15 17:24 svr-setup.tar.
    

    Compared with pigz multi-threaded gzip run took 54.774s
    Code (Text):
    time tar -I pigz -cf /home/speedtest/svr-setup-pigz.tar.gz svr-setup
    
    real    0m54.774s
    user    1m46.527s
    sys     0m2.268s
    


    Final compressed sizes show zstd compression which defaults to level 3 compress the svr-setup tar archive better and was much faster in doing it in 16.74s versus gzip at 107.782s versus pigz gzip at 54.774s.
    Code (Text):
    ls -lah /home/speedtest
    total 1.9G
    drwxr-xr-x   2 root root 4.0K Dec 15 17:30 .
    drwxr-xr-x. 13 root root 4.0K Dec 15 17:22 ..
    -rw-r--r--   1 root root 632M Dec 15 17:31 svr-setup-pigz.tar.gz
    -rw-r--r--   1 root root 633M Dec 15 17:28 svr-setup.tar.gz
    -rw-r--r--   1 root root 621M Dec 15 17:24 svr-setup.tar.zst
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Above tests are on 2 cpu thread VPSes. This time test on Intel Core i7 4790K with 4 cpu core/8 cpu threaded dedicated server to see how zstd handles more multi-threaded compression runs.
    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    2
    Core(s) per socket:    4
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 60
    Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
    Stepping:              3
    CPU MHz:               4199.707
    CPU max MHz:           4400.0000
    CPU min MHz:           800.0000
    BogoMIPS:              7981.85
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0-7
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts spec_ctrl intel_stibp flush_l1d
    

    test against /svr-setup directory of 3.7GB in size
    Code (Text):
    du -sh /svr-setup/
    3.7G    /svr-setup/
    

    tar uncompressed
    Code (Text):
    time tar -cf /home/speedtest/svr-setup.tar svr-setup
    
    real    0m10.270s
    user    0m0.347s
    sys     0m2.804s
    

    First is tar + standard gzip which has default level 6 compression which took 1mi 57.043s or 117.043 seconds
    Code (Text):
    time tar -caf /home/speedtest/svr-setup.tar.gz svr-setup
    
    real    1m57.043s
    user    1m45.699s
    sys     0m3.690s
    

    Then tar + pigz multi-threaded gzip run with default level 6 compression took 22.374 seconds
    Code (Text):
    time tar -I pigz -cf /home/speedtest/svr-setup-pigz.tar.gz svr-setup
    
    real    0m22.374s
    user    2m34.483s
    sys     0m3.911s
    

    Then tar + zstd single threaded run with default level 3 compression took 13.87 seconds
    Code (Text):
    time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup
    
    real    0m13.870s
    user    0m14.148s
    sys     0m2.830s
    

    Resulting compressed and uncompressed sizes
    Code (Text):
    ls -lAhrt /home/speedtest/                   
    total 11G
    -rw-r--r-- 1 root root 1.4G Dec 16 04:10 svr-setup.tar.gz
    -rw-r--r-- 1 root root 1.4G Dec 16 04:11 svr-setup-pigz.tar.gz
    -rw-r--r-- 1 root root 1.3G Dec 16 04:12 svr-setup.tar.zst
    -rw-r--r-- 1 root root 3.5G Dec 16 04:16 svr-setup.tar
    

    Then try higher levels of zstd compression against svr-setup.tar

    zstd level 6
    Code (Text):
    time zstd -6f svr-setup.tar -o svr-setup-zstd-lvl6.tar.zst
    svr-setup.tar        : 35.89%   (3690383360 => 1324652195 bytes, svr-setup-zstd-lvl6.tar.zst)
    
    real    0m32.766s
    user    0m32.016s
    sys     0m0.966s
    

    zstd level 9
    Code (Text):
    time zstd -9f svr-setup.tar -o svr-setup-zstd-lvl9.tar.zst
    svr-setup.tar        : 35.22%   (3690383360 => 1299591955 bytes, svr-setup-zstd-lvl9.tar.zst)
    
    real    1m5.178s
    user    1m3.793s
    sys     0m1.043s
    

    zstd level 12
    Code (Text):
    time zstd -12f svr-setup.tar -o svr-setup-zstd-lvl12.tar.zst
    svr-setup.tar        : 29.75%   (3690383360 => 1098039642 bytes, svr-setup-zstd-lvl12.tar.zst)
    
    real    1m32.450s
    user    1m32.256s
    sys     0m1.039s
    

    Resulting compressed and uncompressed sizes
    Code (Text):
    ls -lAhrt /home/speedtest/
    total 11G
    -rw-r--r-- 1 root root 1.4G Dec 16 04:10 svr-setup.tar.gz
    -rw-r--r-- 1 root root 1.4G Dec 16 04:11 svr-setup-pigz.tar.gz
    -rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup-zstd-lvl9.tar.zst
    -rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup-zstd-lvl6.tar.zst
    -rw-r--r-- 1 root root 1.1G Dec 16 04:16 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r-- 1 root root 1.3G Dec 16 04:16 svr-setup.tar.zst
    -rw-r--r-- 1 root root 3.5G Dec 16 04:16 svr-setup.tar
    

    Code (Text):
    ls -lArt /home/speedtest/
    total 11359368
    -rw-r--r-- 1 root root 1448823495 Dec 16 04:10 svr-setup.tar.gz
    -rw-r--r-- 1 root root 1445803664 Dec 16 04:11 svr-setup-pigz.tar.gz
    -rw-r--r-- 1 root root 1299591955 Dec 16 04:16 svr-setup-zstd-lvl9.tar.zst
    -rw-r--r-- 1 root root 1324652195 Dec 16 04:16 svr-setup-zstd-lvl6.tar.zst
    -rw-r--r-- 1 root root 1098039642 Dec 16 04:16 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r-- 1 root root 1324652195 Dec 16 04:16 svr-setup.tar.zst
    -rw-r--r-- 1 root root 3690383360 Dec 16 04:16 svr-setup.tar
    


    tar + native zstd multi-threaded tests



    By default zstd used natively by tar uses only a single thread from above tests. That is already fast.

    tar + zstd single threaded run with default level 3 compression took 13.87 seconds
    Code (Text):
    time tar -caf /home/speedtest/svr-setup.tar.zst svr-setup
    
    real    0m13.870s
    user    0m14.148s
    sys     0m2.830s
    

    tar + zstd single threaded run with level 12 compression took 1m 34.634s or 94.634 seconds
    Code (Text):
    time tar "-I zstd -12 -T1" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup
    
    real    1m34.634s
    user    1m34.634s
    sys     0m2.933s
    

    tar + zstd single threaded run with level 19 compression took 14m 17.188s or 94.634 seconds
    Code (Text):
    time tar "-I zstd -19 -T1" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup
    
    real    14m17.188s
    user    14m18.003s
    sys     0m2.763s
    

    However it can be sped up dramatically by using more than one cpu thread via zstd -T flag passed through tar's -I flag.

    4 cpu thread zstd test at level 3 compression
    Code (Text):
    time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup
    
    real    0m7.625s
    user    0m19.528s
    sys     0m3.413s
    

    4 cpu thread zstd test at level 12 compression
    Code (Text):
    time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup
    
    real    0m37.832s
    user    2m23.620s
    sys     0m3.383s
    

    4 cpu thread zstd test at level 19 compression
    Code (Text):
    time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup
    
    real    4m13.144s
    user    16m36.715s
    sys     0m3.370s
    

    bump to 8 cpu thread zstd test at level 3 compression
    Code (Text):
    time tar "-I zstd -3 -T8" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup
    
    real    0m7.918s
    user    0m36.137s
    sys     0m3.482s
    

    bump to 8 cpu thread zstd test at level 12 compression
    Code (Text):
    time tar "-I zstd -12 -T8" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup
    
    real    0m31.399s
    user    3m35.345s
    sys     0m3.627s
    

    bump to 8 cpu thread zstd test at level 19 compression
    Code (Text):
    time tar "-I zstd -19 -T8" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup
    
    real    3m14.418s
    user    24m25.890s
    sys     0m4.215s
    

    Speed up on 8 zstd threads wasn't as much compared to 4 zstd threads seeing as Intel Core i7 4790K only really has 4 physical cpu cores and the additional hyperthreaded virtual 4 threads isn't as fast as physical cpu cores.

    Compared to popular xz single threaded and pxz multi-threaded compression tools as well

    tar + xz took 987.607 seconds
    Code (Text):
    time tar "-I xz -6" -cf /home/speedtest/svr-setup-xz-lvl6.tar.xz svr-setup                    
    
    real    16m27.607s
    user    16m25.924s
    sys     0m4.541s
    

    tar + pxz took 221.896 seconds
    Code (Text):
    time tar "-I pxz -6" -cf /home/speedtest/svr-setup-pxz-lvl6.tar.xz svr-setup
    
    real    3m41.896s
    user    25m9.745s
    sys     0m10.244s
    


    Comparison chart and table

    tar-zstd-table-01.png
    tar-zstd-chart-01.png
     
  4. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Facebook also posted an update on 5 ways Facebook improved compression at scale with Zstandard

    Interesting better compression with zstd long range mode
    from zstd 1.3.4 release notes https://github.com/facebook/zstd/releases/tag/v1.3.4
    zstd lvl 3 compression 4 threads without long range mode
    Code (Text):
    time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup
    
    real    0m7.625s
    user    0m19.528s
    sys     0m3.413s
    

    zstd lvl 3 compression 4 threads long=27
    Code (Text):
    time tar "-I zstd -3 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl3-long27.tar.zst svr-setup
    
    real    0m13.171s
    user    0m25.689s
    sys     0m3.338s
    

    zstd lvl 3 compression 4 threads long=31
    Code (Text):
    time tar "-I zstd -3 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl3-long31.tar.zst svr-setup
    
    real    0m13.317s
    user    0m24.447s
    sys     0m3.647s
    

    zstd lvl 12 compression 4 threads without long range mode
    Code (Text):
    time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup
    
    real    0m37.832s
    user    2m23.620s
    sys     0m3.383s
    

    zstd lvl 12 compression 4 threads long=27
    Code (Text):
    time tar "-I zstd -12 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl12-long27.tar.zst svr-setup
    
    real    0m30.850s
    user    1m59.543s
    sys     0m3.604s
    

    zstd lvl 12 compression 4 threads long=31
    Code (Text):
    time tar "-I zstd -12 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl12-long31.tar.zst svr-setup
    
    real    0m29.371s
    user    1m48.642s
    sys     0m4.176s
    

    zstd lvl 19 compression 4 threads without long range mode
    Code (Text):
    time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup
    
    real    4m13.144s
    user    16m36.715s
    sys     0m3.370s
    

    zstd lvl 19 compression 4 threads long=27
    Code (Text):
    time tar "-I zstd -19 -T4 --long=27" -cf /home/speedtest/svr-setup-zstd-lvl19-long27.tar.zst svr-setup
    
    real    4m13.295s
    user    15m33.865s
    sys     0m3.877s
    

    zstd lvl 19 compression 4 threads long=31
    Code (Text):
    time tar "-I zstd -19 -T4 --long=31" -cf /home/speedtest/svr-setup-zstd-lvl19-long31.tar.zst svr-setup
    
    real    3m32.687s
    user    13m9.800s
    sys     0m4.400s
    

    Code (Text):
    ls -lah /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
    -rw-r--r--   1 root root  911M Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
    -rw-r--r--   1 root root  844M Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
    -rw-r--r--   1 root root  1.1G Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r--   1 root root  813M Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
    -rw-r--r--   1 root root  760M Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
    -rw-r--r--   1 root root  937M Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
    -rw-r--r--   1 root root 1023M Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
    -rw-r--r--   1 root root  943M Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
    -rw-r--r--   1 root root  1.3G Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
    

    Code (Text):
    ls -la /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
    -rw-r--r--   1 root root  954719263 Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
    -rw-r--r--   1 root root  884625200 Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
    -rw-r--r--   1 root root 1098039638 Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r--   1 root root  852163352 Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
    -rw-r--r--   1 root root  795986668 Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
    -rw-r--r--   1 root root  981577668 Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
    -rw-r--r--   1 root root 1072611166 Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
    -rw-r--r--   1 root root  988555457 Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
    -rw-r--r--   1 root root 1395589259 Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
    


    zstd long range mode improves compression ratio and results in smaller compressed sizes :)

    tar-zstd-table-02.png
    tar-zstd-chart-02.png

    Then there's Automatic level determination mode
    zstd lvl 3 compression 4 threads without long range mode without automatic level determination
    Code (Text):
    time tar "-I zstd -3 -T4" -cf /home/speedtest/svr-setup-zstd-lvl3.tar.zst svr-setup
    
    real    0m7.625s
    user    0m19.528s
    sys     0m3.413s
    

    zstd lvl 3 compression 4 threads long=27 with automatic level determination
    Code (Text):
    time tar "-I zstd -3 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl3-long27-adapt.tar.zst svr-setup
    
    real    0m13.063s
    user    0m29.756s
    sys     0m3.363s
    

    zstd lvl 3 compression 4 threads long=31 with automatic level determination
    Code (Text):
    time tar "-I zstd -3 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl3-long31-adapt.tar.zst svr-setup
    
    real    0m13.782s
    user    0m26.946s
    sys     0m3.654s
    

    zstd lvl 12 compression 4 threads without long range mode without automatic level determination
    Code (Text):
    time tar "-I zstd -12 -T4" -cf /home/speedtest/svr-setup-zstd-lvl12.tar.zst svr-setup
    
    real    0m37.832s
    user    2m23.620s
    sys     0m3.383s
    

    zstd lvl 12 compression 4 threads long=27 with automatic level determination
    Code (Text):
    time tar "-I zstd -12 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl12-long27-adapt.tar.zst svr-setup
    
    real    0m30.198s
    user    1m55.895s
    sys     0m3.487s
    

    zstd lvl 12 compression 4 threads long=31 with automatic level determination
    Code (Text):
    time tar "-I zstd -12 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl12-long31-adapt.tar.zst svr-setup
    
    real    0m27.908s
    user    1m42.629s
    sys     0m4.037s
    

    zstd lvl 19 compression 4 threads without long range mode without automatic level determination
    Code (Text):
    time tar "-I zstd -19 -T4" -cf /home/speedtest/svr-setup-zstd-lvl19.tar.zst svr-setup
    
    real    4m13.144s
    user    16m36.715s
    sys     0m3.370s
    

    zstd lvl 19 compression 4 threads long=27 with automatic level determination
    Code (Text):
    time tar "-I zstd -19 -T4 --long=27 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl19-long27-adapt.tar.zst svr-setup
    
    real    4m16.817s
    user    15m54.161s
    sys     0m3.906s
    

    zstd lvl 19 compression 4 threads long=31 with automatic level determination
    Code (Text):
    time tar "-I zstd -19 -T4 --long=31 --adapt" -cf /home/speedtest/svr-setup-zstd-lvl19-long31-adapt.tar.zst svr-setup
    
    real    3m30.615s
    user    13m1.289s
    sys     0m4.204s
    

    Code (Text):
    ls -lah /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
    -rw-r--r--   1 root root  911M Dec 19 23:12 svr-setup-zstd-lvl12-long27-adapt.tar.zst
    -rw-r--r--   1 root root  911M Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
    -rw-r--r--   1 root root  844M Dec 19 23:12 svr-setup-zstd-lvl12-long31-adapt.tar.zst
    -rw-r--r--   1 root root  844M Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
    -rw-r--r--   1 root root  1.1G Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r--   1 root root  813M Dec 19 23:17 svr-setup-zstd-lvl19-long27-adapt.tar.zst
    -rw-r--r--   1 root root  813M Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
    -rw-r--r--   1 root root  760M Dec 19 23:23 svr-setup-zstd-lvl19-long31-adapt.tar.zst
    -rw-r--r--   1 root root  760M Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
    -rw-r--r--   1 root root  937M Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
    -rw-r--r--   1 root root 1020M Dec 19 23:10 svr-setup-zstd-lvl3-long27-adapt.tar.zst
    -rw-r--r--   1 root root 1023M Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
    -rw-r--r--   1 root root  941M Dec 19 23:11 svr-setup-zstd-lvl3-long31-adapt.tar.zst
    -rw-r--r--   1 root root  943M Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
    -rw-r--r--   1 root root  1.3G Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
    

    Code (Text):
    ls -la /home/speedtest/ | egrep 'lvl3|lvl12|lvl19'
    -rw-r--r--   1 root root  954949017 Dec 19 23:12 svr-setup-zstd-lvl12-long27-adapt.tar.zst
    -rw-r--r--   1 root root  954719263 Dec 19 22:38 svr-setup-zstd-lvl12-long27.tar.zst
    -rw-r--r--   1 root root  884729008 Dec 19 23:12 svr-setup-zstd-lvl12-long31-adapt.tar.zst
    -rw-r--r--   1 root root  884625200 Dec 19 22:39 svr-setup-zstd-lvl12-long31.tar.zst
    -rw-r--r--   1 root root 1098039638 Dec 19 01:10 svr-setup-zstd-lvl12.tar.zst
    -rw-r--r--   1 root root  852163352 Dec 19 23:17 svr-setup-zstd-lvl19-long27-adapt.tar.zst
    -rw-r--r--   1 root root  852163352 Dec 19 22:53 svr-setup-zstd-lvl19-long27.tar.zst
    -rw-r--r--   1 root root  795894243 Dec 19 23:23 svr-setup-zstd-lvl19-long31-adapt.tar.zst
    -rw-r--r--   1 root root  795986668 Dec 19 22:57 svr-setup-zstd-lvl19-long31.tar.zst
    -rw-r--r--   1 root root  981577668 Dec 19 02:31 svr-setup-zstd-lvl19.tar.zst
    -rw-r--r--   1 root root 1068830132 Dec 19 23:10 svr-setup-zstd-lvl3-long27-adapt.tar.zst
    -rw-r--r--   1 root root 1072611166 Dec 19 22:30 svr-setup-zstd-lvl3-long27.tar.zst
    -rw-r--r--   1 root root  986262705 Dec 19 23:11 svr-setup-zstd-lvl3-long31-adapt.tar.zst
    -rw-r--r--   1 root root  988555457 Dec 19 22:30 svr-setup-zstd-lvl3-long31.tar.zst
    -rw-r--r--   1 root root 1395589259 Dec 19 22:29 svr-setup-zstd-lvl3.tar.zst
    


    table and charts with --adapt added for zstd 4 cpu thread level 3, 12 and 19 tests

    tar-zstd-table-03.png

    tar-zstd-chart-03b.png
     
  5. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    tar + zstd backup for Centmin Mod Nginx Site Files



    Once the custom tar zstd supported RPM and zstd are installed, Centmin Mod users can use this version of tar to backup their Centmin Mod Nginx site's vhost directory and files efficiently with zstd compression - speeding up their backup process time + reducing the size of your compressed backups.

    Below is example manual SSH command line commands for both backup and restore of Centmin Mod site Nginx vhost directory and files using tar + zstd.

    backup



    Backup process involves you running the below SSH command line commands. Only variable you need to edit and adjust is vhostname variable change from domain.com to your site domain name that is located on your server. Full outline of Centmin Mod config file structure is outlined here.
    Code (Text):
    DT=$(date +"%d%m%y-%H%M%S")
    vhostname='domain.com'
    backupdir='/home/nginx-backup-sites'
    if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
    pushd /home/nginx/domains/ && mkdir -p "${backupdir}"
    time tar "-I zstd -3 -T0 --long=${long} --adapt" --exclude=log/* --exclude=backup/* -cpf "${backupdir}/${vhostname}-${DT}.tar.zst" ${vhostname}
    fs=$(ls -lAh "${backupdir}/${vhostname}-${DT}.tar.zst" | awk '{print $5,$9}') && echo "$fs"
    popd && echo "backup ${vhostname} files only complete at ${backupdir}/${vhostname}-${DT}.tar.zst"
    

    • 1st line assigns to DT variable a date timestamp which is appended to your tar.zst backup file
    • 2nd line for vhostname variable assigns variable for intended domain name you want to backup
    • 3rd line assigns backupdir variable the directory where you want to save your tar.zst backup file
    • 4th line determines which zstd long range method is used depending on how much free memory you have available. So if you have more than 2,500,000 KB free memory, zstd will use long range method with --long=31 for 2GB window. If you have less than 2,500,00 KB free memory, zstd will use long range method with --long=27 for 128MB window.
    • 5th line uses pushd to change into /home/nginx/domains directory and create the backup directory set on 3rd line's backupdir variable
    • 6th line, timed tar + zstd backup and compression which also uses zstd automatic level determination --adapt flag and -T0 meaning use all cpu cores. Tar also excludes contents of /home/nginx/domains/domain.com/log and /home/nginx/domains/domain.com/backup directories
    • 7th line assigns to fs variable size and full path name of resulting tar.zst backup file and prints that out via echo command
    • 8th line uses popd to return you to whatever directory you were in prior to pushd command on line 5 above and prints out completion message and full path to tar.zst backup file
    Listing of tar.zst backup compressed files
    Code (Text):
    ls -lah /home/nginx-backup-sites           
    total 32K
    drwxr-xr-x   2 root root 4.0K Dec 20 02:03 .
    drwxr-xr-x. 31 root root 4.0K Dec 20 01:01 ..
    -rw-r--r--   1 root root  12K Dec 20 01:08 domain.com-201218-010827.tar.zst
    -rw-r--r--   1 root root  12K Dec 20 01:35 domain.com-201218-013419.tar.zst
    


    restore



    Edit and assign to backupfile variable the full path to the backup file you want to restore from the backup commands you ran above.
    Code (Text):
    DT=$(date +"%d%m%y-%H%M%S")
    backupfile='/home/nginx-backup-sites/domain.com-201218-010827.tar.zst'
    vhostname='domain.com'
    restoredir="/home/nginx/domains/${vhostname}-restored-${DT}"
    mkdir -p ${restoredir}
    if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
    time tar "-I zstd -d -T0 --long=${long}" -xvf ${backupfile} -C ${restoredir}
    echo "diff compare restore versus existing directories"
    diff -r /home/nginx/domains/${vhostname} ${restoredir}/${vhostname}
    echo "restored ${vhostname} files to ${restoredir}"
    

    • 1st line assigns to backupfile variable the full path to the backup file you want to restore from the backup commands you ran above.
    • 2nd line for vhostname variable assigns variable for intended domain name you intend to restore to
    • 3rd line assigns restoredir variable the directory where the backup tar.zst files' contents will be restored to. You can then compare or sync directory/files with the existing version of the nginx vhost or move the restored files to replace existing version of the nginx vhost
    • 4th line creates the restore directory assigned to restoredir variable if it doesn't exist
    • 5th line determines which zstd long range method is used depending on how much free memory you have available. So if you have more than 2,500,000 KB free memory, zstd will use long range method with --long=31 for 2GB window. If you have less than 2,500,00 KB free memory, zstd will use long range method with --long=27 for 128MB window.
    • 6th line run timed tar decompress and extraction with zstd with all cpu cores and extract to restore directory
    • 7th & 8th lines print out the differences only between restored directory from tar.zst backup and the existing nginx vhost site
    • 9th line, restore completion message
    example of 5th line's tar decompress and extraction run
    Code (Text):
    time tar "-I zstd -d -T0" -xvf ${backupfile} -C ${restoredir}
    domain.com/
    domain.com/private/
    domain.com/backup/
    domain.com/public/
    domain.com/public/404.html
    domain.com/public/401.html
    domain.com/public/maintenance.html
    domain.com/public/50x.html
    domain.com/public/index.html
    domain.com/public/503.html
    domain.com/public/503.jpg
    domain.com/public/403.html
    domain.com/public/504.html
    domain.com/public/502.html
    domain.com/public/500.html
    domain.com/log/
    
    real    0m0.003s
    user    0m0.000s
    sys     0m0.003s
    

    Code (Text):
    tree /home/nginx/domains/domain.com-restored-201218-013419/domain.com/
    /home/nginx/domains/domain.com-restored-201218-013419/domain.com/
    ├── backup
    ├── log
    ├── private
    └── public
       ├── 401.html
       ├── 403.html
       ├── 404.html
       ├── 500.html
       ├── 502.html
       ├── 503.html
       ├── 503.jpg
       ├── 504.html
       ├── 50x.html
       ├── index.html
       └── maintenance.html
    
    4 directories, 11 files
    

    listing in /home/nginx/domains/ with existing nginx vhost directory and restored version
    Code (Text):
    ls -lah /home/nginx/domains/ | grep ${vhostname}
    drwxr-s---   6 nginx nginx 4.0K Dec 20 01:04 domain.com
    drwxr-sr-x   3 root  nginx 4.0K Dec 20 01:36 domain.com-restored-201218-013419
    

    differences reported between existing ngixn vhost directory and restored version
    Code (Text):
    diff -r /home/nginx/domains/${vhostname} ${restoredir}/${vhostname}
    Only in /home/nginx/domains/domain.com/log: access.log
    Only in /home/nginx/domains/domain.com/log: error.log
    


    Backup All Nginx Domain Files on Centmin Mod



    Want to backup all files (excluding mysql databases) for all Nginx sites on Centmin Mod located in /home/nginx/domains ?
    Code (Text):
    DT=$(date +"%d%m%y-%H%M%S")
    backupdir='/home/nginx-backup-sites'
    if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
    pushd /home/nginx/ && mkdir -p "${backupdir}"
    time tar "-I zstd -3 -T0 --long=${long} --adapt" --exclude=log/* --exclude=backup/* --exclude=*-all-restored-*/* -cpvvf "${backupdir}/domains-all-${DT}.tar.zst" domains
    fs=$(ls -lAh "${backupdir}/domains-all-${DT}.tar.zst" | awk '{print $5,$9}') && echo "$fs"
    popd && echo "backup all Nginx vhost files at /home/nginx/domains complete at ${backupdir}/domains-all-${DT}.tar.zst"
    

    Code (Text):
    ls -lah /home/nginx-backup-sites/
    total 24K
    drwxr-xr-x   2 root root 4.0K Dec 21 06:30 .
    drwxr-xr-x. 17 root root 4.0K Dec 21 06:29 ..
    -rw-r--r--   1 root root  13K Dec 21 06:30 domains-all-211218-062709.tar.zst
    


    restore
    Code (Text):
    DT=$(date +"%d%m%y-%H%M%S")
    backupfile='/home/nginx-backup-sites/domains-all-211218-062709.tar.zst'
    restoredir="/home/nginx/domains/domains-all-restored-${DT}"
    mkdir -p ${restoredir}
    if [[ "$(awk '/MemFree/ {print $2}' /proc/meminfo)" -ge '2500000' ]]; then long=31; else long=27; fi
    time tar "-I zstd -d -T0 --long=${long}" -xvf ${backupfile} -C ${restoredir}
    echo "diff compare restore versus existing directories"
    diff -r /home/nginx/domains/ ${restoredir}/domains
    echo "restored /home/nginx/domains files to ${restoredir}"
    

    Code (Text):
    tree /home/nginx/domains/domains-all-restored-211218-070634/domains/
    /home/nginx/domains/domains-all-restored-211218-070634/domains/
    ├── demodomain.com
    │   ├── backup
    │   ├── log
    │   ├── private
    │   └── public
    │       ├── 401.html
    │       ├── 403.html
    │       ├── 404.html
    │       ├── 500.html
    │       ├── 502.html
    │       ├── 503.html
    │       ├── 503.jpg
    │       ├── 504.html
    │       ├── 50x.html
    │       ├── index.html
    │       └── maintenance.html
    ├── http2.domain.com
    │   ├── backup
    │   ├── log
    │   ├── private
    │   └── public
    │       ├── 401.html
    │       ├── 403.html
    │       ├── 404.html
    │       ├── 500.html
    │       ├── 502.html
    │       ├── 503.html
    │       ├── 503.jpg
    │       ├── 504.html
    │       ├── 50x.html
    │       ├── index.html
    │       └── maintenance.html
    ├── newdomain.com
    │   └── backup
    └── testdb.com
        ├── backup
        ├── log
        ├── private
        └── public
            ├── 401.html
            ├── 403.html
            ├── 404.html
            ├── 500.html
            ├── 502.html
            ├── 503.html
            ├── 503.jpg
            ├── 504.html
            ├── 50x.html
            ├── index.html
            └── maintenance.html
    
    17 directories, 33 files
    

    Code (Text):
    diff -r /home/nginx/domains/ ${restoredir}/domains
    Only in /home/nginx/domains/demodomain.com/log: access.log
    Only in /home/nginx/domains/demodomain.com/log: error.log
    Only in /home/nginx/domains/: domains-all-restored-211218-070634
    Only in /home/nginx/domains/http2.domain.com/log: access.log
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181212.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181213.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181214.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181215.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181216.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181217.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181218.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181219.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181220.gz
    Only in /home/nginx/domains/http2.domain.com/log: access.log-20181221
    Only in /home/nginx/domains/http2.domain.com/log: error.log
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181210.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181211.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181212.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181213.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181214.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181217.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181218.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181219.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181220.gz
    Only in /home/nginx/domains/http2.domain.com/log: error.log-20181221
    Only in /home/nginx/domains/newdomain.com/backup: mysql
    Only in /home/nginx/domains/newdomain.com/backup: mysql-logs
    Only in /home/nginx/domains/testdb.com/backup: mysql
    Only in /home/nginx/domains/testdb.com/backup: mysql-logs
    Only in /home/nginx/domains/testdb.com/log: access.log
    Only in /home/nginx/domains/testdb.com/log: error.log
    
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Official Tar 1.31 stable release was announced with native zstd support so updated my above listed custom Tar RPM for CentOS 6 and 7 64 bit OSes.
    Code (Text):
    tar --version
    tar (GNU tar) 1.31
    Copyright (C) 2019 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    

    Code (Text):
    yum -q info tar-zstd-gcc7
    Installed Packages
    Name        : tar-zstd-gcc7
    Arch        : x86_64
    Version     : 1.31
    Release     : 1.el7
    Size        : 3.5 M
    Repo        : installed
    Summary     : tar 1.31 zstd supported for centminmod.com LEMP stack
    URL         : https://centminmod.com
    License     : unknown
    Description : tar 1.31 zstd supported for centminmod.com LEMP stack
    

    Code (Text):
    yum -q info tar-zstd-gcc7
    Installed Packages
    Name        : tar-zstd-gcc7
    Arch        : x86_64
    Version     : 1.31
    Release     : 1.el6
    Size        : 3.5 M
    Repo        : installed
    From repo   : /tar-zstd-gcc7-1.31-1.el6.x86_64
    Summary     : tar 1.31 zstd supported for centminmod.com LEMP stack
    URL         : https://centminmod.com
    License     : unknown
    Description : tar 1.31 zstd supported for centminmod.com LEMP stack
    
     
  7. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    tar 1.31 + zstd compression & decompression benchmarks



    I wrote a quick tar-zstd-test.sh script to test out my tar 1.31 custom RPM which has native zstd compression support and ran the script against uncompressed silesia corpus to compare tar 1.31 pairing with gzip, pigz, zstd, xz and pxz compression algorithms. Test on Intel Core i7 4790K 4C/8T 32GB 2x 240GB Samsung PM863 SSD raid 1 server running CentOS 7.6 64bit with Centmin Mod 123.09beta01 LEMP stack with tar 1.31 installed.

    tar-zstd-test.sh usage


    Code (Text):
    ./tar-zstd-test.sh
    
    ./tar-zstd-test.sh {prep|all|gzip|pigz|zstd|xz|pxz}
    

    Testing tar with gzip vs pigz vs zstd vs xz vs pxz compression algorithms
    Code (Text):
    gzip -V
    gzip 1.5
    
    pigz -V
    pigz 2.4
    
    zstd -V
    *** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
    
    xz -V
    xz (XZ Utils) 5.2.2
    liblzma 5.2.2
    
    pxz -V
    Parallel PXZ 4.999.9beta (build 20160426)
    xz (XZ Utils) 5.2.2
    liblzma 5.2.2
    

    gzip level 1 to 9


    Code (Text):
    ./tar-zstd-test.sh gzip
    ### gzip test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I gzip -1 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 3.51s cpu: 101% maxmem: 1388 KB cswaits: 6470
     compression ratio: 2.6976 211942676 / 78565167 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.33s cpu: 110% maxmem: 1320 KB cswaits: 6472
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I gzip -2 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 3.67s cpu: 101% maxmem: 1388 KB cswaits: 6469
     compression ratio: 2.7780 211942676 / 76290537 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.30s cpu: 110% maxmem: 1320 KB cswaits: 6473
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I gzip -3 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 4.28s cpu: 101% maxmem: 1388 KB cswaits: 6470
     compression ratio: 2.8546 211942676 / 74243600 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.40s cpu: 100% maxmem: 1320 KB cswaits: 6475
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I gzip -4 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 4.60s cpu: 101% maxmem: 1384 KB cswaits: 6470
     compression ratio: 2.9325 211942676 / 72271408 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.36s cpu: 104% maxmem: 1312 KB cswaits: 6475
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I gzip -5 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 5.78s cpu: 101% maxmem: 1388 KB cswaits: 6470
     compression ratio: 3.0099 211942676 / 70413987 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.26s cpu: 110% maxmem: 1316 KB cswaits: 6474
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I gzip -6 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 7.74s cpu: 100% maxmem: 1388 KB cswaits: 6469
     compression ratio: 3.0508 211942676 / 69470849 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.25s cpu: 111% maxmem: 1320 KB cswaits: 6471
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I gzip -7 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 9.08s cpu: 100% maxmem: 1388 KB cswaits: 6470
     compression ratio: 3.0634 211942676 / 69183567 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.26s cpu: 110% maxmem: 1316 KB cswaits: 6474
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I gzip -8 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 13.43s cpu: 100% maxmem: 1388 KB cswaits: 6470
     compression ratio: 3.0745 211942676 / 68934484 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.33s cpu: 103% maxmem: 1320 KB cswaits: 6478
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I gzip -9 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 16.71s cpu: 100% maxmem: 1388 KB cswaits: 6470
     compression ratio: 3.0765 211942676 / 68890462 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.24s cpu: 111% maxmem: 1316 KB cswaits: 6473
    

    pigz level 1 to 9


    • notice cpu utilisation for pigz almost used all 8 cpu threads on Intel Core i7 4790K cpu as such it ended up faster than zstd compression but at a lower compression ratio
    Code (Text):
    ./tar-zstd-test.sh pigz
    ### pigz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I pigz -1 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.63s cpu: 725% maxmem: 7656 KB cswaits: 9902
     compression ratio: 2.7085 211942676 / 78248308 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.66s cpu: 155% maxmem: 1320 KB cswaits: 21809
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I pigz -2 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.67s cpu: 732% maxmem: 7604 KB cswaits: 11371
     compression ratio: 2.7903 211942676 / 75956751 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.65s cpu: 157% maxmem: 1316 KB cswaits: 21719
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I pigz -3 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.78s cpu: 742% maxmem: 7764 KB cswaits: 10880
     compression ratio: 2.8702 211942676 / 73840861 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.63s cpu: 157% maxmem: 1316 KB cswaits: 21648
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I pigz -4 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.84s cpu: 748% maxmem: 7688 KB cswaits: 10766
     compression ratio: 2.9413 211942676 / 72057108 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.75s cpu: 136% maxmem: 1316 KB cswaits: 21622
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I pigz -5 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 1.06s cpu: 771% maxmem: 7656 KB cswaits: 11742
     compression ratio: 3.0187 211942676 / 70208692 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.64s cpu: 158% maxmem: 1320 KB cswaits: 21613
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I pigz -6 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 1.36s cpu: 794% maxmem: 7512 KB cswaits: 11518
     compression ratio: 3.0598 211942676 / 69266472 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.61s cpu: 158% maxmem: 1316 KB cswaits: 21603
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I pigz -7 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 1.67s cpu: 761% maxmem: 7808 KB cswaits: 13123
     compression ratio: 3.0724 211942676 / 68982378 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.61s cpu: 159% maxmem: 1316 KB cswaits: 21588
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I pigz -8 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.24s cpu: 791% maxmem: 7500 KB cswaits: 11178
     compression ratio: 3.0835 211942676 / 68733463 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.61s cpu: 160% maxmem: 1316 KB cswaits: 21490
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I pigz -9 --rsyncable" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.80s cpu: 769% maxmem: 7656 KB cswaits: 11361
     compression ratio: 3.0855 211942676 / 68688684 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.61s cpu: 159% maxmem: 1316 KB cswaits: 21563
    

    zstd level 1 to 19


    • notice cpu utilisation for zstd does not seem to be using all 8 cpu threads on Intel Core i7 4790K cpu and after zstd level 13 compression, cpu utilisation falls.
    • zstd level 11 compression times at 7.37s were equivalent to gzip level 6 default 7.74s but had higher compression ratio 3.6035 vs 3.0508
    • parallel xz, pxz level 2 compression times at 6.56s were faster than zstd level 11 at 7.37s and gzip level 6 at 7.74s and had a higher compression ratio at 3.7188 than zstd level 11 at 3.6035 and gzip level 6 at 3.0508
    • however pxz level 2 decompression times were slower at 2.93s compared to zstd level 11 at 1.93s
    Code (Text):
    ./tar-zstd-test.sh zstd                      
    ### zstd test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I zstd -1 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 2.92s cpu: 308% maxmem: 162036 KB cswaits: 8492
     compression ratio: 2.9241 211942676 / 72479417 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.97s cpu: 100% maxmem: 132464 KB cswaits: 12855
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I zstd -2 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 2.83s cpu: 321% maxmem: 166780 KB cswaits: 8485
     compression ratio: 3.0652 211942676 / 69143240 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.03s cpu: 104% maxmem: 132460 KB cswaits: 12854
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I zstd -3 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 3.18s cpu: 427% maxmem: 172560 KB cswaits: 8327
     compression ratio: 3.1925 211942676 / 66387130 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.22s cpu: 101% maxmem: 132436 KB cswaits: 12858
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I zstd -4 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 3.48s cpu: 453% maxmem: 196756 KB cswaits: 7915
     compression ratio: 3.2549 211942676 / 65114223 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.15s cpu: 103% maxmem: 132448 KB cswaits: 12887
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I zstd -5 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 3.76s cpu: 517% maxmem: 204400 KB cswaits: 7353
     compression ratio: 3.3322 211942676 / 63603510 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.16s cpu: 104% maxmem: 132444 KB cswaits: 12895
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I zstd -6 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 4.41s cpu: 490% maxmem: 235220 KB cswaits: 8031
     compression ratio: 3.3960 211942676 / 62409187 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.12s cpu: 104% maxmem: 132452 KB cswaits: 12913
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I zstd -7 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 4.68s cpu: 551% maxmem: 237472 KB cswaits: 7654
     compression ratio: 3.4796 211942676 / 60909269 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.01s cpu: 104% maxmem: 132448 KB cswaits: 12918
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I zstd -8 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 5.14s cpu: 571% maxmem: 238724 KB cswaits: 7566
     compression ratio: 3.5162 211942676 / 60275135 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.97s cpu: 104% maxmem: 132440 KB cswaits: 12911
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I zstd -9 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 5.58s cpu: 614% maxmem: 251772 KB cswaits: 7241
     compression ratio: 3.5482 211942676 / 59732045 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.94s cpu: 104% maxmem: 132440 KB cswaits: 12911
    ------------------------------------------------------------------------------
     [tar compress lvl: 10 ] tar "-I zstd -10 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 6.30s cpu: 593% maxmem: 328888 KB cswaits: 7091
     compression ratio: 3.5797 211942676 / 59206641 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.95s cpu: 105% maxmem: 132532 KB cswaits: 12918
    ------------------------------------------------------------------------------
     [tar compress lvl: 11 ] tar "-I zstd -11 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 7.37s cpu: 536% maxmem: 473992 KB cswaits: 8304
     compression ratio: 3.6035 211942676 / 58815270 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 104% maxmem: 132496 KB cswaits: 12931
    ------------------------------------------------------------------------------
     [tar compress lvl: 12 ] tar "-I zstd -12 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 9.07s cpu: 566% maxmem: 473572 KB cswaits: 8539
     compression ratio: 3.6289 211942676 / 58403470 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.92s cpu: 105% maxmem: 132500 KB cswaits: 12940
    ------------------------------------------------------------------------------
     [tar compress lvl: 13 ] tar "-I zstd -13 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 11.80s cpu: 575% maxmem: 473004 KB cswaits: 7580
     compression ratio: 3.6661 211942676 / 57810870 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.94s cpu: 101% maxmem: 132500 KB cswaits: 12933
    ------------------------------------------------------------------------------
     [tar compress lvl: 14 ] tar "-I zstd -14 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 18.68s cpu: 360% maxmem: 520832 KB cswaits: 8783
     compression ratio: 3.6990 211942676 / 57297060 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.84s cpu: 105% maxmem: 132504 KB cswaits: 12935
    ------------------------------------------------------------------------------
     [tar compress lvl: 15 ] tar "-I zstd -15 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 42.39s cpu: 151% maxmem: 470792 KB cswaits: 8790
     compression ratio: 3.7293 211942676 / 56831267 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.84s cpu: 105% maxmem: 132564 KB cswaits: 12934
    ------------------------------------------------------------------------------
     [tar compress lvl: 16 ] tar "-I zstd -16 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 32.66s cpu: 346% maxmem: 437868 KB cswaits: 8521
     compression ratio: 3.8056 211942676 / 55690893 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.86s cpu: 105% maxmem: 132528 KB cswaits: 12935
    ------------------------------------------------------------------------------
     [tar compress lvl: 17 ] tar "-I zstd -17 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 68.33s cpu: 163% maxmem: 420208 KB cswaits: 9269
     compression ratio: 3.8555 211942676 / 54971383 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.88s cpu: 105% maxmem: 132564 KB cswaits: 12940
    ------------------------------------------------------------------------------
     [tar compress lvl: 18 ] tar "-I zstd -18 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 106.88s cpu: 156% maxmem: 420952 KB cswaits: 9103
     compression ratio: 3.9234 211942676 / 54019608 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 104% maxmem: 132556 KB cswaits: 12938
    ------------------------------------------------------------------------------
     [tar compress lvl: 19 ] tar "-I zstd -19 -T0 --long --rsyncable" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 158.96s cpu: 147% maxmem: 434764 KB cswaits: 8389
     compression ratio: 3.9783 211942676 / 53273587 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T0" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.95s cpu: 104% maxmem: 132572 KB cswaits: 12938
    

    xz level 1 to 9



    Code (Text):
    ./tar-zstd-test.sh xz
    ### xz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I xz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 11.94s cpu: 100% maxmem: 9616 KB cswaits: 25870
     compression ratio: 3.6282 211942676 / 58414444 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.06s cpu: 105% maxmem: 2036 KB cswaits: 25873
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I xz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 18.73s cpu: 100% maxmem: 17296 KB cswaits: 25866
     compression ratio: 3.7371 211942676 / 56712980 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.00s cpu: 103% maxmem: 3060 KB cswaits: 25834
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I xz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 30.85s cpu: 100% maxmem: 32652 KB cswaits: 25870
     compression ratio: 3.8016 211942676 / 55749668 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.84s cpu: 106% maxmem: 5104 KB cswaits: 25869
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I xz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 48.43s cpu: 100% maxmem: 49228 KB cswaits: 25872
     compression ratio: 4.0672 211942676 / 52109368 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.82s cpu: 106% maxmem: 5104 KB cswaits: 25861
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I xz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 64.46s cpu: 100% maxmem: 96328 KB cswaits: 25875
     compression ratio: 4.2401 211942676 / 49984712 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.74s cpu: 106% maxmem: 9204 KB cswaits: 25859
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I xz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 75.01s cpu: 100% maxmem: 96332 KB cswaits: 25874
     compression ratio: 4.3044 211942676 / 49237796 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.70s cpu: 106% maxmem: 9200 KB cswaits: 25861
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I xz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 79.00s cpu: 100% maxmem: 190540 KB cswaits: 25876
     compression ratio: 4.3297 211942676 / 48950732 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.73s cpu: 107% maxmem: 17396 KB cswaits: 25867
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I xz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 82.92s cpu: 100% maxmem: 378952 KB cswaits: 25877
     compression ratio: 4.3472 211942676 / 48752848 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.82s cpu: 106% maxmem: 33780 KB cswaits: 25836
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I xz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 84.48s cpu: 100% maxmem: 690248 KB cswaits: 25874
     compression ratio: 4.3500 211942676 / 48721676 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.74s cpu: 106% maxmem: 66548 KB cswaits: 25847
    

    pxz level 1 to 9



    Code (Text):
    ./tar-zstd-test.sh pxz
    ### pxz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I pxz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 4.29s cpu: 526% maxmem: 94300 KB cswaits: 4968
     compression ratio: 3.6086 211942676 / 58731740 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.17s cpu: 103% maxmem: 2032 KB cswaits: 25846
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I pxz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 6.56s cpu: 536% maxmem: 180152 KB cswaits: 4071
     compression ratio: 3.7188 211942676 / 56992000 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.93s cpu: 106% maxmem: 3056 KB cswaits: 25862
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I pxz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 10.49s cpu: 538% maxmem: 351176 KB cswaits: 2253
     compression ratio: 3.7820 211942676 / 56039192 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.86s cpu: 106% maxmem: 5108 KB cswaits: 25858
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I pxz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 14.48s cpu: 514% maxmem: 481120 KB cswaits: 3044
     compression ratio: 4.0367 211942676 / 52503612 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.96s cpu: 103% maxmem: 5108 KB cswaits: 25853
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I pxz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 18.11s cpu: 539% maxmem: 924368 KB cswaits: 341
     compression ratio: 4.2126 211942676 / 50310532 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.84s cpu: 103% maxmem: 9204 KB cswaits: 25837
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I pxz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 20.61s cpu: 555% maxmem: 952936 KB cswaits: 434
     compression ratio: 4.2753 211942676 / 49573244 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.98s cpu: 98% maxmem: 9200 KB cswaits: 25841
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I pxz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 27.15s cpu: 354% maxmem: 965744 KB cswaits: 127
     compression ratio: 4.3057 211942676 / 49223036 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.73s cpu: 106% maxmem: 17392 KB cswaits: 25866
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I pxz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 47.40s cpu: 181% maxmem: 964016 KB cswaits: 88
     compression ratio: 4.3316 211942676 / 48929028 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.74s cpu: 106% maxmem: 33780 KB cswaits: 25861
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I pxz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 80.17s cpu: 104% maxmem: 897328 KB cswaits: 62
     compression ratio: 4.3415 211942676 / 48816968 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.77s cpu: 107% maxmem: 66544 KB cswaits: 25861
     


    Updated benchmarks



    I noticed zstd with -T0 --long --rsyncable benchmarks above were slower than pigz multi-threaded gzip with less cpu utilisation as well. So decided to find which zstd options were causing the slowest and re-run tests with --rsyncable disabled as well as with zstd -8 with and without --long. It seems for silesia corpus tar compression tests, zstd with -T0 was using less cpu tilisation than -T8 and --long was the culprit slowing zstd compression. The --rsyncable option for gzip, pigz slightly slowed down compression times but for zstd --rsyncable had a more dramatic slow down for compression times/speed.

    tar-zstd-tests-01.png tar-zstd-tests-02.png tar-zstd-tests-03.png tar-zstd-tests-04.png
     
    • Like Like x 1
  8. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    tar + zstd versus tar alone



    Discussion came up about how disk space is cheap these days so backups can be done uncompressed so zstd isn't as useful. So decided to test that theory with tar v1.31 + zstd 1.3.8 versus tar only without compression and check speeds and resulting tar.zst compressed and tar uncompressed file sizes.

    Testing on low end ImpactVPS 2 cpu KVM VPS with Intel Xeon Silver 4114 Skylake cpu

    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                2
    On-line CPU(s) list:   0,1
    Thread(s) per core:    1
    Core(s) per socket:    1
    Socket(s):             2
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 85
    Model name:            Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    Stepping:              4
    CPU MHz:               2199.992
    BogoMIPS:              4399.98
    Virtualization:        VT-x
    Hypervisor vendor:     KVM
    Virtualization type:   full
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              4096K
    NUMA node0 CPU(s):     0,1
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 arat spec_ctrl
    


    All tests to be done using tar v1.31 custom RPM with native zstd support
    Code (Text):
    tar --version    
    tar (GNU tar) 1.31
    Copyright (C) 2019 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Written by John Gilmore and Jay Fenlason.
    

    First the /svr-setup directory to be tar'd up is 1.70GB in size
    Code (Text):
    du -sh /svr-setup/
    1.7G    /svr-setup/
    

    Timed tar without compression speed at 27.582 seconds
    Code (Text):
    time tar cf svr-setup.tar svr-setup/
    
    real    0m27.582s
    user    0m1.190s
    sys     0m10.934s
    

    resulting in /svr-setup.tar file 1.60GB in size
    Code (Text):
    ls -lah /svr-setup.tar
    -rw-r--r-- 1 root root 1.6G Jan 10 18:18 /svr-setup.tar
    

    tar with zstd level 3 default compression single threaded default
    Code (Text):
    time tar cfa svr-setup.tar.zst svr-setup
    
    real    1m58.883s
    user    1m58.203s
    sys     0m7.051s
    

    resulting in 644MB compressed svr-setup.tar.zst file
    Code (Text):
    ls -lah /svr-setup.tar.zst          
    -rw-r--r-- 1 root root 664M Jan 10 18:22 /svr-setup.tar.zst
    

    now test tar v1.31 with zstd level 1 compression 2 cores = 44.765s
    Code (Text):
    time tar "-I zstd -1 -T2" -cf svr-setup.tar.zst svr-setup      
    
    real    0m44.765s
    user    1m9.498s
    sys     0m6.019s
    

    724MB compressed svr-setup.tar.zst file
    Code (Text):
    ls -lah /svr-setup.tar.zst                                    
    -rw-r--r-- 1 root root 724M Jan 10 18:32 /svr-setup.tar.zst
    

    zstd v1.3.8 also has negative compression levels which sacrifice compression ratio size for speed.

    tar v1.31 with zstd level -1 compression 2 cores = 38.634s getting closer to tar no-compress speeds
    Code (Text):
    time tar "-I zstd --fast=1 -T2" -cf svr-setup.tar.zst svr-setup
    
    real    0m38.634s
    user    0m57.896s
    sys     0m6.269s
    

    but file size for compressed svr-setup.tar.zst is 781MB much smaller than tar no-compress 1.60GB size
    Code (Text):
    ls -lah /svr-setup.tar.zst                                    
    -rw-r--r-- 1 root root 781M Jan 10 18:34 /svr-setup.tar.zst
    

    tar v1.31 with zstd level -2 compression 2 cores at 32.454s
    Code (Text):
    time tar "-I zstd --fast=2 -T2" -cf svr-setup.tar.zst svr-setup
    
    real    0m32.454s
    user    0m48.458s
    sys     0m5.587s
    

    compressed svr-setup.tar.zst is 834MB
    Code (Text):
    ls -lah /svr-setup.tar.zst                                    
    -rw-r--r-- 1 root root 834M Jan 10 18:35 /svr-setup.tar.zst
    

    tar v1.31 with zstd level -3 compression 2 cores at 29.190s
    Code (Text):
    time tar "-I zstd --fast=3 -T2" -cf svr-setup.tar.zst svr-setup
    
    real    0m29.190s
    user    0m42.845s
    sys     0m4.946s
    

    compressed svr-setup.tar.zst is 868MB
    Code (Text):
    ls -lah /svr-setup.tar.zst                                    
    -rw-r--r-- 1 root root 868M Jan 10 18:36 /svr-setup.tar.zst
    

    tar v1.31 with zstd level -4 compression 2 cores at 27.461s which is faster than tar no-compress time of 27.582s
    Code (Text):
    time tar "-I zstd --fast=4 -T2" -cf svr-setup.tar.zst svr-setup
    
    real    0m27.461s
    user    0m39.677s
    sys     0m4.809s
    

    compressed svr-setup.tar.zst is 908MB smaller than tar no-compress size of 1.60GB !
    Code (Text):
    ls -lah /svr-setup.tar.zst                                    
    -rw-r--r-- 1 root root 908M Jan 10 18:37 /svr-setup.tar.zst
    


    So yes it's possible for tar + zstd to compress files at speed close to tar uncompressed sizes when using zstd 1.3.8+ with negative compression levels. In this case it took zstd -4 compression level to achieve tar + zstd compression speeds faster than tar uncompressed :)

    Also it would be even faster if you had more cpu cores to work with and set -TX where X is number of cpu cores. This server only had 2 cpu threads.

    Edit: now compare that tar + gzip level 1 & 6 and tar + pigz level 1 & 6 tests

    pigz multi-threaded gzip default level 6 tar
    Code (Text):
    time tar "-I pigz" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz                   
    
    real    1m8.257s
    user    1m56.274s
    sys     0m7.501s
    -rw-r--r-- 1 root root 692M Jan 11 00:47 /svr-setup.tar.gz
    

    pigz multi-threaded gzip default level 1 tar
    Code (Text):
    time tar "-I pigz -1" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz
    
    real    0m34.844s
    user    0m53.258s
    sys     0m4.954s
    -rw-r--r-- 1 root root 742M Jan 11 00:48 /svr-setup.tar.gz
    

    gzip single threaded default level 6 tar
    Code (Text):
    time tar "-I gzip" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz       
    
    real    1m58.683s
    user    1m55.101s
    sys     0m7.383s
    -rw-r--r-- 1 root root 692M Jan 11 00:51 /svr-setup.tar.gz
    

    gzip single threaded level 1 tar
    Code (Text):
    time tar "-I gzip -1" -cf svr-setup.tar.gz svr-setup  ; ls -lah /svr-setup.tar.gz
    
    real    0m57.323s
    user    0m53.631s
    sys     0m6.508s
    -rw-r--r-- 1 root root 744M Jan 11 00:52 /svr-setup.tar.gz
    
     
    • Like Like x 1
  9. eva2000

    eva2000 Administrator Staff Member

    37,710
    8,254
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,710
    Local Time:
    7:09 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    tar 1.31 + zstd compression & decompression benchmarks part 2



    I wrote a quick tar-zstd-test.sh script to test out my tar 1.31 custom RPM which has native zstd compression support and ran the script against uncompressed silesia corpus to compare tar 1.31 pairing with gzip, pigz, zstd, xz, pxz & lz4 compression algorithms. I have since added pigz level 11 for zopfli compression and added all 32 levels of compression for zstd - from negative levels (-10 to -1) which focus on compression speed, normal levels 1 to 19 and then the 3 ultra levels 20 to 22 which focus on better compression ratios.

    Test System
    • OVH MC-32 Intel Core i7 4790K
    • 32GB Memory
    • 2x240GB SSD Samsung PM863 SSD raid 1
    • 250Mbit Network Bandwidth
    • CentOS 7.6 64bit
    • Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.8, MariaDB 10.1.37 MySQL, + CSF Firewall
    • BHS, Canada
    • tar 1.31 installed

    tar-zstd-test.sh usage


    Code (Text):
    ./tar-zstd-test.sh
    
    ./tar-zstd-test.sh {prep|all|gzip|pigz|zstd|xz|pxz|lz4}
    

    Testing tar with gzip vs pigz vs zstd vs xz vs pxz vs lz4 compression algorithms
    Code (Text):
    gzip -V
    gzip 1.5
    
    pigz -V
    pigz 2.4
    
    zstd -V
    *** zstd command line interface 64-bits v1.3.8, by Yann Collet ***
    
    xz -V
    xz (XZ Utils) 5.2.2
    liblzma 5.2.2
    
    pxz -V
    Parallel PXZ 4.999.9beta (build 20160426)
    xz (XZ Utils) 5.2.2
    liblzma 5.2.2
    
    lz4 -V
    *** LZ4 command line interface 64-bits v1.8.3, by Yann Collet ***
    

    I will be running with all flag to test all compression algorithms in same session.
    Code (Text):
    ./tar-zstd-test.sh all
    

    Also to compare with tar uncompressed times, here's a tar uncompressed run
    Code (Text):
    tar -cf /home/tar-zstd/silesia.tar silesia
    real: 0.38s user: 0.01s sys: 0.11s cpu: 31% maxmem: 1308 KB cswaits: 43
    
    ls -lah /home/tar-zstd/silesia.tar
    -rw-r--r-- 1 root root 203M Jan 11 20:44 /home/tar-zstd/silesia.tar
    
    tar -xf /home/tar-zstd/silesia.tar -C /home/tar-zstd-extract
    real: 0.39s user: 0.00s sys: 0.15s cpu: 39% maxmem: 1312 KB cswaits: 22
    


    tar compression results
    Code (Text):
    ./tar-zstd-test.sh all                            
    ### gzip test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I gzip -1" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.55s cpu: 102% maxmem: 1392 KB cswaits: 6470
     [compress speed:] 79.26 MB/s
     compression ratio: 2.7394 211942676 / 77365465 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.13s cpu: 112% maxmem: 1320 KB cswaits: 6472
     [decompress speed:] 178.87 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I gzip -2" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.76s cpu: 102% maxmem: 1392 KB cswaits: 6469
     [compress speed:] 73.23 MB/s
     compression ratio: 2.8216 211942676 / 75113751 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.10s cpu: 112% maxmem: 1320 KB cswaits: 6471
     [decompress speed:] 183.74 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I gzip -3" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 3.45s cpu: 101% maxmem: 1392 KB cswaits: 6470
     [compress speed:] 58.58 MB/s
     compression ratio: 2.9013 211942676 / 73050277 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.09s cpu: 114% maxmem: 1316 KB cswaits: 6473
     [decompress speed:] 185.43 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I gzip -4" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 3.57s cpu: 101% maxmem: 1392 KB cswaits: 6468
     [compress speed:] 56.61 MB/s
     compression ratio: 2.9826 211942676 / 71057874 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.08s cpu: 113% maxmem: 1320 KB cswaits: 6472
     [decompress speed:] 187.15 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I gzip -5" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 4.77s cpu: 101% maxmem: 1392 KB cswaits: 6469
     [compress speed:] 42.37 MB/s
     compression ratio: 3.0648 211942676 / 69153440 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.08s cpu: 112% maxmem: 1320 KB cswaits: 6472
     [decompress speed:] 187.15 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I gzip -6" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 6.66s cpu: 100% maxmem: 1392 KB cswaits: 6469
     [compress speed:] 30.34 MB/s
     compression ratio: 3.1066 211942676 / 68222450 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.04s cpu: 113% maxmem: 1320 KB cswaits: 6473
     [decompress speed:] 194.35 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I gzip -7" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 8.04s cpu: 100% maxmem: 1392 KB cswaits: 6469
     [compress speed:] 25.13 MB/s
     compression ratio: 3.1195 211942676 / 67941131 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.07s cpu: 115% maxmem: 1316 KB cswaits: 6473
     [decompress speed:] 188.90 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I gzip -8" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 12.36s cpu: 100% maxmem: 1388 KB cswaits: 6471
     [compress speed:] 16.35 MB/s
     compression ratio: 3.1308 211942676 / 67693882 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.10s cpu: 113% maxmem: 1316 KB cswaits: 6473
     [decompress speed:] 183.74 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I gzip -9" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 15.72s cpu: 100% maxmem: 1388 KB cswaits: 6470
     [compress speed:] 12.85 MB/s
     compression ratio: 3.1329 211942676 / 67649708 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I gzip" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 1.03s cpu: 113% maxmem: 1316 KB cswaits: 6473
     [decompress speed:] 196.23 MB/s
    ### pigz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I pigz -1" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.50s cpu: 770% maxmem: 7676 KB cswaits: 11309
     [compress speed:] 404.24 MB/s
     compression ratio: 2.7463 211942676 / 77171874 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.59s cpu: 163% maxmem: 1320 KB cswaits: 21748
     [decompress speed:] 342.58 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I pigz -2" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.54s cpu: 782% maxmem: 7760 KB cswaits: 11187
     [compress speed:] 374.30 MB/s
     compression ratio: 2.8298 211942676 / 74894231 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.58s cpu: 163% maxmem: 1320 KB cswaits: 21686
     [decompress speed:] 348.49 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I pigz -3" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.66s cpu: 779% maxmem: 7652 KB cswaits: 13136
     [compress speed:] 306.24 MB/s
     compression ratio: 2.9111 211942676 / 72802779 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.60s cpu: 165% maxmem: 1320 KB cswaits: 21608
     [decompress speed:] 336.87 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I pigz -4" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.72s cpu: 786% maxmem: 7640 KB cswaits: 11574
     [compress speed:] 280.72 MB/s
     compression ratio: 2.9848 211942676 / 71005642 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.56s cpu: 165% maxmem: 1316 KB cswaits: 21569
     [decompress speed:] 360.93 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I pigz -5" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 0.93s cpu: 793% maxmem: 7468 KB cswaits: 11783
     [compress speed:] 217.33 MB/s
     compression ratio: 3.0642 211942676 / 69167137 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.58s cpu: 164% maxmem: 1320 KB cswaits: 21566
     [decompress speed:] 348.49 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I pigz -6" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 1.30s cpu: 764% maxmem: 7640 KB cswaits: 15437
     [compress speed:] 155.48 MB/s
     compression ratio: 3.1061 211942676 / 68234196 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.55s cpu: 167% maxmem: 1320 KB cswaits: 21511
     [decompress speed:] 367.49 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I pigz -7" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 1.49s cpu: 792% maxmem: 7480 KB cswaits: 12014
     [compress speed:] 135.65 MB/s
     compression ratio: 3.1192 211942676 / 67946718 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.55s cpu: 166% maxmem: 1320 KB cswaits: 21540
     [decompress speed:] 367.49 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I pigz -8" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.16s cpu: 788% maxmem: 7684 KB cswaits: 12500
     [compress speed:] 93.57 MB/s
     compression ratio: 3.1306 211942676 / 67699404 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.54s cpu: 170% maxmem: 1316 KB cswaits: 21425
     [decompress speed:] 374.30 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I pigz -9" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 2.69s cpu: 784% maxmem: 7736 KB cswaits: 11938
     [compress speed:] 75.13 MB/s
     compression ratio: 3.1327 211942676 / 67653907 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.54s cpu: 167% maxmem: 1320 KB cswaits: 21536
     [decompress speed:] 374.30 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 11 ] tar "-I pigz -11" -cf /home/tar-zstd/silesia.tar.gz silesia
     [compress stats:] real: 237.85s cpu: 789% maxmem: 55756 KB cswaits: 12005
     [compress speed:] .84 MB/s
     compression ratio: 3.2845 211942676 / 64527580 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pigz" -xf /home/tar-zstd/silesia.tar.gz -C /home/tar-zstd-extract
     [decompress stats:] real: 0.57s cpu: 168% maxmem: 1320 KB cswaits: 21216
     [decompress speed:] 354.60 MB/s
    ### zstd test
    ------------------------------------------------------------------------------
     [tar compress lvl: -10 ] tar "-I zstd --fast=10 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.49s cpu: 506% maxmem: 44516 KB cswaits: 8121
     [compress speed:] 412.49 MB/s
     compression ratio: 1.7824 211942676 / 118907221 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.21s cpu: 100% maxmem: 1856 KB cswaits: 12944
     [decompress speed:] 167.04 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -9 ] tar "-I zstd --fast=9 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.42s cpu: 618% maxmem: 43712 KB cswaits: 7743
     [compress speed:] 481.24 MB/s
     compression ratio: 1.8016 211942676 / 117639597 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.12s cpu: 103% maxmem: 1856 KB cswaits: 12943
     [decompress speed:] 180.46 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -8 ] tar "-I zstd --fast=8 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.43s cpu: 624% maxmem: 43912 KB cswaits: 7750
     [compress speed:] 470.05 MB/s
     compression ratio: 1.8477 211942676 / 114706049 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.06s cpu: 109% maxmem: 1848 KB cswaits: 12938
     [decompress speed:] 190.68 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -7 ] tar "-I zstd --fast=7 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.45s cpu: 622% maxmem: 44420 KB cswaits: 8220
     [compress speed:] 449.16 MB/s
     compression ratio: 1.8675 211942676 / 113488957 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.11s cpu: 109% maxmem: 1848 KB cswaits: 12925
     [decompress speed:] 182.09 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -6 ] tar "-I zstd --fast=6 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.54s cpu: 533% maxmem: 43236 KB cswaits: 7497
     [compress speed:] 374.30 MB/s
     compression ratio: 1.9370 211942676 / 109417899 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.14s cpu: 108% maxmem: 1848 KB cswaits: 12939
     [decompress speed:] 177.30 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -5 ] tar "-I zstd --fast=5 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.48s cpu: 632% maxmem: 44048 KB cswaits: 7772
     [compress speed:] 421.09 MB/s
     compression ratio: 1.9947 211942676 / 106252061 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.17s cpu: 108% maxmem: 1844 KB cswaits: 12939
     [decompress speed:] 172.75 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -4 ] tar "-I zstd --fast=4 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.51s cpu: 632% maxmem: 42444 KB cswaits: 7522
     [compress speed:] 396.32 MB/s
     compression ratio: 2.0672 211942676 / 102525133 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.21s cpu: 107% maxmem: 1840 KB cswaits: 12936
     [decompress speed:] 167.04 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -3 ] tar "-I zstd --fast=3 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.54s cpu: 669% maxmem: 41696 KB cswaits: 7356
     [compress speed:] 374.30 MB/s
     compression ratio: 2.1507 211942676 / 98541355 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.28s cpu: 107% maxmem: 1836 KB cswaits: 12938
     [decompress speed:] 157.90 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -2 ] tar "-I zstd --fast=2 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.60s cpu: 677% maxmem: 43516 KB cswaits: 8361
     [compress speed:] 336.87 MB/s
     compression ratio: 2.2641 211942676 / 93609955 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.38s cpu: 107% maxmem: 1832 KB cswaits: 12937
     [decompress speed:] 146.46 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: -1 ] tar "-I zstd --fast=1 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.76s cpu: 645% maxmem: 42996 KB cswaits: 7773
     [compress speed:] 265.95 MB/s
     compression ratio: 2.4296 211942676 / 87232776 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.43s cpu: 106% maxmem: 1824 KB cswaits: 12936
     [decompress speed:] 141.34 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I zstd -1 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.82s cpu: 691% maxmem: 40776 KB cswaits: 7889
     [compress speed:] 246.49 MB/s
     compression ratio: 2.8766 211942676 / 73678095 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.89s cpu: 105% maxmem: 1996 KB cswaits: 12931
     [decompress speed:] 106.94 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I zstd -2 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 0.91s cpu: 666% maxmem: 73532 KB cswaits: 8217
     [compress speed:] 222.11 MB/s
     compression ratio: 3.0314 211942676 / 69914803 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.16s cpu: 101% maxmem: 2508 KB cswaits: 12941
     [decompress speed:] 93.57 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I zstd -3 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 1.69s cpu: 626% maxmem: 137708 KB cswaits: 8586
     [compress speed:] 119.60 MB/s
     compression ratio: 3.1722 211942676 / 66811326 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.11s cpu: 105% maxmem: 3504 KB cswaits: 12935
     [decompress speed:] 95.79 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I zstd -4 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 2.05s cpu: 591% maxmem: 147360 KB cswaits: 7922
     [compress speed:] 98.59 MB/s
     compression ratio: 3.2259 211942676 / 65699482 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.10s cpu: 104% maxmem: 3500 KB cswaits: 12940
     [decompress speed:] 96.24 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I zstd -5 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 2.77s cpu: 600% maxmem: 155204 KB cswaits: 8109
     [compress speed:] 72.96 MB/s
     compression ratio: 3.3088 211942676 / 64052950 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.16s cpu: 104% maxmem: 3492 KB cswaits: 12939
     [decompress speed:] 93.57 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I zstd -6 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 3.14s cpu: 592% maxmem: 162916 KB cswaits: 8451
     [compress speed:] 64.37 MB/s
     compression ratio: 3.3660 211942676 / 62964179 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.10s cpu: 104% maxmem: 3496 KB cswaits: 12938
     [decompress speed:] 96.24 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I zstd -7 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 3.66s cpu: 625% maxmem: 161644 KB cswaits: 8228
     [compress speed:] 55.22 MB/s
     compression ratio: 3.4503 211942676 / 61426311 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.09s cpu: 101% maxmem: 3492 KB cswaits: 12940
     [decompress speed:] 96.71 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I zstd -8 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 4.06s cpu: 633% maxmem: 159200 KB cswaits: 7846
     [compress speed:] 49.78 MB/s
     compression ratio: 3.4856 211942676 / 60804851 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.06s cpu: 99% maxmem: 3488 KB cswaits: 12939
     [decompress speed:] 98.11 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I zstd -9 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 4.91s cpu: 604% maxmem: 174712 KB cswaits: 8002
     [compress speed:] 41.16 MB/s
     compression ratio: 3.5182 211942676 / 60241424 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 105% maxmem: 3488 KB cswaits: 12941
     [decompress speed:] 104.72 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 10 ] tar "-I zstd -10 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 5.72s cpu: 586% maxmem: 331796 KB cswaits: 8624
     [compress speed:] 35.33 MB/s
     compression ratio: 3.5579 211942676 / 59569376 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.91s cpu: 105% maxmem: 5536 KB cswaits: 12939
     [decompress speed:] 105.82 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 11 ] tar "-I zstd -11 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 6.21s cpu: 587% maxmem: 429872 KB cswaits: 8508
     [compress speed:] 32.54 MB/s
     compression ratio: 3.5748 211942676 / 59286517 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 2.04s cpu: 104% maxmem: 5536 KB cswaits: 12937
     [decompress speed:] 99.08 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 12 ] tar "-I zstd -12 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 7.97s cpu: 615% maxmem: 429420 KB cswaits: 8521
     [compress speed:] 25.36 MB/s
     compression ratio: 3.6030 211942676 / 58823794 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.95s cpu: 103% maxmem: 5536 KB cswaits: 12941
     [decompress speed:] 103.65 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 13 ] tar "-I zstd -13 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 10.25s cpu: 671% maxmem: 424356 KB cswaits: 8800
     [compress speed:] 19.71 MB/s
     compression ratio: 3.6464 211942676 / 58122395 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.88s cpu: 102% maxmem: 5540 KB cswaits: 12941
     [decompress speed:] 107.51 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 14 ] tar "-I zstd -14 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 11.70s cpu: 655% maxmem: 620516 KB cswaits: 8928
     [compress speed:] 17.27 MB/s
     compression ratio: 3.6775 211942676 / 57632128 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.81s cpu: 105% maxmem: 5540 KB cswaits: 12936
     [decompress speed:] 111.67 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 15 ] tar "-I zstd -15 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 13.67s cpu: 685% maxmem: 751272 KB cswaits: 8219
     [compress speed:] 14.78 MB/s
     compression ratio: 3.7008 211942676 / 57269219 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.80s cpu: 105% maxmem: 5540 KB cswaits: 12937
     [decompress speed:] 112.29 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 16 ] tar "-I zstd -16 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 20.88s cpu: 652% maxmem: 493176 KB cswaits: 8092
     [compress speed:] 9.68 MB/s
     compression ratio: 3.8044 211942676 / 55709560 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.83s cpu: 105% maxmem: 5540 KB cswaits: 12939
     [decompress speed:] 110.45 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 17 ] tar "-I zstd -17 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 28.45s cpu: 540% maxmem: 608544 KB cswaits: 9523
     [compress speed:] 7.10 MB/s
     compression ratio: 3.8802 211942676 / 54620629 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.86s cpu: 105% maxmem: 9652 KB cswaits: 12940
     [decompress speed:] 108.66 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 18 ] tar "-I zstd -18 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 42.73s cpu: 554% maxmem: 611368 KB cswaits: 8969
     [compress speed:] 4.73 MB/s
     compression ratio: 3.9480 211942676 / 53683348 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 105% maxmem: 9628 KB cswaits: 12911
     [decompress speed:] 104.72 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 19 ] tar "-I zstd -19 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 61.23s cpu: 513% maxmem: 840380 KB cswaits: 9803
     [compress speed:] 3.30 MB/s
     compression ratio: 3.9796 211942676 / 53256215 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 105% maxmem: 9628 KB cswaits: 12938
     [decompress speed:] 104.72 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 20 ] tar "-I zstd --ultra -20 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 140.72s cpu: 163% maxmem: 589608 KB cswaits: 8424
     [compress speed:] 1.43 MB/s
     compression ratio: 4.0115 211942676 / 52832599 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.93s cpu: 105% maxmem: 34204 KB cswaits: 12940
     [decompress speed:] 104.72 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 21 ] tar "-I zstd --ultra -21 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 243.44s cpu: 100% maxmem: 588488 KB cswaits: 8578
     [compress speed:] .83 MB/s
     compression ratio: 4.0210 211942676 / 52708823 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.95s cpu: 105% maxmem: 66976 KB cswaits: 12938
     [decompress speed:] 103.65 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 22 ] tar "-I zstd --ultra -22 -T8" -cf /home/tar-zstd/silesia.tar.zst silesia
     [compress stats:] real: 280.86s cpu: 100% maxmem: 916256 KB cswaits: 8895
     [compress speed:] .71 MB/s
     compression ratio: 4.0228 211942676 / 52684381 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I zstd -T8" -xf /home/tar-zstd/silesia.tar.zst -C /home/tar-zstd-extract
     [decompress stats:] real: 1.95s cpu: 105% maxmem: 132508 KB cswaits: 12939
     [decompress speed:] 103.65 MB/s
    ### pxz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I pxz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 4.21s cpu: 529% maxmem: 94300 KB cswaits: 5869
     [compress speed:] 48.01 MB/s
     compression ratio: 3.6086 211942676 / 58731740 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.10s cpu: 106% maxmem: 2036 KB cswaits: 25850
     [decompress speed:] 65.20 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I pxz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 6.56s cpu: 533% maxmem: 180152 KB cswaits: 3513
     [compress speed:] 30.81 MB/s
     compression ratio: 3.7188 211942676 / 56992000 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.93s cpu: 106% maxmem: 3056 KB cswaits: 25852
     [decompress speed:] 68.98 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I pxz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 10.70s cpu: 527% maxmem: 351180 KB cswaits: 2571
     [compress speed:] 18.89 MB/s
     compression ratio: 3.7820 211942676 / 56039192 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.87s cpu: 106% maxmem: 5108 KB cswaits: 25849
     [decompress speed:] 70.42 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I pxz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 14.36s cpu: 504% maxmem: 481288 KB cswaits: 5207
     [compress speed:] 14.07 MB/s
     compression ratio: 4.0367 211942676 / 52503612 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.89s cpu: 107% maxmem: 5108 KB cswaits: 25857
     [decompress speed:] 69.93 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I pxz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 18.08s cpu: 538% maxmem: 907716 KB cswaits: 357
     [compress speed:] 11.17 MB/s
     compression ratio: 4.2126 211942676 / 50310532 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.84s cpu: 103% maxmem: 9200 KB cswaits: 25848
     [decompress speed:] 71.17 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I pxz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 21.04s cpu: 545% maxmem: 952864 KB cswaits: 369
     [compress speed:] 9.60 MB/s
     compression ratio: 4.2753 211942676 / 49573244 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.95s cpu: 98% maxmem: 9204 KB cswaits: 25837
     [decompress speed:] 68.51 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I pxz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 28.05s cpu: 348% maxmem: 965896 KB cswaits: 146
     [compress speed:] 7.20 MB/s
     compression ratio: 4.3057 211942676 / 49223036 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.80s cpu: 104% maxmem: 17392 KB cswaits: 25843
     [decompress speed:] 72.18 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I pxz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 47.66s cpu: 181% maxmem: 963984 KB cswaits: 94
     [compress speed:] 4.24 MB/s
     compression ratio: 4.3316 211942676 / 48929028 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.80s cpu: 103% maxmem: 33780 KB cswaits: 25852
     [decompress speed:] 72.18 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I pxz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 80.69s cpu: 104% maxmem: 897324 KB cswaits: 70
     [compress speed:] 2.50 MB/s
     compression ratio: 4.3415 211942676 / 48816968 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I pxz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.73s cpu: 106% maxmem: 66548 KB cswaits: 25861
     [decompress speed:] 74.03 MB/s
    ### xz test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I xz -1" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 12.13s cpu: 100% maxmem: 9616 KB cswaits: 25870
     [compress speed:] 16.66 MB/s
     compression ratio: 3.6282 211942676 / 58414444 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.08s cpu: 105% maxmem: 2036 KB cswaits: 25864
     [decompress speed:] 65.62 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I xz -2" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 18.77s cpu: 100% maxmem: 17300 KB cswaits: 25869
     [compress speed:] 10.76 MB/s
     compression ratio: 3.7371 211942676 / 56712980 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 3.05s cpu: 105% maxmem: 3056 KB cswaits: 25852
     [decompress speed:] 66.27 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I xz -3" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 30.73s cpu: 100% maxmem: 32656 KB cswaits: 25870
     [compress speed:] 6.57 MB/s
     compression ratio: 3.8016 211942676 / 55749668 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.89s cpu: 105% maxmem: 5108 KB cswaits: 25840
     [decompress speed:] 69.93 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I xz -4" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 48.74s cpu: 100% maxmem: 49224 KB cswaits: 25873
     [compress speed:] 4.14 MB/s
     compression ratio: 4.0672 211942676 / 52109368 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.83s cpu: 106% maxmem: 5108 KB cswaits: 25853
     [decompress speed:] 71.42 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I xz -5" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 64.54s cpu: 100% maxmem: 96328 KB cswaits: 25875
     [compress speed:] 3.13 MB/s
     compression ratio: 4.2401 211942676 / 49984712 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.74s cpu: 106% maxmem: 9204 KB cswaits: 25847
     [decompress speed:] 73.76 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I xz -6" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 75.45s cpu: 100% maxmem: 96332 KB cswaits: 25876
     [compress speed:] 2.67 MB/s
     compression ratio: 4.3044 211942676 / 49237796 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.88s cpu: 103% maxmem: 9204 KB cswaits: 25856
     [decompress speed:] 70.18 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I xz -7" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 79.00s cpu: 100% maxmem: 190540 KB cswaits: 25871
     [compress speed:] 2.55 MB/s
     compression ratio: 4.3297 211942676 / 48950732 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.71s cpu: 106% maxmem: 17392 KB cswaits: 25840
     [decompress speed:] 74.58 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I xz -8" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 83.42s cpu: 100% maxmem: 378956 KB cswaits: 25878
     [compress speed:] 2.42 MB/s
     compression ratio: 4.3472 211942676 / 48752848 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.81s cpu: 106% maxmem: 33776 KB cswaits: 25852
     [decompress speed:] 71.93 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I xz -9" -cf /home/tar-zstd/silesia.tar.xz silesia
     [compress stats:] real: 85.14s cpu: 100% maxmem: 690248 KB cswaits: 25881
     [compress speed:] 2.37 MB/s
     compression ratio: 4.3500 211942676 / 48721676 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I xz" -xf /home/tar-zstd/silesia.tar.xz -C /home/tar-zstd-extract
     [decompress stats:] real: 2.82s cpu: 103% maxmem: 66548 KB cswaits: 25845
     [decompress speed:] 71.67 MB/s
    ### lz4 test
    ------------------------------------------------------------------------------
     [tar compress lvl: 1 ] tar "-I lz4 -1" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 2.44s cpu: 97% maxmem: 8844 KB cswaits: 9043
     [compress speed:] 82.83 MB/s
     compression ratio: 2.0999 211942676 / 100925719 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.86s cpu: 100% maxmem: 8940 KB cswaits: 18036
     [decompress speed:] 235.02 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 2 ] tar "-I lz4 -2" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 2.35s cpu: 102% maxmem: 8844 KB cswaits: 9037
     [compress speed:] 86.01 MB/s
     compression ratio: 2.0999 211942676 / 100925719 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.79s cpu: 107% maxmem: 8940 KB cswaits: 18102
     [decompress speed:] 255.85 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 3 ] tar "-I lz4 -3" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 7.02s cpu: 100% maxmem: 8600 KB cswaits: 8160
     [compress speed:] 28.79 MB/s
     compression ratio: 2.6023 211942676 / 81444275 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.87s cpu: 102% maxmem: 8440 KB cswaits: 18304
     [decompress speed:] 232.32 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 4 ] tar "-I lz4 -4" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 8.21s cpu: 100% maxmem: 8588 KB cswaits: 8430
     [compress speed:] 24.61 MB/s
     compression ratio: 2.6520 211942676 / 79916565 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.81s cpu: 107% maxmem: 8428 KB cswaits: 18075
     [decompress speed:] 249.53 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 5 ] tar "-I lz4 -5" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 9.68s cpu: 100% maxmem: 8588 KB cswaits: 9421
     [compress speed:] 20.88 MB/s
     compression ratio: 2.6825 211942676 / 79008581 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.81s cpu: 106% maxmem: 8428 KB cswaits: 17928
     [decompress speed:] 249.53 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 6 ] tar "-I lz4 -6" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 11.34s cpu: 100% maxmem: 8588 KB cswaits: 9298
     [compress speed:] 17.82 MB/s
     compression ratio: 2.6998 211942676 / 78501649 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.81s cpu: 107% maxmem: 8428 KB cswaits: 17804
     [decompress speed:] 249.53 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 7 ] tar "-I lz4 -7" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 13.41s cpu: 100% maxmem: 8588 KB cswaits: 9107
     [compress speed:] 15.07 MB/s
     compression ratio: 2.7095 211942676 / 78219928 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.87s cpu: 101% maxmem: 8428 KB cswaits: 18173
     [decompress speed:] 232.32 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 8 ] tar "-I lz4 -8" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 15.82s cpu: 100% maxmem: 8584 KB cswaits: 8001
     [compress speed:] 12.77 MB/s
     compression ratio: 2.7145 211942676 / 78075840 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.80s cpu: 107% maxmem: 8428 KB cswaits: 18134
     [decompress speed:] 252.65 MB/s
    ------------------------------------------------------------------------------
     [tar compress lvl: 9 ] tar "-I lz4 -9" -cf /home/tar-zstd/silesia.tar.lz4 silesia
     [compress stats:] real: 18.11s cpu: 100% maxmem: 8588 KB cswaits: 7943
     [compress speed:] 11.16 MB/s
     compression ratio: 2.7171 211942676 / 78003178 KB
    ------------------------------------------------------------------------------
     [tar decompress] tar "-I lz4" -xf /home/tar-zstd/silesia.tar.lz4 -C /home/tar-zstd-extract
     [decompress stats:] real: 0.87s cpu: 99% maxmem: 8428 KB cswaits: 17757
     [decompress speed:] 232.32 MB/s
    


    For pure compression speed, seems with 4 core/8 cpu thread server, zstd negative level --9 came closest to pure tar uncompressed speeds of 532MB/s with 481MB/s followed by multi-threaded gzip via pigz level 1 at 404MB/s. Pigz level 1 had much better compression ratio of 2.7463 vs zstd negative level -9 at 1.8016 compressio ratio.

    tar-zstd-tests-01.png tar-zstd-tests-02.png tar-zstd-tests-03.png
     
..
Thread Status:
Not open for further replies.