Beta Branch update installers additional yum packages at install time in 123.09beta01

update installers additional yum packages at install time in 123.09beta01

---

- Update initial Centmin Mod install's default installed YUM packages to include the following packages for future dependency requirements i.e. newer versions of rsync 3.2+ instead of system native rsync 3.1.x. Existing users can install them for CentOS 7 systems via yum install

Code (Text):

yum install acl libacl-devel attr libattr-devel xxhash-devel libzstd-devel lz4-devel

Continue reading...

123.09beta01 branch

• Branch: centminmod/centminmod
• Commit History: centminmod/centminmod

 

This update adds dependencies required for new software I have planned to add. For example, CentOS 7 comes with rsync 3.1.2. But newer rsync 3.2 has added support for faster zstd compression and faster xxHash hashing algorithm. Details at https://download.samba.org/pub/rsync/NEWS.

I am privately testing custom built rsync 3.2.3 RPM binaries for CentOS 7 right now. No available for public consumption yet.

CentOS 7 native rsync 3.1.2

Code (Text):

rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
   64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
   append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Custom built rsync 3.2.3 RPM binaries one for avx2 supported cpus and one for non-avx2 older cpus.

no avx2 support

Code (Text):

rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
   64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
   batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
   symtimes, prealloc, stop-at, no crtimes
Optimizations:
   SIMD, asm, openssl-crypto
Checksum list:
   xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
   zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Code (Text):

yum -q info rsync-custom-noavx2-3.2.3
Installed Packages
Name        : rsync-custom-noavx2
Arch        : x86_64
Version     : 3.2.3
Release     : 1.el7
Size        : 877 k
Repo        : installed
From repo   : /rsync-custom-noavx2-3.2.3-1.el7.x86_64
Summary     : rsync 3.2.3 (noavx2) for centminmod.com LEMP stack
URL         : https://centminmod.com
License     : unknown
Description : rsync 3.2.3 (noavx2) for centminmod.com LEMP stacks

avx2 support

Code (Text):

rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
   64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
   batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
   symtimes, prealloc, stop-at, no crtimes
Optimizations:
   SIMD, asm, openssl-crypto
Checksum list:
   xxh64 (xxhash) md5 md4 none
Compress list:
   zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Code (Text):

yum -q info rsync-custom-avx2-3.2.3
Installed Packages
Name        : rsync-custom-avx2
Arch        : x86_64
Version     : 3.2.3
Release     : 1.el7
Size        : 873 k
Repo        : installed
From repo   : /rsync-custom-avx2-3.2.3-1.el7.x86_64
Summary     : rsync 3.2.3 (avx2) for centminmod.com LEMP stack
URL         : https://centminmod.com
License     : unknown
Description : rsync 3.2.3 (avx2) for centminmod.com LEMP stacks

rsync 3.2.3 help

Code (Text):

rsync --help
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none
rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.
Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
--verbose, -v            increase verbosity
--info=FLAGS             fine-grained informational verbosity
--debug=FLAGS            fine-grained debug verbosity
--stderr=e|a|c           change stderr output mode (default: errors)
--quiet, -q              suppress non-error messages
--no-motd                suppress daemon-mode MOTD
--checksum, -c           skip based on checksum, not mod-time & size
--archive, -a            archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION              turn off an implied OPTION (e.g. --no-D)
--recursive, -r          recurse into directories
--relative, -R           use relative path names
--no-implied-dirs        don't send implied dirs with --relative
--backup, -b             make backups (see --suffix & --backup-dir)
--backup-dir=DIR         make backups into hierarchy based in DIR
--suffix=SUFFIX          backup suffix (default ~ w/o --backup-dir)
--update, -u             skip files that are newer on the receiver
--inplace                update destination files in-place
--append                 append data onto shorter files
--append-verify          --append w/old data in file checksum
--dirs, -d               transfer directories without recursing
--mkpath                 create the destination's path component
--links, -l              copy symlinks as symlinks
--copy-links, -L         transform symlink into referent file/dir
--copy-unsafe-links      only "unsafe" symlinks are transformed
--safe-links             ignore symlinks that point outside the tree
--munge-links            munge symlinks to make them safe & unusable
--copy-dirlinks, -k      transform symlink to dir into referent dir
--keep-dirlinks, -K      treat symlinked dir on receiver as dir
--hard-links, -H         preserve hard links
--perms, -p              preserve permissions
--executability, -E      preserve executability
--chmod=CHMOD            affect file and/or directory permissions
--acls, -A               preserve ACLs (implies --perms)
--xattrs, -X             preserve extended attributes
--owner, -o              preserve owner (super-user only)
--group, -g              preserve group
--devices                preserve device files (super-user only)
--specials               preserve special files
-D                       same as --devices --specials
--times, -t              preserve modification times
--atimes, -U             preserve access (use) times
--open-noatime           avoid changing the atime on opened files
--crtimes, -N            preserve create times (newness)
--omit-dir-times, -O     omit directories from --times
--omit-link-times, -J    omit symlinks from --times
--super                  receiver attempts super-user activities
--fake-super             store/recover privileged attrs using xattrs
--sparse, -S             turn sequences of nulls into sparse blocks
--preallocate            allocate dest files before writing them
--write-devices          write to devices as files (implies --inplace)
--dry-run, -n            perform a trial run with no changes made
--whole-file, -W         copy files whole (w/o delta-xfer algorithm)
--checksum-choice=STR    choose the checksum algorithm (aka --cc)
--one-file-system, -x    don't cross filesystem boundaries
--block-size=SIZE, -B    force a fixed checksum block-size
--rsh=COMMAND, -e        specify the remote shell to use
--rsync-path=PROGRAM     specify the rsync to run on remote machine
--existing               skip creating new files on receiver
--ignore-existing        skip updating files that exist on receiver
--remove-source-files    sender removes synchronized files (non-dir)
--del                    an alias for --delete-during
--delete                 delete extraneous files from dest dirs
--delete-before          receiver deletes before xfer, not during
--delete-during          receiver deletes during the transfer
--delete-delay           find deletions during, delete after
--delete-after           receiver deletes after transfer, not during
--delete-excluded        also delete excluded files from dest dirs
--ignore-missing-args    ignore missing source args without error
--delete-missing-args    delete missing source args from destination
--ignore-errors          delete even if there are I/O errors
--force                  force deletion of dirs even if not empty
--max-delete=NUM         don't delete more than NUM files
--max-size=SIZE          don't transfer any file larger than SIZE
--min-size=SIZE          don't transfer any file smaller than SIZE
--max-alloc=SIZE         change a limit relating to memory alloc
--partial                keep partially transferred files
--partial-dir=DIR        put a partially transferred file into DIR
--delay-updates          put all updated files into place at end
--prune-empty-dirs, -m   prune empty directory chains from file-list
--numeric-ids            don't map uid/gid values by user/group name
--usermap=STRING         custom username mapping
--groupmap=STRING        custom groupname mapping
--chown=USER:GROUP       simple username/groupname mapping
--timeout=SECONDS        set I/O timeout in seconds
--contimeout=SECONDS     set daemon connection timeout in seconds
--ignore-times, -I       don't skip files that match size and time
--size-only              skip files that match in size
--modify-window=NUM, -@  set the accuracy for mod-time comparisons
--temp-dir=DIR, -T       create temporary files in directory DIR
--fuzzy, -y              find similar file for basis if no dest file
--compare-dest=DIR       also compare destination files relative to DIR
--copy-dest=DIR          ... and include copies of unchanged files
--link-dest=DIR          hardlink to files in DIR when unchanged
--compress, -z           compress file data during the transfer
--compress-choice=STR    choose the compression algorithm (aka --zc)
--compress-level=NUM     explicitly set compression level (aka --zl)
--skip-compress=LIST     skip compressing files with suffix in LIST
--cvs-exclude, -C        auto-ignore files in the same way CVS does
--filter=RULE, -f        add a file-filtering RULE
-F                       same as --filter='dir-merge /.rsync-filter'
                         repeated: --filter='- .rsync-filter'
--exclude=PATTERN        exclude files matching PATTERN
--exclude-from=FILE      read exclude patterns from FILE
--include=PATTERN        don't exclude files matching PATTERN
--include-from=FILE      read include patterns from FILE
--files-from=FILE        read list of source-file names from FILE
--from0, -0              all *-from/filter files are delimited by 0s
--protect-args, -s       no space-splitting; wildcard chars only
--copy-as=USER[:GROUP]   specify user & optional group for the copy
--address=ADDRESS        bind address for outgoing socket to daemon
--port=PORT              specify double-colon alternate port number
--sockopts=OPTIONS       specify custom TCP options
--blocking-io            use blocking I/O for the remote shell
--outbuf=N|L|B           set out buffering to None, Line, or Block
--stats                  give some file-transfer stats
--8-bit-output, -8       leave high-bit chars unescaped in output
--human-readable, -h     output numbers in a human-readable format
--progress               show progress during transfer
-P                       same as --partial --progress
--itemize-changes, -i    output a change-summary for all updates
--remote-option=OPT, -M  send OPTION to the remote side only
--out-format=FORMAT      output updates using the specified FORMAT
--log-file=FILE          log what we're doing to the specified FILE
--log-file-format=FMT    log updates using the specified FMT
--password-file=FILE     read daemon-access password from FILE
--early-input=FILE       use FILE for daemon's early exec input
--list-only              list the files instead of copying them
--bwlimit=RATE           limit socket I/O bandwidth
--stop-after=MINS        Stop rsync after MINS minutes have elapsed
--stop-at=y-m-dTh:m      Stop rsync at the specified point in time
--write-batch=FILE       write a batched update to FILE
--only-write-batch=FILE  like --write-batch but w/o updating dest
--read-batch=FILE        read a batched update from FILE
--protocol=NUM           force an older protocol version to be used
--iconv=CONVERT_SPEC     request charset conversion of filenames
--checksum-seed=NUM      set block/file checksum seed (advanced)
--ipv4, -4               prefer IPv4
--ipv6, -6               prefer IPv6
--version, -V            print the version + other info and exit
--help, -h (*)           show this help (* -h is help only on its own)
Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See https://rsync.samba.org/ for updates, bug reports, and answers

Choosing zstd compression for rsync via --compress-choice=zstd or --zc zstd flags. Custom rsync 3.2.3 supports the following compression algorithms = zstd, lz4, zlibx, zlib and none

Code (Text):

rsync --help | grep '\-\-z'
--compress-choice=STR    choose the compression algorithm (aka --zc)
--compress-level=NUM     explicitly set compression level (aka --zl)

Choosing checksum method via --checksum-choice= or --cc

Code (Text):

rsync --help | grep '\-\-cc'
--checksum-choice=STR    choose the checksum algorithm (aka --cc)

Examples

with checksum md5 and zstd compression

Code (Text):

rsync -avzin --cc md5 --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 214 bytes  received 37 bytes  502.00 bytes/sec
total size is 263,394,826  speedup is 1,049,381.78 (DRY RUN)

with checksum xxhash (you can't use xxhash64) and zstd

Code (Text):

rsync -avzin --cc xxhash --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 214 bytes  received 37 bytes  502.00 bytes/sec
total size is 263,394,826  speedup is 1,049,381.78 (DRY RUN)

Code (Text):

rsync -avzin --cc xxhash64 --zc zstd zcat-test/ zcat-test-dst/
unknown checksum name: xxhash64
rsync error: requested action not supported (code 4) at checksum.c(86) [client=3.2.3]

rsync 3.2 non-avx2 benchmarks

Testing various combinations of compression algorithms from zstd, zlibx and lz4 and checksums, md5 and xxhash.

Tests done on non-avx2 supported dedicated server with Xeon E3-1230v2 IvyBridge cpu.

Code (Text):

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 58
Model name:            Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
Stepping:              9
CPU MHz:               3496.984
CPU max MHz:           3700.0000
CPU min MHz:           1600.0000
BogoMIPS:              6599.85
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d

md5 + zstd

Code (Text):

time rsync -avzi --cc md5 --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,454,871 bytes  received 133 bytes  526,910,008.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.798s
user    0m1.149s
sys     0m0.195s

md5 + zlibx

Code (Text):

time rsync -avzi --cc md5 --zc zlibx zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,505,107 bytes  received 133 bytes  175,670,160.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m1.406s
user    0m1.726s
sys     0m0.214s

md5 + lz4

Code (Text):

time rsync -avzi --cc md5 --zc lz4 zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 264,571,058 bytes  received 133 bytes  176,380,794.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.666s
user    0m0.980s
sys     0m0.184s

xxhash + zstd

Code (Text):

time rsync -avzi --cc xxhash --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,454,823 bytes  received 133 bytes  526,909,912.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.424s
user    0m0.426s
sys     0m0.181s

xxhash + zlibx

Code (Text):

time rsync -avzi --cc xxhash --zc zlibx zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,505,059 bytes  received 133 bytes  175,670,128.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m1.044s
user    0m1.015s
sys     0m0.198s

xxhash + lz4

Code (Text):

time rsync -avzi --cc xxhash --zc lz4 zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 264,571,010 bytes  received 133 bytes  529,142,286.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.326s
user    0m0.266s
sys     0m0.177s

rsync 3.2 no-avx2 benchmark results where lower time = faster.

• md5 + zstd = 0.798s sent 263,454,871 bytes
• md5 + zlibx = 1.406s sent 263,505,107 bytes
• md5 + lz4 = 0.666s sent 264,571,058 bytes
• xxhash + zstd = 0.424s sent 263,454,823 bytes
• xxhash + zlibx = 1.044s sent 263,505,059 bytes
• xxhash + lz4 = 0.326s sent 264,571,010 bytes

rsync 3.2 avx2 benchmarks

Testing various combinations of compression algorithms from zstd, zlibx and lz4 and checksums, md5 and xxhash.

Tests done on non-avx2 supported dedicated server with Intel Core i7 4790K Haswell cpu.

Code (Text):

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:              3
CPU MHz:               4199.951
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              7982.22
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d

md5 + zstd

Code (Text):

time rsync -avzi --cc md5 --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,454,894 bytes  received 133 bytes  526,910,054.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.670s
user    0m0.898s
sys     0m0.212s

md5 + zlibx

Code (Text):

time rsync -avzi --cc md5 --zc zlibx zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,505,130 bytes  received 133 bytes  175,670,175.33 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m1.184s
user    0m1.415s
sys     0m0.244s

md5 + lz4

Code (Text):

time rsync -avzi --cc md5 --zc lz4 zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 264,571,081 bytes  received 133 bytes  529,142,428.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.583s
user    0m0.786s
sys     0m0.222s

xxhash + zstd

Code (Text):

time rsync -avzi --cc xxhash --zc zstd zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,454,846 bytes  received 133 bytes  526,909,958.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.365s
user    0m0.320s
sys     0m0.206s

xxhash + zlibx

Code (Text):

time rsync -avzi --cc xxhash --zc zlibx zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,505,082 bytes  received 133 bytes  175,670,143.33 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.890s
user    0m0.840s
sys     0m0.230s

xxhash + lz4

Code (Text):

time rsync -avzi --cc xxhash --zc lz4 zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 264,571,033 bytes  received 133 bytes  176,380,777.33 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.322s
user    0m0.207s
sys     0m0.212s

and default CentOS 7 rsync 3.1.2 with -z compress flag

Code (Text):

time /usr/bin/rsync -avzi zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,545,200 bytes  received 133 bytes  40,545,435.85 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m6.095s
user    0m6.392s
sys     0m0.269s

default CentOS 7 rsync 3.1.2 without -z compress flag

Code (Text):

time /usr/bin/rsync -avi zcat-test/ zcat-test-dst/      
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,459,568 bytes  received 133 bytes  526,919,402.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.519s
user    0m0.693s
sys     0m0.251s

and rsync 3.2.3 with no flags to replicate 3.1.2 previous test

Code (Text):

time /usr/local/bin/rsync -avzi zcat-test/ zcat-test-dst/
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst
sent 263,454,894 bytes  received 133 bytes  526,910,054.00 bytes/sec
total size is 263,394,826  speedup is 1.00
real    0m0.364s
user    0m0.313s
sys     0m0.210s

Benchmark comparison for rsync 3.1.2 native CentOS 7 binary versus rsync 3.2.3 built RPM binary on avx2 system with Intel Core i7 4790K Haswell cpu for 6x compressed log file rync on same system disk. Looks like rsync 3.2.3 no flags defaults to xxhash + zstd and is ~29.87% faster than rsync 3.1.2 without -z flag and ~94.03% faster than rsync 3.1.2 with -z flag - making rsync 3.2.3 a good candidate if you're rsync transferring a mix of compressed and uncompressed files together.

rsync 3.2 avx2 benchmark results where lower time = faster.

• default CentOS 7 rsync 3.1.2 with -z compress = 6.095s (sent 263,545,200 bytes)
• default CentOS 7 rsync 3.1.2 without -z compress = 0.519s (sent 263,459,568 bytes)
• rsync 3.2.3 no flags default with -z compress = 0.364s (sent 263,454,894 bytes)
• md5 + zstd with -z compress = 0.670s (sent 263,454,894 bytes)
• md5 + zlibx with -z compress = 1.184s (sent 263,505,130 bytes)
• md5 + lz4 with -z compress = 0.583s (sent 264,571,081 bytes)
• xxhash + zstd with -z compress = 0.365s (sent 263,454,846 bytes)
• xxhash + zlibx with -z compress = 0.890s (sent 263,505,082 bytes)
• xxhash + lz4 with -z compress = 0.322s (sent 264,571,033 bytes)

size of source directory is 252MB

Code (Text):

du -sh zcat-test
252M    zcat-test

 

Added extended Rsync 3.1.2 vs Rsync 3.2.3 benchmarks at CentOS 7.x - Custom Rsync 3.2.3 RPM Builds With xxHash/zstd Support For CentOS 7