Sysadmin Round 3: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

This is round 3 comparison compression & decompression test benchmarks. You can read up on round 1 benchmarks here and also tar gzip vs tar zstd benchmarks here and round 2 benchmarks here. Round 4 updates are here.

In round 3, updated the following versions:

• updated zstd from 1.3.8 to 1.4.0 facebook/zstd
• updated brotli from 1.0.5 to 1.07 google/brotli
• added zstd --long (for 128MB window size), --long --adapt (for 128MB window size + dynamically adaptive compression level based on perceived disk I/O conditions), --format=gzip, --format=lz4 and --format=xz compression tests. Zstd supports compression using gzip, lz4 and xz if detected support is available. lz4 1.9.0 is installed for zstd to use.
• added zstd negative compression levels -4 to -1 for faster compression speed at the expense of compress ratio

Compression Algorithms Tested

• zstd v1.4.0 - Facebook developed realtime compression algorithm here. For --format=lz4 using lz4 v1.9.0
• brotli v1.0.7 - Google developed Brotli compression algorithm
• gzip v1.5
• bzip2 v1.06
• pigz v2.4 - multi-threaded version of gzip
• pbzip2 v1.1.13 – multi-threaded version of bzip2
• lbzip2 v2.5– multi-threaded version of bzip2
• lzip v1.21 – based on LZMA compression algorithm
• plzip v1.8 – multi-threaded version of lzip
• xz v5.2.2
• pxz v5.2.2 - multi-threaded version of xz

Test Data Files

The test data set was taken from Silesia Compression Corpus zip file here turned into a tar archive for compression tests.

Code (Text):

mkdir -p /home/gziptest/silesia
cd /home/gziptest/silesia
wget http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
unzip silesia.zip
cd /home/gziptest
rm -rf /home/gziptest/silesia/silesia.zip
tar -cvf silesia.tar /home/gziptest/silesia

~203 MB tar archive

Code (Text):

ls -la /home/gziptest/silesia.tar
-rw-r--r-- 1 root root 211957760 Sep  3 00:41 /home/gziptest/silesia.tar

---

Test System Configuration

System:

• OVH MC-32 Intel Core i7 4790K
• 32GB Memory
• 2x240GB SSD
• 250Mbit Network Bandwidth
• CentOS 7.6 64bit
• Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.12, MariaDB 10.1.38 MySQL, + CSF Firewall
• BHS, Canada

Compression Comparison Results

Below are the comparison results for compression tests with links to the raw data as well.

• Round 3: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

For a Intel Core i7 4790K with 4 cpus + 4 hyperthreaded cpus = 8 cpu threads total the findings are as follows:

• For gzip compression, multithreaded gzip via pigz was fastest. Pigz default level 6 compression had 143.36MB/s compression speed and 284.70MB/s decompression speed with compression ratio = 3.060 for original to compressed ratio size.
• For bzip2 compression, multithreaded bzip2 via pbzip2 was fastest. Pbzip2 default level 6 compression had 62.77MB/s compression speed and 172.76MB/s decompression speed with compression ratio = 3.792 for original to compressed ratio size.
• For xz compression, multithreaded xz via pxz was fastest. Pxz default level 6 compression had 9.45MB/s compression speed and 70.43MB/s decompression speed with compression ratio = 4.277 for original to compressed ratio size.
• For lzip compression, multithreaded lzip via plzip was fastest. Plzip default level 6 compression had 11.51MB/s compression speed and 206.26MB/s decompression speed with compression ratio = 4.225 for original to compressed ratio size.
• For zstd compression with default level 3 compression had 122.50MB/s compression speed and 90.24MB/s decompression speed with compression ratio = 3.172 for original to compressed ratio size.
• Multithreaded gzip via pigz default compression level 6 just edges out zstd default compression level 3 in terms of compression speed 143.36MB/s vs 122.50MB/s and is much faster for decompression at 284.70MB/s vs 90.24MB/s. While pigz compression ratio at 3.060 is just behind zstd at 3.172.
• For compression ratios better than gzip/pigz, multithreaded bzip2 via pbzip2 seems to have the best balance of compression and decompression speed and memory usage for compression ratios between 3.499 to 3.878. Pbzip2 level 9 compression can be done at 50.03MB/s with decompression at 132.11MB/s with compression ratio of 3.878 and only using max of 73,608 bytes of memory. However, if you need better compression ratio than gzip/pigz but faster than pbzip2 speed, zstd at level 3 to 6 are the sweet spot. Or lower pbzip2 compression level to 1 to 4.
• For compression ratios above pbzip2's level 9, 3.878 compression ratio, we have the following options in order of fastest compression to slowest: plzip level 4, pxz level 4 and zstd level 18 and zstd --format=xz level 4
• For pure compression speed, we have pigz level 1 to 4 or zstd level -4 to 2 which are all above 200MB/s compression speed. And pigz level 1 to 2 or zstd level -4 to -2 which are all above 300MB/s compression speed.
• Seems zstd --format=gzip is faster than single threaded gzip but still slower than pigz multithreaded gzip.

Compression speed = MB/s and compression time = seconds

 

Linux Kernel Compression Test

Updated tests this time using linux 5.1-rc5 tarball as the source of the compression algorithm tests. Additional options and test parameters enabled now include:

• add pigz level 11 for zopfli based compression test
• add zstd fast negative compression levels -10 to -1
• add zstd --long=31 for 2GB window size test
• add zstd ultra level 20, 21 and 22 compression level tests
• add brotli level 11 compression test
• add brotli large compression window at 30bits test for 1GB window size instead of default

Compression Algorithms Tested

• zstd v1.4.0 - Facebook developed realtime compression algorithm here. For --format=lz4 using lz4 v1.9.0
• brotli v1.0.7 - Google developed Brotli compression algorithm
• gzip v1.5
• bzip2 v1.06
• pigz v2.4 - multi-threaded version of gzip
• pbzip2 v1.1.13 – multi-threaded version of bzip2
• lbzip2 v2.5– multi-threaded version of bzip2
• lzip v1.21 – based on LZMA compression algorithm
• plzip v1.8 – multi-threaded version of lzip
• xz v5.2.2
• pxz v5.2.2 - multi-threaded version of xz

Test Data Files

The test data set was taken from kernel.org

Code (Text):

mkdir -p /home/gziptest
wget -O /home/gziptest/linux-5.1-rc5.tar.gz https://git.kernel.org/torvalds/t/linux-5.1-rc5.tar.gz
cd /home/gziptest
pigz -d linux-5.1-rc5.tar.gz

~832 MB tar archive

Code (Text):

ls -la linux-5.1-rc5.tar 
-rw-r--r-- 1 root root 871557120 Apr 20 20:15 linux-5.1-rc5.tar

Test System Configuration

System:

• OVH MC-32 Intel Core i7 4790K
• 32GB Memory
• 2x240GB SSD
• 250Mbit Network Bandwidth
• CentOS 7.6 64bit
• Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.12, MariaDB 10.1.38 MySQL, + CSF Firewall
• BHS, Canada

Compression Comparison Results

Below are the comparison results for compression tests with links to the raw data as well.

• Part 4 - kernel tarball Round 3: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

Working with 832MB tar file for compression/decompression, you can see for the more performant multi-threaded compression algorithms that they use alot more cpu & memory (compress/decompress max mem in kilobytes). So you're choice of compression algorithm and compression level maybe dictated by how much free memory available you have on your server as well.



Managed to figure out how to plot a chart for compression speed vs compression ratio but had to switch from Google Sheet to LibreOffice Charts and had to leave off some algorithms like xz, lzip, plzip, lbzip2 as the overlap was making it hard to read the chart.

Tune Zstd Advanced Options

With regards to zstd memory usage, you can tune it to reduce memory usage as well.

zstd compression algorithm has many advanced options which can tune how fast compression is or how compressed the resulting files are at each level compression. These are outlined here. An example would be --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters which refer to:

So take above example of zstd level 3 compression test for Linux kernel tarball

Code (Text):

zstd -T8 -q -f -3 /home/gziptest/linux-5.1-rc5.tar; ls -lah linux-5.1-rc5.tar*
real: 3.96s cpu: 729% maxmem: 120520 KB cswaits: 458
-rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
-rw-r--r-- 1 root root 152M Apr 20 20:15 linux-5.1-rc5.tar.zst

With --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters reduced max memory usage from 120,520KB to 80,276KB

Code (Text):

zstd -T8 --zstd=wlog=19,clog=26,hlog=26,slog=9 -q -f -3 /home/gziptest/linux-5.1-rc5.tar ; ls -lah linux-5.1-rc5.tar*
real: 4.50s cpu: 707% maxmem: 80276 KB cswaits: 1103
-rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
-rw-r--r-- 1 root root 155M Apr 20 20:15 linux-5.1-rc5.tar.zst

example of zstd level 9 compression test for Linux kernel tarball

Code (Text):

zstd -T8 -q -f -9 /home/gziptest/linux-5.1-rc5.tar; ls -lah linux-5.1-rc5.tar*                   
real: 12.15s cpu: 751% maxmem: 160428 KB cswaits: 441
-rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
-rw-r--r-- 1 root root 128M Apr 20 20:15 linux-5.1-rc5.tar.zst

With --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters reduced max memory usage from 160,428KB to 79,480KB

Code (Text):

zstd -T8 --zstd=wlog=19,clog=26,hlog=26,slog=9 -q -f -9 /home/gziptest/linux-5.1-rc5.tar ; ls -lah linux-5.1-rc5.tar*
real: 34.04s cpu: 753% maxmem: 79480 KB cswaits: 1019
-rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
-rw-r--r-- 1 root root 131M Apr 20 20:15 linux-5.1-rc5.tar.zst

Restest zstd -10 to 19 levels with advance tuned settings = --zstd=wlog=19,clog=18,hlog=17,slog=3 to cap memory usage which will reduce compression efficiency (larger compressed sizes)

first table is zstd default



second is zstd with zstd=wlog=19,clog=18,hlog=17,slog=3 where max compressed/decompress memory is much lower. zstd level 3, 7 and 8 are comparable to zstd default in terms of speed and compression ratio but with much lower memory usage.

 

Round 4 compression benchmark updates Sysadmin - Round 4: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc