Sysadmin Round 4: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

This is round 4 comparison compression & decompression test benchmarks. You can read up on round 1 benchmarks here and also tar gzip vs tar zstd benchmarks here and rsync 3.23 zstd benchmarks and round 2 benchmarks here and round 3 benchmarks here. In round 4, updated the following versions:

• updated zstd from 1.4.0 to 1.4.4 facebook/zstd. Zstd 1.4.4 has huge improvements for both compression and decompression speeds compared to previous versions and previous tests on this system.
• add pigz level 11 for zopfli based compression test
  
• add brotli large compression window at 30bits test for 1GB window size instead of default
• add zstd negative compression levels -10 to -1 for faster compression speed at the expense of compress ratio
• add zstd ultra level 20, 21 and 22 compression level tests

Compression Algorithms Tested

• zstd v1.4.4 - Facebook developed realtime compression algorithm here. Run with multi-threaded mode.
• brotli v1.0.7 - Google developed Brotli compression algorithm
• gzip v1.5
• bzip2 v1.06
• pigz v2.4 - multi-threaded version of gzip
• pbzip2 v1.1.13 – multi-threaded version of bzip2
• lbzip2 v2.5– multi-threaded version of bzip2
• lzip v1.21 – based on LZMA compression algorithm
• plzip v1.8 – multi-threaded version of lzip
• xz v5.2.2
• pxz v5.2.2 - multi-threaded version of xz

Test Data Files

The test data set was taken from Silesia Compression Corpus zip file here turned into a tar archive for compression tests.

Code (Text):

mkdir -p /home/gziptest/silesia
cd /home/gziptest/silesia
wget http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
unzip silesia.zip
cd /home/gziptest
rm -rf /home/gziptest/silesia/silesia.zip
tar -cvf silesia.tar /home/gziptest/silesia

~203 MB tar archive

Code (Text):

ls -la /home/gziptest/silesia.tar
-rw-r--r-- 1 root root 211957760 Sep  3 00:41 /home/gziptest/silesia.tar

Test System Configuration

---

System:

• OVH MC-32 Intel Core i7 4790K
• 32GB Memory
• 2x240GB SSD
• 250Mbit Network Bandwidth
• CentOS 7.7 64bit
• Centmin Mod 123.09beta01 LEMP stack - Nginx 1.17.15, MariaDB 10.1.43 MySQL, + CSF Firewall
• BHS, Canada

Compression Comparison Results

Below are the comparison results for compression tests with links to the raw data as well.

• Raw result data here.

 

Additional chart with only the more important compression algorithms charted for easier visualisation.