Join the community today
Become a Member

Sysadmin Round 4: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

Discussion in 'System Administration' started by eva2000, Nov 15, 2019.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    55,164
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    This is round 4 comparison compression & decompression test benchmarks. You can read up on round 1 benchmarks here and also tar gzip vs tar zstd benchmarks here and rsync 3.23 zstd benchmarks and round 2 benchmarks here and round 3 benchmarks here. In round 4, updated the following versions:
    • updated zstd from 1.4.0 to 1.4.4 facebook/zstd. Zstd 1.4.4 has huge improvements for both compression and decompression speeds compared to previous versions and previous tests on this system.
    • add pigz level 11 for zopfli based compression test
    • add brotli large compression window at 30bits test for 1GB window size instead of default
    • add zstd negative compression levels -10 to -1 for faster compression speed at the expense of compress ratio
    • add zstd ultra level 20, 21 and 22 compression level tests

    Compression Algorithms Tested


    • zstd v1.4.4 - Facebook developed realtime compression algorithm here. Run with multi-threaded mode.
    • brotli v1.0.7 - Google developed Brotli compression algorithm
    • gzip v1.5
    • bzip2 v1.06
    • pigz v2.4 - multi-threaded version of gzip
    • pbzip2 v1.1.13 – multi-threaded version of bzip2
    • lbzip2 v2.5– multi-threaded version of bzip2
    • lzip v1.21 – based on LZMA compression algorithm
    • plzip v1.8 – multi-threaded version of lzip
    • xz v5.2.2
    • pxz v5.2.2 - multi-threaded version of xz

    Test Data Files




    The test data set was taken from Silesia Compression Corpus zip file here turned into a tar archive for compression tests.
    Code (Text):
    mkdir -p /home/gziptest/silesia
    cd /home/gziptest/silesia
    wget http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
    unzip silesia.zip
    cd /home/gziptest
    rm -rf /home/gziptest/silesia/silesia.zip
    tar -cvf silesia.tar /home/gziptest/silesia
    

    ~203 MB tar archive
    Code (Text):
    ls -la /home/gziptest/silesia.tar
    -rw-r--r-- 1 root root 211957760 Sep  3 00:41 /home/gziptest/silesia.tar
    

    Test System Configuration



    System:
    • OVH MC-32 Intel Core i7 4790K
    • 32GB Memory
    • 2x240GB SSD
    • 250Mbit Network Bandwidth
    • CentOS 7.7 64bit
    • Centmin Mod 123.09beta01 LEMP stack - Nginx 1.17.15, MariaDB 10.1.43 MySQL, + CSF Firewall
    • BHS, Canada

    Compression Comparison Results



    Below are the comparison results for compression tests with links to the raw data as well.
    • Raw result data here.

    compression-test-compress-speed-vs-ratio-01.png compression-test-compress-table-01.png compression-test-compress-table-02.png compression-test-compress-table-03.png compression-test-compress-table-04.png
     
    Last edited: Jan 24, 2021
  2. eva2000

    eva2000 Administrator Staff Member

    55,164
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    7:26 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Additional chart with only the more important compression algorithms charted for easier visualisation.

    compression-test-compress-speed-vs-ratio-02.png
     
Thread Status:
Not open for further replies.