Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin Round 3: Compression Comparison Benchmarks: zstd vs brotli vs pigz vs bzip2 vs xz etc

Discussion in 'System Administration' started by eva2000, Apr 18, 2019.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    41,079
    9,178
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,086
    Local Time:
    4:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    This is round 3 comparison compression & decompression test benchmarks. You can read up on round 1 benchmarks here and also tar gzip vs tar zstd benchmarks here and round 2 benchmarks here.. In round 3, updated the following versions:
    • updated zstd from 1.3.8 to 1.4.0 facebook/zstd
    • updated brotli from 1.0.5 to 1.07 google/brotli
    • added zstd --long (for 128MB window size), --long --adapt (for 128MB window size + dynamically adaptive compression level based on perceived disk I/O conditions), --format=gzip, --format=lz4 and --format=xz compression tests. Zstd supports compression using gzip, lz4 and xz if detected support is available. lz4 1.9.0 is installed for zstd to use.
    • added zstd negative compression levels -4 to -1 for faster compression speed at the expense of compress ratio

    Compression Algorithms Tested


    Test Data Files



    The test data set was taken from Silesia Compression Corpus zip file here turned into a tar archive for compression tests.
    Code (Text):
    mkdir -p /home/gziptest/silesia
    cd /home/gziptest/silesia
    wget http://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
    unzip silesia.zip
    cd /home/gziptest
    rm -rf /home/gziptest/silesia/silesia.zip
    tar -cvf silesia.tar /home/gziptest/silesia
    

    ~203 MB tar archive
    Code (Text):
    ls -la /home/gziptest/silesia.tar
    -rw-r--r-- 1 root root 211957760 Sep  3 00:41 /home/gziptest/silesia.tar
    


    Test System Configuration



    System:
    • OVH MC-32 Intel Core i7 4790K
    • 32GB Memory
    • 2x240GB SSD
    • 250Mbit Network Bandwidth
    • CentOS 7.6 64bit
    • Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.12, MariaDB 10.1.38 MySQL, + CSF Firewall
    • BHS, Canada

    Compression Comparison Results



    Below are the comparison results for compression tests with links to the raw data as well.
    For a Intel Core i7 4790K with 4 cpus + 4 hyperthreaded cpus = 8 cpu threads total the findings are as follows:
    • For gzip compression, multithreaded gzip via pigz was fastest. Pigz default level 6 compression had 143.36MB/s compression speed and 284.70MB/s decompression speed with compression ratio = 3.060 for original to compressed ratio size.
    • For bzip2 compression, multithreaded bzip2 via pbzip2 was fastest. Pbzip2 default level 6 compression had 62.77MB/s compression speed and 172.76MB/s decompression speed with compression ratio = 3.792 for original to compressed ratio size.
    • For xz compression, multithreaded xz via pxz was fastest. Pxz default level 6 compression had 9.45MB/s compression speed and 70.43MB/s decompression speed with compression ratio = 4.277 for original to compressed ratio size.
    • For lzip compression, multithreaded lzip via plzip was fastest. Plzip default level 6 compression had 11.51MB/s compression speed and 206.26MB/s decompression speed with compression ratio = 4.225 for original to compressed ratio size.
    • For zstd compression with default level 3 compression had 122.50MB/s compression speed and 90.24MB/s decompression speed with compression ratio = 3.172 for original to compressed ratio size.
    • Multithreaded gzip via pigz default compression level 6 just edges out zstd default compression level 3 in terms of compression speed 143.36MB/s vs 122.50MB/s and is much faster for decompression at 284.70MB/s vs 90.24MB/s. While pigz compression ratio at 3.060 is just behind zstd at 3.172.
    • For compression ratios better than gzip/pigz, multithreaded bzip2 via pbzip2 seems to have the best balance of compression and decompression speed and memory usage for compression ratios between 3.499 to 3.878. Pbzip2 level 9 compression can be done at 50.03MB/s with decompression at 132.11MB/s with compression ratio of 3.878 and only using max of 73,608 bytes of memory. However, if you need better compression ratio than gzip/pigz but faster than pbzip2 speed, zstd at level 3 to 6 are the sweet spot. Or lower pbzip2 compression level to 1 to 4.
    • For compression ratios above pbzip2's level 9, 3.878 compression ratio, we have the following options in order of fastest compression to slowest: plzip level 4, pxz level 4 and zstd level 18 and zstd --format=xz level 4
    • For pure compression speed, we have pigz level 1 to 4 or zstd level -4 to 2 which are all above 200MB/s compression speed. And pigz level 1 to 2 or zstd level -4 to -2 which are all above 300MB/s compression speed.
    • Seems zstd --format=gzip is faster than single threaded gzip but still slower than pigz multithreaded gzip.

    Compression speed = MB/s and compression time = seconds

    compress-benchmarks-01.png compress-benchmarks-02.png compress-benchmarks-03.png compress-benchmarks-04.png compress-benchmarks-05.png compress-benchmarks-06.png compress-benchmarks-07.png
     
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    41,079
    9,178
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,086
    Local Time:
    4:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Linux Kernel Compression Test



    Updated tests this time using linux 5.1-rc5 tarball as the source of the compression algorithm tests. Additional options and test parameters enabled now include:
    • add pigz level 11 for zopfli based compression test
    • add zstd fast negative compression levels -10 to -1
    • add zstd --long=31 for 2GB window size test
    • add zstd ultra level 20, 21 and 22 compression level tests
    • add brotli level 11 compression test
    • add brotli large compression window at 30bits test for 1GB window size instead of default

    Compression Algorithms Tested


    Test Data Files



    The test data set was taken from kernel.org
    Code (Text):
    mkdir -p /home/gziptest
    wget -O /home/gziptest/linux-5.1-rc5.tar.gz https://git.kernel.org/torvalds/t/linux-5.1-rc5.tar.gz
    cd /home/gziptest
    pigz -d linux-5.1-rc5.tar.gz
    

    ~832 MB tar archive
    Code (Text):
    ls -la linux-5.1-rc5.tar 
    -rw-r--r-- 1 root root 871557120 Apr 20 20:15 linux-5.1-rc5.tar
    


    Test System Configuration



    System:
    • OVH MC-32 Intel Core i7 4790K
    • 32GB Memory
    • 2x240GB SSD
    • 250Mbit Network Bandwidth
    • CentOS 7.6 64bit
    • Centmin Mod 123.09beta01 LEMP stack - Nginx 1.15.12, MariaDB 10.1.38 MySQL, + CSF Firewall
    • BHS, Canada

    Compression Comparison Results



    Below are the comparison results for compression tests with links to the raw data as well.
    Working with 832MB tar file for compression/decompression, you can see for the more performant multi-threaded compression algorithms that they use alot more cpu & memory (compress/decompress max mem in kilobytes). So you're choice of compression algorithm and compression level maybe dictated by how much free memory available you have on your server as well.

    compress-benchmarks-01.png compress-benchmarks-02.png compress-benchmarks-03.png compress-benchmarks-04.png compress-benchmarks-05.png compress-benchmarks-06.png compress-benchmarks-07.png compress-benchmarks-08.png compress-benchmarks-09.png compress-benchmarks-010.png

    Managed to figure out how to plot a chart for compression speed vs compression ratio but had to switch from Google Sheet to LibreOffice Charts and had to leave off some algorithms like xz, lzip, plzip, lbzip2 as the overlap was making it hard to read the chart.

    round3-part4-charted-shorter-02.png

    Tune Zstd Advanced Options



    With regards to zstd memory usage, you can tune it to reduce memory usage as well.

    zstd compression algorithm has many advanced options which can tune how fast compression is or how compressed the resulting files are at each level compression. These are outlined here. An example would be --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters which refer to:
    So take above example of zstd level 3 compression test for Linux kernel tarball
    Code (Text):
    zstd -T8 -q -f -3 /home/gziptest/linux-5.1-rc5.tar; ls -lah linux-5.1-rc5.tar*
    real: 3.96s cpu: 729% maxmem: 120520 KB cswaits: 458
    -rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
    -rw-r--r-- 1 root root 152M Apr 20 20:15 linux-5.1-rc5.tar.zst
    

    With --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters reduced max memory usage from 120,520KB to 80,276KB
    Code (Text):
    zstd -T8 --zstd=wlog=19,clog=26,hlog=26,slog=9 -q -f -3 /home/gziptest/linux-5.1-rc5.tar ; ls -lah linux-5.1-rc5.tar*
    real: 4.50s cpu: 707% maxmem: 80276 KB cswaits: 1103
    -rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
    -rw-r--r-- 1 root root 155M Apr 20 20:15 linux-5.1-rc5.tar.zst
    

    example of zstd level 9 compression test for Linux kernel tarball
    Code (Text):
    zstd -T8 -q -f -9 /home/gziptest/linux-5.1-rc5.tar; ls -lah linux-5.1-rc5.tar*                   
    real: 12.15s cpu: 751% maxmem: 160428 KB cswaits: 441
    -rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
    -rw-r--r-- 1 root root 128M Apr 20 20:15 linux-5.1-rc5.tar.zst
    

    With --zstd=wlog=19,clog=26,hlog=26,slog=9 parameters reduced max memory usage from 160,428KB to 79,480KB
    Code (Text):
    zstd -T8 --zstd=wlog=19,clog=26,hlog=26,slog=9 -q -f -9 /home/gziptest/linux-5.1-rc5.tar ; ls -lah linux-5.1-rc5.tar*
    real: 34.04s cpu: 753% maxmem: 79480 KB cswaits: 1019
    -rw-r--r-- 1 root root 832M Apr 20 20:15 linux-5.1-rc5.tar
    -rw-r--r-- 1 root root 131M Apr 20 20:15 linux-5.1-rc5.tar.zst
    


    Restest zstd -10 to 19 levels with advance tuned settings = --zstd=wlog=19,clog=18,hlog=17,slog=3 to cap memory usage which will reduce compression efficiency (larger compressed sizes)

    first table is zstd default

    compress-benchmarks-zstd-default-01.png

    second is zstd with zstd=wlog=19,clog=18,hlog=17,slog=3 where max compressed/decompress memory is much lower. zstd level 3, 7 and 8 are comparable to zstd default in terms of speed and compression ratio but with much lower memory usage.

    compress-benchmarks-zstd-tuned-01.png
     
    • Like Like x 1
Thread Status:
Not open for further replies.