Featured Nginx Nginx with Cloudflare zlib fork VS nxg_brotli compression level tests

Quick wrk load testing comparison benchmarks between Centmin Mod Nginx 1.13.8 using cloudflare zlib performance forked library for HTTPS based compressed static index file tests versus Nginx 1.13.8 with Google ngx_brotli module for brotli compression instead. First set of compression level 1-9 tests are for cloudflare zlib and second set are for ngx_brotli based. You can see ngx_brotli scales badly at high compression levels. FYI, Centmin Mod defaults to compression level 5 for both zlib and brotli usually. Though ngx_brotli has smaller file sizes than zlib/gzip based compression in Nginx.

---

Tests done on OVH Core i7 4790K server with Centmin Mod compiled with GCC 8 20180114 snapshot.

Centmin Mod Ngixn 1.13.8 + cloudflare zlib

Code (Text):

domain=https://domain.com
for l in {1..9}; do
sleep 5;
echo -e "\ntest gzip_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='n'|" /usr/local/bin/curltest
/usr/local/bin/curltest $domain;
sed -ie "s|gzip_comp_level .*|gzip_comp_level $l;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|gzip_comp_level .*|gzip_comp_level 5;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level

Code (Text):

test gzip_comp_level 1
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.64ms    2.78ms  91.47ms   82.86%
--
Requests/sec:  57328.78
Transfer/sec:    117.00MB

test gzip_comp_level 2
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.60 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.72ms    2.80ms  89.40ms   80.65%
--
Requests/sec:  56372.51
Transfer/sec:    114.14MB

test gzip_comp_level 3
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.59 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.68ms    2.31ms  23.55ms   76.99%
--
Requests/sec:  55660.12
Transfer/sec:    112.11MB

test gzip_comp_level 4
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.58 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.14ms    2.51ms  40.27ms   80.86%
--
Requests/sec:  49335.99
Transfer/sec:     97.63MB

test gzip_comp_level 5
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.54 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.38ms    2.88ms  28.23ms   75.97%
--
Requests/sec:  46941.20
Transfer/sec:     92.31MB

test gzip_comp_level 6
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.47ms    3.63ms 109.23ms   87.82%
--
Requests/sec:  46594.92
Transfer/sec:     91.67MB

test gzip_comp_level 7
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.45ms    2.74ms  34.57ms   79.50%
--
Requests/sec:  46192.65
Transfer/sec:     90.88MB

test gzip_comp_level 8
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.50ms    2.88ms  30.34ms   75.30%
--
Requests/sec:  45833.67
Transfer/sec:     90.18MB

test gzip_comp_level 9
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.37ms    2.76ms  31.79ms   75.67%
--
Requests/sec:  46975.52
Transfer/sec:     92.42MB

 

Centmin Mod Ngixn 1.13.8 ngx_brotli

Code (Text):

domain=https://domain.com
for l in {1..9}; do
sleep 5;
echo -e "\ntest brotli_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='y'|" /usr/local/bin/curltest
/usr/local/bin/curltest $domain;
sed -ie "s|brotli_comp_level .*|brotli_comp_level $l;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|brotli_comp_level .*|brotli_comp_level 5;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level

Code (Text):

test brotli_comp_level 1
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.10ms    2.36ms  21.26ms   80.09%
--
Requests/sec:  69216.96
Transfer/sec:    149.85MB

test brotli_comp_level 2
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.73 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.15ms    3.62ms 109.56ms   88.92%
--
Requests/sec:  51881.04
Transfer/sec:    106.28MB

test brotli_comp_level 3
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.61 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.89ms    2.91ms  38.35ms   81.21%
--
Requests/sec:  41836.26
Transfer/sec:     84.74MB

test brotli_comp_level 4
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.59 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.12ms    3.29ms  47.08ms   81.60%
--
Requests/sec:  33020.04
Transfer/sec:     59.58MB

test brotli_comp_level 5
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.36 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.99ms    3.24ms  31.60ms   75.34%
--
Requests/sec:  28453.59
Transfer/sec:     48.84MB

test brotli_comp_level 6
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.18ms    3.40ms  70.95ms   84.31%
--
Requests/sec:  27902.55
Transfer/sec:     47.90MB

test brotli_comp_level 7
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.48ms    2.79ms  29.54ms   77.60%
--
Requests/sec:  26428.62
Transfer/sec:     45.24MB

test brotli_comp_level 8
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.53ms    2.56ms  36.67ms   82.18%
--
Requests/sec:  26221.07
Transfer/sec:     44.89MB

test brotli_comp_level 9
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.16ms    7.13ms  69.63ms   58.05%
--
Requests/sec:   9278.82
Transfer/sec:     15.87MB

 

Specifically comparing levels 5 & 6 compression for Cloudflare zlib and Google ngx_brotli

Cloudflare zlib

Code (Text):

test gzip_comp_level 5
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.54 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     4.38ms    2.88ms  28.23ms   75.97%
--
Requests/sec:  46941.20
Transfer/sec:     92.31MB

test gzip_comp_level 6
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     4.47ms    3.63ms 109.23ms   87.82%
--
Requests/sec:  46594.92
Transfer/sec:     91.67MB

Google ngx_brotli

Code (Text):

test brotli_comp_level 5
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.36 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     6.99ms    3.24ms  31.60ms   75.34%
--
Requests/sec:  28453.59
Transfer/sec:     48.84MB

test brotli_comp_level 6
URI: https://domain.com
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     7.18ms    3.40ms  70.95ms   84.31%
--
Requests/sec:  27902.55
Transfer/sec:     47.90MB

 

see the latency numbers in above benchmarks, probably not enough to notice really though throughput has zlib winning

 

most like they they take backend brotli compressed assets and store them temporarily so using brotli pre-compressed served files which don't add as much overhead as on the fly brotli compression. Just like pre-gzipped static files vs on the fly gzip compressed files. So yes maybe for Cloudflare enabled Brotli compression, then it would be fine.

 

From above Centmin Mod Nginx zlib is much faster in terms of scalability than ngx_brotli. But with Cloudflare's brotli implementation it matches and slightly exceeds their zlib scalability performance centminmod.com DNS update.

So looks like with Centmin Mod Nginx brotli usage you choose between page load speed which it does better than zlib but at expense of scalability performance. While with Cloudflare they seem to have engineered brotli to scale as well as zlib so that both brotli and zlib are good at both page load speed (due to higher compression = smaller file size) and scalability in terms of higher concurrency loads.

Or I should look at using Cloudflare's ngx_brotli implementation cloudflare/ngx_brotli_module instead of the current one I use at eustas/ngx_brotli which is maintained by official Google Brotli developer ? But Cloudflare no longer solely use Nginx for web server so given their ngx_brotli code hasn't been updated in years it might not be that implementation which is allowing Cloudflare's Brotli performance to scale to match zlib ?

Update: looks like Centmin Mod used ngx_brotli module at eustas/ngx_brotli was completely rewritten after April 2018 and updated to use Google Brotli 1.0.4. So performance of above tests are using old version of ngx_brotli and will need to be redone against the newly rewritten ngx_brotli version.

 

Centmin Mod Nginx Cloudflare zlib VS nxg_brotli Update

Looks like Centmin Mod used ngx_brotli module at eustas/ngx_brotli was completely rewritten after April 2018 and updated to use Google Brotli 1.0.4. So performance of above tests are using old version of ngx_brotli and will need to be redone against the newly rewritten ngx_brotli version

The tests will be done CentOS 7 with master branch Nginx 1.15.0 with ngx_brotli rewritten module with Google Brotli 1.0.4 and Cloudflare zlib performance patched Nginx. Nginx 1.15.0 will be compiled against latest GCC 8.1.0 compiler.

Code (Text):

brotli -V
brotli 1.0.4

Centmin Mod Nginx 1.15.0 + Cloudflare Zlib Compression

Code (Text):

domain=https://domain.com
for l in {1..9}; do
sleep 5;
echo -e "\ntest gzip_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='n'|" /usr/local/bin/curltest
/usr/local/bin/curltest gzip $domain;
sed -ie "s|gzip_comp_level .*|gzip_comp_level $l;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|gzip_comp_level .*|gzip_comp_level 5;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level

Code (Text):

test gzip_comp_level 1
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.24ms    2.99ms  34.01ms   81.71%
--
Requests/sec:  49358.76
Transfer/sec:    100.73MB

test gzip_comp_level 2
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.60 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.48ms    3.25ms  41.12ms   77.17%
--
Requests/sec:  46669.40
Transfer/sec:     94.49MB

test gzip_comp_level 3
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.59 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.58ms    3.45ms  57.45ms   80.06%
--
Requests/sec:  45950.19
Transfer/sec:     92.55MB

test gzip_comp_level 4
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.58 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.92ms    3.45ms  52.51ms   78.37%
--
Requests/sec:  42095.78
Transfer/sec:     83.31MB

test gzip_comp_level 5
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.54 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.18ms    3.40ms  43.19ms   77.18%
--
Requests/sec:  39559.15
Transfer/sec:     77.79MB

test gzip_comp_level 6
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.23ms    4.85ms 116.98ms   91.59%
--
Requests/sec:  40064.29
Transfer/sec:     78.82MB

test gzip_comp_level 7
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.17ms    3.04ms  31.59ms   81.79%
--
Requests/sec:  39017.16
Transfer/sec:     76.76MB

test gzip_comp_level 8
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.17ms    3.38ms  35.73ms   80.64%
--
Requests/sec:  39925.48
Transfer/sec:     78.55MB

test gzip_comp_level 9
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.31ms    3.85ms  76.48ms   82.75%
--
Requests/sec:  39348.09
Transfer/sec:     77.42MB

Centmin Mod Nginx 1.15.0 + Brotli Compression

Code (Text):

domain=https://domain.com
for l in {1..9}; do
sleep 5;
echo -e "\ntest brotli_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='y'|" /usr/local/bin/curltest
/usr/local/bin/curltest br $domain;
sed -ie "s|brotli_comp_level .*|brotli_comp_level $l;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|brotli_comp_level .*|brotli_comp_level 5;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level

Code (Text):

test brotli_comp_level 1
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.67ms    3.33ms  41.49ms   82.83%
--
Requests/sec:  35996.20
Transfer/sec:     77.93MB

test brotli_comp_level 2
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.73 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.52ms    4.43ms  48.01ms   81.59%
--
Requests/sec:  20866.94
Transfer/sec:     42.81MB

test brotli_comp_level 3
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.62 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.64ms    2.59ms  30.90ms   77.40%
--
Requests/sec:  22791.34
Transfer/sec:     46.12MB

test brotli_comp_level 4
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.59 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    12.16ms    2.51ms  56.29ms   72.57%
--
Requests/sec:  16154.26
Transfer/sec:     29.13MB

test brotli_comp_level 5
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.36 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.80ms    9.02ms 127.80ms   86.60%
--
Requests/sec:  11960.71
Transfer/sec:     20.53MB

test brotli_comp_level 6
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    13.65ms    3.09ms  37.01ms   73.62%
--
Requests/sec:  14376.27
Transfer/sec:     24.68MB

test brotli_comp_level 7
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    14.78ms    3.09ms  94.52ms   93.45%
--
Requests/sec:  13290.59
Transfer/sec:     22.75MB

test brotli_comp_level 8
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.00ms    8.88ms 181.58ms   86.28%
--
Requests/sec:  11041.09
Transfer/sec:     18.90MB

test brotli_comp_level 9
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    29.72ms    8.35ms 108.39ms   79.39%
--
Requests/sec:   6606.58
Transfer/sec:     11.30MB

Cloudflare Zlib vs Brotli Level 1, 5 & 6

Comparing the default Nginx zlib and Brotli compression levels set in Centmin Mod Nginx

Cloudflare zlib 5 and 6

Code (Text):

test gzip_comp_level 5
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.54 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.18ms    3.40ms  43.19ms   77.18%
--
Requests/sec:  39559.15
Transfer/sec:     77.79MB

test gzip_comp_level 6
URI: https://domain.com (gzip)
Uncompressed size : 3.71 KiB
Compressed size   : 1.53 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.23ms    4.85ms 116.98ms   91.59%
--
Requests/sec:  40064.29
Transfer/sec:     78.82MB

Brotli level 1, 5 and 6 - - level 1 has same compressed size 1.27KB as level 6 !

Code (Text):

test brotli_comp_level 1
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     5.67ms    3.33ms  41.49ms   82.83%
--
Requests/sec:  35996.20
Transfer/sec:     77.93MB

test brotli_comp_level 5
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.36 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.80ms    9.02ms 127.80ms   86.60%
--
Requests/sec:  11960.71
Transfer/sec:     20.53MB

test brotli_comp_level 6
URI: https://domain.com (br)
Uncompressed size : 3.71 KiB
Compressed size   : 1.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    13.65ms    3.09ms  37.01ms   73.62%
--
Requests/sec:  14376.27
Transfer/sec:     24.68MB

Seems both Cloudflare zlib and brotli have a dip in performance at level 5 to jump back up in performance at level 6 ! But still like before Centmin Mod Nginx with Cloudflare zlib performed better than Nginx Brotli for scalability.

 

Centmin Mod Nginx zlib/brotli vs Cloudflare zlib/brotli

Next test is direct comparison against Cloudflare reverse proxied zlib/brotli and Centmin Mod Nginx with Cloudflare zlib fork and ngx_brotli. As Cloudflare only caches static files can't test against index.html normally, so change to testing against css file = pure-min.css.

Centmin Mod Nginx 1.15.0

Code (Text):

curl -I $domain/pure-min.css
HTTP/2 200
date: Fri, 18 May 2018 21:29:01 GMT
content-type: text/css
content-length: 17286
last-modified: Tue, 27 Sep 2016 15:04:57 GMT
vary: Accept-Encoding
etag: "57ea8a99-4386"
server: nginx centminmod
x-powered-by: centminmod
expires: Sun, 17 Jun 2018 21:29:01 GMT
cache-control: max-age=2592000
access-control-allow-origin: *
cache-control: public, must-revalidate, proxy-revalidate
accept-ranges: bytes

Centmin Mod Nginx 1.15.0 + Cloudflare zlib fork

Code (Text):

for l in {1..9}; do
sleep 5;
echo -e "\ntest gzip_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='n'|" /usr/local/bin/curltest
/usr/local/bin/curltest gzip $domain/pure-min.css;
sed -ie "s|gzip_comp_level .*|gzip_comp_level $l;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|gzip_comp_level .*|gzip_comp_level 5;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level

Code (Text):

test gzip_comp_level 1
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.00 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.02ms    3.66ms  59.75ms   82.90%
--
Requests/sec:  28579.66
Transfer/sec:    139.31MB

test gzip_comp_level 2
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.52 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.78ms    3.44ms  32.14ms   80.32%
--
Requests/sec:  29648.73
Transfer/sec:    141.49MB

test gzip_comp_level 3
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.42 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.46ms    4.18ms  50.52ms   81.48%
--
Requests/sec:  23588.88
Transfer/sec:    111.36MB

test gzip_comp_level 4
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.36 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.76ms    5.00ms  72.75ms   78.03%
--
Requests/sec:  20516.19
Transfer/sec:     92.06MB

test gzip_comp_level 5
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.12 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.07ms    3.75ms  74.32ms   84.27%
--
Requests/sec:  21906.21
Transfer/sec:     95.56MB

test gzip_comp_level 6
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.00 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.67ms    5.36ms  49.06ms   80.21%
--
Requests/sec:  16996.54
Transfer/sec:     73.80MB

test gzip_comp_level 7
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.98 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.23ms    3.28ms  34.06ms   69.40%
--
Requests/sec:  19185.81
Transfer/sec:     83.27MB

test gzip_comp_level 8
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.97 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.70ms    3.34ms  54.31ms   74.89%
--
Requests/sec:  16812.23
Transfer/sec:     72.90MB

test gzip_comp_level 9
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.97 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    14.24ms    6.72ms  57.65ms   77.27%
--
Requests/sec:  13975.82
Transfer/sec:     60.60MB

Centmin Mod Nginx 1.15.0 + Nginx Brotli

Code (Text):

for l in {1..9}; do
sleep 5;
echo -e "\ntest brotli_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='y'|" /usr/local/bin/curltest
/usr/local/bin/curltest br $domain/pure-min.css;
sed -ie "s|brotli_comp_level .*|brotli_comp_level $l;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|brotli_comp_level .*|brotli_comp_level 5;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level

Code (Text):

test brotli_comp_level 1
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    12.28ms    5.31ms  76.79ms   80.91%
--
Requests/sec:  16190.71
Transfer/sec:     82.07MB

test brotli_comp_level 2
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 4.72 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    19.99ms    5.38ms 134.65ms   88.46%
--
Requests/sec:   9843.13
Transfer/sec:     45.59MB

test brotli_comp_level 3
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 4.27 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.23ms    4.19ms  60.41ms   67.43%
--
Requests/sec:   9225.87
Transfer/sec:     41.83MB

test brotli_comp_level 4
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 4.17 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.90ms    7.26ms  55.69ms   74.15%
--
Requests/sec:   6769.71
Transfer/sec:     29.09MB

test brotli_comp_level 5
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.93 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.61ms    8.43ms 167.43ms   85.94%
--
Requests/sec:   5059.95
Transfer/sec:     20.38MB

test brotli_comp_level 6
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    40.12ms    5.15ms 119.51ms   90.46%
--
Requests/sec:   4874.63
Transfer/sec:     19.50MB

test brotli_comp_level 7
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.63 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    46.61ms   13.16ms 157.93ms   75.22%
--
Requests/sec:   4190.27
Transfer/sec:     16.76MB

test brotli_comp_level 8
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.63 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    50.76ms   15.73ms 213.52ms   83.17%
--
Requests/sec:   3855.11
Transfer/sec:     15.41MB

test brotli_comp_level 9
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.62 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    70.71ms   17.86ms 229.45ms   82.74%
--
Requests/sec:   2755.16
Transfer/sec:     10.99MB

Interesting Brotli level 1 vs level 5 & 6 compression size. Brotli level 1 and 6 result in same compressed size 3.66KB !

Code (Text):

test brotli_comp_level 1
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency    12.28ms    5.31ms  76.79ms   80.91%
--
Requests/sec:  16190.71
Transfer/sec:     82.07MB

test brotli_comp_level 5
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.93 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.61ms    8.43ms 167.43ms   85.94%
--
Requests/sec:   5059.95
Transfer/sec:     20.38MB

test brotli_comp_level 6
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    40.12ms    5.15ms 119.51ms   90.46%
--
Requests/sec:   4874.63
Transfer/sec:     19.50MB

Cloudflare Reverse Proxy Zlib & Brotli Performance

Next compare with Cloudflare enabled as reverse proxy in front of Centmin Mod Nginx 1.15.0 server from above tests to compare Cloudflare's edge server Zlib and Brotli performance against pure-min.css file.

Code (Text):

curl -I $domain/pure-min.css
HTTP/2 200
date: Fri, 18 May 2018 21:40:03 GMT
content-type: text/css
set-cookie: __cfduid=d80ebaa41be4bf47e2feb95071a1b4c221526679603; expires=Sat, 18-May-19 21:40:03 GMT; path=/; domain=.domain.com; HttpOnly
last-modified: Tue, 27 Sep 2016 15:04:57 GMT
vary: Accept-Encoding
etag: W/"57ea8a99-4386"
x-powered-by: centminmod
expires: Sun, 17 Jun 2018 21:40:03 GMT
cache-control: public, max-age=2592000
access-control-allow-origin: *
cf-cache-status: MISS
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 41d17ae1bff64728-EWR

Cloudflare Reverse Proxy Zlib

Code (Text):

echo -e "\ntest cloudflare proxy zlib/gzip compression";
/usr/local/bin/curltest gzip $domain/pure-min.css;
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';

Code (Text):

URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.00 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   133.71ms  278.11ms   2.00s    88.93%
--
Requests/sec:   3693.35
Transfer/sec:     16.96MB

Code (Text):

URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 4.00 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.08ms  135.60ms   1.93s    97.09%
--
Requests/sec:   8223.68
Transfer/sec:     37.77MB

Cloudflare Reverse Proxy Brotli

Need to purge pure-min.css from Cloudflare cache for retesting next test.

Code (Text):

echo -e "\ntest cloudflare proxy brotli compression";
/usr/local/bin/curltest br $domain/pure-min.css;
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';

Code (Text):

/usr/local/bin/curltest br $domain/pure-min.css;
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.45ms   34.95ms 778.50ms   96.29%
--
Requests/sec:   6634.20
Transfer/sec:     28.30MB

retest cached performance

Code (Text):

/usr/local/bin/curltest br $domain/pure-min.css;
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.66 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    24.99ms   60.59ms   1.40s    97.81%
--
Requests/sec:   7293.99
Transfer/sec:     31.12MB

Summary

• Seems pretty clear that Cloudflare has further optimised their Brotli compression compared to Zlib/gzip even on their own Cloudflare edge servers for uncached runs. Cached runs zlib seems to fair better. But this could just depend on the Cloudflare edge servers you hit as there would be many edge servers in their cluster so variance would be expected.
• However seems local Centmin Mod zlib (cloudflare forked) seem to have performed best in this test skewed by fact that local tests were done on local network where average wrk-cmm latency was ~11ms versus Cloudflare reverse proxy which had to transverse a network with wrk0cmm latency ~38ms so more than 3x times the latency as local network tests.

Centmin Mod Nginx tests were do via local network so latency is much less than Cloudflare reverse proxy and thus skews results. But this is a comparison of relative zlib and brotli implementations as well to see how they fair on their respective web servers - Centmin Mod Nginx and Cloudflare edge servers.

config	compressed size	req/s	avg latency	max latency
Centmin Mod Nginx zlib level 5 (cf fork)	4.12KB	21906	9.07ms	74.32ms
Centmin Mod Nginx zlib level 6 (cf fork)	4.00KB	16997	11.67ms	49.06ms
Centmin Mod Nginx brotli level 5	3.93KB	5060	38.61ms	167.43ms
Centmin Mod Nginx brotli level 6	3.66KB	4875	40.12ms	119.51ms
Cloudflare Reverse Proxy zlib 1st run	4.00KB	3693	133.71ms	2000.0ms
Cloudflare Reverse Proxy zlib 2nd run	4.00KB	8224	38.08ms	1930.0ms
Cloudflare Reverse Proxy brotli 1st run	3.66KB	6634	23.45ms	778.5ms
Cloudflare Reverse Proxy brotli 2nd run	3.66KB	7294	24.99ms	1400.0ms

For Cloudflare zlib vs brotli tests

 

Centmin Mod Nginx zlib/brotli vs Cloudflare zlib/brotli - static compression

Another theory is Clouldflare Brotli is using static Brotli compression instead of dynamic Brotli compression by storing pre-Brotli compressed assets on edge servers. So lets test pre-gzip and pre-brotli compressed pure-min.css. I used pigz level 11 which uses better Zopfli based gzip compression level which is higher than standard zlib level 9 compression.

Code (Text):

cd /usr/local/nginx/html/
pigz -11kf /usr/local/nginx/html/pure-min.css
brotli -q 11 --force /usr/local/nginx/html/pure-min.css --output=/usr/local/nginx/html/pure-min.css.br
ls -lh | grep pure-min.css

Code (Text):

ls -lh | grep pure-min.css
-rw-r--r--  1 root  nginx  17K Sep 27  2016 pure-min.css
-rw-r--r--  1 root  nginx 3.4K Sep 27  2016 pure-min.css.br
-rw-r--r--  1 root  nginx 3.8K Sep 27  2016 pure-min.css.gz

Centmin Mod Nginx 1.15.0 + Cloudflare zlib fork (pre-zip compressed pure-min.css)

Code (Text):

for l in {1..9}; do
sleep 5;
echo -e "\ntest gzip_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='n'|" /usr/local/bin/curltest
/usr/local/bin/curltest gzip $domain/pure-min.css;
sed -ie "s|gzip_comp_level .*|gzip_comp_level $l;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|gzip_comp_level .*|gzip_comp_level 5;|" /usr/local/nginx/conf/nginx.conf | grep gzip_comp_level

Code (Text):

test gzip_comp_level 1
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.02ms    2.77ms  30.07ms   85.84%
--
Requests/sec:  75612.32
Transfer/sec:    312.88MB

test gzip_comp_level 2
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.32ms    3.54ms  69.33ms   88.26%
--
Requests/sec:  72866.07
Transfer/sec:    301.55MB

test gzip_comp_level 3
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.41ms    3.38ms  33.86ms   86.28%
--
Requests/sec:  71070.23
Transfer/sec:    294.10MB

test gzip_comp_level 4
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.40ms    3.52ms  64.03ms   86.99%
--
Requests/sec:  71715.11
Transfer/sec:    296.78MB

test gzip_comp_level 5
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.27ms    3.17ms  32.38ms   86.62%
--
Requests/sec:  72443.07
Transfer/sec:    299.79MB

test gzip_comp_level 6
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.36ms    3.39ms  50.43ms   87.24%
--
Requests/sec:  71905.31
Transfer/sec:    297.55MB

test gzip_comp_level 7
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.33ms    3.37ms  39.83ms   87.15%
--
Requests/sec:  71885.71
Transfer/sec:    297.49MB

test gzip_comp_level 8
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.30ms    3.21ms  36.53ms   86.61%
--
Requests/sec:  72355.43
Transfer/sec:    299.43MB

test gzip_comp_level 9
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.27ms    3.16ms  31.73ms   87.18%
--
Requests/sec:  71312.88
Transfer/sec:    295.12MB

Centmin Mod Nginx 1.15.0 + Nginx Brotli (pre-brotli compressed pure-min.css)

Code (Text):

for l in {1..9}; do
sleep 5;
echo -e "\ntest brotli_comp_level $l";
sed -ie "s|BROTLI=.*|BROTLI='y'|" /usr/local/bin/curltest
/usr/local/bin/curltest br $domain/pure-min.css;
sed -ie "s|brotli_comp_level .*|brotli_comp_level $l;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level
ngxrestart >/dev/null 2>&1;
echo "wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';"
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency $domain/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
done;
sed -ie "s|brotli_comp_level .*|brotli_comp_level 5;|" /usr/local/nginx/conf/brotli_inc.conf | grep brotli_comp_level

Code (Text):

test brotli_comp_level 1
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.66ms    2.66ms  51.53ms   87.71%
--
Requests/sec:  89472.49
Transfer/sec:    334.08MB

test brotli_comp_level 2
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.82ms    2.91ms  41.43ms   87.14%
--
Requests/sec:  85518.35
Transfer/sec:    319.32MB

test brotli_comp_level 3
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.70ms    2.52ms  29.44ms   87.35%
--
Requests/sec:  83973.56
Transfer/sec:    313.55MB

test brotli_comp_level 4
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.75ms    2.67ms  33.84ms   86.93%
--
Requests/sec:  85561.93
Transfer/sec:    319.47MB

test brotli_comp_level 5
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.76ms    2.72ms  39.79ms   87.91%
--
Requests/sec:  84643.04
Transfer/sec:    316.05MB

test brotli_comp_level 6
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.86ms    3.39ms  87.54ms   89.78%
--
Requests/sec:  84974.72
Transfer/sec:    317.28MB

test brotli_comp_level 7
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.35ms   12.63ms 165.20ms   97.76%
--
Requests/sec:  84183.69
Transfer/sec:    314.33MB

test brotli_comp_level 8
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.75ms    2.77ms  43.95ms   88.19%
--
Requests/sec:  85118.24
Transfer/sec:    317.82MB

test brotli_comp_level 9
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.73ms    2.61ms  31.51ms   86.12%
--
Requests/sec:  85812.28
Transfer/sec:    320.41MB

Zlib level 5 & 6

Code (Text):

test gzip_comp_level 5
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.27ms    3.17ms  32.38ms   86.62%
--
Requests/sec:  72443.07
Transfer/sec:    299.79MB

test gzip_comp_level 6
URI: https://domain.com/pure-min.css (gzip)
Uncompressed size : 16.88 KiB
Compressed size   : 3.79 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.36ms    3.39ms  50.43ms   87.24%
--
Requests/sec:  71905.31
Transfer/sec:    297.55MB

Brotli level 5 & 6

Code (Text):

test brotli_comp_level 5
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.76ms    2.72ms  39.79ms   87.91%
--
Requests/sec:  84643.04
Transfer/sec:    316.05MB

test brotli_comp_level 6
URI: https://domain.com/pure-min.css (br)
Uncompressed size : 16.88 KiB
Compressed size   : 3.38 KiB
wrk-cmm -t3 -c200 -d5s --breakout -H 'Accept-Encoding: br' -s scripts/setup.lua --latency https://domain.com/pure-min.css 2>&1 | egrep -A1 'Thread Stats|Requests/sec';
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.86ms    3.39ms  87.54ms   89.78%
--
Requests/sec:  84974.72
Transfer/sec:    317.28MB

config	compressed size	req/s	avg latency	max latency
Centmin Mod Nginx zlib level 5 pre-compress static (cf fork)	3.79KB	72443	3.27ms	32.38ms
Centmin Mod Nginx zlib level 6 pre-compress static (cf fork)	3.79KB	71905	3.36ms	50.43ms
Centmin Mod Nginx brotli level 5 pre-compress static	3.38KB	84643	2.76ms	39.79ms
Centmin Mod Nginx brotli level 6 pre-compress static	3.38KB	84975	2.96ms	87.54ms
Centmin Mod Nginx zlib level 5 dynamic (cf fork)	4.12KB	21906	9.07ms	74.32ms
Centmin Mod Nginx zlib level 6 dynamic (cf fork)	4.00KB	16997	11.67ms	49.06ms
Centmin Mod Nginx brotli level 5 dynamic	3.93KB	5060	38.61ms	167.43ms
Centmin Mod Nginx brotli level 6 dynamic	3.66KB	4875	40.12ms	119.51ms
Cloudflare Reverse Proxy zlib 1st run	4.00KB	3693	133.71ms	2000.0ms
Cloudflare Reverse Proxy zlib 2nd run	4.00KB	8224	38.08ms	1930.0ms
Cloudflare Reverse Proxy brotli 1st run	3.66KB	6634	23.45ms	778.5ms
Cloudflare Reverse Proxy brotli 2nd run	3.66KB	7294	24.99ms	1400.0ms

 

Interesting but 2 different locations or do you mean the drop after Nov 26th is ngx_brotli ?

 

Looks like Hetrixtools Free Uptime Monitor & Blacklist Monitor- HetrixTools (affiliate link)