Nginx Nginx HTTP/2 HTTPS Benchmarks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack

eva2000 · Jun 13, 2018

Previously did some non-HTTPS Nginx static html benchmarks comparing several LEMP stacks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack. This time will be comparing Nginx HTTP/2 and HTTP/1.1 HTTPS benchmark performance using wrk-cmm tool for HTTP/1.1 HTTPS and nghttp2's h2load HTTP/2 HTTPS load testing tool for HTTP/2 HTTPS. If you'd like to jump straight to the high user concurrency Nginx HTTP/2 HTTPS benchmarks, you can find them here or PHP comparison benchmarks are posted here.

HTTP/2 HTTPS Benchmark Comparisons are between the following LEMP stack installers installed in Ubuntu 18.04 LTS LXD containers on a ssdnode's 4 CPU, 16GB ram, 80GB disk KVM VPS with Ubuntu 18.04 LTS.

Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)

Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)

OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)

OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)

VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)

Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)

Nginx HTTPS Benchmarking Background

The Previous exact same LXD container based installs are used. The only difference is I wrote a script, lxc.sh which is run on Ubuntu 18.04 LTS LXD host which automates the configuration and setup for self-signed SSL certificates for Nginx dual RSA 2048bit and ECDSA 256bit SSL certificate configuration for all 6x LEMP web stack installs - Centmin Mod Nginx, Easyengine, Webinoly, Vestacp, Oneinstack Nginx, and Oneinstack Openresty Nginx.

You can view the full raw output for lxc.sh script for the automated Nginx HTTP/2 HTTPS setup for all LEMP stacks here.

LEMP Stack HTTP/2 HTTPS Setup Notes

lxc.sh ssl command automates the set up for dual rsa 2048bit/ecdsa 256bit self-signed ssl certificates which has been supported since Nginx 1.11.0 for each respective lemp stacks' nginx vhost named = http2.domain.com as well as inspects each LEMP stacks' default nginx configuration settings i.e. ssl ciphers etc

webinoly default nginx ssl cipher configuration doesn't support ECDSA 256bit ssl ciphers out of the box so only RSA 2048bit ssl ciphers are tested

vestacp and easyengine failed to restart nginx as it uses deprecated ssl on; directive and doesn't use http/2 https by default so had to manually remove ssl on;, add to listen directive ssl http2 and then also manually add ecdsa 256bit ssl cert to existing rsa2048 bit ssl certifcate

Using nghttp2's h2load HTTP/2 HTTPS load testing tool for HTTP/2 HTTPS benchmarks. CentOS 7.5 yum repo nghttp2 is v1.31 while, Ubuntu 18.04 LTS used by webinoly apt package is nghttp2 v1.30. For other LEMP stacks using Ubuntu 16.04 LTS, apt ngttp2 package is very old v1.7.1 and doesn't include h2load binary so needs to be source compiled from nghttp2 github repo. So for CentOS 7.5, Ubuntu 16.04 LTS and 18.04 LTS, nghttp2/h2load v1.33 is installed via source compile.

easyengine 3.8.1 default ssl cipher order choice causes h2load HTTP/2 HTTPS load test to fail due to use of weak unsupported ssl cipher DHE_RSA_AES_128_GCM_SHA256

easyengine 3.8.1 default ssl cipher not supported by HTTP/2 HTTPS for DHE_RSA_AES_128_GCM_SHA256
Code (Text):
curl -sv https://http2.domain.com/
*   Trying 10.71.164.21...
* Connected to http2.domain.com (10.71.164.21) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / DHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
easyengine 3.8.1 failed test of h2load HTTP/2 HTTPS
Code (Text):
h2load -t1 -c1 -n1 https://http2.domain.com/
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests

finished in 4.37ms, 0.00 req/s, 0B/s
requests: 1 total, 0 started, 0 done, 0 succeeded, 1 failed, 1 errored, 0 timeout
status codes: 0 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 0B (0) total, 0B (0) headers (space savings 0.00%), 0B (0) data
                    min         max         mean         sd        +/- sd
time for request:        0us         0us         0us         0us     0.00%
time for connect:        0us         0us         0us         0us     0.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :       0.00        0.00        0.00        0.00   100.00%
easyengine 3.8.1 default ssl ciphers
Code (Text):
lxc exec easyengine-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
       ssl_session_cache shared:SSL:20m;
       ssl_session_timeout 10m;
       ssl_prefer_server_ciphers on;
       ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHADHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!ECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
       ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
easyengine 3.8.1 needs manually fixing for HTTP/2 HTTPS support by changing ssl cipher choices according to Chrome Security Issue
Code (Text):
sed -i 's/ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHADHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!ECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;/ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;/' /etc/nginx/nginx.conf

systemctl restart nginx
recheck easyengine 3.8.1 nginx HTTP/2 HTTPS support via h2load HTTP/2 HTTPS tool = working no failed request
Code (Text):
h2load -t1 -c1 -n1 https://http2.domain.com/index.html
starting benchmark...
spawning thread #0: 1 total client(s). 1 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 100% done

finished in 6.35ms, 157.41 req/s, 1.00MB/s
requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 6.51KB (6665) total, 142B (142) headers (space savings 35.16%), 6.30KB (6456) data
                    min         max         mean         sd        +/- sd
time for request:      506us       506us       506us         0us   100.00%
time for connect:     5.16ms      5.16ms      5.16ms         0us   100.00%
time to 1st byte:     5.71ms      5.71ms      5.71ms         0us   100.00%
req/s           :     173.25      173.25      173.25        0.00   100.00%
Nginx TLS Protocol / SSL Cipher Summary

Summary of all LEMP stacks' Nginx default preferred TLS protocol and SSL ciphers for h2load HTTP/2 HTTPS client are in below table when all Nginx configurations have dual RSA 2048bit + ECDSA 256bit SSL certification configurations setup manually.

Nginx OpenSSL Protocol SSL Cipher Server Temp Key App Protocol

Centmin Mod Nginx 1.15.0 OpenSSL 1.1.0h TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2

EasyEngine Nginx 1.14.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2

OneInStack Std Nginx 1.14.0 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

OneInStack OpenResty Nginx 1.13.6 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

VestaCP Nginx 1.15.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

Webinoly Nginx 1.14.0 OpenSSL 1.1.0g TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 X25519 253 bits h2

Centmin Mod 123.09beta01 Nginx HTTPS settings

default ssl cipher settings
Code (Text):
lxc exec centos75-2 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/nginx/conf/conf.d/http2.domain.com.ssl.conf
  ssl_dhparam /usr/local/nginx/conf/ssl/http2.domain.com/dhparam.pem;
  ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
curl header check
Code (Text):
lxc exec centos75-2 -- curl -Ik https://http2.domain.com/
HTTP/1.1 200 OK
Date: Tue, 12 Jun 2018 23:54:58 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 6457
Last-Modified: Fri, 08 Jun 2018 03:54:47 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b19fe07-1939"
Server: nginx centminmod
X-Powered-By: centminmod
Accept-Ranges: bytes
h2load version
Code (Text):
lxc exec centos75-2 -- h2load --version
h2load nghttp2/1.31.1
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ centmin mod
prefers TLSv1.2 with ECDHE-ECDSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec centos75-2 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
Nginx version

nginx -V
nginx version: nginx/1.15.0 (060618-051249)
built by gcc 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=060618-051249 --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.42 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.0h --with-openssl-opt='enable-ec_nistp_64_gcc_128'
Click to expand...

Easyengine 3.8.1 Nginx HTTPS settings

default ssl cipher settings
Code (Text):
lxc exec easyengine-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
       ssl_session_cache shared:SSL:20m;
       ssl_session_timeout 10m;
       ssl_prefer_server_ciphers on;
       ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
       ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
curl header check
Code (Text):
lxc exec easyengine-ubuntu1604 -- curl -Ik https://http2.domain.com/
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 12 Jun 2018 23:55:11 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Sun, 10 Jun 2018 06:28:01 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1cc4f1-1938"
X-Powered-By: EasyEngine 3.8.1
Accept-Ranges: bytes
h2load version
Code (Text):
lxc exec easyengine-ubuntu1604 -- h2load --version
h2load nghttp2/1.33.0-DEV
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ easyengine
prefers TLSv1.2 with ECDHE-ECDSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec easyengine-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
Nginx version

nginx -V
nginx version: nginx/1.14.0 (EasyEngine)
built with OpenSSL 1.0.2g-fips 1 Mar 2016 (running with OpenSSL 1.0.2g 1 Mar 2016)
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --add-module=/usr/src/packages/BUILD/debian/modules/headers-more-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/nginx-auth-pam --add-module=/usr/src/packages/BUILD/debian/modules/nginx-cache-purge --add-module=/usr/src/packages/BUILD/debian/modules/nginx-echo --add-module=/usr/src/packages/BUILD/debian/modules/ngx-fancyindex --add-module=/usr/src/packages/BUILD/debian/modules/nginx-lua --add-module=/usr/src/packages/BUILD/debian/modules/nginx-upload-progress --add-module=/usr/src/packages/BUILD/debian/modules/nginx-upstream-fair --add-module=/usr/src/packages/BUILD/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/packages/BUILD/debian/modules/memc-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/srcache-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/HttpRedisModule --add-module=/usr/src/packages/BUILD/debian/modules/redis2-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/ngx_devel_kit --add-module=/usr/src/packages/BUILD/debian/modules/set-misc-nginx-module
Click to expand...

VestaCP 0.9.8-21 Nginx HTTPS settings

default ssl cipher settings
Code (Text):
lxc exec vestacp-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
   ssl_session_cache   shared:SSL:10m;
   ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
   ssl_prefer_server_ciphers on;
   ssl_ciphers        "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
curl header check
Code (Text):
lxc exec vestacp-ubuntu1604 -- curl -Ik https://http2.domain.com/
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 12 Jun 2018 23:55:38 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Sun, 10 Jun 2018 23:56:56 GMT
Connection: keep-alive
Keep-Alive: timeout=60
ETag: "5b1dbac8-1938"
Accept-Ranges: bytes
h2load version
Code (Text):
lxc exec vestacp-ubuntu1604 -- h2load --version
h2load nghttp2/1.33.0-DEV
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ vestacp nginx
prefers TLSv1.2 with ECDHE-RSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec vestacp-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
Nginx version

nginx -V
nginx version: nginx/1.15.0
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
Click to expand...

OneInStack v1.7 Standard Nginx HTTPS settings

default ssl cipher settings - seems to use nginx upstream defaults ?
Code (Text):
lxc exec oneinstack-ubuntu16-nginx -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/nginx/conf/nginx.conf.default
   #    ssl_session_cache    shared:SSL:1m;
   #    ssl_session_timeout  5m;
   #    ssl_ciphers  HIGH:!aNULL:!MD5;
   #    ssl_prefer_server_ciphers  on;
curl header check
Code (Text):
lxc exec oneinstack-ubuntu16-nginx -- curl -Ik https://http2.domain.com/
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 12 Jun 2018 23:55:20 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Mon, 11 Jun 2018 21:27:15 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1ee933-1938"
Accept-Ranges: bytes
h2load version
Code (Text):
lxc exec oneinstack-ubuntu16-nginx -- h2load --version
h2load nghttp2/1.33.0-DEV
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ oneinstack nginx
prefers TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec oneinstack-ubuntu16-nginx -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
Nginx version

/usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.14.0
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2o 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_v2_module --with-http_ssl_module --with-http_gzip_static_module --with-http_realip_module --with-http_flv_module --with-http_mp4_module --with-openssl=../openssl-1.0.2o --with-pcre=../pcre-8.42 --with-pcre-jit --with-ld-opt=-ljemalloc
Click to expand...

OneInStack v1.7 OpenResty Nginx HTTPS settings

default ssl cipher settings - seems to use nginx upstream defaults ?
Code (Text):
lxc exec oneinstack-ubuntu16-openresty -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/openresty/nginx/conf/nginx.conf.default
   #    ssl_session_cache    shared:SSL:1m;
   #    ssl_session_timeout  5m;
   #    ssl_ciphers  HIGH:!aNULL:!MD5;
   #    ssl_prefer_server_ciphers  on;
curl header check
Code (Text):
lxc exec oneinstack-ubuntu16-openresty -- curl -Ik https://http2.domain.com/
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 12 Jun 2018 23:55:29 GMT
Content-Type: text/html
Content-Length: 6456
Last-Modified: Mon, 11 Jun 2018 22:28:32 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5b1ef790-1938"
Accept-Ranges: bytes
h2load version
Code (Text):
lxc exec oneinstack-ubuntu16-openresty -- h2load --version
h2load nghttp2/1.33.0-DEV
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ oneinstack openresty nginx
prefers TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec oneinstack-ubuntu16-openresty -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
Nginx version

/usr/local/openresty/nginx/sbin/nginx -V
nginx version: openresty/1.13.6.2
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2o 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -ljemalloc' --user=www --group=www --with-http_stub_status_module --with-http_v2_module --with-http_ssl_module --with-http_gzip_static_module --with-http_realip_module --with-http_flv_module --with-http_mp4_module --with-openssl=/root/oneinstack/src/openresty-1.13.6.2/../openssl-1.0.2o --with-pcre=/root/oneinstack/src/openresty-1.13.6.2/../pcre-8.42 --with-pcre-jit --with-stream --with-stream_ssl_module
Click to expand...

Webinoly v1.4.3 Nginx HTTPS settings

default ssl cipher settings
Code (Text):
lxc exec webinoly -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
       ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
       ssl_session_timeout 10m;
       ssl_session_cache shared:SSL:20m;
       ssl_dhparam /etc/ssl/dhparam.pem;
       ssl_prefer_server_ciphers on;
       ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT;
curl header check
Code (Text):
lxc exec webinoly -- curl -Ik https://http2.domain.com/
HTTP/2 200
server: nginx
date: Tue, 12 Jun 2018 23:55:50 GMT
content-type: text/html
content-length: 6457
last-modified: Sun, 10 Jun 2018 05:51:21 GMT
vary: Accept-Encoding
etag: "5b1cbc59-1939"
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: public, no-cache
referrer-policy: unsafe-url
accept-ranges: bytes
h2load version
Code (Text):
lxc exec webinoly -- h2load --version
h2load nghttp2/1.33.0-DEV
h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ - webinoly prefers TLSv1.2 with
ECDHE-RSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
Code (Text):
lxc exec webinoly -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
Nginx version

nginx -V
nginx version: nginx/1.14.0
built with OpenSSL 1.1.0g 2 Nov 2017
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-6kZD6z/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-echo --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-subs-filter
Click to expand...

eva2000 · Jun 13, 2018

h2load HTTP/2 HTTPS Benchmarks

For CentOS 7.5, Ubuntu 16.04 LTS and 18.04 LTS, nghttp2/h2load v1.33 is installed via source compile with their native OS' GCC compilers due to differing versions in their native repositories.

h2load version and h2load client preferred TLS protocol and cipher handshake preferences

Code (Text):

./lxc.sh sslinfo

-------------------------------------------------------------------------------------------
lxc exec centos75-2 -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec centos75-2 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2

-------------------------------------------------------------------------------------------
lxc exec easyengine-ubuntu1604 -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec easyengine-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2

Code (Text):

-------------------------------------------------------------------------------------------
lxc exec oneinstack-ubuntu16-nginx -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec oneinstack-ubuntu16-nginx -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2

-------------------------------------------------------------------------------------------
lxc exec oneinstack-ubuntu16-openresty -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec oneinstack-ubuntu16-openresty -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2

Code (Text):

-------------------------------------------------------------------------------------------
lxc exec vestacp-ubuntu1604 -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec vestacp-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2

-------------------------------------------------------------------------------------------
lxc exec webinoly -- h2load --version
h2load nghttp2/1.33.0-DEV

lxc exec webinoly -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2

h2load test parameters

The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c300 and -n6000 values were chosen so they're comparable to Centmin Mod Nginx HTTP/2 HTTPS benchmarking thread.

h2load -t1 = 1 thread
--ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
-c300 = 300 concurrent users
-n6000 = 6000 requests

h2load test command used:

Code (Text):

h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c300 -n6000 https://http2.domain.com

h2load HTTP/2 HTTPS test results

Full raw h2load benchmark results are posted here.
Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 20% faster than EasyEngine and OneInStack Standard Nginx, 24.4% faster than OneInStack OpenResty Nginx, 45.3% faster than VestaCP Nginx and 72.2% faster than Webinoly Nginx
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 16.4% faster than OneInStack Standard Nginx, 38.7% faster than OneInStack OpenResty Nginx, 40.96% faster than EasyEngine, 69.33% faster than VestaCP Nginx and 247% faster than Webinoly Nginx

h2load-c300-n6000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

Centmin Mod Nginx LXD container (centos75-2)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 8547.89 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4527.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4899.63 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4531.04 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4194.85 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3940.57 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 7187.32 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4126.79 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 5372.18 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
3940.57  5259  8547.89  1576.95

EasyEngine LXD Container (easyengine-ubtunu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 5073.68 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3992.09 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 5143.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3124.82 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 6063.55 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 4134.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 5135.44 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3244.38 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3505.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
3124.82  4380  6063.55  1019.21

OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 4114.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3379.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 7341.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 4464.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3294.19 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 2997.02 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 4716.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3148.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 5908.97 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
2997.02  4374  7341.39  1453.78

OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 4100.36 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3873.98 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3712.26 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 4332.21 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 6161.56 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3373.14 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3344.59 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3741.93 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 5409.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
3344.59  4228  6161.56  955.22

VestaCP Nginx LXD Container (vestacp-ubuntu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 4030.93 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 2606.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3558.60 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 4174.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 2988.50 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 5017.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3116.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 4007.24 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
300 6000 3073.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
2606.03  3619  5017.79  755.22

Webinoly Nginx LXD Container (webinoly)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
300 6000 3304.84 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3016.89 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 2904.67 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 2957.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3458.85 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 2886.78 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3080.34 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 3013.99 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
300 6000 2857.94 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000

h2load result summary
min:     avg:  max:     stddev:
2857.94  3054  3458.85  202.65

eva2000 · Jun 14, 2018

h2load higher user concurrency tests

Now to test at higher user concurrency level = 600 users with 60,000 requests.

h2load test parameters

The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c600 and -n60000 values were chosen

h2load -t1 = 1 thread
--ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
-c600 = 600 concurrent users
-n60000 = 60000 requests

h2load test command used:

Code (Text):

h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com

h2load HTTP/2 HTTPS test results

Full raw h2load benchmark results are posted here.
Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 14.6% faster than OneInStack Standard Nginx, 19.95% faster than Easyengine, 32.32% faster than OneInStack OpenResty Nginx, 63.17% faster than VestaCP Nginx and 72.07% faster than Webinoly Nginx
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 5.4% faster than OneInStack Standard Nginx, 19.9% faster than Easyengine, 31.4% faster than OneInStack OpenResty Nginx, 75.29% faster than VestaCP Nginx and 243% faster than Webinoly Nginx

h2load-c600-n60000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

Centmin Mod Nginx LXD container (centos75-2)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 6698.91 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 6210.37 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 5594.28 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 7215.18 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4767.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 7078.43 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 6662.67 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 10259.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 6608.53 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000

h2load result summary
min:     avg:  max:      stddev:
4767.78  6788  10259.78  1508.97

EasyEngine LXD Container (easyengine-ubtunu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 3907.37 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 8556.10 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 7715.98 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4208.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3917.30 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3729.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 7901.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 6945.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4037.10 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000

h2load result summary
min:     avg:  max:    stddev:
3729.17  5658  8556.1  2057.52

OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 9731.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 8406.34 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 6101.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 5950.41 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4661.76 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4335.41 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4406.01 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4300.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 5411.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000

h2load result summary
min:     avg:  max:     stddev:
4300.09  5923  9731.03  1936.76

OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 5162.43 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4133.99 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3808.69 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4156.90 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4633.76 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 7808.13 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 5729.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4203.43 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 6530.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000

h2load result summary
min:     avg:  max:     stddev:
3808.69  5130  7808.13  1336.22

VestaCP Nginx LXD Container (vestacp-ubuntu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 5853.35 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3164.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3527.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 5155.85 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3509.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3395.67 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 3601.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 5138.77 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
600 6000 4097.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
h2load result summary
min:     avg:  max:     stddev:
3164.25  4160  5853.35  970.27

Webinoly Nginx LXD Container (webinoly)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
600 6000 3932.95 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3721.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3908.50 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4107.80 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4008.05 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3881.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 4219.72 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3748.31 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
600 6000 3976.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000

h2load result summary
min:     avg:  max:     stddev:
3721.12  3945  4219.72  158.61

eva2000 · Jun 15, 2018

h2load 1000 concurrent user test

Now to test at higher user concurrency level = 1000 users with 100,000 requests.

h2load test parameters

The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c1000 and -n100000 values were chosen

h2load -t1 = 1 thread
--ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
-c1000 = 1000 concurrent users
-n100000 = 100000 requests

h2load test command used:

Code (Text):

h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com

h2load HTTP/2 HTTPS test results

Full raw h2load benchmark results are posted here.
Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 19.6% faster than OneInStack OpenResty Nginx, 27.5% faster than OneInStack Standard Nginx, 27.65% faster than VestaCP Nginx, 31.86% faster than EasyEngine Nginx and 83.15% faster than Webinoly Nginx
On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 4.17% faster than OneInStack OpenResty Nginx, 21.55% faster than OneInStack Standard Nginx, 26.8% faster than VestaCP Nginx, 53.3% faster than EasyEngine Nginx and 276.7% faster than Webinoly Nginx

h2load-c1000-n100000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

Centmin Mod Nginx LXD container (centos75-2)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 7697.12 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5475.08 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6415.33 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4982.01 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 10666.16 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 12028.07 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6173.02 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5066.09 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5266.98 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:      stddev:
4982.01  7086  12028.07  2581.40

EasyEngine LXD Container (easyengine-ubtunu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 5178.64 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3572.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4951.62 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3762.51 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4922.35 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 7713.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 7845.75 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3795.15 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6622.21 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
3572.05  5374  7845.75  1654.51

OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 4517.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4384.28 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4457.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7240.80 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 9895.08 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4873.05 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4326.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3825.86 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6488.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
3825.86  5556  9895.08  1973.80

OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 6103.27 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4030.54 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3970.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5534.97 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4710.35 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5678.26 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4583.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7151.60 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 11545.82 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:      stddev:
3970.79  5923  11545.82  2345.88

VestaCP Nginx LXD Container (vestacp-ubuntu1604)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 3796.12 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 9483.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6836.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3946.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4791.68 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4842.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6802.77 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5430.91 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4024.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
3796.12  5551  9483.75  1862.98

Webinoly Nginx LXD Container (webinoly)

Code (Text):

users requests req/s encoding cipher protocol started succeeded
100 1000 4346.54 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3853.59 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3672.07 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4042.92 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3884.68 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3631.22 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3765.88 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3756.65 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3871.60 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
3631.22  3869  4346.54  217.19

eva2000 · Jun 15, 2018

h2load -t2 1000 concurrent user test

Nginx is multi-threaded and up to now all above tests have only used 1 thread for h2load tests (-t1 option). As the test server is a 4 cpu KVM VPS, this next test is changing h2load to -t2 to test with 2 threads and the same previous 1000 users with 100,000 requests to see how respective LEMP stack's Nginx fairs for HTTP/2 HTTPS performance.

h2load test parameters

The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -t2, -c1000 and -n100000 values were chosen

h2load -t2 = 2 thread

--ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.

Accept-Encoding: gzip = test compressed HTTP requests like a web browser would

-c1000 = 1000 concurrent users

-n100000 = 100000 requests

h2load test command used:
Code (Text):
h2load -t2 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com
h2load HTTP/2 HTTPS test results

Full raw h2load benchmark results are posted here.

Testing with h2load -t2 for 2 threads and thus testing Nginx with 2 threads improved results for all LEMP stacks' Nginx servers compared to previous above test with h2load -t1 - 1 thread.

Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance for both average requests/s and max requests/s. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place but testing with 2 threads has improved Webinoly results slightly.

On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 11.8% faster than OneInStack OpenResty Nginx, 21.89% faster than OneInStack Standard Nginx, 53.49% faster than EasyEngine Nginx, 60.8% faster than VestaCP Nginx and 212.99% faster than Webinoly Nginx

On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 16.45% faster than OneInStack OpenResty Nginx, 25.7% faster than OneInStack Standard Nginx, 27.49% faster than EasyEngine Nginx, 54.49% faster than VestaCP Nginx and 238.45% faster than Webinoly Nginx

Of note, for minimum requests/s results, it's first time I've seen a Nginx version other than Centmin Mod Nginx lead with OneInStack OpenResty Nginx 1.13.6 achieving 7,646.22 requests/s landing in 1st place - 18.6% ahead of Centmin Mod Nginx 1.15.0 with 6,445.9 requests/s landing in 2nd place. If you look carefully, OneInStack OpenResty Nginx and Centmin Mod Nginx's minimum requests/s is actually higher than the average requests/s for all other LEMP stacks' Nginx servers and higher than Webinoly Nginx's max requests/s !

Centmin Mod Nginx LXD container (centos75-2)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 11270.51 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 9691.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6445.90 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 11344.67 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 10306.99 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 12873.71 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 7226.89 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 11773.58 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 7781.15 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:    avg:  max:      stddev:
6445.9  9857  12873.71  2238.22
EasyEngine LXD Container (easyengine-ubtunu1604)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 10097.55 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4660.85 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5364.29 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5935.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 8832.90 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4365.01 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6235.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6142.11 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6169.23 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:      stddev:
4365.01  6422  10097.55  1877.09
OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 6652.98 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 9498.91 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7808.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6401.80 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8442.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8136.59 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 10240.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 9430.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6166.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:      stddev:
6166.96  8087  10240.79  1467.41
OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 9739.53 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8330.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 11054.67 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8411.56 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 9096.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7993.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7646.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8699.65 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8352.57 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:      stddev:
7646.22  8814  11054.67  1037.75
VestaCP Nginx LXD Container (vestacp-ubuntu1604)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 7315.23 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5902.55 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5435.50 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 7775.14 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 8333.15 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4364.15 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5321.69 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4714.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6004.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
4364.15  6130  8333.15  1382.46
Webinoly Nginx LXD Container (webinoly)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 4380.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3862.08 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4338.28 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4761.52 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5398.63 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4245.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4930.35 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4919.26 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4815.79 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000

h2load result summary
min:     avg:  max:     stddev:
3862.08  4628  5398.63  461.21

eva2000 · Jun 15, 2018

h2load -t1 vs -t2 1k concurrent users

Combining results for last 2 runs for h2load -t1 vs h2load -t2 for 1,000 concurrent user tests. You can see which Nginx versions are better optimised for multi-threaded Nginx work loads by comparing their respective h2load -t1 vs -t2 results and seeing which -t2 results continue to scale in terms of performance.

LEMP stack installers installed in Ubuntu 18.04 LTS LXD containers on a ssdnode's 4 CPU, 16GB ram, 80GB disk KVM VPS with Ubuntu 18.04 LTS and their respective performance scaling with from -t1 to -t2 (1 thread to 2 threads)

Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5) = 39.1% increase in average requests/s and 29.39% increase in min requests/s and 7.03% increase in max requests/s

Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6) = 19.5% increase in average requests/s and 22.2% increase in min requests/s and 28.7% increase in max requests/s

OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6) = 45.55% increase in average requests/s and 61.19% increase in min requests/s and 3.5% increase in max requests/s

OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6) = 48.8% increase in average requests/s and 92.55% increase in min requests/s and 4.25% decrease in max requests/s

VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9) = 10.4% increase in average requests/s and 14.96% increase in min requests/s and 12.13% decrease in max requests/s

Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6) = 19.6% increase in average requests/s and 6.36% increase in min requests/s and 24.2% increase in max requests/s

Observations

For average requests/s, Centmin Mod Nginx's 1 thread results are actually faster than EasyEngine (+10.33%), VestaCP (+15.6%) and Webinoly's (+53.11%) 2 thread results and within 80-87% of OneInStack's 2 thread results !

For minimum requests/s, Centmin Mod Nginx's 2 thread results are actually faster than the average requests/s for EasyEngine (+0.37%), VestaCP (+5.15%) and Webinoly (+39.28) !

eva2000 · Jun 15, 2018

Nginx Idle Resource Usage

For those curious about respective Nginx server resource usage, I will be looking at comparison stats at idle and load. But for now here's the base line idle Nginx resource usage from pidstats and smem output on each respective LEMP stack.

For those unfamiliar with Nginx processes and outline is at NGINX Docs | Controlling NGINX Processes at Runtime

Master and Worker Processes
NGINX has one master process and one or more worker processes. If caching is enabled, the cache loader and cache manager processes also run at startup.

The main purpose of the master process is to read and evaluate configuration files, as well as maintain the worker processes.

The worker processes do the actual processing of requests. NGINX relies on OS-dependent mechanisms to efficiently distribute requests among worker processes. The number of worker processes is defined by the worker_processes directive in the nginx.conf configuration file and can either be set to a fixed number or configured to adjust automatically to the number of available CPU cores.
Click to expand...

Notes:

FYI, Centmin Mod 123.09beta01 Nginx (centos75-2 LXD container) doesn't allocate all worker_processes to all cpus but auto detects number of cpus available and only allocates ~1/2 the number to worker_processes (4/2 = 2 cpus allocated) leaving some cpus to better balanced sharing with php-fpm, MariaDB MySQL, memcached and other processes. So that reflects in below pidstats and number of nginx processes reported when compared with other LEMP stacks' Nginx output.

Looks like out of the box EasyEngine, VestaCP and Webinoly do enable caching configuration directives as the nginx cache manager, cache loader processes are showing

Also VestaCP seems to have 2x Nginx instances running - one for VestaCP GUI and one for main Nginx

RSS/VSZ memory size also reflects number of Nginx modules/features built in and default settings configurations Nginx out of the box as you can see from Nginx -V outputs further below. Centmin Mod LEMP stack initial install auto detects server resources including memory available and auto optimises Nginx, PHP-FPM and MariaDB configuration settings to optimally use the available memory, so it's expected memory usage would be higher.

Note smem output totals include python smem process memory usage too.
Code (Text):
-------------------------------------------------------------------------------------------
stats check centos75-2

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (centos75-2)    06/15/18        _x86_64_        (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
 1529033515     0       537    0.00    0.00    0.00    0.00     2      3.43      0.00  125024  26240   0.16  nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
 1529033515  1000       538    0.00    0.05    0.00    0.05     2    261.29      0.00  153696  48980   0.30  nginx: worker process                                     
 1529033515  1000       539    0.00    0.10    0.00    0.10     2    261.19      0.00  153696  48980   0.30  nginx: worker process                                     

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
  816 root     python /usr/bin/smem -tk -P        0     6.4M     6.5M     8.3M 
  537 root     nginx: master process /usr/        0   200.0K     8.4M    25.6M 
  539 nginx    nginx: worker process              0    21.2M    29.8M    48.2M 
  538 nginx    nginx: worker process              0    21.2M    29.9M    48.3M 
-------------------------------------------------------------------------------
    4 2                                           0    49.1M    74.6M   130.4M 
Code (Text):
-------------------------------------------------------------------------------------------
stats check easyengine-ubuntu1604

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (easyengine-ubtunu1604)         06/15/18        _x86_64_        (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
 1529033516     0       432    0.00    0.00    0.00    0.00     0      5.06      0.00  213032    4340   0.03  nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
 1529033516    33       445    0.00    0.00    0.00    0.00     0     32.38      0.00  214824    6464   0.04  nginx: worker process                           
 1529033516    33       446    0.00    0.00    0.00    0.00     3     32.38      0.00  214824    6464   0.04  nginx: worker process                           
 1529033516    33       447    0.00    0.00    0.00    0.00     1     32.38      0.00  214824    6464   0.04  nginx: worker process                           
 1529033516    33       448    0.00    0.00    0.00    0.00     0     32.38      0.00  214824    6464   0.04  nginx: worker process                           
 1529033516    33       449    0.00    0.00    0.00    0.00     0     11.19      0.00  213240    5392   0.03  nginx: cache manager process                     
 1529033516    33       450    0.00    0.00    0.00    0.00     0     10.69      0.00  213240    5392   0.03  nginx: cache loader process                     

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
  584 root     sh -c smem -tk -P 'nginx:'         0    88.0K   208.0K     1.6M 
  432 root     nginx: master process /usr/        0   188.0K   607.0K     4.2M 
  450 www-data nginx: cache loader process        0   476.0K     1.0M     5.4M 
  449 www-data nginx: cache manager proces        0   636.0K     1.2M     5.8M 
  445 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
  446 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
  447 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
  448 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
  585 root     python /usr/bin/smem -tk -P        0     7.6M     7.7M     9.6M 
-------------------------------------------------------------------------------
    9 2                                           0    16.3M    19.9M    53.2M 
Code (Text):
-------------------------------------------------------------------------------------------
stats check oneinstack-ubuntu16-nginx

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (oneinstack-ubuntu16-nginx)     06/15/18        _x86_64_        (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
 1529033517     0       422    0.00    0.00    0.00    0.00     3      8.15      0.00   51688    3996   0.02  nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
 1529033517  1002       423    0.00    0.08    0.00    0.08     3    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
 1529033517  1002       424    0.00    0.08    0.00    0.08     3    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
 1529033517  1002       425    0.00    0.08    0.00    0.08     1    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
 1529033517  1002       426    0.00    0.08    0.00    0.08     3    415.54      0.00   82408   25564   0.16  nginx: worker process                                           

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
  906 root     sh -c smem -tk -P 'nginx:'         0   104.0K   170.0K     1.6M 
  422 root     nginx: master process /usr/        0   208.0K   777.0K     3.9M 
  907 root     /usr/bin/python /usr/bin/sm        0     7.0M     7.1M     8.9M 
  426 www      nginx: worker process              0    20.9M    21.5M    25.3M 
  423 www      nginx: worker process              0    21.0M    21.6M    25.5M 
  424 www      nginx: worker process              0    21.0M    21.6M    25.5M 
  425 www      nginx: worker process              0    21.0M    21.6M    25.5M 
-------------------------------------------------------------------------------
    7 2                                           0    91.1M    94.2M   116.3M 
Code (Text):
-------------------------------------------------------------------------------------------
stats check oneinstack-ubuntu16-openresty

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (oneinstack-ubuntu16-openresty)         06/15/18        _x86_64_        (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
 1529033518     0       447    0.00    0.00    0.00    0.00     2      9.33      0.00   55200    4492   0.03  nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
 1529033518  1002       448    0.00    0.08    0.00    0.08     0    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
 1529033518  1002       449    0.00    0.08    0.00    0.08     1    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
 1529033518  1002       450    0.00    0.08    0.00    0.08     3    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
 1529033518  1002       451    0.00    0.08    0.00    0.08     0    451.08      0.00   85920   26060   0.16  nginx: worker process                                                               

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
  909 root     sh -c smem -tk -P 'nginx:'         0    84.0K   162.0K     1.7M 
  447 root     nginx: master process /usr/        0   224.0K   861.0K     4.4M 
  910 root     /usr/bin/python /usr/bin/sm        0     7.0M     7.2M     8.9M 
  451 www      nginx: worker process              0    21.0M    21.7M    25.9M 
  448 www      nginx: worker process              0    21.0M    21.7M    26.1M 
  449 www      nginx: worker process              0    21.0M    21.7M    26.1M 
  450 www      nginx: worker process              0    21.0M    21.7M    26.1M 
-------------------------------------------------------------------------------
    7 2                                           0    91.2M    94.9M   119.3M 
Code (Text):
-------------------------------------------------------------------------------------------
stats check vestacp-ubuntu1604

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (vestacp-ubuntu1604.example.com)        06/15/18        _x86_64_        (4 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
 1529033518     0       647    0.00    0.00    0.00    0.00     0      4.73      0.00   54040    1400   0.01  nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 1529033518    33       648    0.00    0.00    0.00    0.00     0     16.27      0.00   54044    3164   0.02  nginx: worker process                   
 1529033518    33       649    0.00    0.00    0.00    0.00     1     16.27      0.00   54044    3164   0.02  nginx: worker process                   
 1529033518    33       650    0.00    0.00    0.00    0.00     2     16.27      0.00   54044    3164   0.02  nginx: worker process                   
 1529033518    33       651    0.00    0.00    0.00    0.00     3     16.27      0.00   54044    3164   0.02  nginx: worker process                   
 1529033518    33       652    0.00    0.00    0.00    0.00     0     11.64      0.00   54044    3288   0.02  nginx: cache manager process             
 1529033518    33       653    0.00    0.00    0.00    0.00     2     11.18      0.00   54044    3288   0.02  nginx: cache loader process             
 1529033518     0       667    0.00    0.00    0.00    0.00     2      4.18      0.00   42192     828   0.01  nginx: master process /usr/local/vesta/nginx/sbin/vesta-nginx
 1529033518  1001       669    0.00    0.00    0.00    0.00     2      8.09      0.00   42192    2780   0.02  nginx: worker process                   

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
 1122 root     sh -c smem -tk -P 'nginx:'         0    92.0K   174.0K     1.6M 
  647 root     nginx: master process /usr/        0   148.0K   358.0K     2.2M 
  653 www-data nginx: cache loader process        0   368.0K   617.0K     3.3M 
  667 root     nginx: master process /usr/        0   148.0K   718.0K     2.2M 
  652 www-data nginx: cache manager proces        0   512.0K   758.0K     3.4M 
  648 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
  649 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
  650 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
  651 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
  669 admin    nginx: worker process              0   256.0K   846.0K     3.0M 
 1123 root     python /usr/bin/smem -tk -P        0     6.9M     7.0M     8.8M 
-------------------------------------------------------------------------------
   11 3                                           0    10.7M    13.6M    38.5M 
Code (Text):
-------------------------------------------------------------------------------------------
stats check webinoly

pidstat -urhl -C nginx | grep -v 'pidstat'
Linux 4.15.0-22-generic (webinoly)      06/15/18        _x86_64_        (4 CPU)

# Time        UID       PID    %usr %system  %guest   %wait    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
03:31:58        0       259    0.00    0.00    0.00    0.00    0.00     0      4.78      0.00  115544    1164   0.01  nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
03:31:58       33       260    0.00    0.00    0.00    0.11    0.00     1    107.33      0.00  123392    7012   0.04  nginx: worker process                           
03:31:58       33       261    0.00    0.00    0.00    0.22    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
03:31:58       33       262    0.00    0.00    0.00    0.44    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
03:31:58       33       263    0.00    0.00    0.00    0.33    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
03:31:58       33       264    0.00    0.00    0.00    0.11    0.00     0     17.89      0.00  119924    4080   0.02  nginx: cache manager process                     
03:31:58       33       265    0.00    0.00    0.00    0.11    0.00     0     17.44      0.00  119924    3676   0.02  nginx: cache loader process                     

smem -tk -P 'nginx:'
  PID User     Command                         Swap      USS      PSS      RSS 
  696 root     sh -c smem -tk -P 'nginx:'         0   200.0K   247.0K     1.6M 
  259 root     nginx: master process /usr/        0   180.0K   364.0K     1.9M 
  265 www-data nginx: cache loader process        0   424.0K   700.0K     3.9M 
  264 www-data nginx: cache manager proces        0   568.0K   844.0K     4.0M 
  260 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
  261 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
  262 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
  263 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
  697 root     python /usr/bin/smem -tk -P        0     7.1M     7.3M     9.6M 
-------------------------------------------------------------------------------
    9 2                                           0    22.8M    24.8M    49.0M 
Centmin Mod 123.09beta01 Nginx

Nginx version

nginx -V
nginx version: nginx/1.15.0 (060618-051249)
built by gcc 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -g -O3 -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=060618-051249 --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.4.2 --add-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre=../pcre-8.42 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.0h --with-openssl-opt='enable-ec_nistp_64_gcc_128'
Click to expand...

Easyengine 3.8.1 Nginx

Nginx version

nginx -V
nginx version: nginx/1.14.0 (EasyEngine)
built with OpenSSL 1.0.2g-fips 1 Mar 2016 (running with OpenSSL 1.0.2g 1 Mar 2016)
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --add-module=/usr/src/packages/BUILD/debian/modules/headers-more-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/nginx-auth-pam --add-module=/usr/src/packages/BUILD/debian/modules/nginx-cache-purge --add-module=/usr/src/packages/BUILD/debian/modules/nginx-echo --add-module=/usr/src/packages/BUILD/debian/modules/ngx-fancyindex --add-module=/usr/src/packages/BUILD/debian/modules/nginx-lua --add-module=/usr/src/packages/BUILD/debian/modules/nginx-upload-progress --add-module=/usr/src/packages/BUILD/debian/modules/nginx-upstream-fair --add-module=/usr/src/packages/BUILD/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/packages/BUILD/debian/modules/memc-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/srcache-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/HttpRedisModule --add-module=/usr/src/packages/BUILD/debian/modules/redis2-nginx-module --add-module=/usr/src/packages/BUILD/debian/modules/ngx_devel_kit --add-module=/usr/src/packages/BUILD/debian/modules/set-misc-nginx-module
Click to expand...

VestaCP 0.9.8-21 Nginx

Nginx version

nginx -V
nginx version: nginx/1.15.0
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
Click to expand...

OneInStack v1.7 Standard Nginx

Nginx version

/usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.14.0
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2o 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_v2_module --with-http_ssl_module --with-http_gzip_static_module --with-http_realip_module --with-http_flv_module --with-http_mp4_module --with-openssl=../openssl-1.0.2o --with-pcre=../pcre-8.42 --with-pcre-jit --with-ld-opt=-ljemalloc
Click to expand...

OneInStack v1.7 OpenResty Nginx

Nginx version

/usr/local/openresty/nginx/sbin/nginx -V
nginx version: openresty/1.13.6.2
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
built with OpenSSL 1.0.2o 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -ljemalloc' --user=www --group=www --with-http_stub_status_module --with-http_v2_module --with-http_ssl_module --with-http_gzip_static_module --with-http_realip_module --with-http_flv_module --with-http_mp4_module --with-openssl=/root/oneinstack/src/openresty-1.13.6.2/../openssl-1.0.2o --with-pcre=/root/oneinstack/src/openresty-1.13.6.2/../pcre-8.42 --with-pcre-jit --with-stream --with-stream_ssl_module
Click to expand...

Webinoly v1.4.3 Nginx

Nginx version

nginx -V
nginx version: nginx/1.14.0
built with OpenSSL 1.1.0g 2 Nov 2017
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-6kZD6z/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-echo --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-subs-filter
Click to expand...

eva2000 · Jun 21, 2018

Nginx Amplify For LXD Container Based LEMP Stacks

Next up is install Nginx Amplify monitoring for all LXD container based LEMP stacks

Of course you need working /nginx_status urls for Amplify to gather stats. I wrote a custom lxc.sh script to be able to query all LXD containers via one command

Code (Text):

./lxc.sh query curl -s localhost/nginx_status

--------------------------------------------------------------
centos75-2

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------
--------------------------------------------------------------
easyengine-ubuntu1604

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------
--------------------------------------------------------------
oneinstack-ubuntu16-nginx

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------
--------------------------------------------------------------
oneinstack-ubuntu16-openresty

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------
--------------------------------------------------------------
vestacp-ubuntu1604

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------
--------------------------------------------------------------
webinoly

curl -s localhost/nginx_status      

Active connections: 1
server accepts handled requests
 1 1 1
Reading: 0 Writing: 1 Waiting: 0
--------------------------------------------------------------

Custom Nginx Amplify dashboard showing all LEMP's Nginx stats on one page.

eva2000 · Jun 30, 2018

VestaCP & Webinoly Updates

Some LEMP stacks got version updated

VestaCP 0.9.8.22 for fixing another security bug

Webinoly 1.5.0 with some nginx.conf changes for ssl ciphers to add missing ECDSA ssl cipher support

Webinoly v1.5.0 Nginx HTTPS settings
Code (Text):
egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
        ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_session_timeout 10m;
        ssl_session_cache shared:SSL:20m;
        ssl_dhparam /etc/ssl/dhparam.pem;
        ssl_prefer_server_ciphers on;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACH
Code (Text):
h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
nginx -V
nginx version: nginx/1.14.0
built with OpenSSL 1.1.0g 2 Nov 2017 (running with OpenSSL 1.1.0h 27 Mar 2018)
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-6kZD6z/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-echo --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-6kZD6z/nginx-1.14.0/debian/modules/http-subs-filter
Click to expand...

Nginx TLS Protocol / SSL Cipher Summary

Updated summary of all LEMP stacks' Nginx default preferred TLS protocol and SSL ciphers for h2load HTTP/2 HTTPS client are in below table when all Nginx configurations have dual RSA 2048bit + ECDSA 256bit SSL certification configurations setup manually.

Nginx OpenSSL Protocol SSL Cipher Server Temp Key App Protocol

Centmin Mod Nginx 1.15.0 OpenSSL 1.1.0h TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2

EasyEngine Nginx 1.14.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2

OneInStack Std Nginx 1.14.0 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

OneInStack OpenResty Nginx 1.13.6 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

VestaCP Nginx 1.15.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

Webinoly Nginx 1.14.0 OpenSSL 1.1.0g TLSv1.2 ECDHE-ECDSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2

eva2000 · Jul 1, 2018

h2load -t2 1000 concurrent user test

With VestaCP 0.9.8-22 security update and Webinoly 1.5.0 updated to enable ECDSA ssl ciphers for Nginx HTTP/2 HTTPS, decided to revisit Nginx HTTP/2 HTTPS static HTML file tests at h2load -t2 to test with 2 threads and the same previous 1000 users with 100,000 requests to see how respective LEMP stack's Nginx fairs for HTTP/2 HTTPS performance.

Note: As the underlying server is a KVM VPS, we are subject to the KVM host node's underlying resource usage loads shared with neighbouring KVM VPS customers so results may vary from previously tested. However, relative results this re-run can be compared across all LEMP stack's Nginx servers.

h2load test parameters

The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -t2, -c1000 and -n100000 values were chosen

h2load -t2 = 2 thread

--ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.

Accept-Encoding: gzip = test compressed HTTP requests like a web browser would

-c1000 = 1000 concurrent users

-n100000 = 100000 requests

h2load test command used:
Code (Text):
h2load -t2 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com/index.html
h2load HTTP/2 HTTPS test results

Again Centmin Mod Nginx came out on top and Webinoly still came last. However, EasyEngine came in 2nd place taking the position from OneInStack OpenResty Nginx

Centmin Mod Nginx's minimum requests/s was higher than other LEMP stack's Nginx average requests/s !

On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 38.5% faster than EasyEngine, 54.88% faster than OneInStack OpenResty Nginx, 58.68% faster than VestaCP Nginx, 71.65% faster OneInStack Standard Nginx, and 93.36% faster than Webinoly Nginx

On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 9.4% faster than EasyEngine, 15.6% faster than OneInStack OpenResty Nginx, 16.048% faster than VestaCP Nginx, 7.3% faster OneInStack Standard Nginx, and 45.03% faster than Webinoly Nginx

Centmin Mod Nginx LXD container (centos75-2)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 6357.63 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6808.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6647.53 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6639.08 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6256.79 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6126.83 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6800.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5500.96 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6512.54 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:  requests-succeeded:
5500.960  6405.598  6808.970  412.812  100.00
EasyEngine LXD Container (easyengine-ubtunu1604)
Code (Text):
100 1000 4228.64 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3517.66 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4740.30 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 3832.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 6222.66 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 5790.96 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4284.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4410.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
100 1000 4606.21 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:  requests-succeeded:
3517.660  4625.958  6222.660  873.115  100.00
OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 2328.04 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2249.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4003.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4437.06 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4201.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2889.85 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 6345.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3095.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4038.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:   requests-succeeded:
2249.520  3732.094  6345.610  1274.745  100.00
OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 3603.37 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2576.92 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5573.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3310.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3641.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4809.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5887.63 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3964.42 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3853.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:   requests-succeeded:
2576.920  4135.658  5887.630  1080.359  100.00
VestaCP Nginx LXD Container (vestacp-ubuntu1604)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 3060.54 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4084.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2745.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3662.30 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4530.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4932.57 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3675.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 5868.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3770.66 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:  requests-succeeded:
2745.390  4036.678  5868.090  959.435  100.00
Webinoly Nginx LXD Container (webinoly)
Code (Text):
users requests req/s encoding cipher protocol started succeeded
100 1000 3142.85 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3035.93 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3114.39 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2728.16 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3032.64 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4013.47 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 4695.01 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 2544.65 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
100 1000 3510.57 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
h2load result summary
min:      avg:      max:      stddev:  requests-succeeded:
2544.650  3313.074  4695.010  670.430  100.00

eva2000 · Jul 2, 2018

nghttp HTTP/2 Statistics

A much simpler test is nghttp HTTP/2 statistics test which reports the request start and end times and actual processing times for all HTTPS requests as defined below where lower the request times and processing times = faster it is

Code (Text):

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

Results ordered as follows for:

Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)
Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)
VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)
Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)

Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)

Code (Text):

centos75-2

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13     +4.20ms       +144us   4.06ms  200   2K /index.html
 15     +6.39ms      +4.24ms   2.15ms  200   4K /purecss/pure-min.css
 17     +7.74ms      +4.24ms   3.50ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)

Code (Text):

easyengine-ubuntu1604

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13    +20.80ms       +295us  20.50ms  200   2K /index.html
 15    +31.03ms     +20.86ms  10.18ms  200   3K /purecss/pure-min.css
 17    +31.60ms     +20.86ms  10.74ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)

Code (Text):

oneinstack-ubuntu16-nginx

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13    +17.78ms       +207us  17.58ms  200   2K /index.html
 15    +32.24ms     +17.87ms  14.37ms  200   3K /purecss/pure-min.css
 17    +33.52ms     +17.87ms  15.65ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)

Code (Text):

oneinstack-ubuntu16-openresty

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13    +21.69ms       +769us  20.92ms  200   2K /index.html
 15    +31.86ms     +21.77ms  10.09ms  200   3K /purecss/pure-min.css
 17    +32.57ms     +21.77ms  10.80ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)

Code (Text):

vestacp-ubuntu1604

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13    +18.75ms       +172us  18.58ms  200   2K /index.html
 15    +29.21ms     +18.82ms  10.38ms  200   3K /purecss/pure-min.css
 17    +30.08ms     +18.83ms  11.25ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)

Code (Text):

webinoly

nghttp -ynas https://http2.domain.com/index.html      

***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13    +18.29ms       +194us  18.10ms  200   2K /index.html
 15    +28.29ms     +18.35ms   9.94ms  200   3K /purecss/pure-min.css
 17    +29.17ms     +18.36ms  10.82ms  200   1K /purecss/grids-responsive-min.css
--------------------------------------------------------------

Log in / Register

Useful Searches

Want more timely Centmin Mod News Updates?

Nginx Nginx HTTP/2 HTTPS Benchmarks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack

eva2000 Administrator Staff Member

Nginx HTTPS Benchmarking Background

LEMP Stack HTTP/2 HTTPS Setup Notes

Nginx TLS Protocol / SSL Cipher Summary

Centmin Mod 123.09beta01 Nginx HTTPS settings

Easyengine 3.8.1 Nginx HTTPS settings

VestaCP 0.9.8-21 Nginx HTTPS settings

OneInStack v1.7 Standard Nginx HTTPS settings

OneInStack v1.7 OpenResty Nginx HTTPS settings

Webinoly v1.4.3 Nginx HTTPS settings

eva2000 Administrator Staff Member

h2load HTTP/2 HTTPS Benchmarks

h2load test parameters

h2load HTTP/2 HTTPS test results

eva2000 Administrator Staff Member

h2load higher user concurrency tests

h2load test parameters

h2load HTTP/2 HTTPS test results

eva2000 Administrator Staff Member

h2load 1000 concurrent user test

h2load test parameters

h2load HTTP/2 HTTPS test results

eva2000 Administrator Staff Member

h2load -t2 1000 concurrent user test

h2load test parameters

h2load HTTP/2 HTTPS test results

eva2000 Administrator Staff Member

h2load -t1 vs -t2 1k concurrent users

eva2000 Administrator Staff Member

Nginx Idle Resource Usage

Centmin Mod 123.09beta01 Nginx

Easyengine 3.8.1 Nginx

VestaCP 0.9.8-21 Nginx

OneInStack v1.7 Standard Nginx

OneInStack v1.7 OpenResty Nginx

Webinoly v1.4.3 Nginx

eva2000 Administrator Staff Member

Nginx Amplify For LXD Container Based LEMP Stacks

eva2000 Administrator Staff Member

VestaCP & Webinoly Updates

Webinoly v1.5.0 Nginx HTTPS settings

Nginx TLS Protocol / SSL Cipher Summary

eva2000 Administrator Staff Member

h2load -t2 1000 concurrent user test

h2load test parameters

h2load HTTP/2 HTTPS test results

eva2000 Administrator Staff Member

nghttp HTTP/2 Statistics

.