Welcome to Centmin Mod Community
Become a Member

Nginx Nginx HTTP/2 HTTPS Benchmarks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack

Discussion in 'Nginx and PHP-FPM news & discussions' started by eva2000, Jun 13, 2018.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Previously did some non-HTTPS Nginx static html benchmarks comparing several LEMP stacks - Centmin Mod vs Easyengine vs Webinoly vs VestaCP vs OneInStack. This time will be comparing Nginx HTTP/2 and HTTP/1.1 HTTPS benchmark performance using wrk-cmm tool for HTTP/1.1 HTTPS and nghttp2's h2load HTTP/2 HTTPS load testing tool for HTTP/2 HTTPS. If you'd like to jump straight to the high user concurrency Nginx HTTP/2 HTTPS benchmarks, you can find them here or PHP comparison benchmarks are posted here.

    HTTP/2 HTTPS Benchmark Comparisons are between the following LEMP stack installers installed in Ubuntu 18.04 LTS LXD containers on a ssdnode's 4 CPU, 16GB ram, 80GB disk KVM VPS with Ubuntu 18.04 LTS.
    • Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)
    • Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)
    • Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)

    Nginx HTTPS Benchmarking Background


    • The Previous exact same LXD container based installs are used. The only difference is I wrote a script, lxc.sh which is run on Ubuntu 18.04 LTS LXD host which automates the configuration and setup for self-signed SSL certificates for Nginx dual RSA 2048bit and ECDSA 256bit SSL certificate configuration for all 6x LEMP web stack installs - Centmin Mod Nginx, Easyengine, Webinoly, Vestacp, Oneinstack Nginx, and Oneinstack Openresty Nginx.
    • You can view the full raw output for lxc.sh script for the automated Nginx HTTP/2 HTTPS setup for all LEMP stacks here.

    LEMP Stack HTTP/2 HTTPS Setup Notes



    • lxc.sh ssl command automates the set up for dual rsa 2048bit/ecdsa 256bit self-signed ssl certificates which has been supported since Nginx 1.11.0 for each respective lemp stacks' nginx vhost named = http2.domain.com as well as inspects each LEMP stacks' default nginx configuration settings i.e. ssl ciphers etc
    • webinoly default nginx ssl cipher configuration doesn't support ECDSA 256bit ssl ciphers out of the box so only RSA 2048bit ssl ciphers are tested
    • vestacp and easyengine failed to restart nginx as it uses deprecated ssl on; directive and doesn't use http/2 https by default so had to manually remove ssl on;, add to listen directive ssl http2 and then also manually add ecdsa 256bit ssl cert to existing rsa2048 bit ssl certifcate
    • Using nghttp2's h2load HTTP/2 HTTPS load testing tool for HTTP/2 HTTPS benchmarks. CentOS 7.5 yum repo nghttp2 is v1.31 while, Ubuntu 18.04 LTS used by webinoly apt package is nghttp2 v1.30. For other LEMP stacks using Ubuntu 16.04 LTS, apt ngttp2 package is very old v1.7.1 and doesn't include h2load binary so needs to be source compiled from nghttp2 github repo. So for CentOS 7.5, Ubuntu 16.04 LTS and 18.04 LTS, nghttp2/h2load v1.33 is installed via source compile.
    • easyengine 3.8.1 default ssl cipher order choice causes h2load HTTP/2 HTTPS load test to fail due to use of weak unsupported ssl cipher DHE_RSA_AES_128_GCM_SHA256
    easyengine 3.8.1 default ssl cipher not supported by HTTP/2 HTTPS for DHE_RSA_AES_128_GCM_SHA256
    Code (Text):
    curl -sv https://http2.domain.com/
    *   Trying 10.71.164.21...
    * Connected to http2.domain.com (10.71.164.21) port 443 (#0)
    * found 148 certificates in /etc/ssl/certs/ca-certificates.crt
    * found 594 certificates in /etc/ssl/certs
    * ALPN, offering http/1.1
    * SSL connection using TLS1.2 / DHE_RSA_AES_128_GCM_SHA256
    * server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
    * Closing connection 0
    

    easyengine 3.8.1 failed test of h2load HTTP/2 HTTPS
    Code (Text):
    h2load -t1 -c1 -n1 https://http2.domain.com/
    starting benchmark...
    spawning thread #0: 1 total client(s). 1 total requests
    
    finished in 4.37ms, 0.00 req/s, 0B/s
    requests: 1 total, 0 started, 0 done, 0 succeeded, 1 failed, 1 errored, 0 timeout
    status codes: 0 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 0B (0) total, 0B (0) headers (space savings 0.00%), 0B (0) data
                        min         max         mean         sd        +/- sd
    time for request:        0us         0us         0us         0us     0.00%
    time for connect:        0us         0us         0us         0us     0.00%
    time to 1st byte:        0us         0us         0us         0us     0.00%
    req/s           :       0.00        0.00        0.00        0.00   100.00%
    

    easyengine 3.8.1 default ssl ciphers
    Code (Text):
    lxc exec easyengine-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
           ssl_session_cache shared:SSL:20m;
           ssl_session_timeout 10m;
           ssl_prefer_server_ciphers on;
           ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHADHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!ECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
           ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    

    easyengine 3.8.1 needs manually fixing for HTTP/2 HTTPS support by changing ssl cipher choices according to Chrome Security Issue
    Code (Text):
    sed -i 's/ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHADHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!ECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;/ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;/' /etc/nginx/nginx.conf
    
    systemctl restart nginx
    

    recheck easyengine 3.8.1 nginx HTTP/2 HTTPS support via h2load HTTP/2 HTTPS tool = working no failed request
    Code (Text):
    h2load -t1 -c1 -n1 https://http2.domain.com/index.html
    starting benchmark...
    spawning thread #0: 1 total client(s). 1 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 100% done
    
    finished in 6.35ms, 157.41 req/s, 1.00MB/s
    requests: 1 total, 1 started, 1 done, 1 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 6.51KB (6665) total, 142B (142) headers (space savings 35.16%), 6.30KB (6456) data
                        min         max         mean         sd        +/- sd
    time for request:      506us       506us       506us         0us   100.00%
    time for connect:     5.16ms      5.16ms      5.16ms         0us   100.00%
    time to 1st byte:     5.71ms      5.71ms      5.71ms         0us   100.00%
    req/s           :     173.25      173.25      173.25        0.00   100.00%
    


    Nginx TLS Protocol / SSL Cipher Summary



    Summary of all LEMP stacks' Nginx default preferred TLS protocol and SSL ciphers for h2load HTTP/2 HTTPS client are in below table when all Nginx configurations have dual RSA 2048bit + ECDSA 256bit SSL certification configurations setup manually.

    Nginx OpenSSL Protocol SSL Cipher Server Temp Key App Protocol
    Centmin Mod Nginx 1.15.0 OpenSSL 1.1.0h TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2
    EasyEngine Nginx 1.14.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2
    OneInStack Std Nginx 1.14.0 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    OneInStack OpenResty Nginx 1.13.6 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    VestaCP Nginx 1.15.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    Webinoly Nginx 1.14.0 OpenSSL 1.1.0g TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 X25519 253 bits h2


    Centmin Mod 123.09beta01 Nginx HTTPS settings



    default ssl cipher settings
    Code (Text):
    lxc exec centos75-2 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/nginx/conf/conf.d/http2.domain.com.ssl.conf
      ssl_dhparam /usr/local/nginx/conf/ssl/http2.domain.com/dhparam.pem;
      ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
      ssl_prefer_server_ciphers   on;
    

    curl header check
    Code (Text):
    lxc exec centos75-2 -- curl -Ik https://http2.domain.com/
    HTTP/1.1 200 OK
    Date: Tue, 12 Jun 2018 23:54:58 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 6457
    Last-Modified: Fri, 08 Jun 2018 03:54:47 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5b19fe07-1939"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Accept-Ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec centos75-2 -- h2load --version
    h2load nghttp2/1.31.1
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ centmin mod
    prefers TLSv1.2 with ECDHE-ECDSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec centos75-2 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx version

    Easyengine 3.8.1 Nginx HTTPS settings



    default ssl cipher settings
    Code (Text):
    lxc exec easyengine-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
           ssl_session_cache shared:SSL:20m;
           ssl_session_timeout 10m;
           ssl_prefer_server_ciphers on;
           ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
           ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    

    curl header check
    Code (Text):
    lxc exec easyengine-ubuntu1604 -- curl -Ik https://http2.domain.com/
    HTTP/1.1 200 OK
    Server: nginx
    Date: Tue, 12 Jun 2018 23:55:11 GMT
    Content-Type: text/html
    Content-Length: 6456
    Last-Modified: Sun, 10 Jun 2018 06:28:01 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5b1cc4f1-1938"
    X-Powered-By: EasyEngine 3.8.1
    Accept-Ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec easyengine-ubuntu1604 -- h2load --version
    h2load nghttp2/1.33.0-DEV
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ easyengine
    prefers TLSv1.2 with ECDHE-ECDSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec easyengine-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx version

    VestaCP 0.9.8-21 Nginx HTTPS settings



    default ssl cipher settings
    Code (Text):
    lxc exec vestacp-ubuntu1604 -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
       ssl_session_cache   shared:SSL:10m;
       ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
       ssl_prefer_server_ciphers on;
       ssl_ciphers        "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    

    curl header check
    Code (Text):
    lxc exec vestacp-ubuntu1604 -- curl -Ik https://http2.domain.com/
    HTTP/1.1 200 OK
    Server: nginx
    Date: Tue, 12 Jun 2018 23:55:38 GMT
    Content-Type: text/html
    Content-Length: 6456
    Last-Modified: Sun, 10 Jun 2018 23:56:56 GMT
    Connection: keep-alive
    Keep-Alive: timeout=60
    ETag: "5b1dbac8-1938"
    Accept-Ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec vestacp-ubuntu1604 -- h2load --version
    h2load nghttp2/1.33.0-DEV
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ vestacp nginx
    prefers TLSv1.2 with ECDHE-RSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec vestacp-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx version

    OneInStack v1.7 Standard Nginx HTTPS settings



    default ssl cipher settings - seems to use nginx upstream defaults ?
    Code (Text):
    lxc exec oneinstack-ubuntu16-nginx -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/nginx/conf/nginx.conf.default
       #    ssl_session_cache    shared:SSL:1m;
       #    ssl_session_timeout  5m;
       #    ssl_ciphers  HIGH:!aNULL:!MD5;
       #    ssl_prefer_server_ciphers  on;
    

    curl header check
    Code (Text):
    lxc exec oneinstack-ubuntu16-nginx -- curl -Ik https://http2.domain.com/
    HTTP/1.1 200 OK
    Server: nginx
    Date: Tue, 12 Jun 2018 23:55:20 GMT
    Content-Type: text/html
    Content-Length: 6456
    Last-Modified: Mon, 11 Jun 2018 21:27:15 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5b1ee933-1938"
    Accept-Ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec oneinstack-ubuntu16-nginx -- h2load --version
    h2load nghttp2/1.33.0-DEV
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ oneinstack nginx
    prefers TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec oneinstack-ubuntu16-nginx -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx version

    OneInStack v1.7 OpenResty Nginx HTTPS settings



    default ssl cipher settings - seems to use nginx upstream defaults ?
    Code (Text):
    lxc exec oneinstack-ubuntu16-openresty -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /usr/local/openresty/nginx/conf/nginx.conf.default
       #    ssl_session_cache    shared:SSL:1m;
       #    ssl_session_timeout  5m;
       #    ssl_ciphers  HIGH:!aNULL:!MD5;
       #    ssl_prefer_server_ciphers  on;
    

    curl header check
    Code (Text):
    lxc exec oneinstack-ubuntu16-openresty -- curl -Ik https://http2.domain.com/
    HTTP/1.1 200 OK
    Server: openresty
    Date: Tue, 12 Jun 2018 23:55:29 GMT
    Content-Type: text/html
    Content-Length: 6456
    Last-Modified: Mon, 11 Jun 2018 22:28:32 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5b1ef790-1938"
    Accept-Ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec oneinstack-ubuntu16-openresty -- h2load --version
    h2load nghttp2/1.33.0-DEV
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ oneinstack openresty nginx
    prefers TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec oneinstack-ubuntu16-openresty -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx version

    Webinoly v1.4.3 Nginx HTTPS settings



    default ssl cipher settings
    Code (Text):
    lxc exec webinoly -- egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
           ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
           ssl_session_timeout 10m;
           ssl_session_cache shared:SSL:20m;
           ssl_dhparam /etc/ssl/dhparam.pem;
           ssl_prefer_server_ciphers on;
           ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT;
    

    curl header check
    Code (Text):
    lxc exec webinoly -- curl -Ik https://http2.domain.com/
    HTTP/2 200
    server: nginx
    date: Tue, 12 Jun 2018 23:55:50 GMT
    content-type: text/html
    content-length: 6457
    last-modified: Sun, 10 Jun 2018 05:51:21 GMT
    vary: Accept-Encoding
    etag: "5b1cbc59-1939"
    x-frame-options: SAMEORIGIN
    x-content-type-options: nosniff
    x-xss-protection: 1; mode=block
    cache-control: public, no-cache
    referrer-policy: unsafe-url
    accept-ranges: bytes
    

    h2load version
    Code (Text):
    lxc exec webinoly -- h2load --version
    h2load nghttp2/1.33.0-DEV
    

    h2load tested SSL cipher/protocol when connecting to https://http2.domain.com/ - webinoly prefers TLSv1.2 with
    ECDHE-RSA-AES128-GCM-SHA256 ssl cipher for h2load HTTP/2 HTTPS client
    Code (Text):
    lxc exec webinoly -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: X25519 253 bits
    Application protocol: h2
    

    Nginx version
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load HTTP/2 HTTPS Benchmarks



    For CentOS 7.5, Ubuntu 16.04 LTS and 18.04 LTS, nghttp2/h2load v1.33 is installed via source compile with their native OS' GCC compilers due to differing versions in their native repositories.

    h2load version and h2load client preferred TLS protocol and cipher handshake preferences
    Code (Text):
    ./lxc.sh sslinfo
    
    -------------------------------------------------------------------------------------------
    lxc exec centos75-2 -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec centos75-2 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    
    -------------------------------------------------------------------------------------------
    lxc exec easyengine-ubuntu1604 -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec easyengine-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    lxc exec oneinstack-ubuntu16-nginx -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec oneinstack-ubuntu16-nginx -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    
    -------------------------------------------------------------------------------------------
    lxc exec oneinstack-ubuntu16-openresty -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec oneinstack-ubuntu16-openresty -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    lxc exec vestacp-ubuntu1604 -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec vestacp-ubuntu1604 -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    
    -------------------------------------------------------------------------------------------
    lxc exec webinoly -- h2load --version
    h2load nghttp2/1.33.0-DEV
    
    lxc exec webinoly -- h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: X25519 253 bits
    Application protocol: h2
    


    h2load test parameters



    The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c300 and -n6000 values were chosen so they're comparable to Centmin Mod Nginx HTTP/2 HTTPS benchmarking thread.
    • h2load -t1 = 1 thread
    • --ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
    • Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
    • -c300 = 300 concurrent users
    • -n6000 = 6000 requests
    h2load test command used:
    Code (Text):
    h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c300 -n6000 https://http2.domain.com
    


    h2load HTTP/2 HTTPS test results


    • Full raw h2load benchmark results are posted here.
    • Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 20% faster than EasyEngine and OneInStack Standard Nginx, 24.4% faster than OneInStack OpenResty Nginx, 45.3% faster than VestaCP Nginx and 72.2% faster than Webinoly Nginx
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 16.4% faster than OneInStack Standard Nginx, 38.7% faster than OneInStack OpenResty Nginx, 40.96% faster than EasyEngine, 69.33% faster than VestaCP Nginx and 247% faster than Webinoly Nginx

    h2load-c300-n6000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

    Centmin Mod Nginx LXD container (centos75-2)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 8547.89 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4527.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4899.63 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4531.04 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4194.85 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3940.57 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 7187.32 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4126.79 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 5372.18 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3940.57  5259  8547.89  1576.95
    

    EasyEngine LXD Container (easyengine-ubtunu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 5073.68 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3992.09 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 5143.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3124.82 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 6063.55 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 4134.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 5135.44 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3244.38 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3505.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3124.82  4380  6063.55  1019.21
    

    OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 4114.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3379.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 7341.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 4464.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3294.19 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 2997.02 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 4716.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3148.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 5908.97 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    2997.02  4374  7341.39  1453.78
    

    OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 4100.36 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3873.98 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3712.26 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 4332.21 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 6161.56 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3373.14 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3344.59 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3741.93 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 5409.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3344.59  4228  6161.56  955.22
    

    VestaCP Nginx LXD Container (vestacp-ubuntu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 4030.93 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 2606.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3558.60 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 4174.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 2988.50 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 5017.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3116.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 4007.24 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    300 6000 3073.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    2606.03  3619  5017.79  755.22
    

    Webinoly Nginx LXD Container (webinoly)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 3304.84 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3016.89 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 2904.67 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 2957.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3458.85 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 2886.78 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3080.34 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 3013.99 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 2857.94 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    
    h2load result summary
    min:     avg:  max:     stddev:
    2857.94  3054  3458.85  202.65
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load higher user concurrency tests



    Now to test at higher user concurrency level = 600 users with 60,000 requests.

    h2load test parameters



    The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c600 and -n60000 values were chosen
    • h2load -t1 = 1 thread
    • --ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
    • Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
    • -c600 = 600 concurrent users
    • -n60000 = 60000 requests
    h2load test command used:
    Code (Text):
    h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com
    


    h2load HTTP/2 HTTPS test results


    • Full raw h2load benchmark results are posted here.
    • Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 14.6% faster than OneInStack Standard Nginx, 19.95% faster than Easyengine, 32.32% faster than OneInStack OpenResty Nginx, 63.17% faster than VestaCP Nginx and 72.07% faster than Webinoly Nginx
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 5.4% faster than OneInStack Standard Nginx, 19.9% faster than Easyengine, 31.4% faster than OneInStack OpenResty Nginx, 75.29% faster than VestaCP Nginx and 243% faster than Webinoly Nginx

    h2load-c600-n60000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

    Centmin Mod Nginx LXD container (centos75-2)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 6698.91 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 6210.37 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 5594.28 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 7215.18 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4767.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 7078.43 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 6662.67 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 10259.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 6608.53 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    
    h2load result summary
    min:     avg:  max:      stddev:
    4767.78  6788  10259.78  1508.97
    

    EasyEngine LXD Container (easyengine-ubtunu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 3907.37 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 8556.10 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 7715.98 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4208.78 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3917.30 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3729.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 7901.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 6945.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4037.10 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    
    h2load result summary
    min:     avg:  max:    stddev:
    3729.17  5658  8556.1  2057.52
    

    OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 9731.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 8406.34 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 6101.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 5950.41 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4661.76 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4335.41 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4406.01 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4300.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 5411.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    
    h2load result summary
    min:     avg:  max:     stddev:
    4300.09  5923  9731.03  1936.76
    

    OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 5162.43 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4133.99 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3808.69 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4156.90 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4633.76 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 7808.13 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 5729.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4203.43 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 6530.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3808.69  5130  7808.13  1336.22
    

    VestaCP Nginx LXD Container (vestacp-ubuntu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 5853.35 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3164.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3527.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 5155.85 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3509.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3395.67 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 3601.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 5138.77 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    600 6000 4097.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 60000 60000
    h2load result summary
    min:     avg:  max:     stddev:
    3164.25  4160  5853.35  970.27
    

    Webinoly Nginx LXD Container (webinoly)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    600 6000 3932.95 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3721.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3908.50 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4107.80 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4008.05 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3881.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 4219.72 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3748.31 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    600 6000 3976.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 60000 60000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3721.12  3945  4219.72  158.61
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load 1000 concurrent user test



    Now to test at higher user concurrency level = 1000 users with 100,000 requests.

    h2load test parameters



    The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -c1000 and -n100000 values were chosen
    • h2load -t1 = 1 thread
    • --ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
    • Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
    • -c1000 = 1000 concurrent users
    • -n100000 = 100000 requests
    h2load test command used:
    Code (Text):
    h2load -t1 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com
    


    h2load HTTP/2 HTTPS test results


    • Full raw h2load benchmark results are posted here.
    • Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place.
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 19.6% faster than OneInStack OpenResty Nginx, 27.5% faster than OneInStack Standard Nginx, 27.65% faster than VestaCP Nginx, 31.86% faster than EasyEngine Nginx and 83.15% faster than Webinoly Nginx
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 4.17% faster than OneInStack OpenResty Nginx, 21.55% faster than OneInStack Standard Nginx, 26.8% faster than VestaCP Nginx, 53.3% faster than EasyEngine Nginx and 276.7% faster than Webinoly Nginx

    h2load-c1000-n100000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

    Centmin Mod Nginx LXD container (centos75-2)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 7697.12 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5475.08 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6415.33 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4982.01 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 10666.16 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 12028.07 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6173.02 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5066.09 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5266.98 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:      stddev:
    4982.01  7086  12028.07  2581.40
    

    EasyEngine LXD Container (easyengine-ubtunu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 5178.64 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3572.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4951.62 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3762.51 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4922.35 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 7713.77 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 7845.75 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3795.15 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6622.21 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3572.05  5374  7845.75  1654.51
    

    OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 4517.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4384.28 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4457.03 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7240.80 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 9895.08 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4873.05 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4326.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3825.86 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6488.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3825.86  5556  9895.08  1973.80
    

    OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 6103.27 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4030.54 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3970.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5534.97 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4710.35 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5678.26 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4583.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7151.60 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 11545.82 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:      stddev:
    3970.79  5923  11545.82  2345.88
    

    VestaCP Nginx LXD Container (vestacp-ubuntu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 3796.12 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 9483.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6836.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3946.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4791.68 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4842.64 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6802.77 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5430.91 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4024.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3796.12  5551  9483.75  1862.98
    

    Webinoly Nginx LXD Container (webinoly)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 4346.54 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3853.59 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3672.07 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4042.92 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3884.68 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3631.22 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3765.88 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3756.65 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3871.60 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3631.22  3869  4346.54  217.19
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load -t2 1000 concurrent user test



    Nginx is multi-threaded and up to now all above tests have only used 1 thread for h2load tests (-t1 option). As the test server is a 4 cpu KVM VPS, this next test is changing h2load to -t2 to test with 2 threads and the same previous 1000 users with 100,000 requests to see how respective LEMP stack's Nginx fairs for HTTP/2 HTTPS performance.

    h2load test parameters



    The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -t2, -c1000 and -n100000 values were chosen
    • h2load -t2 = 2 thread
    • --ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
    • Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
    • -c1000 = 1000 concurrent users
    • -n100000 = 100000 requests
    h2load test command used:
    Code (Text):
    h2load -t2 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com
    


    h2load HTTP/2 HTTPS test results


    • Full raw h2load benchmark results are posted here.
    • Testing with h2load -t2 for 2 threads and thus testing Nginx with 2 threads improved results for all LEMP stacks' Nginx servers compared to previous above test with h2load -t1 - 1 thread.
    • Like the non-HTTPS results, Centmin Mod Nginx came out on top for Nginx HTTP/2 HTTPS performance for both average requests/s and max requests/s. Interestingly, Webinoly Nginx for Nginx HTTP/2 HTTPS performance came in last place unlike Webinoly Nginx's non-HTTPS results which had them coming in 2nd place but testing with 2 threads has improved Webinoly results slightly.
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 11.8% faster than OneInStack OpenResty Nginx, 21.89% faster than OneInStack Standard Nginx, 53.49% faster than EasyEngine Nginx, 60.8% faster than VestaCP Nginx and 212.99% faster than Webinoly Nginx
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 16.45% faster than OneInStack OpenResty Nginx, 25.7% faster than OneInStack Standard Nginx, 27.49% faster than EasyEngine Nginx, 54.49% faster than VestaCP Nginx and 238.45% faster than Webinoly Nginx
    • Of note, for minimum requests/s results, it's first time I've seen a Nginx version other than Centmin Mod Nginx lead with OneInStack OpenResty Nginx 1.13.6 achieving 7,646.22 requests/s landing in 1st place - 18.6% ahead of Centmin Mod Nginx 1.15.0 with 6,445.9 requests/s landing in 2nd place. If you look carefully, OneInStack OpenResty Nginx and Centmin Mod Nginx's minimum requests/s is actually higher than the average requests/s for all other LEMP stacks' Nginx servers and higher than Webinoly Nginx's max requests/s !

    h2load-t2-c1000-n100000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

    Centmin Mod Nginx LXD container (centos75-2)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 11270.51 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 9691.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6445.90 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 11344.67 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 10306.99 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 12873.71 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 7226.89 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 11773.58 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 7781.15 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:    avg:  max:      stddev:
    6445.9  9857  12873.71  2238.22
    


    EasyEngine LXD Container (easyengine-ubtunu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 10097.55 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4660.85 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5364.29 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5935.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 8832.90 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4365.01 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6235.25 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6142.11 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6169.23 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:      stddev:
    4365.01  6422  10097.55  1877.09
    


    OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 6652.98 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 9498.91 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7808.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6401.80 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8442.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8136.59 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 10240.79 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 9430.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6166.96 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:      stddev:
    6166.96  8087  10240.79  1467.41
    


    OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 9739.53 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8330.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 11054.67 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8411.56 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 9096.40 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7993.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7646.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8699.65 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8352.57 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:      stddev:
    7646.22  8814  11054.67  1037.75
    


    VestaCP Nginx LXD Container (vestacp-ubuntu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 7315.23 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5902.55 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5435.50 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 7775.14 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 8333.15 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4364.15 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5321.69 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4714.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6004.07 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    4364.15  6130  8333.15  1382.46
    


    Webinoly Nginx LXD Container (webinoly)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 4380.12 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3862.08 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4338.28 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4761.52 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5398.63 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4245.30 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4930.35 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4919.26 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4815.79 gzip ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    
    h2load result summary
    min:     avg:  max:     stddev:
    3862.08  4628  5398.63  461.21
    
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load -t1 vs -t2 1k concurrent users



    Combining results for last 2 runs for h2load -t1 vs h2load -t2 for 1,000 concurrent user tests. You can see which Nginx versions are better optimised for multi-threaded Nginx work loads by comparing their respective h2load -t1 vs -t2 results and seeing which -t2 results continue to scale in terms of performance.

    LEMP stack installers installed in Ubuntu 18.04 LTS LXD containers on a ssdnode's 4 CPU, 16GB ram, 80GB disk KVM VPS with Ubuntu 18.04 LTS and their respective performance scaling with from -t1 to -t2 (1 thread to 2 threads)
    • Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5) = 39.1% increase in average requests/s and 29.39% increase in min requests/s and 7.03% increase in max requests/s
    • Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6) = 19.5% increase in average requests/s and 22.2% increase in min requests/s and 28.7% increase in max requests/s
    • OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6) = 45.55% increase in average requests/s and 61.19% increase in min requests/s and 3.5% increase in max requests/s
    • OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6) = 48.8% increase in average requests/s and 92.55% increase in min requests/s and 4.25% decrease in max requests/s
    • VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9) = 10.4% increase in average requests/s and 14.96% increase in min requests/s and 12.13% decrease in max requests/s
    • Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6) = 19.6% increase in average requests/s and 6.36% increase in min requests/s and 24.2% increase in max requests/s
    Observations
    • For average requests/s, Centmin Mod Nginx's 1 thread results are actually faster than EasyEngine (+10.33%), VestaCP (+15.6%) and Webinoly's (+53.11%) 2 thread results and within 80-87% of OneInStack's 2 thread results !
    • For minimum requests/s, Centmin Mod Nginx's 2 thread results are actually faster than the average requests/s for EasyEngine (+0.37%), VestaCP (+5.15%) and Webinoly (+39.28) !

    h2load-t1vs2-c1000-n100000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png
     
    • Like Like x 1
  7. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Nginx Idle Resource Usage



    For those curious about respective Nginx server resource usage, I will be looking at comparison stats at idle and load. But for now here's the base line idle Nginx resource usage from pidstats and smem output on each respective LEMP stack.

    For those unfamiliar with Nginx processes and outline is at NGINX Docs | Controlling NGINX Processes at Runtime
    Notes:
    • FYI, Centmin Mod 123.09beta01 Nginx (centos75-2 LXD container) doesn't allocate all worker_processes to all cpus but auto detects number of cpus available and only allocates ~1/2 the number to worker_processes (4/2 = 2 cpus allocated) leaving some cpus to better balanced sharing with php-fpm, MariaDB MySQL, memcached and other processes. So that reflects in below pidstats and number of nginx processes reported when compared with other LEMP stacks' Nginx output.
    • Looks like out of the box EasyEngine, VestaCP and Webinoly do enable caching configuration directives as the nginx cache manager, cache loader processes are showing
    • Also VestaCP seems to have 2x Nginx instances running - one for VestaCP GUI and one for main Nginx
    • RSS/VSZ memory size also reflects number of Nginx modules/features built in and default settings configurations Nginx out of the box as you can see from Nginx -V outputs further below. Centmin Mod LEMP stack initial install auto detects server resources including memory available and auto optimises Nginx, PHP-FPM and MariaDB configuration settings to optimally use the available memory, so it's expected memory usage would be higher.
    • Note smem output totals include python smem process memory usage too.

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check centos75-2
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (centos75-2)    06/15/18        _x86_64_        (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
     1529033515     0       537    0.00    0.00    0.00    0.00     2      3.43      0.00  125024  26240   0.16  nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
     1529033515  1000       538    0.00    0.05    0.00    0.05     2    261.29      0.00  153696  48980   0.30  nginx: worker process                                     
     1529033515  1000       539    0.00    0.10    0.00    0.10     2    261.19      0.00  153696  48980   0.30  nginx: worker process                                     
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
      816 root     python /usr/bin/smem -tk -P        0     6.4M     6.5M     8.3M 
      537 root     nginx: master process /usr/        0   200.0K     8.4M    25.6M 
      539 nginx    nginx: worker process              0    21.2M    29.8M    48.2M 
      538 nginx    nginx: worker process              0    21.2M    29.9M    48.3M 
    -------------------------------------------------------------------------------
        4 2                                           0    49.1M    74.6M   130.4M 
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check easyengine-ubuntu1604
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (easyengine-ubtunu1604)         06/15/18        _x86_64_        (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
     1529033516     0       432    0.00    0.00    0.00    0.00     0      5.06      0.00  213032    4340   0.03  nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
     1529033516    33       445    0.00    0.00    0.00    0.00     0     32.38      0.00  214824    6464   0.04  nginx: worker process                           
     1529033516    33       446    0.00    0.00    0.00    0.00     3     32.38      0.00  214824    6464   0.04  nginx: worker process                           
     1529033516    33       447    0.00    0.00    0.00    0.00     1     32.38      0.00  214824    6464   0.04  nginx: worker process                           
     1529033516    33       448    0.00    0.00    0.00    0.00     0     32.38      0.00  214824    6464   0.04  nginx: worker process                           
     1529033516    33       449    0.00    0.00    0.00    0.00     0     11.19      0.00  213240    5392   0.03  nginx: cache manager process                     
     1529033516    33       450    0.00    0.00    0.00    0.00     0     10.69      0.00  213240    5392   0.03  nginx: cache loader process                     
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
      584 root     sh -c smem -tk -P 'nginx:'         0    88.0K   208.0K     1.6M 
      432 root     nginx: master process /usr/        0   188.0K   607.0K     4.2M 
      450 www-data nginx: cache loader process        0   476.0K     1.0M     5.4M 
      449 www-data nginx: cache manager proces        0   636.0K     1.2M     5.8M 
      445 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
      446 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
      447 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
      448 www-data nginx: worker process              0     1.8M     2.3M     6.7M 
      585 root     python /usr/bin/smem -tk -P        0     7.6M     7.7M     9.6M 
    -------------------------------------------------------------------------------
        9 2                                           0    16.3M    19.9M    53.2M 
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check oneinstack-ubuntu16-nginx
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (oneinstack-ubuntu16-nginx)     06/15/18        _x86_64_        (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
     1529033517     0       422    0.00    0.00    0.00    0.00     3      8.15      0.00   51688    3996   0.02  nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
     1529033517  1002       423    0.00    0.08    0.00    0.08     3    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
     1529033517  1002       424    0.00    0.08    0.00    0.08     3    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
     1529033517  1002       425    0.00    0.08    0.00    0.08     1    416.08      0.00   82408   25564   0.16  nginx: worker process                                           
     1529033517  1002       426    0.00    0.08    0.00    0.08     3    415.54      0.00   82408   25564   0.16  nginx: worker process                                           
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
      906 root     sh -c smem -tk -P 'nginx:'         0   104.0K   170.0K     1.6M 
      422 root     nginx: master process /usr/        0   208.0K   777.0K     3.9M 
      907 root     /usr/bin/python /usr/bin/sm        0     7.0M     7.1M     8.9M 
      426 www      nginx: worker process              0    20.9M    21.5M    25.3M 
      423 www      nginx: worker process              0    21.0M    21.6M    25.5M 
      424 www      nginx: worker process              0    21.0M    21.6M    25.5M 
      425 www      nginx: worker process              0    21.0M    21.6M    25.5M 
    -------------------------------------------------------------------------------
        7 2                                           0    91.1M    94.2M   116.3M 
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check oneinstack-ubuntu16-openresty
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (oneinstack-ubuntu16-openresty)         06/15/18        _x86_64_        (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
     1529033518     0       447    0.00    0.00    0.00    0.00     2      9.33      0.00   55200    4492   0.03  nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
     1529033518  1002       448    0.00    0.08    0.00    0.08     0    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
     1529033518  1002       449    0.00    0.08    0.00    0.08     1    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
     1529033518  1002       450    0.00    0.08    0.00    0.08     3    451.58      0.00   85920   26060   0.16  nginx: worker process                                                               
     1529033518  1002       451    0.00    0.08    0.00    0.08     0    451.08      0.00   85920   26060   0.16  nginx: worker process                                                               
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
      909 root     sh -c smem -tk -P 'nginx:'         0    84.0K   162.0K     1.7M 
      447 root     nginx: master process /usr/        0   224.0K   861.0K     4.4M 
      910 root     /usr/bin/python /usr/bin/sm        0     7.0M     7.2M     8.9M 
      451 www      nginx: worker process              0    21.0M    21.7M    25.9M 
      448 www      nginx: worker process              0    21.0M    21.7M    26.1M 
      449 www      nginx: worker process              0    21.0M    21.7M    26.1M 
      450 www      nginx: worker process              0    21.0M    21.7M    26.1M 
    -------------------------------------------------------------------------------
        7 2                                           0    91.2M    94.9M   119.3M 
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check vestacp-ubuntu1604
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (vestacp-ubuntu1604.example.com)        06/15/18        _x86_64_        (4 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
     1529033518     0       647    0.00    0.00    0.00    0.00     0      4.73      0.00   54040    1400   0.01  nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
     1529033518    33       648    0.00    0.00    0.00    0.00     0     16.27      0.00   54044    3164   0.02  nginx: worker process                   
     1529033518    33       649    0.00    0.00    0.00    0.00     1     16.27      0.00   54044    3164   0.02  nginx: worker process                   
     1529033518    33       650    0.00    0.00    0.00    0.00     2     16.27      0.00   54044    3164   0.02  nginx: worker process                   
     1529033518    33       651    0.00    0.00    0.00    0.00     3     16.27      0.00   54044    3164   0.02  nginx: worker process                   
     1529033518    33       652    0.00    0.00    0.00    0.00     0     11.64      0.00   54044    3288   0.02  nginx: cache manager process             
     1529033518    33       653    0.00    0.00    0.00    0.00     2     11.18      0.00   54044    3288   0.02  nginx: cache loader process             
     1529033518     0       667    0.00    0.00    0.00    0.00     2      4.18      0.00   42192     828   0.01  nginx: master process /usr/local/vesta/nginx/sbin/vesta-nginx
     1529033518  1001       669    0.00    0.00    0.00    0.00     2      8.09      0.00   42192    2780   0.02  nginx: worker process                   
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
     1122 root     sh -c smem -tk -P 'nginx:'         0    92.0K   174.0K     1.6M 
      647 root     nginx: master process /usr/        0   148.0K   358.0K     2.2M 
      653 www-data nginx: cache loader process        0   368.0K   617.0K     3.3M 
      667 root     nginx: master process /usr/        0   148.0K   718.0K     2.2M 
      652 www-data nginx: cache manager proces        0   512.0K   758.0K     3.4M 
      648 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
      649 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
      650 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
      651 www-data nginx: worker process              0   592.0K   839.0K     3.5M 
      669 admin    nginx: worker process              0   256.0K   846.0K     3.0M 
     1123 root     python /usr/bin/smem -tk -P        0     6.9M     7.0M     8.8M 
    -------------------------------------------------------------------------------
       11 3                                           0    10.7M    13.6M    38.5M 
    

    Code (Text):
    -------------------------------------------------------------------------------------------
    stats check webinoly
    
    pidstat -urhl -C nginx | grep -v 'pidstat'
    Linux 4.15.0-22-generic (webinoly)      06/15/18        _x86_64_        (4 CPU)
    
    # Time        UID       PID    %usr %system  %guest   %wait    %CPU   CPU  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
    03:31:58        0       259    0.00    0.00    0.00    0.00    0.00     0      4.78      0.00  115544    1164   0.01  nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
    03:31:58       33       260    0.00    0.00    0.00    0.11    0.00     1    107.33      0.00  123392    7012   0.04  nginx: worker process                           
    03:31:58       33       261    0.00    0.00    0.00    0.22    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
    03:31:58       33       262    0.00    0.00    0.00    0.44    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
    03:31:58       33       263    0.00    0.00    0.00    0.33    0.00     0    107.33      0.00  123392    7012   0.04  nginx: worker process                           
    03:31:58       33       264    0.00    0.00    0.00    0.11    0.00     0     17.89      0.00  119924    4080   0.02  nginx: cache manager process                     
    03:31:58       33       265    0.00    0.00    0.00    0.11    0.00     0     17.44      0.00  119924    3676   0.02  nginx: cache loader process                     
    
    smem -tk -P 'nginx:'
      PID User     Command                         Swap      USS      PSS      RSS 
      696 root     sh -c smem -tk -P 'nginx:'         0   200.0K   247.0K     1.6M 
      259 root     nginx: master process /usr/        0   180.0K   364.0K     1.9M 
      265 www-data nginx: cache loader process        0   424.0K   700.0K     3.9M 
      264 www-data nginx: cache manager proces        0   568.0K   844.0K     4.0M 
      260 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
      261 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
      262 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
      263 www-data nginx: worker process              0     3.6M     3.8M     7.0M 
      697 root     python /usr/bin/smem -tk -P        0     7.1M     7.3M     9.6M 
    -------------------------------------------------------------------------------
        9 2                                           0    22.8M    24.8M    49.0M 
    


    Centmin Mod 123.09beta01 Nginx



    Nginx version

    Easyengine 3.8.1 Nginx



    Nginx version

    VestaCP 0.9.8-21 Nginx



    Nginx version

    OneInStack v1.7 Standard Nginx



    Nginx version

    OneInStack v1.7 OpenResty Nginx



    Nginx version

    Webinoly v1.4.3 Nginx



    Nginx version
     
  8. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    Nginx Amplify For LXD Container Based LEMP Stacks



    Next up is install Nginx Amplify monitoring for all LXD container based LEMP stacks :D

    Of course you need working /nginx_status urls for Amplify to gather stats. I wrote a custom lxc.sh script to be able to query all LXD containers via one command :)

    Code (Text):
    ./lxc.sh query curl -s localhost/nginx_status
    
    --------------------------------------------------------------
    centos75-2
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    --------------------------------------------------------------
    easyengine-ubuntu1604
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    --------------------------------------------------------------
    oneinstack-ubuntu16-nginx
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    --------------------------------------------------------------
    oneinstack-ubuntu16-openresty
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    --------------------------------------------------------------
    vestacp-ubuntu1604
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    --------------------------------------------------------------
    webinoly
    
    curl -s localhost/nginx_status      
    
    Active connections: 1
    server accepts handled requests
     1 1 1
    Reading: 0 Writing: 1 Waiting: 0
    --------------------------------------------------------------
    


    nginx-amplify-lxd-01.png

    Custom Nginx Amplify dashboard showing all LEMP's Nginx stats on one page.

    nginx-amplify-lxd-lempstack-compared-01.png
     
  9. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    VestaCP & Webinoly Updates



    Some LEMP stacks got version updated

    Webinoly v1.5.0 Nginx HTTPS settings


    Code (Text):
    egrep 'ssl_session_cache|ssl_session_timeout|ssl_prefer_server_ciphers|ssl_ciphers|ssl_protocols|ssl_dhparam' /etc/nginx/nginx.conf
            ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
            ssl_session_timeout 10m;
            ssl_session_cache shared:SSL:20m;
            ssl_dhparam /etc/ssl/dhparam.pem;
            ssl_prefer_server_ciphers on;
            ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACH
    

    Code (Text):
    h2load -t1 -c1 -n1 https://http2.domain.com/ | egrep 'TLS Protocol:|Cipher:|Server Temp Key:|Application protocol:'
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    

    Nginx TLS Protocol / SSL Cipher Summary



    Updated summary of all LEMP stacks' Nginx default preferred TLS protocol and SSL ciphers for h2load HTTP/2 HTTPS client are in below table when all Nginx configurations have dual RSA 2048bit + ECDSA 256bit SSL certification configurations setup manually.

    Nginx OpenSSL Protocol SSL Cipher Server Temp Key App Protocol
    Centmin Mod Nginx 1.15.0 OpenSSL 1.1.0h TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2
    EasyEngine Nginx 1.14.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-ECDSA-AES128-GCM-SHA256 ECDH P-256 256 bits h2
    OneInStack Std Nginx 1.14.0 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    OneInStack OpenResty Nginx 1.13.6 OpenSSL 1.0.2o TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    VestaCP Nginx 1.15.0 OpenSSL 1.0.2g TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
    Webinoly Nginx 1.14.0 OpenSSL 1.1.0g TLSv1.2 ECDHE-ECDSA-AES256-GCM-SHA384 ECDH P-256 256 bits h2
     
  10. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    h2load -t2 1000 concurrent user test



    With VestaCP 0.9.8-22 security update and Webinoly 1.5.0 updated to enable ECDSA ssl ciphers for Nginx HTTP/2 HTTPS, decided to revisit Nginx HTTP/2 HTTPS static HTML file tests at h2load -t2 to test with 2 threads and the same previous 1000 users with 100,000 requests to see how respective LEMP stack's Nginx fairs for HTTP/2 HTTPS performance.

    Note: As the underlying server is a KVM VPS, we are subject to the KVM host node's underlying resource usage loads shared with neighbouring KVM VPS customers so results may vary from previously tested. However, relative results this re-run can be compared across all LEMP stack's Nginx servers.

    h2load test parameters



    The h2load tests were run 9x times to derive the below results. The following h2load test parameters were used where -t2, -c1000 and -n100000 values were chosen
    • h2load -t2 = 2 thread
    • --ciphers = the list is the default h2load client preferred ciphers specifically mentioned. If you ran h2load without --ciphers, these would be same defaults used.
    • Accept-Encoding: gzip = test compressed HTTP requests like a web browser would
    • -c1000 = 1000 concurrent users
    • -n100000 = 100000 requests
    h2load test command used:
    Code (Text):
    h2load -t2 --ciphers=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -H 'Accept-Encoding: gzip' -c600 -n60000 https://http2.domain.com/index.html
    


    h2load HTTP/2 HTTPS test results


    • Again Centmin Mod Nginx came out on top and Webinoly still came last. However, EasyEngine came in 2nd place taking the position from OneInStack OpenResty Nginx
    • Centmin Mod Nginx's minimum requests/s was higher than other LEMP stack's Nginx average requests/s !
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for average requests/s, Centmin Mod Nginx was 38.5% faster than EasyEngine, 54.88% faster than OneInStack OpenResty Nginx, 58.68% faster than VestaCP Nginx, 71.65% faster OneInStack Standard Nginx, and 93.36% faster than Webinoly Nginx
    • On an average of 9x h2load HTTP/2 HTTPS benchmark runs for max requests/s, Centmin Mod Nginx was 9.4% faster than EasyEngine, 15.6% faster than OneInStack OpenResty Nginx, 16.048% faster than VestaCP Nginx, 7.3% faster OneInStack Standard Nginx, and 45.03% faster than Webinoly Nginx
    h2load-t2-c1000-n100000-lxd-centminmod-vs-easyengine-vs-webinoly-vs-vestacp-vs-oneinstack-01.png

    Centmin Mod Nginx LXD container (centos75-2)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 6357.63 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6808.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6647.53 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6639.08 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6256.79 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6126.83 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6800.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5500.96 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6512.54 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:  requests-succeeded:
    5500.960  6405.598  6808.970  412.812  100.00
    


    EasyEngine LXD Container (easyengine-ubtunu1604)
    Code (Text):
    100 1000 4228.64 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3517.66 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4740.30 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 3832.97 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 6222.66 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 5790.96 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4284.05 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4410.17 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    100 1000 4606.21 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:  requests-succeeded:
    3517.660  4625.958  6222.660  873.115  100.00
    


    OneInStack Standard Nginx LXD Container (oneinstack-ubuntu16-nginx)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 2328.04 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2249.52 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4003.75 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4437.06 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4201.71 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2889.85 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 6345.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3095.22 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4038.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:   requests-succeeded:
    2249.520  3732.094  6345.610  1274.745  100.00
    


    OneInStack OpenResty Nginx LXD Container (oneinstack-ubuntu16-openresty)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 3603.37 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2576.92 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5573.48 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3310.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3641.25 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4809.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5887.63 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3964.42 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3853.73 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:   requests-succeeded:
    2576.920  4135.658  5887.630  1080.359  100.00
    


    VestaCP Nginx LXD Container (vestacp-ubuntu1604)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 3060.54 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4084.74 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2745.39 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3662.30 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4530.20 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4932.57 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3675.61 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 5868.09 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3770.66 gzip ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:  requests-succeeded:
    2745.390  4036.678  5868.090  959.435  100.00
    


    Webinoly Nginx LXD Container (webinoly)
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    100 1000 3142.85 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3035.93 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3114.39 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2728.16 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3032.64 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4013.47 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 4695.01 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 2544.65 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    100 1000 3510.57 gzip ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 100000 100000
    h2load result summary
    min:      avg:      max:      stddev:  requests-succeeded:
    2544.650  3313.074  4695.010  670.430  100.00
    
     
  11. eva2000

    eva2000 Administrator Staff Member

    36,884
    8,074
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,435
    Local Time:
    9:36 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x

    nghttp HTTP/2 Statistics



    A much simpler test is nghttp HTTP/2 statistics test which reports the request start and end times and actual processing times for all HTTPS requests as defined below where lower the request times and processing times = faster it is
    Code (Text):
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model


    Results ordered as follows for:
    • Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)
    • Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)
    • VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)
    • Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)

    nghttp-stats-020718.png

    Centmin Mod 123.09beta01 beta Nginx 1.15.0 on CentOS 7.5 64bit (default gzip compression = 5)
    Code (Text):
    centos75-2
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13     +4.20ms       +144us   4.06ms  200   2K /index.html
     15     +6.39ms      +4.24ms   2.15ms  200   4K /purecss/pure-min.css
     17     +7.74ms      +4.24ms   3.50ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    

    Easyengine 3.8.1 using Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    Code (Text):
    easyengine-ubuntu1604
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13    +20.80ms       +295us  20.50ms  200   2K /index.html
     15    +31.03ms     +20.86ms  10.18ms  200   3K /purecss/pure-min.css
     17    +31.60ms     +20.86ms  10.74ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    

    OneInStack Nginx 1.14.0 on Ubuntu 16.04 LTS (default gzip compression = 6)
    Code (Text):
    oneinstack-ubuntu16-nginx
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13    +17.78ms       +207us  17.58ms  200   2K /index.html
     15    +32.24ms     +17.87ms  14.37ms  200   3K /purecss/pure-min.css
     17    +33.52ms     +17.87ms  15.65ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    

    OneInStack OpenResty Nginx 1.13.6 on Ubuntu 16.04 LTS (default gzip compression = 6)
    Code (Text):
    oneinstack-ubuntu16-openresty
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13    +21.69ms       +769us  20.92ms  200   2K /index.html
     15    +31.86ms     +21.77ms  10.09ms  200   3K /purecss/pure-min.css
     17    +32.57ms     +21.77ms  10.80ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    

    VestaCP 0.9.8-21 using Nginx 1.15.0 on Ubuntu 16.04 LTS (default gzip compression = 9)
    Code (Text):
    vestacp-ubuntu1604
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13    +18.75ms       +172us  18.58ms  200   2K /index.html
     15    +29.21ms     +18.82ms  10.38ms  200   3K /purecss/pure-min.css
     17    +30.08ms     +18.83ms  11.25ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    

    Webinoly 1.4.3 using Nginx 1.14.0 on Ubuntu 18.04 LTS (default gzip compression = 6)
    Code (Text):
    webinoly
    
    nghttp -ynas https://http2.domain.com/index.html      
    
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
     requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
     13    +18.29ms       +194us  18.10ms  200   2K /index.html
     15    +28.29ms     +18.35ms   9.94ms  200   3K /purecss/pure-min.css
     17    +29.17ms     +18.36ms  10.82ms  200   1K /purecss/grids-responsive-min.css
    --------------------------------------------------------------
    
    
     
..
Thread Status:
Not open for further replies.