Welcome to Centmin Mod Community
Become a Member

SSL Caddy HTTP/2 server & benchmarks - Part 2

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, Sep 16, 2017.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    This is Part 2 continuation from Caddy 0.10.9 benchmarks here.

    h2load Load Testing



    nghttp2's h2load load tester is a HTTP/2 based HTTPS load testing tool which fully supports HTTP/2 unlike other benchmarking tools (siege, apache bench, wkr, etc). I am using h2load version 1.26.0-dev via my Ubuntu Docker image using native networking to remove the Docker networking overhead. Only testing against source built Caddy 0.10.9 binary as the new Caddy EULA license agreement prohibits sharing benchmark results for official Caddy provided binaries https://caddy.community/t/caddy-eula-section-3-1-h-benchmarking-info-clarification/2727/.

    h2load test server:

    The h2load test server is a dedicated server from OVH BHS, Canada
    • OVH BHS Dedicated
    • Intel i7 4790K @4.0-4.40Ghz
    • 32GB RAM
    • 2x240GB SSD
    • CentOS 7.4 64bit
    • Centmin Mod 123.09beta01
    Test server for Nginx/Caddy:

    The Nginx/Caddy test server is a 2 cpu core OpenVZ VPS server from HostUS.us. So the h2load tests are testing how well Centmin Mod Nginx and Caddy handle load on a low end OpenVZ VPS server.

    • HostUS.us Washington, DC
    • OpenVZ 2 CPU Core @Intel Xeon E5-1650v3 @3.5-3.8Ghz
    • 2GB RAM
    • 50GB SSD
    • CentOS 7.4
    • CentOS 7.4 64bit
    • Centmin Mod 123.09beta01
    Code (Text):
    h2load --version
    h2load nghttp2/1.26.0-DEV
    


    Configuration



    I setup Centmin Mod 123.09beta01 Nginx 1.13.5 HTTP/2 vhost using Letsencrypt integration and defaults to RSA 2048bit SSL certs to match Caddy's default RSA 2048bit SSL certificate issuance and to match the negotiated SSL cipher. Check negotiated SSL ciphers for h2load tests with low load 2 thread, 2 concurrent user and 2 request test. Ensuring both Centmin Mod Nginx HTTP/2 HTTPS and Caddy HTTP/2 HTTPS negotiate using same SSL ciphers = ECDHE-RSA-AES256-GCM-SHA384 and test over HTTP/2 via = Application protocol: h2

    Vhost Config Files

    For Centmin Mod Nginx 1.13.5
    Code (Text):
    curl -I -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    

    Code (Text):
    curl -I -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    HTTP/2 200
    date: Sat, 16 Sep 2017 11:19:25 GMT
    content-type: text/html; charset=utf-8
    last-modified: Sat, 16 Sep 2017 09:26:40 GMT
    vary: Accept-Encoding
    etag: W/"59bcee50-17cf"
    server: nginx centminmod
    x-powered-by: centminmod
    x-frame-options: SAMEORIGIN
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    content-encoding: gzip
    

    Code (Text):
    /usr/local/bin/h2load -t2 -c2 -m100 -n2 -v https://nginxbench.domain.com:443
    

    Code (Text):
    /usr/local/bin/h2load -t2 -c2 -m100 -n2 -v https://nginxbench.domain.com:443
    starting benchmark...
    spawning thread #0: 1 total client(s). 1 total requests
    spawning thread #1: 1 total client(s). 1 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 100% done
    
    finished in 134.55ms, 14.86 req/s, 92.83KB/s
    requests: 2 total, 2 started, 2 done, 2 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 2 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 12.49KB (12790) total, 466B (466) headers (space savings 27.64%), 11.90KB (12190) data
                         min         max         mean         sd        +/- sd
    time for request:    31.95ms     32.59ms     32.27ms       452us   100.00%
    time for connect:    98.22ms    101.60ms     99.91ms      2.39ms   100.00%
    time to 1st byte:   129.85ms    133.85ms    131.85ms      2.83ms   100.00%
    req/s           :       7.45        7.68        7.57        0.16   100.00%
    


    For Caddy 0.10.9

    Code (Text):
    curl -I -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443
    

    Code (Text):
    curl -I -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443
    HTTP/2 200
    accept-ranges: bytes
    content-encoding: gzip
    content-type: text/html; charset=utf-8
    etag: W/"owd8wg4pb"
    last-modified: Sat, 16 Sep 2017 09:26:40 GMT
    server: Caddy
    vary: Accept-Encoding
    x-content-type-options: nosniff
    x-frame-options: SAMEORIGIN
    x-powered-by: Caddy via CentminMod
    x-xss-protection: 1; mode=block
    content-length: 23
    date: Sat, 16 Sep 2017 11:21:06 GMT
    

    Code (Text):
    /usr/local/bin/h2load -t2 -c2 -m100 -n2 -v https://caddybench.domain.com:443
    

    Code (Text):
    /usr/local/bin/h2load -t2 -c2 -m100 -n2 -v https://caddybench.domain.com:443
    starting benchmark...
    spawning thread #0: 1 total client(s). 1 total requests
    spawning thread #1: 1 total client(s). 1 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 100% done
    
    finished in 183.77ms, 10.88 req/s, 67.11KB/s
    requests: 2 total, 2 started, 2 done, 2 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 2 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 12.33KB (12628) total, 274B (274) headers (space savings 43.15%), 11.90KB (12190) data
                         min         max         mean         sd        +/- sd
    time for request:    35.46ms     63.10ms     49.28ms     19.54ms   100.00%
    time for connect:   120.24ms    123.52ms    121.88ms      2.32ms   100.00%
    time to 1st byte:   158.77ms    183.35ms    171.06ms     17.38ms   100.00%
    req/s           :       5.45        6.29        5.87        0.59   100.00%
    
     
    Last edited: Sep 17, 2017
  2. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    h2load Load Testing Results



    Now for HTTP/2 based load testing via nghttp2's h2load load testing tool.

    Did 4 sets of h2load tests are different levels of load all with h2load passing Accept-Encoding: gzip header for gzip compressed requests for:
    1. 2 threads, 500 concurrent users, 50,000 requests with max concurrent HTTP/2 streams set at 100
    2. 2 threads, 400 concurrent users, 40,000 requests with max concurrent HTTP/2 streams set at 100
    3. 2 threads, 300 concurrent users, 30,000 requests with max concurrent HTTP/2 streams set at 100
    4. 2 threads, 200 concurrent users, 10,000 requests with max concurrent HTTP/2 streams set at 100
    caddy-vs-nginx-h2load-http2-loadtests-160917-01b.png

    Centmin Mod Nginx 1.13.5 HTTP/2



    7x runs of h2load test with 2 threads, 500 concurrent users, 50,000 requests with max concurrent HTTP/2 streams set at 100 with average of 7412.84 reques/sec with 100% completion rate

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test
    8001.94 req/s 100% completed
    6268.60 req/s 100% completed
    7545.21 req/s 100% completed
    8088.38 req/s 100% completed
    7660.50 req/s 100% completed
    5787.04 req/s 100% completed
    8538.20 req/s 100% completed
    

    Caddy 0.10.9 Source Built HTTP/2



    h2load test with 2 threads, 500 concurrent users, 50,000 requests with max concurrent HTTP/2 streams set at 100 with average of 2,793.13 reques/sec with 87.36% completion rate.

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    3510.35 req/s 95.6% completed
    3407.05 req/s 89.8% completed
    2456.65 req/s 82.496% completed
    2191.76 req/s 75.6% completed
    2367.59 req/s 95% completed
    2540.23 req/s 90.2% completed
    3078.25 req/s 82.8% completed
    


    Centmin Mod Nginx 1.13.5 HTTP/2



    7x runs of h2load test with 2 threads, 400 concurrent users, 40,000 requests with max concurrent HTTP/2 streams set at 100 with average of 7,315.63 reques/sec with 100% completion rate

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c400 -m100 -n40000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c400 -m100 -n40000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test
    7188.68 req/s 100% completed
    7904.49 req/s 100% completed
    6016.62 req/s 100% completed
    6876.74 req/s 100% completed
    7905.02 req/s 100% completed
    7336.11 req/s 100% completed
    7981.76 req/s 100% completed
    

    Caddy 0.10.9 Source Built HTTP/2



    h2load test with 2 threads, 400 concurrent users, 40,000 requests with max concurrent HTTP/2 streams set at 100 with average of 2,867.45 reques/sec with 91.84% completion rate.

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c400 -m100 -n40000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c400 -m100 -n40000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    3257.08 req/s 91% completed
    2831.80 req/s 91.5% completed
    2176.95 req/s 89.25% completed
    3172.83 req/s 98% completed
    3214.47 req/s 97.5% completed
    2730.83 req/s 79.61% completed
    2688.22 req/s 96% completed
    


    Centmin Mod Nginx 1.13.5 HTTP/2



    7x runs of h2load test with 2 threads, 300 concurrent users, 30,000 requests with max concurrent HTTP/2 streams set at 100 with average of 7,174.14 reques/sec with 100% completion rate

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c300 -m100 -n30000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c300 -m100 -n30000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test
    7651.89 req/s 100% completed
    7913.35 req/s 100% completed
    5572.42 req/s 100% completed
    6957.35 req/s 100% completed
    7260.92 req/s 100% completed
    7536.88 req/s 100% completed
    7326.19 req/s 100% completed
    

    Caddy 0.10.9 Source Built HTTP/2



    h2load test with 2 threads, 300 concurrent users, 30,000 requests with max concurrent HTTP/2 streams set at 100 with average of 3,335.63 reques/sec with 92.61% completion rate.

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c300 -m100 -n30000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c300 -m100 -n30000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    3266.61 req/s 90.3333% completed
    3300.61 req/s 97.3333% completed
    3564.28 req/s 96% completed
    3327.11 req/s 93.6667% completed
    3545.27 req/s 85.62% completed
    3328.88 req/s 92% completed
    3016.64 req/s 93.3333% completed
    


    Centmin Mod Nginx 1.13.5 HTTP/2



    7x runs of h2load test with 2 threads, 200 concurrent users, 10,000 requests with max concurrent HTTP/2 streams set at 100 with average of 4,604.68 reques/sec with 100% completion rate

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test
    4342.71 req/s 100% completed
    4489.34 req/s 100% completed
    4735.29 req/s 100% completed
    4543.58 req/s 100% completed
    4872.51 req/s 100% completed
    4666.24 req/s 100% completed
    4583.11 req/s 100% completed
    


    example log for 5th run
    Code (Text):
    cat h2load.5.nginx.log
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.05s, 4872.51 req/s, 10.22MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 20.97MB (21989800) total, 2.19MB (2300000) headers (space savings 25.32%), 18.60MB (19500000) data
                         min         max         mean         sd        +/- sd
    time for request:    32.02ms    752.52ms    384.73ms    137.39ms    65.07%
    time for connect:    99.23ms       1.79s    812.66ms    623.51ms    37.50%
    time to 1st byte:   131.28ms       1.84s    964.01ms    581.92ms    51.00%
    req/s           :      24.38       79.40       46.59       20.69    53.50%
    


    Caddy 0.10.9 Source Built HTTP/2



    h2load test with 2 threads, 200 concurrent users, 10,000 requests with max concurrent HTTP/2 streams set at 100 with average of 2,786.89 reques/sec with 100% completion rate.

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    2736.89 req/s 100% completed
    2700.29 req/s 100% completed
    2441.27 req/s 100% completed
    2993.02 req/s 100% completed
    3244.75 req/s 100% completed
    2439.18 req/s 100% completed
    2952.80 req/s 100% completed
    


    example log for 5th run
    Code (Text):
    cat h2load.5.caddy.log
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 3.08s, 3244.75 req/s, 6.13MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.89MB (19812113) total, 176.87KB (181113) headers (space savings 94.66%), 18.54MB (19440000) data
                         min         max         mean         sd        +/- sd
    time for request:   125.10ms       2.37s       1.05s    337.91ms    64.39%
    time for connect:   133.30ms       2.69s       1.02s    719.40ms    69.00%
    time to 1st byte:   282.48ms       2.95s       1.74s    795.92ms    63.00%
    req/s           :      16.23       50.30       25.20        9.63    75.50%
    


    Summary



    caddy-vs-nginx-h2load-http2-loadtests-160917-01b.png

    Nginx completed all h2load HTTP/2 HTTPS tests with 100% completion while at higher concurrency and number of requests, it seems Caddy choked with incomplete requests. Could be related to OpenVZ VPS being tested against as you can't do any TCP level tuning like you can on non-OpenVZ VPS/Servers.

    Doing a manual run against Caddy to see what's going on.
    Code (Text):
    /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443
    

    h2load Results
    Code (Text):
    /usr/local/bin/h2load -t2 -c500 -m100 -n50000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443
    starting benchmark...
    spawning thread #0: 250 total client(s). 25000 total requests
    spawning thread #1: 250 total client(s). 25000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    
    finished in 13.97s, 3384.84 req/s, 6.39MB/s
    requests: 50000 total, 50000 started, 47300 done, 47300 succeeded, 2700 failed, 2700 errored, 0 timeout
    status codes: 47300 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 89.26MB (93592801) total, 745.06KB (762944) headers (space savings 95.24%), 87.69MB (91951200) data
                         min         max         mean         sd        +/- sd
    time for request:    73.14ms       8.27s       2.88s       1.93s    64.71%
    time for connect:   138.59ms      13.35s       5.50s       4.12s    56.60%
    time to 1st byte:   327.98ms      13.60s       7.28s       3.76s    60.68%
    req/s           :       0.00       70.44       11.48        7.22    86.60%
    

    From requests line, doesn't seem that any requests timed out though, they just failed.
    Code (Text):
    requests: 50000 total, 50000 started, 47300 done, 47300 succeeded, 2700 failed, 2700 errored, 0 timeout
    
     
    Last edited: Sep 17, 2017
  3. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy & Nginx Vhost Creation



    For creating Caddy vhost config file, used my caddy.sh script which is in development for Caddy integration into Centmin Mod eventually. It runs Caddy on default HTTP port 8888 and HTTPS on port 448 self-signed SSL cert for testing, so need to manually change it to HTTP 80 and HTTPS 443 for testing after Caddy vhost is created and restart Caddy for it to issue a valid Letsencrypt SSL certificate. The caddy.sh script can auto generate both Caddy vhost + Centmin Mod Nginx vhost config files for the same domain - allowing users to switch between Nginx and Caddy web servers. With planned integrations for OpenLiteSpeed, Litespeed, Apache 2.4 and Caddy/H2O web servers with existing Nginx web servers, the end goal is to be able auto generate one domain Vhost with has equivalent vhosts for each respective web server - sharing the same the same web roots. This will allow you to switch between web servers or concurrently run different web servers, once Haproxy load balancer integration is added.

    output snippet from caddy.sh addsite command
    Code (Text):
    ./caddy.sh addsite
    
    Add a new Caddy site domain vhost
    Enter domain name wihtout www.: caddybench.domain.com
    
    Caddy integration into Centmin Mod LEMP runs Nginx on port 80 & 443
    Caddy server runs on 8888 & 448
    Do you want to create the Nginx vhost for caddybench.domain.com as well ? [y/n]: y
    Create self-signed SSL cert for Nginx vhost caddybench.domain.com ? recommended [y/n]: y
    
    -------------------------------------------------------------
    ok: /usr/local/src/centminmod/addons/acmetool.sh
    /usr/local/src/centminmod/addons/acmetool.sh issue caddybench.domain.com
    


    Using Centmin Mod 123.09beta01 which has optional Letsencrypt integration which is disabled by default. To enable Letsencrypt integration, set in /etc/centminmod/custom_config.inc following settings to enable letsencrypt detectioion.
    Code (Text):
    LETSENCRYPT_DETECT='y'
    

    Install or update underlying acme.sh client via addons/acmetool.sh
    Code (Text):
    /usr/local/src/centminmod/addons/acmetool.sh acmeupdate
    

    Then create Centmin Mod Nginx HTTP/2 based HTTPS default site with free Letsencrypt SSL certificate using addons/acmetool.sh via /usr/bin/nv command line setting your own desired Pure-FTPD virtual ftp username i.e. DESIREDFTPUSERNAME
    Code (Text):
    nv
    
    Usage: /bin/nv [-d yourdomain.com] [-s y|n|yd|le|led|lelive|lelived] [-u ftpusername]
    
      -d  yourdomain.com or subdomain.yourdomain.com
      -s  ssl self-signed create = y or n or https only vhost = yd
      -s  le - letsencrypt test cert or led test cert with https default
      -s  lelive - letsencrypt live cert or lelived live cert with https default
      -u  your FTP username
    
      example:
    
      /bin/nv -d yourdomain.com -s y -u ftpusername
      /bin/nv -d yourdomain.com -s n -u ftpusername
      /bin/nv -d yourdomain.com -s yd -u ftpusername
      /bin/nv -d yourdomain.com -s le -u ftpusername
      /bin/nv -d yourdomain.com -s led -u ftpusername
      /bin/nv -d yourdomain.com -s lelive -u ftpusername
      /bin/nv -d yourdomain.com -s lelived -u ftpusername
    

    So SSH command to create a Nginx HTTP/2 HTTPS default vhost site with Letsencrypt SSL certificate is as follows:
    Code (Text):
    nv -d nginxbench.domain.com -s lelived -u DESIREDFTPUSERNAME
    

    output snippet shows all the auto generated config files and letsencrypt SSL issued certificate
    Code (Text):
    -------------------------------------------------------------
    FTP hostname : ipaddress
    FTP port : 21
    FTP mode : FTP (explicit SSL)
    FTP Passive (PASV) : ensure is checked/enabled
    FTP username created for nginxbench.domain.com : DESIREDFTPUSERNAME
    FTP password created for nginxbench.domain.com : ****
    -------------------------------------------------------------
    vhost for nginxbench.domain.com created successfully
    
    domain: http://nginxbench.domain.com
    vhost conf file for nginxbench.domain.com created: /usr/local/nginx/conf/conf.d/nginxbench.domain.com.conf
    
    vhost ssl for nginxbench.domain.com created successfully
    
    domain: https://nginxbench.domain.com
    vhost ssl conf file for nginxbench.domain.com created: /usr/local/nginx/conf/conf.d/nginxbench.domain.com.ssl.conf
    /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.crt.key.conf created
    /usr/local/nginx/conf/ssl_include.conf created
    Self-signed SSL Certificate: /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.crt
    SSL Private Key: /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.key
    SSL CSR File: /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.csr
    Backup SSL Private Key: /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com-backup.key
    Backup SSL CSR File: /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com-backup.csr
    
    upload files to /home/nginx/domains/nginxbench.domain.com/public
    vhost log files directory is /home/nginx/domains/nginxbench.domain.com/log
    
    -------------------------------------------------------------
    Current vhost listing at: /usr/local/nginx/conf/conf.d/
                         
    Sep 12  10:59   1.1K   demodomain.com.conf
    Sep 15  17:53   1.6K   virtual.conf
    Sep 16  09:28   3.4K   nginxbench.domain.com.ssl.conf
    
    -------------------------------------------------------------
    Current vhost ssl files listing at: /usr/local/nginx/conf/ssl/nginxbench.domain.com
    
                         
    Sep 16  09:26   1.7K   nginxbench.domain.com.key
    Sep 16  09:26   997    nginxbench.domain.com.csr
    Sep 16  09:26   1.2K   nginxbench.domain.com.crt
    Sep 16  09:26   1.7K   nginxbench.domain.com-backup.key
    Sep 16  09:26   997    nginxbench.domain.com-backup.csr
    Sep 16  09:27   45     hpkp-info-primary-pin.txt
    Sep 16  09:27   45     hpkp-info-secondary-pin.txt
    Sep 16  09:27   424    dhparam.pem
    Sep 16  09:28   408    nginxbench.domain.com.crt.key.conf
    Sep 16  09:28   408    acme-vhost-config.txt
    Sep 16  09:28   3.2K   nginxbench.domain.com-acme.cer
    Sep 16  09:28   302    nginxbench.domain.com-acme.key
    Sep 16  09:28   3.2K   nginxbench.domain.com-fullchain-acme.key
    
    -------------------------------------------------------------
    Commands to remove nginxbench.domain.com
    
     pure-pw userdel DESIREDFTPUSERNAME
     rm -rf /usr/local/nginx/conf/conf.d/nginxbench.domain.com.conf
     rm -rf /usr/local/nginx/conf/conf.d/nginxbench.domain.com.ssl.conf
     rm -rf /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.crt
     rm -rf /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.key
     rm -rf /usr/local/nginx/conf/ssl/nginxbench.domain.com/nginxbench.domain.com.csr
     rm -rf /usr/local/nginx/conf/ssl/nginxbench.domain.com
     rm -rf /home/nginx/domains/nginxbench.domain.com
     rm -rf /root/.acme.sh/nginxbench.domain.com
     service nginx restart
    
    -------------------------------------------------------------
    vhost for nginxbench.domain.com setup successfully
    nginxbench.domain.com setup info log saved at:
    /root/centminlogs/centminmod_160917-092640_nginx_addvhost_nv.log
    -------------------------------------------------------------
    


    Vhost Config Files
     
  4. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    HTTP/2 HTTPS Header Compression Testing



    Doing more HTTP/2 HTTPS tests for Centmin Mod Nginx and Caddy source compiled binaries. Not all HTTP/2 feature sets are equal.
    • Caddy has full HTTP/2 HPACK encoding / header compression support, while Nginx out of box only has partial HTTP/2 HPACK encoding support. So over subsequent HTTP/2 HTTPS requests, Caddy will have more header size savings due to higher levels of compression compared to Nginx. However, Cloudflare folks have open sourced a Nginx patch to enable full HTTP/2 HPACK encoding support in Nginx. You can see Cloudflare's blog article about the benefits here.
    • Caddy has HTTP/2 Server Push support, while Nginx doesn't have HTTP/2 Server Push officially as yet.
    Centmin Mod 123.09beta01's Nginx server has optional support to enable the Cloudflare HTTP/2 HPACK full encoding support patch. So let's see how h2load tests fair when both Centmin Mod Nginx and Caddy are using full HTTP/2 HPACK encoding support :)

    If you look carefully, h2load tester also reports the amount of header savings due to HTTP/2 HPACK encoding support / header compression. For Nginx defaults it's usually around 20-25% due to partial support. Here, you see headers (space savings 25.32%). With full HTTP/2 HPACK encoding support, header savings should be around 90-97% - getting closer to 99% as you handle more HTTP/2 HTTPS requests.
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.05s, 4872.51 req/s, 10.22MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 20.97MB (21989800) total, 2.19MB (2300000) headers (space savings 25.32%), 18.60MB (19500000) data
                        min         max         mean         sd        +/- sd
    time for request:    32.02ms    752.52ms    384.73ms    137.39ms    65.07%
    time for connect:    99.23ms       1.79s    812.66ms    623.51ms    37.50%
    time to 1st byte:   131.28ms       1.84s    964.01ms    581.92ms    51.00%
    req/s           :      24.38       79.40       46.59       20.69    53.50%
    


    Before Cloudflare patch, Nginx config is compiled against LibreSSL 2.5.5 (which actually is less performant than OpenSSL 1.0.2/1.1), so will do tests against OpenSSL as well later.
    Then enable Cloudflare patch, setting NGINX_HPACK='y' in persistent config file at /etc/centminmod/custom_config.inc and then running centmin.sh menu option 4 to recompile Nginx.

    After Cloudflare patch applied, Nginx config - notice --with-http_v2_hpack_enc indicating that Cloudflare's HTTP/2 HPACK full encoding patch has been added.
    retesting h2load testing and notice the header space saving is now increased from ~25% to 94.83%
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.14s, 4676.12 req/s, 8.85MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.93MB (19849000) total, 155.47KB (159200) headers (space savings 94.83%), 18.60MB (19500000) data
                         min         max         mean         sd        +/- sd
    time for request:    32.39ms    571.10ms    237.62ms    115.17ms    69.44%
    time for connect:   105.86ms       1.90s       1.00s    564.89ms    59.00%
    time to 1st byte:   143.36ms       1.98s       1.09s    540.39ms    57.00%
    req/s           :      23.39       83.86       44.78       19.24    61.50%
    


    Let's switch Centmin Mod Nginx from using LibreSSL 2.5.5 to OpenSSL 1.1.0f setting LIBRESSL_SWITCH='n' in persistent config file at /etc/centminmod/custom_config.inc so the persistent config file now looks like this
    Code (Text):
    NGINX_HPACK='y'
    LIBRESSL_SWITCH='n'
    LETSENCRYPT_DETECT='y'
    

    Run centmin.sh menu option 4 to recompile Nginx with OpenSSL 1.1.0f
    retesting h2load testing with header space saving is still around 94.83% though req/s is down a bit.
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.18s, 4593.27 req/s, 8.69MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.93MB (19849000) total, 155.47KB (159200) headers (space savings 94.83%), 18.60MB (19500000) data
                         min         max         mean         sd        +/- sd
    time for request:    32.48ms    574.35ms    296.40ms    115.78ms    67.03%
    time for connect:   103.28ms       1.93s    942.41ms    575.89ms    50.00%
    time to 1st byte:   138.22ms       2.05s       1.06s    571.18ms    56.50%
    req/s           :      22.98       96.53       47.19       22.95    72.00%
    


    Centmin Mod Nginx was compiled using Clang 3.4.2 compiler, but the VPS is using Intel Xeon E5-1650v3 2 cpu cores so may lend well to using a newer compiler for the new cpu. So switching Nginx compile from Clang 3.4.2 to GCC 6.3.1 instead setting in persistent config file at /etc/centminmod/custom_config.inc additional 3 options at bottom to switch from Clang to GCC compiler and use GCC 6.3.1 via devtoolset-6. Centmin Mod 123.09beta01 can compile Nginx via additional persistent config file options to support Clang 3.4, 4.0.1, 5.0 and GCC native 4.4/4.8, 5.3, 6.3 and 7.1.
    Code (Text):
    NGINX_HPACK='y'
    LIBRESSL_SWITCH='n'
    LETSENCRYPT_DETECT='y'
    
    DEVTOOLSETSIX='y'
    NGINX_DEVTOOLSETGCC='y'
    CLANG='n'
    

    Run centmin.sh menu option 4 to recompile Nginx with OpenSSL 1.1.0f + GCC 6.3.1 + Cloudflare HTTP/2 HPACK Full encoding patch.
    retesting h2load testing with header space saving is still around 94.83% though req/s is up a bit at 4,796.58 req/s
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.08s, 4796.58 req/s, 9.08MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.93MB (19849000) total, 155.47KB (159200) headers (space savings 94.83%), 18.60MB (19500000) data
                         min         max         mean         sd        +/- sd
    time for request:    31.78ms    683.64ms    377.83ms    143.25ms    67.21%
    time for connect:   103.95ms       1.71s    876.70ms    542.49ms    48.50%
    time to 1st byte:   140.36ms       1.89s       1.04s    573.50ms    54.00%
    req/s           :      24.00      100.69       46.48       23.81    81.00%
    

    Switching from GCC 6.3.1 to GCC 7.1 compiled Nginx is still in private testing but can be used if devtoolset-7 SCL is installed and setting in persistent config file at /etc/centminmod/custom_config.inc the following
    Code (Text):
    NGINX_HPACK='y'
    LIBRESSL_SWITCH='n'
    LETSENCRYPT_DETECT='y'
    
    #DEVTOOLSETSIX='y'
    DEVTOOLSETSEVEN='y'
    NGINX_DEVTOOLSETGCC='y'
    CLANG='n'
    

    Run centmin.sh menu option 4 to recompile Nginx with OpenSSL 1.1.0f + GCC 7.1 + Cloudflare HTTP/2 HPACK Full encoding patch.
    retesting h2load testing with header space saving is still around 94.83% but req/s is higher at 4929.19 req/s
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 2.03s, 4929.19 req/s, 9.33MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.93MB (19849000) total, 155.47KB (159200) headers (space savings 94.83%), 18.60MB (19500000) data
                         min         max         mean         sd        +/- sd
    time for request:    31.72ms    794.90ms    468.84ms    170.18ms    64.76%
    time for connect:    97.66ms       1.71s    778.32ms    576.91ms    56.50%
    time to 1st byte:   130.49ms       1.83s    967.94ms    565.99ms    55.50%
    req/s           :      24.66       73.87       42.20       16.71    67.50%
    


    Compared to Caddy's HTTP/2 header compression savings ~94.64%
    Code (Text):
    /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443
    
    starting benchmark...
    spawning thread #0: 100 total client(s). 5000 total requests
    spawning thread #1: 100 total client(s). 5000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 3.77s, 2650.73 req/s, 5.01MB/s
    requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 18.89MB (19812711) total, 177.45KB (181711) headers (space savings 94.64%), 18.54MB (19440000) data
                         min         max         mean         sd        +/- sd
    time for request:   378.20ms       3.40s       1.69s    635.10ms    71.59%
    time for connect:   261.01ms       2.70s       1.24s    958.16ms    58.50%
    time to 1st byte:   641.85ms       3.61s       2.58s    784.14ms    60.50%
    req/s           :      13.26       30.88       16.27        3.41    84.50%
    


    Also will look into compiling Caddy binaries with Clang and GCC versions to see if there's a difference :)

    Summary

    caddy-vs-nginx-h2load-http2-loadtests-160917-02.png

    Updated:
    Built Caddy custom source binary with GCC 7.1.1 instead of system GCC 4.8.5 to compare. Nice gains :)

    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -v -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;   
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    2836.69 req/s 100% completed
    2992.75 req/s 100% completed
    2751.78 req/s 100% completed
    2886.93 req/s 100% completed
    2996.70 req/s 100% completed
    3004.66 req/s 100% completed
    2474.80 req/s 99% completed
    


    caddy-vs-nginx-h2load-http2-loadtests-160917-03.png
     
    Last edited: Sep 18, 2017
  5. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy http.cache Proxy Caching



    Caddy also has a http.cache plugin for proxy caching. Testing Caddy source built with GCC 7.1.1 binary with the following http.cache config
    Code (Text):
        proxy / caddybench.domain.com:80
        cache {
            match_path /
            match_header Content-Type image/jpg image/png text/html
            status_header X-Cache-Status
            default_max_age 5m
            path /tmp/caddy-cache
        }
    

    Caddy proxy cache miss
    Code (Text):
    curl -I https://caddybench.domain.com
    HTTP/2 200
    accept-ranges: bytes
    content-type: text/html; charset=utf-8
    date: Sun, 17 Sep 2017 23:06:08 GMT
    etag: "owd8wg4pb"
    last-modified: Sat, 16 Sep 2017 09:26:40 GMT
    server: Caddy
    server: Caddy
    x-cache-status: miss
    x-content-type-options: nosniff
    x-frame-options: SAMEORIGIN
    x-powered-by: Caddy via CentminMod
    x-xss-protection: 1; mode=block
    content-length: 6095
    

    Caddy proxy cache hit
    Code (Text):
    curl -I https://caddybench.domain.com
    HTTP/2 200
    accept-ranges: bytes
    content-type: text/html; charset=utf-8
    date: Sun, 17 Sep 2017 23:06:08 GMT
    etag: "owd8wg4pb"
    last-modified: Sat, 16 Sep 2017 09:26:40 GMT
    server: Caddy
    server: Caddy
    x-cache-status: hit
    x-content-type-options: nosniff
    x-frame-options: SAMEORIGIN
    x-powered-by: Caddy via CentminMod
    x-xss-protection: 1; mode=block
    content-length: 6095
    


    h2load test against HTTP/2 HTTPS proxy cached site
    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/2 h2load stress test
    
    4436.63 req/s 100% completed
    4838.55 req/s 100% completed
    4737.82 req/s 100% completed
    5068.30 req/s 100% completed
    3881.18 req/s 100% completed
    4568.86 req/s 100% completed
    4099.06 req/s 100% completed
    

    h2load test against non-HTTPS HTTP/1.1 backend directly on port 80
    Code (Text):
    echo "Caddy 0.10.9 Source Build HTTP/1.1 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load --h1 -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' http://caddybench.domain.com:80 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Build HTTP/1.1 h2load stress test
    2884.80 req/s 100% completed
    3753.52 req/s 100% completed
    3985.83 req/s 100% completed
    4374.14 req/s 100% completed
    3670.77 req/s 100% completed
    3881.24 req/s 100% completed
    3813.97 req/s 100% completed
    

    Retest Nginx 1.13.5 built with GCC 7.1.1 for 7x h2load runs = 4,956 req/s average
    Code (Text):
    echo "CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://nginxbench.domain.com:443 > h2load.$i.nginx.log; cat h2load.$i.nginx.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.13.5 HTTP/2 h2load stress test
    5171.32 req/s 100% completed
    4654.93 req/s 100% completed
    5028.61 req/s 100% completed
    4695.41 req/s 100% completed
    5101.92 req/s 100% completed
    5039.42 req/s 100% completed
    5001.25 req/s 100% completed
    


    Summary compared to previous results in the following table. So Caddy http.cache proxy caching improves non-HTTPS requests by ~20% and HTTP/2 HTTPS requests by 58.6%. However, even with Caddy proxy caching via http.cache plugin, Caddy is still slower than Nginx without any caching.

    caddy-vs-nginx-h2load-http2-loadtests-160917-06.png
     
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy Source Build: GCC 7.1.1 vs Clang 4.0.1



    Previously Caddy was tested with GCC 7.1.1 compiler as well. This time also built Caddy 0.10.9 binary with Clang 4.0.1 compiler to compare. From the results, Caddy built with GCC 7.1.1 was overall faster especially for non-HTTPS port 80 backend and Caddy http.cache proxy caching. While for straight Caddy HTTP/2 HTTPS tests, Caddy built with GCC 7.1.1 and Clang 4.0.1 were pretty close.

    Caddy h2load HTTP/2 test with GCC 7.1.1
    Code (Text):
    echo "Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    avg: 2812 req/s
    Code (Text):
    echo "Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test
    2696.12 req/s 100% completed
    2906.17 req/s 100% completed
    2719.78 req/s 100% completed
    2718.14 req/s 100% completed
    2912.49 req/s 100% completed
    2736.60 req/s 100% completed
    2995.76 req/s 100% completed
    

    Caddy h2load HTTP/2 test with Clang 4.0.1
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    avg: 2813 req/s
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test
    3148.24 req/s 100% completed
    2964.38 req/s 100% completed
    2836.92 req/s 100% completed
    2984.65 req/s 100% completed
    2347.34 req/s 100% completed
    2843.19 req/s 100% completed
    2569.00 req/s 100% completed
    

    Caddy port 80 backend with Clang 4.0.1
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/1.1 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load --h1 -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' http://caddybench.domain.com:80 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    avg: 2543 req/s
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/1.1 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load --h1 -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' http://caddybench.domain.com:80 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    2499.96 req/s 100% completed
    2995.45 req/s 100% completed
    1135.85 req/s 100% completed
    2985.12 req/s 100% completed
    2368.40 req/s 100% completed
    3091.10 req/s 100% completed
    2727.03 req/s 100% completed
    

    Caddy HTTP/2 HTTPS http.cache proxy caching to 80 with Clang 4.0.1
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    avg: 3962 req/s
    Code (Text):
    echo "Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source Clang 4.0.1 Build HTTP/2 h2load stress test
    4476.26 req/s 100% completed
    3759.10 req/s 100% completed
    3205.71 req/s 100% completed
    3684.59 req/s 100% completed
    4043.06 req/s 100% completed
    4655.74 req/s 100% completed
    3906.90 req/s 100% completed
    


    caddy-vs-nginx-h2load-http2-loadtests-190917-07.png
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,606
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,297
    Local Time:
    9:48 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy + Varnish Cache 5.2 Sandwiched Proxies



    Testing Caddy 0.10.9 GCC 7.1.1 build binary with Varnish Cache 5.2 where Caddy HTTPS frontend acts as a SSL proxy for SSL termination (port 443) to Varnish Cache backend (port 6081) and then Varnish Cache talks to Caddy HTTP (port 80) as backend. This is following Caddy + Varnish guide setup on official forums here. It seems it worked well for users testing non-HTTP/2 HTTPS loads i.e. with wrk load testing tool. Unfortunately, for h2load HTTP/2 HTTPS load testing was pretty bad - much worse than Caddy http.cache proxy caching performance. HTTP/2 issues might be related to how Varnish Cache is handling HTTP/2 H/2 threads with these bugs:
    Varnish Cache has experimental HTTP/2 support since 5.0 but disabled by default so need to enable it.
    Code (Text):
    varnishadm param.show feature
    feature
            Value is: +http2
            Default is: none
    
            Enable/Disable various minor features.
               none                       Disable all features.
    
            Use +/- prefix to enable/disable individual feature:
               short_panic                Short panic message.
               wait_silo                  Wait for persistent silo.
               no_coredump                No coredumps.
               esi_ignore_https           Treat HTTPS as HTTP in
                                          ESI:includes
               esi_disable_xml_check      Don't check of body looks like
                                          XML
               esi_ignore_other_elements  Ignore non-esi XML-elements
               esi_remove_bom             Remove UTF-8 BOM
               https_scheme               Also split https URIs
               http2                      Support HTTP/2 protocol
    

    Varnish Cache 5.2 was ran with following options
    Code (Text):
    DAEMON_OPTS="-p feature=+http2 -p thread_pools=2 -p thread_pool_add_delay=0.02 -p listen_depth=4096 -p lru_interval=2 -p cli_timeout=10 -p thread_pool_min=500 -p thread_pool_max=1500 -p thread_pool_timeout=300 -p tcp_fastopen=on"
    


    Caddy proxy to Varnish 6081 port
    Code (Text):
         proxy / localhost:6081 {
             transparent
         }
    

    Code (Text):
    curl -I https://caddybench.domain.com
    HTTP/2 200
    accept-ranges: bytes
    age: 6
    content-type: text/html; charset=utf-8
    date: Thu, 21 Sep 2017 01:09:21 GMT
    etag: W/"owd8wg4pb"
    last-modified: Sat, 16 Sep 2017 09:26:40 GMT
    server: Caddy
    server: Caddy
    vary: Accept-Encoding
    via: 1.1 varnish (Varnish/5.2)
    x-cache: HIT
    x-cache-hits: 1
    x-content-type-options: nosniff
    x-frame-options: SAMEORIGIN
    x-powered-by: Caddy via CentminMod
    x-varnish: 32770 3
    x-xss-protection: 1; mode=block
    

    Code (Text):
    echo "Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test + Varnish 5.2"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    avg: 145.57 req/s 31.99%
    Code (Text):
    echo "Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test + Varnish 5.2"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test + Varnish 5.2
    699.98 req/s 94.49% completed
    166.57 req/s 46.03% completed
    146.46 req/s 59.46% completed
    56.99 req/s 23.93% completed
    0.00 req/s 0% completed
    0.00 req/s 0% completed
    0.00 req/s 0% completed
    

    caddy-vs-nginx-h2load-http2-loadtests-varnish-210917-01.png
     
    Last edited: Sep 21, 2017
Thread Status:
Not open for further replies.