SSL Caddy 0.11.5 TLS 1.3 HTTP/2 HTTPS Benchmarks Part 1

Caddy 0.11.5 was released with TLS 1.3 support when built with Go 1.12 latest release. This thread will cover some HTTP/2 HTTPS TLS 1.3 benchmarks comparing Centmin Mod Nginx 1.15.9 with Caddy 0.11.5 custom built binary with Go 1.12.

• You can find older comparison Caddy vs Nginx benchmarks here part 1 and part 2.
• Caddy EULA license agreement prohibits sharing benchmark results for official Caddy provided binaries Caddy EULA section 3.1 h - benchmarking info clarification?. So I built my own Caddy 0.11.5 custom built binary with Go 1.12 on CentOS 7.6 64bit VPS system. FYI, my custom built Caddy binaries are already faster than official Caddy provided binary

h2load test server:

The h2load test server is ImpactVPS KVM VPS

• 2 CPU Core - Intel Xeon Silver 4114 Skylake cpu
• 4GB ram
• 60GB NVMe SSD
• CentOS 7.6 64bit
• Centmin Mod 123.09beta01 LEMP Stack

Code (Text):

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
Stepping:              4
CPU MHz:               2199.992
BogoMIPS:              4399.98
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 arat spec_ctrl

Test Parameters

I created Centmin Mod Nginx HTTP/2 HTTPS vhost site = test.com via nv command line with autogenerated self-signed RSA 2048bit SSL certificate.

Code (Text):

nv -d test.com -s y -u MYFTPUSERNAME

This will create Nginx HTTP/HTTPS vhost with HTTP/2 HTTPS and self-signed RSA 2048bit SSL certificate automatcially on HTTPS port 443

Then I created Caddy HTTP/2 HTTPS site with TLS 1.3 on HTTPS port 444 and used Caddy's own internal self-signed SSL certificate generation using the following configs:

For Caddy generated self-signed SSL certificate generation

Code (Text):

   tls self_signed

Centmin Mod Nginx 1.15.9 curl header check on port 443 and version info

Code (Text):

curl -Isk https://test.com:443/
HTTP/2 200
date: Tue, 05 Mar 2019 10:23:16 GMT
content-type: text/html; charset=utf-8
content-length: 6552
last-modified: Tue, 05 Mar 2019 03:17:21 GMT
vary: Accept-Encoding
etag: "5c7dea41-1998"
server: nginx centminmod
x-powered-by: centminmod
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
accept-ranges: bytes

Caddy 0.11.5 curl header check on port 444 and version info

Code (Text):

curl -Isk https://test.com:444/
HTTP/2 200
accept-ranges: bytes
content-type: text/html; charset=utf-8
etag: "pnvigx520"
last-modified: Tue, 05 Mar 2019 03:17:21 GMT
server: Caddy
x-content-type-options: nosniff
x-powered-by: Caddy via CentminMod
x-xss-protection: 1; mode=block
content-length: 6552
date: Tue, 05 Mar 2019 10:23:33 GMT

Code (Text):

caddy -version
Caddy 0.11.5 (+fda7350 Tue Mar 05 06:04:26 UTC 2019) (unofficial)
1 file changed, 46 insertions(+), 1 deletion(-)
caddy/caddymain/run.go

Code (Text):

caddy -plugins
Server types:
  http

Caddyfile loaders:
  short
  flag
  default

Other plugins:
  http.authz
  http.awses
  http.awslambda
  http.basicauth
  http.bind
  http.browse
  http.cache
  http.cgi
  http.cors
  http.datadog
  http.errors
  http.expires
  http.expvar
  http.ext
  http.fastcgi
  http.filter
  http.forwardproxy
  http.geoip
  http.git
  http.gopkg
  http.grpc
  http.gzip
  http.header
  http.index
  http.internal
  http.ipfilter
  http.jwt
  http.limits
  http.locale
  http.log
  http.login
  http.mailout
  http.markdown
  http.mime
  http.minify
  http.nobots
  http.pprof
  http.prometheus
  http.proxy
  http.proxyprotocol
  http.push
  http.ratelimit
  http.realip
  http.reauth
  http.redir
  http.request_id
  http.restic
  http.rewrite
  http.root
  http.s3browser
  http.secrets
  http.status
  http.templates
  http.timeouts
  http.upload
  http.webdav
  http.websocket
  on
  tls
  tls.cluster.file
  tls.dns.azure
  tls.dns.cloudflare
  tls.dns.digitalocean
  tls.dns.dnsmadeeasy
  tls.dns.dyn
  tls.dns.godaddy
  tls.dns.googlecloud
  tls.dns.linode
  tls.dns.namecheap
  tls.dns.ns1
  tls.dns.ovh
  tls.dns.powerdns
  tls.dns.rackspace
  tls.dns.route53
  tls.dns.vultr

h2load HTTP/2 HTTPS Benchmarks

h2load HTTP/2 HTTPS benchmark tool version used

Code (Text):

h2load --version
h2load nghttp2/1.37.0-DEV

h2load test parameters 200 users and 25,000 requests for gzip encoded HTTP/2 HTTPS request/load testing. Each server had 3x h2load test runs.

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:443
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:444

---

Centmin Mod Nginx 1.15.9 HTTP/2 HTTPS run 1

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:443
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 4.72s, 5299.41 req/s, 11.73MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 55.35MB (58034800) total, 4.94MB (5175000) headers (space savings 26.86%), 49.97MB (52400000) data
                    min         max         mean         sd        +/- sd
time for request:     2.50ms    282.07ms     35.11ms     16.66ms    92.66%
time for connect:    42.63ms    326.84ms    158.68ms     77.33ms    58.50%
time to 1st byte:   181.01ms    383.96ms    318.63ms     35.77ms    64.00%
req/s           :      26.53       28.83       27.50        0.77    55.00%

Centmin Mod Nginx 1.15.9 HTTP/2 HTTPS run 2

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:443
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 4.66s, 5362.61 req/s, 11.87MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 55.35MB (58034800) total, 4.94MB (5175000) headers (space savings 26.86%), 49.97MB (52400000) data
                    min         max         mean         sd        +/- sd
time for request:     3.68ms    282.83ms     34.26ms     17.93ms    95.15%
time for connect:    47.09ms    332.00ms    160.37ms     80.32ms    60.00%
time to 1st byte:   101.30ms    429.20ms    349.01ms     43.27ms    68.00%
req/s           :      26.87       30.41       28.18        1.19    58.50%

Centmin Mod Nginx 1.15.9 HTTP/2 HTTPS run 3

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:443
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 4.79s, 5223.73 req/s, 11.56MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 55.35MB (58034800) total, 4.94MB (5175000) headers (space savings 26.86%), 49.97MB (52400000) data
                     min         max         mean         sd        +/- sd
time for request:     1.67ms    302.63ms     34.59ms     14.58ms    89.50%
time for connect:    52.32ms    357.80ms    173.24ms     82.30ms    59.00%
time to 1st byte:   218.43ms    401.12ms    310.55ms     61.72ms    55.00%
req/s           :      26.17       29.99       27.88        1.59    56.50%

Caddy 0.11.5 HTTP/2 HTTPS run 1

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:444
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 13.42s, 1862.89 req/s, 3.79MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 50.81MB (53279316) total, 408.51KB (418316) headers (space savings 94.67%), 49.97MB (52400000) data
                    min         max         mean         sd        +/- sd
time for request:      951us    460.43ms    100.43ms     47.72ms    70.94%
time for connect:    43.01ms       1.29s    506.25ms    414.93ms    65.50%
time to 1st byte:   273.81ms       1.54s    674.25ms    408.92ms    82.00%
req/s           :       9.32       10.48        9.58        0.23    76.50%

Caddy 0.11.5 HTTP/2 HTTPS run 2

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:444
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 14.53s, 1721.05 req/s, 3.50MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 50.82MB (53283985) total, 413.07KB (422985) headers (space savings 94.61%), 49.97MB (52400000) data
                    min         max         mean         sd        +/- sd
time for request:     1.09ms    556.96ms    110.05ms     57.95ms    78.02%
time for connect:    40.10ms       1.04s    446.79ms    325.63ms    62.00%
time to 1st byte:   343.31ms       1.29s    656.70ms    275.09ms    74.50%
req/s           :       8.61        9.56        8.80        0.18    77.00%

Caddy 0.11.5 HTTP/2 HTTPS run 3

Code (Text):

users=200
requests=25000
h2load -t1 -H 'Accept-Encoding: gzip' -c${users} -n${requests} https://test.com:444
starting benchmark...
spawning thread #0: 200 total client(s). 25000 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 12.83s, 1949.13 req/s, 3.96MB/s
requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 50.81MB (53276786) total, 406.04KB (415786) headers (space savings 94.70%), 49.97MB (52400000) data
                     min         max         mean         sd        +/- sd
time for request:      995us    593.22ms     97.70ms     47.46ms    75.41%
time for connect:    47.90ms    652.23ms    322.09ms    164.51ms    60.00%
time to 1st byte:   401.57ms    810.19ms    659.48ms     84.67ms    64.00%
req/s           :       9.75       10.85        9.98        0.21    79.50%

3x h2load HTTP/2 HTTPS TLSv1.3 load test runs' individual and averages.