Learn about Centmin Mod LEMP Stack today
Become a Member

SSL Caddy HTTP/2 server & benchmarks

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, Dec 7, 2015.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5
    Lets check out Caddy HTTP/2 web server which also integrates Letsencrypt SSL :) Centmin Mod has plans to integrate OpenLiteSpeed HTTP/2 web server, Apache 2.4 + PHP-FPM and H20 HTTP/2 server. I've already been playing with H2O and OpenLiteSpeed. So I will also evaluate the viability of Caddy integration into Centmin Mod stack :)

    Right now I am using Centmin Mod 123.09beta01le branch's generated free Letsencrypt SSL certificates for both Centmin Mod Nginx and Caddy's HTTP/2 SSL deployment so not using Caddy's Letsencrypt auto setup and deployment.

    Caddy Documentation


    Site Links



    Set up on my le12.http2ssl.xyz test Letsencrypt server at the following urls and ports. Server is an OpenVZ 1GB VPS with 4 cpu threads using Intel Xeon E3-1230v2 @3.30Ghz

    Server Versions



    Code:
    caddy -version
    Caddy 0.8
    

    Notes



    • OCSP stapling is currently only enabled on Caddy auto generated Letsencrypt SSL certificates [issue]. If you try to define your own SSL certificates for Caddy via tls directive, the OCSP stapling (and Ticket sessions) is disabled.
      Code:
      cipherscan le12.http2ssl.xyz:443
      ............................
      Target: le12.http2ssl.xyz:443
      
      prio  ciphersuite                  protocols              pfs                 curves
      1     ECDHE-RSA-CHACHA20-POLY1305  TLSv1.2                ECDH,P-256,256bits  prime256v1
      2     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
      3     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
      4     DHE-RSA-AES128-GCM-SHA256    TLSv1.2                DH,2048bits         None
      5     DHE-RSA-AES256-GCM-SHA384    TLSv1.2                DH,2048bits         None
      6     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
      7     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
      8     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
      9     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
      10    DHE-RSA-AES128-SHA256        TLSv1.2                DH,2048bits         None
      11    DHE-RSA-AES128-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
      12    DHE-RSA-AES256-SHA256        TLSv1.2                DH,2048bits         None
      13    DHE-RSA-AES256-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
      14    AES128-GCM-SHA256            TLSv1.2                None                None
      15    AES256-GCM-SHA384            TLSv1.2                None                None
      16    AES128-SHA256                TLSv1.2                None                None
      17    AES256-SHA256                TLSv1.2                None                None
      18    AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
      19    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
      
      Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature
      TLS ticket lifetime hint: 3600
      OCSP stapling: supported
      Cipher ordering: server
      Curves ordering: server - fallback: no
      Server supports secure renegotiation
      Server supported compression methods: NONE
      TLS Tolerance: yes  
      Code:
      cipherscan le12.http2ssl.xyz:445        
      .......................
      Target: le12.http2ssl.xyz:445
      
      prio  ciphersuite                  protocols              pfs                 curves
      1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1,secp384r1,secp521r1
      2     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1,secp384r1,secp521r1
      3     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1,secp384r1,secp521r1
      4     AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
      5     AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
      6     ECDHE-RSA-DES-CBC3-SHA       TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1,secp384r1,secp521r1
      7     DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None
      
      Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature
      TLS ticket lifetime hint: None
      OCSP stapling: not supported
      Cipher ordering: server
      Curves ordering: server - fallback: no
      Server supports secure renegotiation
      Server supported compression methods: NONE
      TLS Tolerance: yes
    • Default log file format doesn't seem to include a user agent string
      Code:
      IP - [07/Dec/2015:00:29:55 +0000] "HEAD / HTTP/1.1" 200 0
      IP - [07/Dec/2015:00:30:22 +0000] "HEAD / HTTP/2.0" 200 0
      Update: there's a hidden user-agent variable too so updated my Caddyfile below.

    Curl Headers



    For Centmin Mod Nginx

    over HTTP/1.1
    Code:
    curl -I http://le12.http2ssl.xyz:80
    HTTP/1.1 200 OK
    Server: nginx centminmod
    Date: Mon, 07 Dec 2015 00:29:25 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 1832
    Last-Modified: Sun, 06 Dec 2015 23:40:18 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5664c762-728"
    Expires: Tue, 08 Dec 2015 00:29:25 GMT
    Cache-Control: max-age=86400
    Accept-Ranges: bytes
    over HTTP/1.1
    Code:
    curl -I https://le12.http2ssl.xyz:443
    HTTP/1.1 200 OK
    Server: nginx centminmod
    Date: Mon, 07 Dec 2015 00:29:52 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 1832
    Last-Modified: Sun, 06 Dec 2015 23:40:18 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5664c762-728"
    Expires: Tue, 08 Dec 2015 00:29:52 GMT
    Cache-Control: max-age=86400
    Accept-Ranges: bytes
    over HTTP/2
    Code:
    curl --http2 -I https://le12.http2ssl.xyz:443
    HTTP/2.0 200
    server:nginx
    date:Mon, 07 Dec 2015 00:30:19 GMT
    content-type:text/html; charset=utf-8
    content-length:1832
    last-modified:Sun, 06 Dec 2015 23:40:18 GMT
    vary:Accept-Encoding
    etag:"5664c762-728"
    expires:Tue, 08 Dec 2015 00:30:19 GMT
    cache-control:max-age=86400
    accept-ranges:bytes
    For Caddy

    over HTTP/1.1
    Code:
    curl -I http://le12.http2ssl.xyz:81
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1832
    Content-Type: text/html; charset=utf-8
    Last-Modified: Sun, 06 Dec 2015 23:40:18 GMT
    Server: Caddy
    Date: Mon, 07 Dec 2015 00:29:28 GMT
    over HTTP/1.1
    Code:
    curl -I https://le12.http2ssl.xyz:445
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1832
    Content-Type: text/html; charset=utf-8
    Last-Modified: Sun, 06 Dec 2015 23:40:18 GMT
    Server: Caddy
    Date: Mon, 07 Dec 2015 00:29:55 GMT
    over HTTP/2
    Code:
    curl --http2 -I https://le12.http2ssl.xyz:445
    HTTP/2.0 200
    content-length:1832
    server:Caddy
    cache-control:max-age=86400
    last-modified:Sun, 06 Dec 2015 23:40:18 GMT
    content-type:text/html; charset=utf-8
    accept-ranges:bytes

    VPS Specs



    Server 1GB OpenVZ using Xeon E3-1230v2
    Code:
    cat /proc/cpuinfo
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 58
    model name      : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
    stepping        : 9
    cpu MHz         : 3300.117
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
    bogomips        : 6600.23
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 58
    model name      : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
    stepping        : 9
    cpu MHz         : 3300.117
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 1
    cpu cores       : 4
    apicid          : 2
    initial apicid  : 2
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
    bogomips        : 6600.23
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    processor       : 2
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 58
    model name      : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
    stepping        : 9
    cpu MHz         : 3300.117
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 2
    cpu cores       : 4
    apicid          : 4
    initial apicid  : 4
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
    bogomips        : 6600.23
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    processor       : 3
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 58
    model name      : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
    stepping        : 9
    cpu MHz         : 3300.117
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 3
    cpu cores       : 4
    apicid          : 6
    initial apicid  : 6
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
    bogomips        : 6600.23
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
     
    Last edited: Dec 8, 2015
    • Informative Informative x 1
  2. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5
    My Caddyfile at /opt/caddy/Caddyfile

    Updated with custom log format that closely resembles Nginx format with added user-agent
    Code:
    le12.http2ssl.xyz:81 {
        gzip {
            level 6
        }
        browse
        header / {
            Cache-Control "max-age=86400"
        }  
        tls off
        root /home/nginx/domains/le12.http2ssl.xyz/public
        fastcgi / 127.0.0.1:9000 {
            ext   .php
            split .php
            index index.php
        }
        log / /opt/caddy/le12.http2ssl.xyz-access.log "{remote} {when} {method} {uri} {proto} {status} {size} {>User-Agent} {latency}" {
            rotate {
                    size 100 # Rotate after 100 MB
                    age  14  # Keep log files for 14 days
                    keep 10  # Keep at most 10 log files
            }
        }
    }
    
    le12.http2ssl.xyz:445 {
        gzip {
            level 6
        }
        browse
        header / {
            Cache-Control "max-age=86400"
        }  
        tls /etc/letsencrypt/live/le12.http2ssl.xyz/fullchain.pem /etc/letsencrypt/live/le12.http2ssl.xyz/privkey.pem {
            protocols tls1.0 tls1.2
            ciphers ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-CBC-SHA ECDHE-RSA-AES256-CBC-SHA ECDHE-ECDSA-AES256-CBC-SHA ECDHE-ECDSA-AES128-CBC-SHA RSA-AES128-CBC-SHA RSA-AES256-CBC-SHA ECDHE-RSA-3DES-EDE-CBC-SHA RSA-3DES-EDE-CBC-SHA
        }
        root /home/nginx/domains/le12.http2ssl.xyz/public
        fastcgi / 127.0.0.1:9000 {
            ext   .php
            split .php
            index index.php
        }
        log / /opt/caddy/le12.http2ssl.xyz-access.log "{remote} {when} {method} {uri} {proto} {status} {size} {>User-Agent} {latency}" {
            rotate {
                    size 100 # Rotate after 100 MB
                    age  14  # Keep log files for 14 days
                    keep 10  # Keep at most 10 log files
            }
        }
    }
    
     
    Last edited: Aug 5, 2016
    • Informative Informative x 1
  3. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    testssl tests



    Caddy
    Code:
    testssl le12.http2ssl.xyz:445
    
    Testing protocols (via sockets except TLS 1.2 and SPDY/NPN)
    
    SSLv2      not offered (OK)
    SSLv3      not offered (OK)
    TLS 1      offered
    TLS 1.1    offered
    TLS 1.2    offered (OK)
    SPDY/NPN   please check manually, server response was ambigious ...
    
    Testing ~standard cipher lists
    
    Null Ciphers                 not offered (OK)
    Anonymous NULL Ciphers       not offered (OK)
    Anonymous DH Ciphers         not offered (OK)
    40 Bit encryption            not offered (OK)
    56 Bit encryption            not offered (OK)
    Export Ciphers (general)     not offered (OK)
    Low (<=64 Bit)               not offered (OK)
    DES Ciphers                  not offered (OK)
    Medium grade encryption      not offered (OK)
    Triple DES Ciphers           offered (NOT ok)
    High grade encryption        offered (OK)
    
    Testing (perfect) forward secrecy, (P)FS -- omitting 3DES, RC4 and Null Encryption here
    
    PFS is offered (OK)  ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA
    
    Testing server preferences
    
    Has server cipher order?     yes (OK)
    Negotiated protocol          TLSv1.2
    Negotiated cipher            ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH
    Cipher order
         TLSv1:     ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA AES128-SHA AES256-SHA ECDHE-RSA-DES-CBC3-SHA DES-CBC3-SHA
         TLSv1.1:   ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA AES128-SHA AES256-SHA ECDHE-RSA-DES-CBC3-SHA DES-CBC3-SHA
         TLSv1.2:   ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA AES128-SHA AES256-SHA ECDHE-RSA-DES-CBC3-SHA DES-CBC3-SHA
         h2:        ECDHE-RSA-AES128-GCM-SHA256
         h2-14:     ECDHE-RSA-AES128-GCM-SHA256
    
    Testing server defaults (Server Hello)
    
    TLS server extensions (std)  "next protocol" "session ticket" "renegotiation info"
    Session Tickets RFC 5077     (none)
    SSL Session ID support       yes
    Server key size              2048 bit
    Signature Algorithm          SHA256 with RSA
    Fingerprint / Serial         SHA1 815EB565DAC210C4CECAFE125899374786BFE2D1 / 0195E067A220D2083468AA50F1FAD06977FB
                                  SHA256 8804577F5FBC9B2F8A3F71693EE02FA8032B00E58F6EA3F94DA8044728ACE041
    Common Name (CN)             "le12.http2ssl.xyz" (works w/o SNI)
    subjectAltName (SAN)         "le12.http2ssl.xyz"
    Issuer                       "Let's Encrypt Authority X1" ("Let's Encrypt" from "US")
    EV cert (experimental)       no
    Certificate Expiration       >= 60 days (2015-12-03 17:39 --> 2016-03-02 17:39 +0000)
    # of certificates provided   2
    Chain of trust (experim.)    "/usr/bin/etc/*.pem" cannot be found / not readable
    Certificate Revocation List  --
    OCSP URI                     http://ocsp.int-x1.letsencrypt.org/
    OCSP stapling                not offered
    TLS timestamp                random values, no fingerprinting possible
    
    Testing HTTP header response @ "/"
    
    HTTP Status Code             200 OK
    HTTP clock skew              0 sec from localtime
    Strict Transport Security    --
    Public Key Pinning           --
    Server banner                Caddy
    Application banner           --
    Cookie(s)                    (none issued at "/")
    Security headers             --
    Reverse Proxy banner         --
    
    Testing vulnerabilities
    
    Heartbleed (CVE-2014-0160)                not vulnerable (OK) (no heartbeat extension)
    CCS (CVE-2014-0224)                       not vulnerable (OK)
    Secure Renegotiation (CVE-2009-3555)      not vulnerable (OK)
    Secure Client-Initiated Renegotiation     not vulnerable (OK)
    CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)
    BREACH (CVE-2013-3587)                    potentially NOT ok, uses gzip HTTP compression. - only supplied "/" tested
                                               Can be ignored for static pages or if no secrets in the page
    POODLE, SSL (CVE-2014-3566)               not vulnerable (OK)
    TLS_FALLBACK_SCSV (RFC 7507), experim.    Downgrade attack prevention supported (OK)
    FREAK (CVE-2015-0204)                     not vulnerable (OK)
    LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK), common primes not checked. See below for any DH ciphers + bit size
    BEAST (CVE-2011-3389)                     TLS1: DES-CBC3-SHA AES128-SHA
                                                     AES256-SHA ECDHE-RSA-DES-CBC3-SHA ECDHE-RSA-AES128-SHA
                                                     ECDHE-RSA-AES256-SHA
                                               VULNERABLE -- but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
    RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)
    
    
    Testing all 181 locally available ciphers against the server, ordered by encryption strength
    
    Hexcode  Cipher Suite Name (OpenSSL)    KeyExch.   Encryption Bits
    -------------------------------------------------------------------------
    xc014   ECDHE-RSA-AES256-SHA           ECDH 256   AES        256                                                                                   
    x35     AES256-SHA                     RSA        AES        256                                                                                   
    xc02f   ECDHE-RSA-AES128-GCM-SHA256    ECDH 256   AESGCM     128                                                                                   
    xc013   ECDHE-RSA-AES128-SHA           ECDH 256   AES        128                                                                                   
    x2f     AES128-SHA                     RSA        AES        128                                                                                   
    xc012   ECDHE-RSA-DES-CBC3-SHA         ECDH 256   3DES       168                                                                                   
    x0a     DES-CBC3-SHA                   RSA        3DES       168  
    Centmin Mod Nginx
    Code:
    testssl le12.http2ssl.xyz:443   
    
    Testing protocols (via sockets except TLS 1.2 and SPDY/NPN)
    
    SSLv2      not offered (OK)
    SSLv3      not offered (OK)
    TLS 1      offered
    TLS 1.1    offered
    TLS 1.2    offered (OK)
    SPDY/NPN   h2, http/1.1 (advertised)
    
    Testing ~standard cipher lists
    
    Null Ciphers                 not offered (OK)
    Anonymous NULL Ciphers       not offered (OK)
    Anonymous DH Ciphers         not offered (OK)
    40 Bit encryption            not offered (OK)
    56 Bit encryption            not offered (OK)
    Export Ciphers (general)     not offered (OK)
    Low (<=64 Bit)               not offered (OK)
    DES Ciphers                  not offered (OK)
    Medium grade encryption      not offered (OK)
    Triple DES Ciphers           not offered (OK)
    High grade encryption        offered (OK)
    
    Testing (perfect) forward secrecy, (P)FS -- omitting 3DES, RC4 and Null Encryption here
    
    PFS is offered (OK)  ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA ECDHE-RSA-CHACHA20-POLY1305 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA ECDHE-RSA-AES128-SHA
    
    Testing server preferences
    
    Has server cipher order?     yes (OK)
    Negotiated protocol          TLSv1.2
    Negotiated cipher            ECDHE-RSA-CHACHA20-POLY1305, 256 bit ECDH
    Cipher order
         TLSv1:     ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
         TLSv1.1:   ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
         TLSv1.2:   ECDHE-RSA-CHACHA20-POLY1305 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA AES128-GCM-SHA256 AES256-GCM-SHA384 AES128-SHA256 AES256-SHA256 AES128-SHA AES256-SHA
         h2:        ECDHE-RSA-CHACHA20-POLY1305 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA AES128-GCM-SHA256 AES256-GCM-SHA384 AES128-SHA256 AES256-SHA256 AES128-SHA AES256-SHA
         http/1.1:  ECDHE-RSA-CHACHA20-POLY1305 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA AES128-GCM-SHA256 AES256-GCM-SHA384 AES128-SHA256 AES256-SHA256 AES128-SHA AES256-SHA
    
    Testing server defaults (Server Hello)
    
    TLS server extensions (std)  "server name" "renegotiation info" "EC point formats" "session ticket" "status request" "next protocol"
    Session Tickets RFC 5077     3600 seconds (PFS requires session ticket keys to be rotated <= daily)
    SSL Session ID support       yes
    Server key size              2048 bit
    Signature Algorithm          SHA256 with RSA
    Fingerprint / Serial         SHA1 815EB565DAC210C4CECAFE125899374786BFE2D1 / 0195E067A220D2083468AA50F1FAD06977FB
                                  SHA256 8804577F5FBC9B2F8A3F71693EE02FA8032B00E58F6EA3F94DA8044728ACE041
    Common Name (CN)             "le12.http2ssl.xyz" (CN in response to request w/o SNI: "le10.http2ssl.xyz")
    subjectAltName (SAN)         "le12.http2ssl.xyz"
    Issuer                       "Let's Encrypt Authority X1" ("Let's Encrypt" from "US")
    EV cert (experimental)       no
    Certificate Expiration       >= 60 days (2015-12-03 17:39 --> 2016-03-02 17:39 +0000)
    # of certificates provided   2
    Chain of trust (experim.)    "/usr/bin/etc/*.pem" cannot be found / not readable
    Certificate Revocation List  --
    OCSP URI                     http://ocsp.int-x1.letsencrypt.org/
    OCSP stapling                offered
    TLS timestamp                random values, no fingerprinting possible
    
    Testing HTTP header response @ "/"
    
    HTTP Status Code             200 OK
    HTTP clock skew              0 sec from localtime
    Strict Transport Security    --
    Public Key Pinning           --
    Server banner                nginx centminmod
    Application banner           --
    Cookie(s)                    (none issued at "/")
    Security headers             --
    Reverse Proxy banner         --
    
    Testing vulnerabilities
    
    Heartbleed (CVE-2014-0160)                not vulnerable (OK) (no heartbeat extension)
    CCS (CVE-2014-0224)                       not vulnerable (OK)
    Secure Renegotiation (CVE-2009-3555)      not vulnerable (OK)
    Secure Client-Initiated Renegotiation     not vulnerable (OK)
    CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)
    BREACH (CVE-2013-3587)                    potentially NOT ok, uses gzip HTTP compression. - only supplied "/" tested
                                               Can be ignored for static pages or if no secrets in the page
    POODLE, SSL (CVE-2014-3566)               not vulnerable (OK)
    TLS_FALLBACK_SCSV (RFC 7507), experim.    Downgrade attack prevention supported (OK)
    FREAK (CVE-2015-0204)                     not vulnerable (OK)
    LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK), common primes not checked. See below for any DH ciphers + bit size
    BEAST (CVE-2011-3389)                     TLS1: AES128-SHA DHE-RSA-AES128-SHA
                                                     AES256-SHA DHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA
                                                     ECDHE-RSA-AES256-SHA
                                               VULNERABLE -- but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
    RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)
    
    Testing all 181 locally available ciphers against the server, ordered by encryption strength
    
    Hexcode  Cipher Suite Name (OpenSSL)    KeyExch.   Encryption Bits
    -------------------------------------------------------------------------
    xcc13   ECDHE-RSA-CHACHA20-POLY1305    ECDH 256   ChaCha20   256                                                                                   
    xc030   ECDHE-RSA-AES256-GCM-SHA384    ECDH 256   AESGCM     256                                                                                   
    xc028   ECDHE-RSA-AES256-SHA384        ECDH 256   AES        256                                                                                   
    xc014   ECDHE-RSA-AES256-SHA           ECDH 256   AES        256                                                                                   
    x9f     DHE-RSA-AES256-GCM-SHA384      DH 2048    AESGCM     256                                                                                   
    x6b     DHE-RSA-AES256-SHA256          DH 2048    AES        256                                                                                   
    x39     DHE-RSA-AES256-SHA             DH 2048    AES        256                                                                                   
    x9d     AES256-GCM-SHA384              RSA        AESGCM     256                                                                                   
    x3d     AES256-SHA256                  RSA        AES        256                                                                                   
    x35     AES256-SHA                     RSA        AES        256                                                                                   
    xc02f   ECDHE-RSA-AES128-GCM-SHA256    ECDH 256   AESGCM     128                                                                                   
    xc027   ECDHE-RSA-AES128-SHA256        ECDH 256   AES        128                                                                                   
    xc013   ECDHE-RSA-AES128-SHA           ECDH 256   AES        128                                                                                   
    x9e     DHE-RSA-AES128-GCM-SHA256      DH 2048    AESGCM     128                                                                                   
    x67     DHE-RSA-AES128-SHA256          DH 2048    AES        128                                                                                   
    x33     DHE-RSA-AES128-SHA             DH 2048    AES        128                                                                                   
    x9c     AES128-GCM-SHA256              RSA        AESGCM     128                                                                                   
    x3c     AES128-SHA256                  RSA        AES        128                                                                                   
    x2f     AES128-SHA                     RSA        AES        128                                                                                   
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    nghttp2 HTTP/2 client test



    nghttp2 test over HTTP/2 connections

    Centmin Mod Nginx
    Code:
    nghttp -nv https://le12.http2ssl.xyz:443
    [  0.020] Connected
    The negotiated protocol: h2
    [  0.057] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
              (niv=3)
              [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
              [SETTINGS_INITIAL_WINDOW_SIZE(0x04):2147483647]
              [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
    [  0.058] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
              (window_size_increment=2147418112)
    [  0.058] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
              (niv=2)
              [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
              [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.058] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
              ; ACK
              (niv=0)
    [  0.058] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
              (dep_stream_id=0, weight=201, exclusive=0)
    [  0.058] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
              (dep_stream_id=0, weight=101, exclusive=0)
    [  0.058] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
              (dep_stream_id=0, weight=1, exclusive=0)
    [  0.058] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
              (dep_stream_id=7, weight=1, exclusive=0)
    [  0.058] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
              (dep_stream_id=3, weight=1, exclusive=0)
    [  0.058] send HEADERS frame <length=44, flags=0x25, stream_id=13>
              ; END_STREAM | END_HEADERS | PRIORITY
              (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
              ; Open new stream
              :method: GET
              :path: /
              :scheme: https
              :authority: le12.http2ssl.xyz
              accept: */*
              accept-encoding: gzip, deflate
              user-agent: nghttp2/1.5.1-DEV
    [  0.073] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
              ; ACK
              (niv=0)
    [  0.074] recv (stream_id=13) :status: 200
    [  0.074] recv (stream_id=13) server: nginx
    [  0.074] recv (stream_id=13) date: Mon, 07 Dec 2015 00:35:16 GMT
    [  0.074] recv (stream_id=13) content-type: text/html; charset=utf-8
    [  0.074] recv (stream_id=13) last-modified: Sun, 06 Dec 2015 23:40:18 GMT
    [  0.074] recv (stream_id=13) vary: Accept-Encoding
    [  0.074] recv (stream_id=13) etag: W/"5664c762-728"
    [  0.074] recv (stream_id=13) expires: Tue, 08 Dec 2015 00:35:16 GMT
    [  0.074] recv (stream_id=13) cache-control: max-age=86400
    [  0.074] recv (stream_id=13) content-encoding: gzip
    [  0.074] recv HEADERS frame <length=227, flags=0x04, stream_id=13>
              ; END_HEADERS
              (padlen=0)
              ; First response header
    [  0.074] recv DATA frame <length=543, flags=0x01, stream_id=13>
              ; END_STREAM
    [  0.074] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
              (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])
    Caddy
    Code:
    nghttp -nv https://le12.http2ssl.xyz:445
    [  0.035] Connected
    The negotiated protocol: h2
    [  0.072] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
              (niv=2)
              [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
              [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.072] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
              (dep_stream_id=0, weight=201, exclusive=0)
    [  0.072] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
              (dep_stream_id=0, weight=101, exclusive=0)
    [  0.072] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
              (dep_stream_id=0, weight=1, exclusive=0)
    [  0.072] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
              (dep_stream_id=7, weight=1, exclusive=0)
    [  0.072] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
              (dep_stream_id=3, weight=1, exclusive=0)
    [  0.072] send HEADERS frame <length=47, flags=0x25, stream_id=13>
              ; END_STREAM | END_HEADERS | PRIORITY
              (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
              ; Open new stream
              :method: GET
              :path: /
              :scheme: https
              :authority: le12.http2ssl.xyz:445
              accept: */*
              accept-encoding: gzip, deflate
              user-agent: nghttp2/1.5.1-DEV
    [  0.085] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
              (niv=3)
              [SETTINGS_MAX_FRAME_SIZE(0x05):1048576]
              [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):250]
              [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):1048896]
    [  0.085] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
              ; ACK
              (niv=0)
    [  0.085] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
              ; ACK
              (niv=0)
    [  0.086] recv (stream_id=13) :status: 200
    [  0.086] recv (stream_id=13) vary: Accept-Encoding
    [  0.086] recv (stream_id=13) cache-control: max-age=86400
    [  0.086] recv (stream_id=13) last-modified: Sun, 06 Dec 2015 23:40:18 GMT
    [  0.086] recv (stream_id=13) content-type: text/html; charset=utf-8
    [  0.086] recv (stream_id=13) accept-ranges: bytes
    [  0.086] recv (stream_id=13) server: Caddy
    [  0.086] recv (stream_id=13) content-encoding: gzip
    [  0.086] recv (stream_id=13) content-length: 548
    [  0.086] recv HEADERS frame <length=92, flags=0x04, stream_id=13>
              ; END_HEADERS
              (padlen=0)
              ; First response header
    [  0.086] recv DATA frame <length=548, flags=0x01, stream_id=13>
              ; END_STREAM
    [  0.087] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
              (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

    nghttp2 statistics



    HTTP/2 connection timings

    for Centmin Mod Nginx
    Code:
    nghttp -nas https://le12.http2ssl.xyz:443
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
    requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
    13    +16.95ms       +351us  16.60ms  200  543 /
    for Caddy
    Code:
    nghttp -nas https://le12.http2ssl.xyz:445
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
    requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
    13    +16.97ms       +118us  16.85ms  200  548 /
     
  5. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Pingdom FPT Tests



    Pingdom FPT uses Chrome 39 browser so HTTP/2 isn't supported so these tests would of fallen back to HTTP/1.1

    Centmin Mod Nginx HTTPS over Chrome 39 which doesn't support HTTP/2
    upload_2015-12-7_11-11-57.png

    Caddy HTTPS over Chrome 39 which doesn't support HTTP/2
    upload_2015-12-7_11-12-31.png
     
  6. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Webpagetest 3G Mobile Dulles



    The test server is in Los Angeles, so using a worse case situation where test WPT location is on other side of USA in Dulles, VA with 3G Mobile Fast connection speeds and Chrome.

    upload_2015-12-7_11-21-21.png
    upload_2015-12-7_11-21-50.png

    Nginx served page is slightly larger at 8KB vs Caddy 7KB maybe due to some additional headers in Nginx ? Both Nginx and Caddy are set to gzip level 6 compression so maybe differences in gzip implementation ?

    upload_2015-12-7_11-23-18.png

    upload_2015-12-7_11-23-50.png
     
    Last edited: Dec 7, 2015
    • Informative Informative x 1
  7. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5
  8. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Blitz.io Benchmarks 1000 concurrent users



    Blitz.io 1000 user testing results

    For Centmin Mod Nginx HTTP/2 where nginx worker_processes = 2

    upload_2015-12-7_11-38-18.png

    upload_2015-12-7_11-38-48.png

    For Caddy HTTP/2

    upload_2015-12-7_11-39-15.png

    upload_2015-12-7_11-39-36.png
     
    Last edited: Dec 7, 2015
  9. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Blitz.io Benchmarks 2000 concurrent users



    Blitz.io 2000 user testing results

    For Centmin Mod Nginx HTTP/2 where nginx worker_processes = 2

    upload_2015-12-7_11-43-11.png

    upload_2015-12-7_11-43-42.png

    For Caddy HTTP/2

    upload_2015-12-7_11-44-6.png

    upload_2015-12-7_11-44-31.png
     
    Last edited: Dec 7, 2015
    • Informative Informative x 1
  10. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    HTTP/2 Load Testing via h2load client



    Using my Docker based nghttp2 client/library image tool box, I ran a h2load HTTP/2 client tests against Centmin Mod Nginx HTTP/2 on port 443 and Caddy 0.80 on port 445. Sites are using Letsencrypt issued SSL certificates for both Nginx and Caddy servers :)

    h2load tests summary for single 1 h2load threads

    h2load test defaults to using 1 thread for testing, so below results are for single thread based h2load load testing.

    10 concurrent users, 10 requests
    • Centmin Mod Nginx HTTP/2 = 126.76 req/s, 263.42KB/s
    • Caddy HTTP/2 = 95.01 req/s, 181.85KB/s
    100 concurrent users, 100 requests
    • Centmin Mod Nginx HTTP/2 = 536.59 req/s, 1.09MB/s
    • Caddy HTTP/2 = 220.95 req/s, 422.91KB/s
    250 concurrent users, 1000 requests
    • Centmin Mod Nginx HTTP/2 = 2439.27 req/s, 4.86MB/s
    • Caddy HTTP/2 = 1049.99 req/s, 1.89MB/s
    1000 concurrent users, 50000 requests
    • Centmin Mod Nginx HTTP/2 = 15580.02 req/s, 30.90MB/s
    • Caddy HTTP/2 = 6139.60 req/s, 10.89MB/s
    upload_2015-12-10_9-56-45.png

    Baseline ping times from h2load invoked server (Digital Ocean San Francisco KVM VPS) to le12.http2ssl.xyz OpenVZ VPS in Los Angeles

    Code:
    ping -c4 le12.http2ssl.xyz
    PING le12.http2ssl.xyz (104.152.214.220) 56(84) bytes of data.
    64 bytes from 220-214-152-104-static.reverse.queryfoundry.net (104.152.214.220): icmp_seq=1 ttl=53 time=12.2 ms
    64 bytes from 220-214-152-104-static.reverse.queryfoundry.net (104.152.214.220): icmp_seq=2 ttl=53 time=12.1 ms
    64 bytes from 220-214-152-104-static.reverse.queryfoundry.net (104.152.214.220): icmp_seq=3 ttl=53 time=12.1 ms
    64 bytes from 220-214-152-104-static.reverse.queryfoundry.net (104.152.214.220): icmp_seq=4 ttl=53 time=12.1 ms
    
    --- le12.http2ssl.xyz ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3004ms
    rtt min/avg/max/mdev = 12.127/12.176/12.277/0.059 ms

    Centmin Mod Nginx HTTP/2 Tests



    h2load test against my Letsencrypt free SSL enabled Centmin Mod Nginx HTTP/2 server at le12.http2ssl.xyz:443
    Code:
    /usr/local/bin/h2load -c10 -m100 -n10 -v https://le12.http2ssl.xyz:443
    starting benchmark...
    spawning thread #0: 10 total client(s). 10 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 78.89ms, 126.76 req/s, 263.42KB/s
    requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 21280 bytes total, 2290 bytes headers (space savings 14.23%), 18320 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    12.18ms     15.28ms     13.97ms      1.27ms    60.00%
    time for connect:    44.10ms     62.14ms     52.38ms      6.90ms    60.00%
    time to 1st byte:    56.30ms     77.42ms     66.36ms      8.11ms    60.00%
    req/s (client)  :      12.91       17.74       15.28        1.86    60.00%

    Caddy HTTP/2 Tests



    h2load test against my Letsencrypt free SSL enabled Caddy 0.80 HTTP/2 server at le12.http2ssl.xyz:445
    Code:
    /usr/local/bin/h2load -c10 -m100 -n10 -v https://le12.http2ssl.xyz:445
    starting benchmark...
    spawning thread #0: 10 total client(s). 10 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 105.26ms, 95.01 req/s, 181.85KB/s
    requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 19600 bytes total, 740 bytes headers (space savings 54.04%), 18320 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    12.46ms     15.64ms     13.57ms      1.07ms    70.00%
    time for connect:    71.42ms     88.76ms     78.97ms      5.69ms    70.00%
    time to 1st byte:    84.71ms    104.41ms     92.55ms      6.44ms    60.00%
    req/s (client)  :       9.57       11.79       10.85        0.74    60.00%
    note I had updated Centmin Mod Nginx to 1.9.8 too

    h2load 100 user concurrency + 100 requests



    Bumping up concurrency from 10 to 100 for h2load tests

    Centmin Mod Nginx HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c100 -m100 -n100 -v https://le12.http2ssl.xyz:443
    starting benchmark...
    spawning thread #0: 100 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 186.36ms, 536.59 req/s, 1.09MB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 212800 bytes total, 22900 bytes headers (space savings 14.23%), 183200 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    12.82ms     20.79ms     16.61ms      1.63ms    72.00%
    time for connect:    44.65ms    162.87ms    105.29ms     35.55ms    54.00%
    time to 1st byte:    61.19ms    178.16ms    121.91ms     35.14ms    54.00%
    req/s (client)  :       5.61       16.32        9.02        2.97    66.00%
    

    Caddy HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c100 -m100 -n100 -v https://le12.http2ssl.xyz:445
    starting benchmark...
    spawning thread #0: 100 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 452.59ms, 220.95 req/s, 422.91KB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 196000 bytes total, 7400 bytes headers (space savings 54.04%), 183200 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    11.53ms    109.40ms     20.61ms     15.39ms    97.00%
    time for connect:    66.70ms    437.34ms    274.33ms    100.08ms    51.00%
    time to 1st byte:   165.23ms    449.83ms    294.95ms     94.69ms    47.00%
    req/s (client)  :       2.22        6.05        3.79        1.27    49.00%

    h2load 250 user concurrency + 1000 requests



    Bumping up concurrency to 250 + 1000 requests for h2load tests

    Centmin Mod Nginx HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c250 -m100 -n1000 -v https://le12.http2ssl.xyz:443
    starting benchmark...
    spawning thread #0: 250 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 409.96ms, 2439.27 req/s, 4.86MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 2091250 bytes total, 229000 bytes headers (space savings 14.23%), 1832000 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    11.13ms     42.11ms     18.73ms      4.33ms    85.00%
    time for connect:    46.62ms    384.38ms    213.22ms     99.91ms    57.20%
    time to 1st byte:    62.25ms    399.59ms    231.86ms     99.15ms    57.00%
    req/s (client)  :      10.00       64.17       22.20       12.93    82.40%
    

    Caddy HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c250 -m100 -n1000 -v https://le12.http2ssl.xyz:445
    starting benchmark...
    spawning thread #0: 250 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 952.39ms, 1049.99 req/s, 1.89MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 1882750 bytes total, 23750 bytes headers (space savings 85.25%), 1832000 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    11.63ms    242.21ms     25.68ms     18.26ms    93.80%
    time for connect:   134.45ms    928.42ms    610.27ms    203.87ms    49.60%
    time to 1st byte:   338.60ms    944.03ms    635.53ms    199.10ms    50.40%
    req/s (client)  :       4.24       11.80        7.03        2.44    60.80%
    

    h2load 1000 user concurrency + 50000 requests



    Bumping up concurrency to 1000 + 50000 requests for h2load tests

    Centmin Mod Nginx HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://le12.http2ssl.xyz:443 
    starting benchmark...
    spawning thread #0: 1000 total client(s). 50000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 3.21s, 15580.02 req/s, 30.90MB/s
    requests: 50000 total, 50000 started, 50000 done, 50000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 50000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 103999000 bytes total, 11450000 bytes headers (space savings 14.23%), 91600000 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    15.67ms       2.34s       1.05s    476.85ms    69.38%
    time for connect:   532.55ms       2.71s       1.29s    394.21ms    59.00%
    time to 1st byte:   810.47ms       2.97s       1.73s    459.23ms    64.60%
    req/s (client)  :      15.82       30.53       20.55        5.34    70.80%
    

    Caddy HTTP/2 Tests



    Code:
    /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://le12.http2ssl.xyz:445
    starting benchmark...
    spawning thread #0: 1000 total client(s). 50000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 8.14s, 6139.60 req/s, 10.89MB/s
    requests: 50000 total, 50000 started, 50000 done, 50000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 50000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 92953000 bytes total, 417000 bytes headers (space savings 94.82%), 91600000 bytes data
                         min         max         mean         sd        +/- sd
    time for request:    14.51ms       3.08s       1.18s    827.53ms    52.96%
    time for connect:   285.69ms       7.49s       3.86s       2.28s    50.00%
    time to 1st byte:   422.20ms       8.02s       4.66s       2.33s    54.70%
    req/s (client)  :       6.15      103.05       15.66       19.05    90.60%
    
     
    Last edited: Dec 10, 2015
    • Informative Informative x 1
  11. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy Integration Into Centmin Mod LEMP Stack



    Just a bit of a preview of what Caddy server integration into Centmin Mod LEMP Stack might look like :)

    Caddy serving Centmin Mod's main hostname virtualhost off of port 8888
    Code (Text):
    curl -I localhost:8888
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 3801
    Content-Type: text/html; charset=utf-8
    Etag: W/"579b144d-ed9"
    Last-Modified: Fri, 29 Jul 2016 08:31:09 GMT
    Server: Caddy
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-Xss-Protection: 1; mode=block
    Date: Sat, 06 Aug 2016 01:41:41 GMT

    Caddy commands
    Code (Text):
    /root/tools/caddy.sh              
    
    Command Usage:
    /root/tools/caddy.sh {install|update|uninstall|addsite|start|stop|restart|status|domains|certs|version|help}

    Caddy status, stop, start and restart commands
    Code (Text):
    /root/tools/caddy.sh status
    caddy is running (4991)
    
    /root/tools/caddy.sh stop
    
    /root/tools/caddy.sh status
    caddy is not running
    
    /root/tools/caddy.sh start
    
    /root/tools/caddy.sh status
    caddy is running (5084)

    Caddy addsite function
    Code (Text):
    /root/tools/caddy.sh addsite      
    
    enter domain name wihtout www.: newdomain.com
    
    /home/nginx/domains/newdomain.com/public directory does not exist
    
    --------------------------------------------------------------------
    caddy.sh was written for centminmod.com LEMP stack environments and
    requires a Centmin Mod Nginx generated vhost directory to work as
    Caddy HTTP vhost domains are setup on port 8888 and Caddy HTTPS
    vhost domains are setup on port 448 leaving Centmin Mod Nginx to run
    on HTTP port 80 and HTTPS port 443
    --------------------------------------------------------------------

    Generating Caddy site vhosts via addsite with self-signed ssl certs
    Code (Text):
    /root/tools/caddy.sh addsite      
    
    enter domain name wihtout www.: domain1.com
    
    /etc/caddy/conf.d/domain1.com.conf created on 8888
    /etc/caddy/conf.d/domain1.com.ssl.conf created on 448
    

    Code (Text):
    curl -I http://domain1.com:8888
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1399
    Content-Type: text/html; charset=utf-8
    Etag: W/"57a542e8-577"
    Last-Modified: Sat, 06 Aug 2016 01:52:40 GMT
    Server: Caddy
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-Xss-Protection: 1; mode=block
    Date: Sat, 06 Aug 2016 05:33:38 GMT

    Code (Text):
    curl -Ik https://domain1.com:448
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1399
    Content-Type: text/html; charset=utf-8
    Etag: W/"57a542e8-577"
    Last-Modified: Sat, 06 Aug 2016 01:52:40 GMT
    Server: Caddy
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-Xss-Protection: 1; mode=block
    Date: Sat, 06 Aug 2016 05:33:41 GMT
    
     
    Last edited: Aug 6, 2016
  12. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy Integration Revised - caddy.sh



    Updated caddy.sh integration script so now addsite option can automatically generate the site domain vhost for both Caddy and Nginx servers simultaneously - including issuing of self-signed SSL certificates + Letsencrypt SSL certificates for both Caddy and Nginx to run HTTP/2 based HTTPS sites and generation of pure-ftpd virtual FTP user which can access the web root for site for both Caddy and Nginx servers :)

    Web Server Binary Sizes
    Code (Text):
    ls -lah $(which nginx)
    -rwxr-xr-x 1 root root 3.1M Aug  6 15:44 /usr/local/sbin/nginx
    
    ls -lah $(which caddy)
    -rwxrwxr-x 1 nginx nginx 29M Aug  5 04:08 /usr/local/bin/caddy
    


    Command Usage options
    Code (Text):
    /root/tools/caddy/caddy.sh
    
    Command Usage:
    /root/tools/caddy/caddy.sh {install|update|uninstall|addsite|start|stop|restart|status|domains|certs|version|help}

    Install run on CentOS 7.2 64bit Virtualbox Instance
    Code (Text):
    /root/tools/caddy/caddy.sh install
    Downloading Caddy for linux/amd64...
    https://caddyserver.com/download/build?os=linux&arch=amd64&arm=&features=filemanager%2Cgit%2Chugo%2Cipfilter%2Clocale%2Cmailout%2Cminify%2Cprometheus%2Cratelimit%2Crealip%2Csearch%2Ccloudflare%2Cdigitalocean%2Cgooglecloud%2Cnamecheap%2Croute53%2Cvultr
    Extracting...
    Putting caddy in /usr/local/bin (may require password)
    Caddy 0.9.0
    Successfully installed
    Server types:
      http
    
    Caddyfile loaders:
      short
      flag
      default
    
    Other plugins:
      http.basicauth
      http.bind
      http.browse
      http.errors
      http.expvar
      http.ext
      http.fastcgi
      http.filemanager
      http.git
      http.gzip
      http.header
      http.hugo
      http.internal
      http.ipfilter
      http.locale
      http.log
      http.mailout
      http.markdown
      http.mime
      http.minify
      http.pprof
      http.prometheus
      http.proxy
      http.ratelimit
      http.realip
      http.redir
      http.rewrite
      http.root
      http.search
      http.templates
      http.websocket
      shutdown
      startup
      tls
      tls.dns.cloudflare
      tls.dns.digitalocean
      tls.dns.googlecloud
      tls.dns.namecheap
      tls.dns.route53
      tls.dns.vultr
    

    Starting Caddy via caddy.sh and checking status
    Code (Text):
    /root/tools/caddy/caddy.sh status
    caddy is not running

    Code (Text):
    /root/tools/caddy/caddy.sh start
    /root/tools/caddy/caddy.sh status
    caddy is running (3070)
    -----------------------------------------------
    Max processes     8192     8192     processes
    Max open files    262144   262144   files
    -----------------------------------------------

    Code (Text):
    ps aufxw | grep caddy | grep -v grep
    root      3070  0.0  0.4  24552  7864 pts/2    Sl   19:16   0:00 /usr/local/bin/caddy -log /usr/local/nginx/logs/caddy.log -quiet=true -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp -pidfile=/var/run/caddy/caddy.pid


    caddy.sh addsite usage



    Simultaneous Caddy + Nginx vhost generation with self-signed SSL certificates for both servers for domain1.com testing. This will allow me to properly do side by side comparison benchmarks of Caddy and Centmin Mod's Nginx server :D
    Code (Text):
    /root/tools/caddy/caddy.sh addsite
    
    Add a new Caddy site domain vhost
    Enter domain name wihtout www.: domain1.com
    
    Caddy integration into Centmin Mod LEMP runs Nginx on port 80 & 443
    Caddy server runs on 8888 & 448
    Do you want to create the Nginx vhost for domain1.com as well ? [y/n]: y
    Create self-signed SSL cert for Nginx vhost domain1.com ? recommended [y/n]: y
    
    -------------------------------------------------------------
    Generating Nginx vhost for domain1.com
    -------------------------------------------------------------
    
    -------------------------------------------------------------
    /usr/bin/nv -d domain1.com -s y -u ftpusername
    -------------------------------------------------------------
    
    ---------------------------------------------------------------
    Nginx Vhost Setup...
    ---------------------------------------------------------------
    
    
    FTP password auto generated: **************
    
    Password:
    Enter it again:
    ---------------------------------------------------------------
    SSL Vhost Setup...
    ---------------------------------------------------------------
    
    ---------------------------------------------------------------
    Generating self signed SSL certificate...
    CSR file can also be used to be submitted for paid SSL certificates
    If using for paid SSL certificates be sure to keep both private key and CSR safe
    creating CSR File: domain1.com.csr
    creating private key: domain1.com.key
    creating self-signed SSL certificate: domain1.com.crt
    Generating a 2048 bit RSA private key
    ......................................................................+++
    ........................+++
    writing new private key to 'domain1.com.key'
    -----
    No value provided for Subject Attribute C, skipped
    No value provided for Subject Attribute ST, skipped
    No value provided for Subject Attribute L, skipped
    Signature ok
    subject=/O=domain1.com/OU=domain1.com/CN=domain1.com
    Getting Private key
    
    ---------------------------------------------------------------
    Generating backup CSR and private key for HTTP Public Key Pinning...
    creating CSR File: domain1.com-backup.csr
    creating private key: domain1.com-backup.key
    Generating a 2048 bit RSA private key
    ..................................................................................................+++
    ..............+++
    writing new private key to 'domain1.com-backup.key'
    -----
    No value provided for Subject Attribute C, skipped
    No value provided for Subject Attribute ST, skipped
    No value provided for Subject Attribute L, skipped
    
    ---------------------------------------------------------------
    Extracting Base64 encoded information for primary and secondary
    private key's SPKI - Subject Public Key Information
    Primary private key - domain1.com.key
    Backup private key - domain1.com-backup.key
    For HPKP - HTTP Public Key Pinning hash generation...
    
    extracting SPKI Base64 encoded hash for primary private key = domain1.com.key ...
    writing RSA key
    urcdM0QCH7avbVN0dvPZG70I7Y6nWkTpcQ0V7af0+2s=
    
    extracting SPKI Base64 encoded hash for backup private key = domain1.com-backup.key ...
    writing RSA key
    fpNNHgiU21nnUXd5ZLMaew0MjEcDXedaezrQzSGHLHg=
    
    HTTP Public Key Pinning Header for Nginx
    
    for 7 days max-age including subdomains
    
    add_header Public-Key-Pins 'pin-sha256="urcdM0QCH7avbVN0dvPZG70I7Y6nWkTpcQ0V7af0+2s="; pin-sha256="fpNNHgiU21nnUXd5ZLMaew0MjEcDXedaezrQzSGHLHg="; max-age=86400; includeSubDomains';
    
    for 7 days max-age excluding subdomains
    
    add_header Public-Key-Pins 'pin-sha256="urcdM0QCH7avbVN0dvPZG70I7Y6nWkTpcQ0V7af0+2s="; pin-sha256="fpNNHgiU21nnUXd5ZLMaew0MjEcDXedaezrQzSGHLHg="; max-age=86400';
    
    ---------------------------------------------------------------
    Generating dhparam.pem file - can take a few minutes...
    Generating DH parameters, 2048 bit long safe prime, generator 2
    This is going to take a long time
    .........+..............++*++*
    dhparam file generation time: 113.662816389
    
    -------------------------------------------------------------
    
    -------------------------------------------------------------
    /usr/local/src/centminmod/tools/autoprotect.sh
    generated nginx include file: /usr/local/nginx/conf/autoprotect/domain1.com/autoprotect-domain1.com.conf
    
    autoprotect.sh run completed...
    
    Restarting nginx (via systemctl):  [  OK  ]
    Restarting nginx (via systemctl):  [  OK  ]
    systemctl restart pure-ftpd.service
    
    -------------------------------------------------------------
    FTP hostname : IPADDR
    FTP port : 21
    FTP mode : FTP (explicit SSL)
    FTP Passive (PASV) : ensure is checked/enabled
    FTP username created for domain1.com : ftpusername
    FTP password created for domain1.com : **************
    -------------------------------------------------------------
    vhost for domain1.com created successfully
    
    domain: http://domain1.com
    vhost conf file for domain1.com created: /usr/local/nginx/conf/conf.d/domain1.com.conf
    
    vhost ssl for domain1.com created successfully
    
    domain: https://domain1.com
    vhost ssl conf file for domain1.com created: /usr/local/nginx/conf/conf.d/domain1.com.ssl.conf
    /usr/local/nginx/conf/ssl_include.conf created
    Self-signed SSL Certificate: /usr/local/nginx/conf/ssl/domain1.com/domain1.com.crt
    SSL Private Key: /usr/local/nginx/conf/ssl/domain1.com/domain1.com.key
    SSL CSR File: /usr/local/nginx/conf/ssl/domain1.com/domain1.com.csr
    Backup SSL Private Key: /usr/local/nginx/conf/ssl/domain1.com/domain1.com-backup.key
    Backup SSL CSR File: /usr/local/nginx/conf/ssl/domain1.com/domain1.com-backup.csr
    
    upload files to /home/nginx/domains/domain1.com/public
    vhost log files directory is /home/nginx/domains/domain1.com/log
    
    -------------------------------------------------------------
    Current vhost listing at: /usr/local/nginx/conf/conf.d/
    
        
    Aug 6   15:44   845    ssl.conf
    Aug 6   15:44   1.1K   demodomain.com.conf
    Aug 6   15:52   1.6K   virtual.conf
    Aug 6   19:20   2.0K   domain1.com.conf
    Aug 6   19:20   3.8K   domain1.com.ssl.conf
    
    -------------------------------------------------------------
    Current vhost ssl files listing at: /usr/local/nginx/conf/ssl/domain1.com
    
        
    Aug 6   19:18   1.7K   domain1.com.key
    Aug 6   19:18   952    domain1.com.csr
    Aug 6   19:18   1.1K   domain1.com.crt
    Aug 6   19:18   1.7K   domain1.com-backup.key
    Aug 6   19:18   952    domain1.com-backup.csr
    Aug 6   19:18   45     hpkp-info-primary-pin.txt
    Aug 6   19:18   45     hpkp-info-secondary-pin.txt
    Aug 6   19:20   424    dhparam.pem
    
    -------------------------------------------------------------
    Commands to remove domain1.com
    
    pure-pw userdel ftpusername
    rm -rf /usr/local/nginx/conf/conf.d/domain1.com.conf
    rm -rf /usr/local/nginx/conf/conf.d/domain1.com.ssl.conf
    rm -rf /usr/local/nginx/conf/ssl/domain1.com/domain1.com.crt
    rm -rf /usr/local/nginx/conf/ssl/domain1.com/domain1.com.key
    rm -rf /usr/local/nginx/conf/ssl/domain1.com/domain1.com.csr
    rm -rf /usr/local/nginx/conf/ssl/domain1.com
    rm -rf /home/nginx/domains/domain1.com
    service nginx restart
    
    -------------------------------------------------------------
    vhost for domain1.com setup successfully
    domain1.com setup info log saved at:
    /root/centminlogs/centminmod_060816-191811_nginx_addvhost_nv.log
    -------------------------------------------------------------
    
    
    /etc/caddy/conf.d/domain1.com.conf created on 8888
    /etc/caddy/conf.d/domain1.com.ssl.conf created on 448
    
    caddy restarted and is running (3479)
    -----------------------------------------------
    Max processes     8192     8192     processes
    Max open files    262144   262144   files
    -----------------------------------------------

    curl domain1.com headers for Caddy and Nginx HTTP and HTTPS sites

    For HTTP on port 80 and port 8888 for Nginx and Caddy respectively
    Code (Text):
    curl -I http://domain1.com
    HTTP/1.1 200 OK
    Date: Sat, 06 Aug 2016 19:24:33 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 1362
    Last-Modified: Sat, 06 Aug 2016 19:18:11 GMT
    Connection: keep-alive
    ETag: "57a637f3-552"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Sun, 07 Aug 2016 19:24:33 GMT
    Cache-Control: max-age=86400
    Cache-Control: public, must-revalidate, proxy-revalidate
    Accept-Ranges: bytes
    

    Code (Text):
    curl -I http://domain1.com:8888
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1362
    Content-Type: text/html; charset=utf-8
    Etag: W/"57a637f3-552"
    Last-Modified: Sat, 06 Aug 2016 19:18:11 GMT
    Server: Caddy
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-Powered-By: Caddy via CentminMod
    X-Xss-Protection: 1; mode=block
    Date: Sat, 06 Aug 2016 19:24:36 GMT
    

    For HTTPS on port 443 and port 448 for Nginx and Caddy respectively
    Code (Text):
    curl -Ik https://domain1.com:443
    HTTP/1.1 200 OK
    Date: Sat, 06 Aug 2016 19:24:46 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 1362
    Last-Modified: Sat, 06 Aug 2016 19:18:11 GMT
    Connection: keep-alive
    ETag: "57a637f3-552"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Sun, 07 Aug 2016 19:24:46 GMT
    Cache-Control: max-age=86400
    Cache-Control: public, must-revalidate, proxy-revalidate
    Accept-Ranges: bytes
    

    Code (Text):
    curl -Ik https://domain1.com:448
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 1362
    Content-Type: text/html; charset=utf-8
    Etag: W/"57a637f3-552"
    Last-Modified: Sat, 06 Aug 2016 19:18:11 GMT
    Server: Caddy
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-Powered-By: Caddy via CentminMod
    X-Xss-Protection: 1; mode=block
    Date: Sat, 06 Aug 2016 19:24:51 GMT
    


    HTTPS over HTTP/2 curl'd headers
    Code (Text):
    curl -Ik https://domain1.com:443
    HTTP/2 200
    date: Sat, 06 Aug 2016 19:30:23 GMT
    content-type: text/html; charset=utf-8
    content-length: 1362
    last-modified: Sat, 06 Aug 2016 19:18:11 GMT
    etag: "57a637f3-552"
    server: nginx centminmod
    x-powered-by: centminmod
    expires: Sun, 07 Aug 2016 19:30:23 GMT
    cache-control: max-age=86400
    cache-control: public, must-revalidate, proxy-revalidate
    accept-ranges: bytes

    Code (Text):
    curl -Ik https://domain1.com:448
    HTTP/2 200
    accept-ranges: bytes
    cache-control: max-age=86400
    content-type: text/html; charset=utf-8
    etag: W/"57a637f3-552"
    last-modified: Sat, 06 Aug 2016 19:18:11 GMT
    server: Caddy
    x-content-type-options: nosniff
    x-frame-options: SAMEORIGIN
    x-powered-by: Caddy via CentminMod
    x-xss-protection: 1; mode=block
    content-length: 1362
    date: Sat, 06 Aug 2016 19:30:28 GMT


    Server Version Info



    Note Nginx is setup out of box to handle 2 worker_processes on the 4 cpu VirtualBox CentOS 7.2 guest server. While I believe Caddy is setup to utilise all 4 cpus.
    Code (Text):
    user              nginx nginx;
    worker_processes 2;
    worker_priority -10;
    


    Code (Text):
    caddy -version
    Caddy 0.9.0

    Code (Text):
    caddy -plugins
    Server types:
      http
    
    Caddyfile loaders:
      short
      flag
      default
    
    Other plugins:
      http.basicauth
      http.bind
      http.browse
      http.errors
      http.expvar
      http.ext
      http.fastcgi
      http.filemanager
      http.git
      http.gzip
      http.header
      http.hugo
      http.internal
      http.ipfilter
      http.locale
      http.log
      http.mailout
      http.markdown
      http.mime
      http.minify
      http.pprof
      http.prometheus
      http.proxy
      http.ratelimit
      http.realip
      http.redir
      http.rewrite
      http.root
      http.search
      http.templates
      http.websocket
      shutdown
      startup
      tls
      tls.dns.cloudflare
      tls.dns.digitalocean
      tls.dns.googlecloud
      tls.dns.namecheap
      tls.dns.route53
      tls.dns.vultr

    Nginx & Caddy HTTPS Vhost Files



    As I am more concerned with HTTP/2 HTTPS performance, l'll focus on HTTPS vhost config files for both Nginx and Caddy. Nginx one defaults to LibreSSL 2.4.2 with chacha20_poly1305 as higher preference or can be switched to OpenSSL 1.0.2h with Cloudflare patches for chacha20 which only serve chacha20 over mobile. I'll edit this to prefer AES to match Caddy.

    Caddy HTTP/2 HTTPS Vhost at /etc/caddy/conf.d/domain1.com.ssl.conf auto generated via caddy.sh addsite
    Code (Text):
    domain1.com:448 {
        gzip {
            level 5
            min_length 1400
        }
        #browse
        header / {
            #Strict-Transport-Security "max-age=31536000"
            Cache-Control "max-age=86400"
            X-Content-Type-Options "nosniff"
            X-Frame-Options "SAMEORIGIN"
            X-XSS-Protection "1; mode=block"
            X-Powered-By "Caddy via CentminMod"
            #-Server
        }
        tls self_signed {
            protocols tls1.0 tls1.2
            ciphers ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-CBC-SHA ECDHE-RSA-AES256-CBC-SHA ECDHE-ECDSA-AES256-CBC-SHA ECDHE-ECDSA-AES128-CBC-SHA RSA-AES128-CBC-SHA RSA-AES256-CBC-SHA ECDHE-RSA-3DES-EDE-CBC-SHA RSA-3DES-EDE-CBC-SHA
        }
        root /home/nginx/domains/domain1.com/public
        fastcgi / 127.0.0.1:9000 {
            ext   .php
            split .php
            index index.php
        }
        # for wordpress
        #rewrite {
        #    if {path} not_match ^\/wp-admin
        #    to {path} {path}/ /index.php?_url={uri}
        #}
        errors {
          log /home/nginx/domains/domain1.com/log/caddy-domain1.com-errors.log {
            size 100 # Rotate after 50 MB
            age  14 # Keep rotated files for 30 days
            keep 10 # Keep at most 5 log files
          }
          #404 404.html # Not Found
          #500 500.html # Internal Server Error
        }
        log / /home/nginx/domains/domain1.com/log/caddy-domain1.com-access.ssl.log "{remote} {when} {method} {uri} {proto} {status} {size} {>User-Agent} {latency}" {
            rotate {
                    size 100 # Rotate after 100 MB
                    age  14  # Keep log files for 14 days
                    keep 10  # Keep at most 10 log files
            }
        }
    }
    

    Nginx HTTP/2 HTTPS Vhost at /usr/local/nginx/conf/conf.d/domain1.com.ssl.conf auto generated via caddy.sh addsite. Include sites are added by default for various Nginx security and performance features including the autoprotect generated file, site maintenance status mode, nginx vhost traffic stats include and optional cloudflare realip + csf firewall whitelisting. Nginx ngx_pagespeed is disabled for Nginx vhost as well.

    FYI, Centmin Mod Nginx's ssl_buffer_size is set to 1369 due to integration of Cloudflare's dynamic TLS record size patch.
    Code (Text):
    # Centmin Mod Getting Started Guide
    # must read http://centminmod.com/getstarted.html
    # For HTTP/2 SSL Setup
    # read http://centminmod.com/nginx_configure_https_ssl_spdy.html
    
    # redirect from www to non-www  forced SSL
    # uncomment, save file and restart Nginx to enable
    # if unsure use return 302 before using return 301
    # server {
    #   server_name domain1.com www.domain1.com;
    #    return 302 https://$server_name$request_uri;
    # }
    
    server {
      listen 443 ssl http2;
      server_name domain1.com www.domain1.com;
    
      ssl_dhparam /usr/local/nginx/conf/ssl/domain1.com/dhparam.pem;
      ssl_certificate      /usr/local/nginx/conf/ssl/domain1.com/domain1.com.crt;
      ssl_certificate_key  /usr/local/nginx/conf/ssl/domain1.com/domain1.com.key;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      http2_max_field_size 16k;
      http2_max_header_size 32k;
      # mozilla recommended
      ssl_ciphers EECDH+CHACHA20:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+ECDSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+SHA384:EECDH+AES128:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!CAMELLIA;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
      # HTTP Public Key Pinning Header uncomment only one that applies include or exclude domains.
      # You'd want to include subdomains if you're using SSL wildcard certificates
      # include subdomain
      #add_header Public-Key-Pins 'pin-sha256="urcdM0QCH7avbVN0dvPZG70I7Y6nWkTpcQ0V7af0+2s="; pin-sha256="fpNNHgiU21nnUXd5ZLMaew0MjEcDXedaezrQzSGHLHg="; max-age=86400; includeSubDomains';
      # exclude subdomains
      #add_header Public-Key-Pins 'pin-sha256="urcdM0QCH7avbVN0dvPZG70I7Y6nWkTpcQ0V7af0+2s="; pin-sha256="fpNNHgiU21nnUXd5ZLMaew0MjEcDXedaezrQzSGHLHg="; max-age=86400';
      #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
      #add_header X-Frame-Options SAMEORIGIN;
      #add_header X-Xss-Protection "1; mode=block" always;
      #add_header X-Content-Type-Options "nosniff" always;
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
      # enable ocsp stapling
      #resolver 8.8.8.8 8.8.4.4 valid=10m;
      #resolver_timeout 10s;
      #ssl_stapling on;
      #ssl_stapling_verify on;
      #ssl_trusted_certificate /usr/local/nginx/conf/ssl/domain1.com/domain1.com-trusted.crt;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/domain1.com/log/access.log main_ext buffer=256k flush=60m;
      error_log /home/nginx/domains/domain1.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/domain1.com/autoprotect-domain1.com.conf;
      root /home/nginx/domains/domain1.com/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
    
    # block common exploits, sql injections etc
    #include /usr/local/nginx/conf/block.conf;
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # Shows file listing times as local time
      #autoindex_localtime on;
    
      # Enable for vBulletin usage WITHOUT vbSEO installed
      # More example Nginx vhost configurations at
      # http://centminmod.com/nginx_configure.html
      #try_files    $uri $uri/ /index.php;
    
      }
    
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/php.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    


    SSL Ciphers



    Seems Caddy defaults to ECC 256 bit ECDSA based SSL certificates for self-signed option. While Centmin Mod Nginx defaults to RSA 2048bit standard SSL certificates for self-signed option. ECDSA is more performant than RSA 2048bit so will see how things fair. Caddy also exchange defaults to AES128 vs Centmin Mod Nginx AES256.

    caddy-domain1.com-opera39-devtools-01.png

    cmm-nginx-domain1.com-opera39-devtools-01.png

    caddy-domain1.com-opera39-devtools-00.png

    cmm-nginx-domain1.com-opera39-devtools-00.png

    nghttp2 tests



    Using nghttp2 HTTP/2 library/client for HTTP/2 tests.

    Centmin Mod Nginx HTTP/2 on port 443
    Code (Text):
    nghttp -nas https://domain1.com:443
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
    requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
    13      +521us        +91us    429us  200   1K /
    
    

    Caddy HTTP/2 on port 448
    Code (Text):
    nghttp -nas https://domain1.com:448
    ***** Statistics *****
    
    Request timing:
      responseEnd: the  time  when  last  byte of  response  was  received
                   relative to connectEnd
    requestStart: the time  just before  first byte  of request  was sent
                   relative  to connectEnd.   If  '*' is  shown, this  was
                   pushed by server.
          process: responseEnd - requestStart
             code: HTTP status code
             size: number  of  bytes  received as  response  body  without
                   inflation.
              URI: request URI
    
    see http://www.w3.org/TR/resource-timing/#processing-model
    
    sorted by 'complete'
    
    id  responseEnd requestStart  process code size request path
    13     +5.73ms       +247us   5.48ms  200   1K /
    
    


    Cipherscan Tests



    Centmin Mod Nginx HTTP/2 on port 443
    Code (Text):
    cipherscan https://domain1.com:443 
    ............
    Target: domain1.com:443
    
    prio  ciphersuite                  protocols              pfs                 curves
    1     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
    2     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
    3     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
    4     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
    5     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
    
    Certificate: untrusted, 2048 bits, sha256WithRSAEncryption signature
    TLS ticket lifetime hint: 3600
    OCSP stapling: not supported
    Cipher ordering: server
    Curves ordering: server - fallback: no
    Server supports secure renegotiation
    Server supported compression methods: NONE
    TLS Tolerance: yes
    

    Caddy HTTP/2 on port 448
    Code (Text):
    cipherscan https://domain1.com:448
    ........
    Target: domain1.com:448
    
    prio  ciphersuite                    protocols              pfs                 curves
    1     ECDHE-ECDSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
    2     ECDHE-ECDSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
    3     ECDHE-ECDSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
    
    Certificate: untrusted, 256 bits, ecdsa-with-SHA256 signature
    TLS ticket lifetime hint: None
    OCSP stapling: not supported
    Cipher ordering: server
    Curves ordering: server - fallback: no
    Server supports secure renegotiation
    Server supported compression methods: NONE
    TLS Tolerance: yes
    


    Other HTTP/2 Checks



    Code (Text):
    is-http2 https://domain1.com:443
    ✓ HTTP/2 supported by https://domain1.com:443
    Supported protocols: h2 http/1.1
    

    Code (Text):
    is-http2 https://domain1.com:448
    ✓ HTTP/2 supported by https://domain1.com:448
    Supported protocols: h2 http/1.1


    Check for ALPN protocol
    Code (Text):
    echo | /usr/local/http2-15/bin/openssl s_client -alpn h2 -host domain1.com -port 443
    
    New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    ALPN protocol: h2
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    


    Code (Text):
    echo | /usr/local/http2-15/bin/openssl s_client -alpn h2 -host domain1.com -port 448
    
    New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES128-GCM-SHA256
    Server public key is 256 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    ALPN protocol: h2
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-ECDSA-AES128-GCM-SHA256
    


    h2load HTTP/2 Load Testing



    Using nghttp2 bundled h2load HTTP/2 load tester h2load - HTTP/2 benchmarking tool - HOW-TO — nghttp2 1.14.0-DEV documentation
    Code (Text):
    h2load --version
    h2load nghttp2/1.14.0-DEV
    


    Low concurrency tests at 10 concurrent users and 100 requests
    • Caddy 0.9 HTTP/2 HTTPS = finished in 55.63ms, 1797.46 req/s, 2.43MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS = finished in 25.92ms, 3857.58 req/s, 5.99MB/s
    Caddy 0.9 HTTP/2 HTTPS
    Code (Text):
    /usr/local/bin/h2load -c10 -m100 -n100 -v https://domain1.com:448
    starting benchmark...
    spawning thread #0: 10 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 55.63ms, 1797.46 req/s, 2.43MB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 138.33KB (141650) total, 3.21KB (3290) headers (space savings 90.00%), 133.01KB (136200) data
                         min         max         mean         sd        +/- sd
    time for request:     5.01ms     40.86ms     25.26ms     10.62ms    65.00%
    time for connect:     6.12ms     14.98ms     11.86ms      2.31ms    80.00%
    time to 1st byte:    11.44ms     41.38ms     30.79ms      8.71ms    70.00%
    req/s           :     184.33      828.90      292.50      192.24    90.00%
    

    Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS
    Code (Text):
    /usr/local/bin/h2load -c10 -m100 -n100 -v https://domain1.com:443
    starting benchmark...
    spawning thread #0: 10 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 25.92ms, 3857.58 req/s, 5.99MB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 158.88KB (162690) total, 23.63KB (24200) headers (space savings 27.76%), 133.01KB (136200) data
                         min         max         mean         sd        +/- sd
    time for request:      333us      7.70ms      4.79ms      1.98ms    66.00%
    time for connect:     9.23ms     21.85ms     14.42ms      3.90ms    60.00%
    time to 1st byte:    15.33ms     22.19ms     18.97ms      2.18ms    60.00%
    req/s           :     436.26      604.38      512.96       50.95    50.00%
    


    Higher concurrency tests at 100 concurrent users and 1000 requests
    • Caddy 0.9 HTTP/2 HTTPS = finished in 324.30ms, 3083.56 req/s, 4.17MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS = finished in 228.77ms, 4371.15 req/s, 6.78MB/s
    Caddy 0.9 HTTP/2 HTTPS
    Code (Text):
    /usr/local/bin/h2load -c100 -m100 -n1000 -v https://domain1.com:448
    starting benchmark...
    spawning thread #0: 100 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 324.30ms, 3083.56 req/s, 4.17MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 1.35MB (1417029) total, 32.65KB (33429) headers (space savings 89.84%), 1.30MB (1362000) data
                         min         max         mean         sd        +/- sd
    time for request:    20.81ms    254.72ms    139.09ms     46.84ms    72.30%
    time for connect:    58.87ms    216.23ms     90.95ms     20.44ms    80.00%
    time to 1st byte:    93.30ms    309.86ms    192.95ms     41.66ms    74.00%
    req/s           :      31.34      106.64       37.83        9.49    97.00%
    

    Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS
    Code (Text):
    /usr/local/bin/h2load -c100 -m100 -n1000 -v https://domain1.com:443
    starting benchmark...
    spawning thread #0: 100 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 228.77ms, 4371.15 req/s, 6.78MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 1.55MB (1626900) total, 236.33KB (242000) headers (space savings 27.76%), 1.30MB (1362000) data
                         min         max         mean         sd        +/- sd
    time for request:      324us     61.89ms     38.38ms      8.09ms    85.60%
    time for connect:    58.44ms    174.41ms    140.73ms     22.08ms    61.00%
    time to 1st byte:   120.04ms    221.25ms    178.70ms     26.93ms    57.00%
    req/s           :      44.99       82.82       57.00        9.04    64.00%
    


    Caddy has a performance overhead issue due to header middleware used for serving headers. The more headers you add to your site, the lower the performance you get see Any performance overhead as you add more headers under HTTP/2? - Help - Caddy Community

    So to confirm this I edited Caddy domain1.com HTTPS vhost to remove all the added headers to see re-test performance and see what the numbers give.

    so Caddy vhost for domain1.com headers changes from
    Code (Text):
        header / {
            #Strict-Transport-Security "max-age=31536000"
            Cache-Control "max-age=86400"
            X-Content-Type-Options "nosniff"
            X-Frame-Options "SAMEORIGIN"
            X-XSS-Protection "1; mode=block"
            X-Powered-By "Caddy via CentminMod"
            #-Server
        }

    to which leaves just the default server header in play
    Code (Text):
        header / {
            #Strict-Transport-Security "max-age=31536000"
            #Cache-Control "max-age=86400"
            #X-Content-Type-Options "nosniff"
            #X-Frame-Options "SAMEORIGIN"
            #X-XSS-Protection "1; mode=block"
            #X-Powered-By "Caddy via CentminMod"
            #-Server
        }


    Low concurrency tests at 10 concurrent users and 100 requests
    • Caddy 0.9 HTTP/2 HTTPS with headers = finished in 55.63ms, 1797.46 req/s, 2.43MB/s
    • Caddy 0.9 HTTP/2 HTTPS with without headers = finished in 48.52ms, 2060.92 req/s, 2.75MB/s
    Code (Text):
    /usr/local/bin/h2load -c10 -m100 -n100 -v https://domain1.com:448
    starting benchmark...
    spawning thread #0: 10 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 48.52ms, 2060.92 req/s, 2.75MB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 136.81KB (140090) total, 1.69KB (1730) headers (space savings 90.80%), 133.01KB (136200) data
                         min         max         mean         sd        +/- sd
    time for request:     5.70ms     37.94ms     25.76ms      8.73ms    71.00%
    time for connect:     4.89ms     14.10ms      9.72ms      2.39ms    80.00%
    time to 1st byte:    10.64ms     41.93ms     29.57ms      8.59ms    80.00%
    req/s           :     213.41      920.36      303.60      206.22    90.00%
    

    Higher concurrency tests at 100 concurrent users and 1000 requests
    • Caddy 0.9 HTTP/2 HTTPS with headers = finished in 324.30ms, 3083.56 req/s, 4.17MB/s
    • Caddy 0.9 HTTP/2 HTTPS without headers = finished in 303.17ms, 3298.46 req/s, 4.41MB/s
    Code (Text):
    /usr/local/bin/h2load -c100 -m100 -n1000 -v https://domain1.com:448
    starting benchmark...
    spawning thread #0: 100 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 303.17ms, 3298.46 req/s, 4.41MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 1.34MB (1400900) total, 16.89KB (17300) headers (space savings 90.80%), 1.30MB (1362000) data
                         min         max         mean         sd        +/- sd
    time for request:    11.45ms    238.16ms    172.13ms     43.30ms    81.20%
    time for connect:    55.90ms     94.26ms     81.40ms     14.52ms    71.00%
    time to 1st byte:    88.59ms    285.75ms    226.33ms     47.72ms    82.00%
    req/s           :      33.30      112.68       37.89       12.44    96.00%
    
     
    Last edited: Aug 7, 2016
    • Informative Informative x 1
  13. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5
    Note previous post tests with Centmin Mod Nginx is setup out of box to handle 2 worker_processes on the 4 cpu VirtualBox CentOS 7.2 guest server. While I believe Caddy is setup to utilise all 4 cpus.
    Code (Text):
    user              nginx nginx;
    worker_processes 2;
    worker_priority -10;
    

    Retesting with Centmin Mod Nginx set to 4 worker_processes now. Seeing as it's 4 cpu threads not 4 real cpu cores, the bump from 2 to 4 only meant a marginal increase.

    Low concurrency tests at 10 concurrent users and 100 requests
    • Caddy 0.9 HTTP/2 HTTPS with headers = finished in 55.63ms, 1797.46 req/s, 2.43MB/s
    • Caddy 0.9 HTTP/2 HTTPS with without headers = finished in 48.52ms, 2060.92 req/s, 2.75MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS (2 cpus) = finished in 25.92ms, 3857.58 req/s, 5.99MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS (4 cpus) = finished in 22.39ms, 4465.68 req/s, 6.93MB/s
    Code (Text):
    /usr/local/bin/h2load -c10 -m100 -n100 -v https://domain1.com:443
    starting benchmark...
    spawning thread #0: 10 total client(s). 100 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 22.39ms, 4465.68 req/s, 6.93MB/s
    requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 158.88KB (162690) total, 23.63KB (24200) headers (space savings 27.76%), 133.01KB (136200) data
                         min         max         mean         sd        +/- sd
    time for request:     1.55ms      3.93ms      3.03ms       690us    68.00%
    time for connect:    11.39ms     18.24ms     13.94ms      2.41ms    70.00%
    time to 1st byte:    14.33ms     20.01ms     16.89ms      1.96ms    60.00%
    req/s           :     475.41      675.08      585.28       69.79    60.00%

    Higher concurrency tests at 100 concurrent users and 1000 requests
    • Caddy 0.9 HTTP/2 HTTPS with headers = finished in 324.30ms, 3083.56 req/s, 4.17MB/s
    • Caddy 0.9 HTTP/2 HTTPS without headers = finished in 303.17ms, 3298.46 req/s, 4.41MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS (2 cpus) = finished in 228.77ms, 4371.15 req/s, 6.78MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS (4 cpus) = finished in 195.44ms, 5116.69 req/s, 7.94MB/s
    Code (Text):
    /usr/local/bin/h2load -c100 -m100 -n1000 -v https://domain1.com:443
    starting benchmark...
    spawning thread #0: 100 total client(s). 1000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 195.44ms, 5116.69 req/s, 7.94MB/s
    requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 1.55MB (1626900) total, 236.33KB (242000) headers (space savings 27.76%), 1.30MB (1362000) data
                         min         max         mean         sd        +/- sd
    time for request:    19.62ms     67.80ms     30.76ms      8.76ms    71.30%
    time for connect:    66.91ms    157.08ms    124.66ms     17.20ms    78.00%
    time to 1st byte:   127.22ms    184.74ms    155.12ms     17.22ms    51.00%
    req/s           :      53.82       78.32       65.01        7.37    54.00%
    


    Even Higher Concurrency Tests - 2000 Users



    Even higher concurrency tests at 2000 concurrent users and 25000 requests
    • Caddy 0.9 HTTP/2 HTTPS with headers = finished in 6.16s, 4058.36 req/s, 5.47MB/s
    • Centmin Mod Nginx 1.11.3 HTTP/2 HTTPS (4 cpus) = finished in 3.21s, 7795.35 req/s, 12.09MB/s
    Caddy
    Code (Text):
    /usr/local/bin/h2load -c2000 -m100 -n25000 -v https://domain1.com:448
    starting benchmark...
    spawning thread #0: 2000 total client(s). 25000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 6.16s, 4058.36 req/s, 5.47MB/s
    requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 33.73MB (35363701) total, 773.15KB (791701) headers (space savings 90.37%), 32.47MB (34050000) data
                         min         max         mean         sd        +/- sd
    time for request:   272.24ms       5.16s       2.75s       1.23s    57.35%
    time for connect:   859.65ms       2.64s       1.47s    265.03ms    78.70%
    time to 1st byte:      1.55s       5.00s       3.49s       1.07s    59.25%
    req/s           :       1.99        8.26        2.93        1.28    87.25%
    

    Nginx
    Code (Text):
    /usr/local/bin/h2load -c2000 -m100 -n25000 -v https://domain1.com:443
    starting benchmark...
    spawning thread #0: 2000 total client(s). 25000 total requests
    TLS Protocol: TLSv1.2
    Cipher: ECDHE-RSA-AES256-GCM-SHA384
    Server Temp Key: ECDH P-256 256 bits
    Application protocol: h2
    progress: 10% done
    progress: 20% done
    progress: 30% done
    progress: 40% done
    progress: 50% done
    progress: 60% done
    progress: 70% done
    progress: 80% done
    progress: 90% done
    progress: 100% done
    
    finished in 3.21s, 7795.35 req/s, 12.09MB/s
    requests: 25000 total, 25000 started, 25000 done, 25000 succeeded, 0 failed, 0 errored, 0 timeout
    status codes: 25000 2xx, 0 3xx, 0 4xx, 0 5xx
    traffic: 38.76MB (40648000) total, 5.77MB (6050000) headers (space savings 27.76%), 32.47MB (34050000) data
                         min         max         mean         sd        +/- sd
    time for request:     1.05ms       1.12s    641.55ms    193.01ms    76.68%
    time for connect:   690.11ms       2.26s       1.82s    351.88ms    84.00%
    time to 1st byte:   913.11ms       3.07s       2.46s    497.12ms    84.90%
    req/s           :       3.91       14.22        5.51        2.21    91.85%
    

    Current system memory usage totals via smem
    Code (Text):
    smem -tk
      PID User     Command                         Swap      USS      PSS      RSS 
      686 avahi    avahi-daemon: chroot helpe         0    84.0K   182.0K   400.0K 
     2897 root     /sbin/agetty --noclear tty1        0   156.0K   182.0K   840.0K 
      640 root     /sbin/rngd -f                      0   180.0K   193.0K   672.0K 
      643 root     /usr/sbin/irqbalance --fore        0   288.0K   352.0K     1.1M 
     5569 root     /usr/sbin/anacron -s               0   332.0K   365.0K   788.0K 
     2594 root     /usr/lib/systemd/systemd-ma        0   556.0K   580.0K     1.3M 
      506 root     /usr/sbin/lvmetad -f               0   536.0K   599.0K     1.5M 
      644 avahi    avahi-daemon: running [cent        0   428.0K   609.0K     1.7M 
      618 root     /sbin/auditd -n                    0   552.0K   610.0K     1.7M 
      698 root     /usr/sbin/crond -n                 0   672.0K   705.0K     1.5M 
      697 root     /usr/lib/systemd/systemd-lo        0   776.0K   808.0K     1.7M 
     1659 nobody   /usr/local/bin/memcached -d        0   800.0K   814.0K     1.3M 
      516 root     /usr/lib/systemd/systemd-ud        0   792.0K   856.0K     1.8M 
      685 dbus     /bin/dbus-daemon --system -        0  1016.0K     1.0M     1.9M 
      732 root     /usr/sbin/wpa_supplicant -u        0   896.0K     1.1M     2.7M 
      692 ntp      /usr/sbin/ntpd -u ntp:ntp -        0     1.0M     1.1M     2.3M 
     2582 root     /bin/bash                          0   488.0K     1.2M     1.9M 
     3004 root     /bin/bash                          0   512.0K     1.2M     2.0M 
      956 root     /usr/sbin/sshd -D                  0   860.0K     1.3M     3.5M 
     2402 root     /usr/libexec/postfix/master        0     1.2M     1.3M     2.2M 
     1288 root     pure-ftpd (SERVER)                 0     1.4M     1.5M     2.7M 
     4823 postfix  pickup -l -t unix -u               0     1.2M     1.7M     4.0M 
     2449 postfix  qmgr -l -t unix -u                 0     1.3M     1.8M     4.2M 
     2913 root     -bash                              0     1.4M     1.8M     3.0M 
     3179 root     -bash                              0     1.4M     1.9M     3.1M 
      642 root     /usr/sbin/rsyslogd -n              0     1.5M     2.2M     4.0M 
     2911 root     sshd: root@pts/0,pts/              0     2.2M     2.7M     5.3M 
        1 root     /usr/lib/systemd/systemd --        0     2.6M     2.7M     3.7M 
      645 root     /usr/sbin/haveged -w 4067 -        0     5.8M     5.8M     6.4M 
      639 root     /usr/sbin/NetworkManager --        0     4.9M     5.9M     8.7M 
     5649 root     python /usr/bin/smem -tk           0     5.5M     6.2M     7.6M 
      492 root     /usr/lib/systemd/systemd-jo        0     7.0M     7.7M     9.2M 
     4390 root     nginx: master process /usr/        0     3.4M     8.3M    28.0M 
     4417 root     php-fpm: master process (/u        0     8.6M     8.6M     9.2M 
      733 polkitd  /usr/lib/polkit-1/polkitd -        0     8.6M     9.5M    11.9M 
      945 root     /usr/bin/python -Es /usr/sb        0    11.6M    12.7M    16.0M 
     4392 nginx    nginx: worker process              0     8.0M    13.0M    33.7M 
     2993 root     /usr/bin/docker-current exe        0     9.5M    13.1M    17.5M 
     4393 nginx    nginx: worker process              0     7.9M    13.1M    33.9M 
      743 root     /sbin/dhclient -d -q -sf /u        0    13.1M    13.4M    15.4M 
     4394 nginx    nginx: worker process              0     8.4M    13.5M    34.1M 
     4395 nginx    nginx: worker process              0     8.7M    13.7M    34.4M 
     3059 root     lfd - sleepin                      0    22.6M    22.6M    23.3M 
     1065 root     /usr/bin/docker-current dae        0    19.2M    22.8M    27.4M 
     4334 root     /usr/local/bin/caddy -log /        0    45.2M    45.2M    45.2M 
     1477 mysql    /usr/sbin/mysqld                   0    93.8M    94.1M    96.2M 
    -------------------------------------------------------------------------------
       46 9                                           0   316.8M   360.4M   521.2M
    
    
     
    Last edited: Aug 7, 2016
    • Informative Informative x 1
  14. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    System Resource Usage



    Next up is testing system resource usage under load. For this I'll install Centmin Mod's alpha addons/netdata.sh addon for system resource monitoring :)

    I chose not to use nginx to proxy the netdata gui as that would add to nginx's usage loads and skew nginx usage stats :)

    Available netdata system metrics to monitor
    Code (Text):
    cat /etc/netdata/netdata.conf | grep "^\[" | cut -d '[' -f 2 | cut -d ']' -f 1 | grep -v ^global$ | grep -v "^plugin" | sort -u
    apps.cpu
    apps.cpu_system
    apps.cpu_user
    apps.files
    apps.lreads
    apps.lwrites
    apps.major_faults
    apps.mem
    apps.minor_faults
    apps.pipes
    apps.preads
    apps.processes
    apps.pwrites
    apps.sockets
    apps.threads
    cgroup_34c80afbfaa5.cpu
    cgroup_34c80afbfaa5.cpu_per_core
    cgroup_34c80afbfaa5.io
    cgroup_34c80afbfaa5.mem
    cgroup_34c80afbfaa5.mem_activity
    cgroup_34c80afbfaa5.pgfaults
    cgroup_34c80afbfaa5.queued_ops
    cgroup_34c80afbfaa5.serviced_ops
    cgroup_34c80afbfaa5.throttle_serviced_ops
    cgroup_34c80afbfaa5.writeback
    cpu.cpu0
    cpu.cpu0_interrupts
    cpu.cpu0_softirqs
    cpu.cpu1
    cpu.cpu1_interrupts
    cpu.cpu1_softirqs
    cpu.cpu2
    cpu.cpu2_interrupts
    cpu.cpu2_softirqs
    cpu.cpu3
    cpu.cpu3_interrupts
    cpu.cpu3_softirqs
    disk_avgsz.dm-1
    disk_avgsz.sda
    disk_await.dm-1
    disk_await.sda
    disk_backlog.dm-1
    disk_backlog.sda
    disk.dm-1
    disk_inodes.dm-1
    disk_inodes.sda1
    disk_iotime.dm-1
    disk_iotime.sda
    disk_mops.sda
    disk_ops.dm-1
    disk_ops.sda
    disk.sda
    disk_space.dm-1
    disk_space.sda1
    disk_svctm.dm-1
    disk_svctm.sda
    disk_util.dm-1
    disk_util.sda
    groups.cpu
    groups.cpu_system
    groups.cpu_user
    groups.files
    groups.lreads
    groups.lwrites
    groups.major_faults
    groups.mem
    groups.minor_faults
    groups.pipes
    groups.preads
    groups.processes
    groups.pwrites
    groups.sockets
    groups.threads
    ipv4.bcast
    ipv4.bcastpkts
    ipv4.errors
    ipv4.fragsin
    ipv4.fragsout
    ipv4.mcast
    ipv4.mcastpkts
    ipv4.packets
    ipv4.tcperrors
    ipv4.tcphandshake
    ipv4.tcppackets
    ipv4.tcpsock
    ipv4.udperrors
    ipv4.udppackets
    ipv6.ect
    ipv6.icmp
    ipv6.icmpmldv2
    ipv6.icmpneighbor
    ipv6.icmprouter
    ipv6.icmptypes
    ipv6.mcast
    ipv6.mcastpkts
    ipv6.packets
    mem.committed
    mem.kernel
    mem.pgfaults
    mem.slab
    mem.writeback
    netdata.apps_children_fix
    netdata.apps_cpu
    netdata.apps_files
    netdata.apps_fix
    netdata.clients
    netdata.compression_ratio
    netdata.net
    netdata.plugin_cgroups_cpu
    netdata.plugin_proc_cpu
    netdata.plugin_tc_cpu
    netdata.plugin_tc_time
    netdata.requests
    netdata.response_time
    netdata.server_cpu
    net.docker0
    net.enp0s3
    netfilter.conntrack_changes
    netfilter.conntrack_errors
    netfilter.conntrack_expect
    netfilter.conntrack_new
    netfilter.conntrack_search
    netfilter.conntrack_sockets
    net_packets.docker0
    net_packets.enp0s3
    net_packets.vethc135f09
    net.vethc135f09
    registry
    system.active_processes
    system.cpu
    system.ctxt
    system.entropy
    system.forks
    system.idlejitter
    system.interrupts
    system.intr
    system.io
    system.ipv4
    system.ipv6
    system.load
    system.processes
    system.ram
    system.softirqs
    system.swap
    system.swapio
    users.cpu
    users.cpu_system
    users.cpu_user
    users.files
    users.lreads
    users.lwrites
    users.major_faults
    users.mem
    users.minor_faults
    users.pipes
    users.preads
    users.processes
    users.pwrites
    users.sockets
    users.threads
    
     
    Last edited: Aug 7, 2016
    • Winner Winner x 1
  15. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    h2load netdata charts



    With addons/netdata.sh installing and configuring netdata for system monitoring, I re-tested h2load stress test with higher loads at 1,000 concurrent users and 50,0000 requests. But this time I ran each test 3 times and filtered just the requests per second and percentage of completed requests.

    Commands used:

    For Centmin Mod Nginx 1.11.3 HTTP/2 port 443
    Code (Text):
    echo "CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://domain1.com:443 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    For Caddy 0.9 HTTP/2 port 448
    Code (Text):
    echo "Caddy 0.9 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://domain1.com:448 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Resulting output:

    For Centmin Mod Nginx 1.11.3 HTTP/2 port 443 = 11,322.50 req/s average
    Code (Text):
    echo "CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://domain1.com:443 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test
    8896.39 req/s 100% completed
    17788.95 req/s 100% completed
    7282.18 req/s 100% completed
    

    For Caddy 0.9 HTTP/2 port 448 = 3,751.71 req/s average
    Code (Text):
    echo "Caddy 0.9 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -c1000 -m100 -n50000 -v https://domain1.com:448 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    Caddy 0.9 HTTP/2 h2load stress test
    3908.97 req/s 100% completed
    3504.25 req/s 100% completed
    3841.91 req/s 100% completed
    

    Netdata recorded the system cpu utilisation, cpu load, disk and memory usage. Below netdata charts show 2 sets of 3 grouped spikes. The set of 3 spikes on the left is Centmin Mod Nginx h2load run and the set of 3 spikes on the right is Caddy h2load run. As you can see Caddy used more memory than Nginx and spent more time in cpu. Though both didn't max cpu beyond <100% or a 4 cpu thread system so 100/400 = 0.25 cpu load so not that much. The h2load by default only tests with a single thread. So I guess next test is to use 2 or 4 h2load threads for testing.

    • CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test cpu load peaked at 0.60 and 548MB total memory used
    • Caddy 0.9 HTTP/2 h2load stress test cpu load peaked at 1.57 and 1289MB total memory used

    cmm-nginx-vs-caddy-h2load-1000-50000-netdata-080816-00.png cmm-nginx-vs-caddy-h2load-1000-50000-netdata-080816-01.png

    Looks like Caddy's cpu load could also be partially disk related though still very light at ~1.47% disk utilisation

    cmm-nginx-vs-caddy-h2load-1000-50000-netdata-080816-02.png

    Netdata has Nginx support for monitoring but no support for Caddy. Nginx peaked at around 619 active connections per second and 335 requests/second

    cmm-nginx-vs-caddy-h2load-1000-50000-netdata-080816-03.png
     
    Last edited: Aug 8, 2016
    • Informative Informative x 1
  16. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    h2load threads



    Next is to repeat the h2load tests in previous post but instead of default 1 h2load thread, set to 2 threads as per h2load(1) — nghttp2 1.14.0-DEV documentation
    Commands used:

    For Centmin Mod Nginx 1.11.3 HTTP/2 port 443
    Code (Text):
    echo "CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -t2 -c1000 -m100 -n50000 -v https://domain1.com:443 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    For Caddy 0.9 HTTP/2 port 448
    Code (Text):
    echo "Caddy 0.9 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -t2 -c1000 -m100 -n50000 -v https://domain1.com:448 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    

    Resulting output:

    For Centmin Mod Nginx 1.11.3 HTTP/2 port 443 = 10,850.84 req/s average
    Code (Text):
    echo "CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -t2 -c1000 -m100 -n50000 -v https://domain1.com:443 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test
    10320.74 req/s 100% completed
    10585.68 req/s 100% completed
    11646.10 req/s 100% completed
    

    For Caddy 0.9 HTTP/2 port 448 = 3,942.75 req/s average
    Code (Text):
    echo "Caddy 0.9 HTTP/2 h2load stress test"; sleep 30; for i in {1..3}; do /usr/local/bin/h2load -t2 -c1000 -m100 -n50000 -v https://domain1.com:448 > h2load.$i.log; cat h2load.$i.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
    4879.98 req/s 100% completed
    3438.50 req/s 100% completed
    3509.76 req/s 100% completed
    


    Again 3 spikes on left is for Centmin Mod Nginx and 3 spikes on right is for Caddy.
    • CentminMod.com Nginx 1.11.3 HTTP/2 h2load stress test cpu load peaked at 0.62 and 551MB total memory used
    • Caddy 0.9 HTTP/2 h2load stress test cpu load peaked at 1.86 and 1393MB total memory used
    h2load 2 thread test with 1000 users and 50,000 requests

    cmm-nginx-vs-caddy-h2load-t2-1000-50000-netdata-080816-00.png
    cmm-nginx-vs-caddy-h2load-t2-1000-50000-netdata-080816-01.png

    Nginx peaked at 949 active connections and requests/sec

    cmm-nginx-vs-caddy-h2load-t2-1000-50000-netdata-080816-02.png

    Note, Nginx was with LibreSSL 2.4.2 compiled but seems OpenSSL 1.0.2h and 1.1.0 would have even better performance SSL - OpenSSL 1.1.0-pre6 vs OpenSSL 1.0.2h vs OpenSSL 1.0.1e vs LibreSSL 2.4.2 vs LibreSSL 2.3.6 | Centmin Mod Community
     
    Last edited: Aug 10, 2016
  17. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy Commercial Licensing & Changes



    Caddy has announced changes to it's licensing and introduced a paid commercial licensing option. Full details at https://caddyserver.com/blog/accouncing-caddy-commercial-licenses.

    But for me using HTTP header as a means for it isn't my cup of tea when there are known performance overhead hit as you add more headers to Caddy. Though you can build from source yourself to get around the HTTP sponsor header requirement https://caddy.community/t/caddy-commercial-sponsor-header-clarification/2716. But right now building from source is sort of broken too Local build instructions from README don't work · Issue #1843 · mholt/caddy · GitHub.

    I think most personal usage folks using Caddy wouldn't care about that as they'd just use free version/personal with HTTP sponsor header intact anyway not knowing or caring about the performance hit as you add more HTTP headers.

    For HTTP/2 HTTPS loads, Caddy is ~1/3rd the performance of Nginx. So you'd need 3-4x Caddy servers to match the performance of 1x Nginx server for HTTP/2 based HTTPS. So for Commercial licensing you'd need 5/server license at discounted US$250/month which reverts to 4x250 = US$1,000 month once introductory licensing ends. That equates to $3,000/yr discounted or $12,000/yr for 5 commercial licenses if you only need 4 ?

    Compare Nginx commercial licensing for 1-4 servers Pricing - Application Delivery for the Modern Web | NGINX
    • Nginx Basic $2,500/yr per server
    • Nginx Pro $3,500/yr per server
    • Nginx Enterprise $5,000/yr per server
    If I need 4x Caddy servers to match 1x Nginx servers performance the comparative cost is:
    • Caddy 5 instance license = $3,000/yr discounted or $12,000/yr normal price
    • Nginx 1 server Basic license = $2,500/yr, $3,500/yr Pro or $5,000/yr Enterprise.
    If I need 8x Caddy servers to match 2x Nginx servers performance the comparative cost is:
    • Caddy 10 instance license = $6,000/yr discounted or $24,000/yr normal price
    • Nginx 2 server Basic license = $5,000/yr, $7,000/yr Pro or $10,000/yr Enterprise.
    From financial and performance perspective, doesn't make sense unfortunately.

    Edit: oh mis-read that as intro price was 1/4 of the full price in above numbers. The non-intro price isn't actually mentioned yet

    Update: Started building my own Caddy binaries without the HTTP Sponsor header, though Caddy has already reverted and removed the HTTP Sponsor header in upstream source. Guess it was an learning exercise for me to figure out how to build my own custom Caddy binaries and integrate Caddy plugins.

    Below is custom built Caddy binary + plugins source compiled with Go 1.9

    Code (Text):
    go version
    go version go1.9 linux/amd64
    

    Code (Text):
    caddy -version
    Caddy 0.10.9 (+545fa84 Fri Sep 15 14:36:05 UTC 2017)
    1 file changed, 32 insertions(+)
    caddy/caddymain/run.go

    Code (Text):
    ./caddy.sh status
    caddy is running (13188)
    -----------------------------------------------
    Max processes     8192     8192     processes
    Max open files    262144   262144   files
    -----------------------------------------------

    Code (Text):
    curl -I http://$(hostname):8888         
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 3801
    Content-Type: text/html; charset=utf-8
    Etag: "ow5yjt2xl"
    Last-Modified: Tue, 12 Sep 2017 10:59:53 GMT
    Server: Caddy
    X-Powered-By: Caddy via CentminMod
    Date: Fri, 15 Sep 2017 14:44:28 GMT

    Code (Text):
    caddy -plugins
    Server types:
      http
    
    Caddyfile loaders:
      short
      flag
      default
    
    Other plugins:
      http.authz
      http.awses
      http.awslambda
      http.basicauth
      http.bind
      http.browse
      http.cache
      http.cgi
      http.cors
      http.datadog
      http.errors
      http.expires
      http.expvar
      http.ext
      http.fastcgi
      http.filter
      http.git
      http.gopkg
      http.grpc
      http.gzip
      http.header
      http.hugo
      http.index
      http.internal
      http.ipfilter
      http.jekyll
      http.jsonp
      http.jwt
      http.limits
      http.locale
      http.log
      http.login
      http.mailout
      http.markdown
      http.mime
      http.minify
      http.multipass
      http.nobots
      http.pprof
      http.prometheus
      http.proxy
      http.proxyprotocol
      http.push
      http.ratelimit
      http.realip
      http.reauth
      http.redir
      http.request_id
      http.restic
      http.rewrite
      http.root
      http.search
      http.status
      http.templates
      http.timeouts
      http.upload
      http.webdav
      http.websocket
      shutdown
      startup
      tls
      tls.storage.file


    Update: September 16th, 2017

    Just sat down to read Caddy EULA which applies to their official Caddy binaries and conclusion for me is I probably need to build my own source binaries anyway caddy/EULA.txt at master · mholt/caddy · GitHub
    so I shouldn't have published benchmarks for official Caddy binaries ? Never came across a EULA that prohibits benchmark info sharing ?
    I'm trying to combine/integrate several web servers under the one roof/server - nginx, openlitespeed, litespeed, apache 2.4, caddy and/or h2o.
     
    Last edited: Sep 15, 2017
  18. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5

    Caddy Official vs Caddy Source Binary vs Centmin Mod Nginx



    Testing performance of official Caddy v0.10.9 binary versus custom source compiled Caddy v0.10.9 vs Centmin Mod Nginx 1.13.5. Caddy official has intact HTTP sponsor header while custom source compiled have removed HTTP Sponsor headers. However, these quick tests were done on OpenVZ VPS so couldn't install my nghttp2 h2load Docker image. So not able to see the HTTP header overhead with h2load HTTP/2 based HTTPS tests.

    Below tests are just non-HTTPS quick tests.

    Test System
    • 2GB OpenVZ VPS
    • 2 CPU Threads @E5-1650v3 @3.50Ghz
    • 50GB SSD
    • Washington, DC
    • CentOS 7.4 64bit
    • Centmin Mod 123.09beta01
    Official Caddy v0.10.9 binary
    Code (Text):
    curl -I http://$(hostname):8888
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Caddy-Sponsors: This free web server is licensed only for non-commercial use, made possible by its sponsors: Minio, Uptime Robot, and Sourcegraph
    Content-Length: 3801
    Content-Type: text/html; charset=utf-8
    Etag: "ow5yjt2xl"
    Last-Modified: Tue, 12 Sep 2017 10:59:53 GMT
    Server: Caddy
    X-Powered-By: Caddy via CentminMod
    Date: Fri, 15 Sep 2017 16:02:30 GMT
    

    Custom Caddy source built v0.10.9 binary
    Code (Text):
    curl -I http://$(hostname):8888     
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Cache-Control: max-age=86400
    Content-Length: 3801
    Content-Type: text/html; charset=utf-8
    Etag: "ow5yjt2xl"
    Last-Modified: Tue, 12 Sep 2017 10:59:53 GMT
    Server: Caddy
    X-Powered-By: Caddy via CentminMod
    Date: Fri, 15 Sep 2017 14:44:28 GMT
    

    Centmin Mod Nginx 1.13.5
    Code (Text):
    curl -I http://$(hostname):80
    HTTP/1.1 200 OK
    Date: Fri, 15 Sep 2017 16:04:26 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 3801
    Last-Modified: Tue, 12 Sep 2017 10:59:53 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "59b7be29-ed9"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Accept-Ranges: bytes
    

    Siege 4.0.2 test parameters
    Code (Text):
    siege -v
    SIEGE 4.0.2
    Usage: siege [options]
          siege [options] URL
          siege -g URL
    Options:
      -V, --version             VERSION, prints the version number.
      -h, --help                HELP, prints this section.
      -C, --config              CONFIGURATION, show the current config.
      -v, --verbose             VERBOSE, prints notification to screen.
      -q, --quiet               QUIET turns verbose off and suppresses output.
      -g, --get                 GET, pull down HTTP headers and display the
                               transaction. Great for application debugging.
      -c, --concurrent=NUM      CONCURRENT users, default is 10
      -r, --reps=NUM            REPS, number of times to run the test.
      -t, --time=NUMm           TIMED testing where "m" is modifier S, M, or H
                               ex: --time=1H, one hour test.
      -d, --delay=NUM           Time DELAY, random delay before each requst
      -b, --benchmark           BENCHMARK: no delays between requests.
      -i, --internet            INTERNET user simulation, hits URLs randomly.
      -f, --file=FILE           FILE, select a specific URLS FILE.
      -R, --rc=FILE             RC, specify an siegerc file
      -l, --log[=FILE]          LOG to FILE. If FILE is not specified, the
                               default is used: PREFIX/var/siege.log
      -m, --mark="text"         MARK, mark the log file with a string.
                               between .001 and NUM. (NOT COUNTED IN STATS)
      -H, --header="text"       Add a header to request (can be many)
      -A, --user-agent="text"   Sets User-Agent in request
      -T, --content-type="text" Sets Content-Type in request
    
    Copyright (C) 2016 by Jeffrey Fulmer, et al.
    This is free software; see the source for copying conditions.
    There is NO warranty; not even for MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.
    

    • siege -b -c200 -t60s http://$(hostname):8888/ -m caddy-0.10.9-official
    • siege -b -c200 -t60s http://$(hostname):8888/ -m caddy-0.10.9-sourcebuilt
    • siege -b -c200 -t60s http://$(hostname):80/ -m cmm-nginx-1.13.5
    Results
    Code (Text):
    head -n1 /usr/local/var/log/siege.log; tail -6 /usr/local/var/log/siege.log
         Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
    **** caddy-0.10.9-official ****
    2017-09-15 16:07:54,  68079,      59.80,         340,       0.17,     1138.44,        5.69,      198.44,   68079,       0
    **** caddy-0.10.9-sourcebuilt ****
    2017-09-15 16:12:40,  67078,      59.07,         335,       0.18,     1135.57,        5.67,      198.77,   67078,       0
    **** cmm-nginx-1.13.5 ****
    2017-09-15 16:14:26, 138406,      59.43,         840,       0.08,     2328.89,       14.13,      192.20,  138465,       0
    


    Wrk 4.0 Test Parameters
    Code (Text):
    wrk -v
    wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
    Usage: wrk <options> <url>                     
      Options:                                     
       -c, --connections <N>  Connections to keep open
       -d, --duration    <T>  Duration of test     
       -t, --threads     <N>  Number of threads to use
                                                   
       -s, --script      <S>  Load Lua script file 
       -H, --header      <H>  Add header to request
           --latency          Print latency statistics
           --timeout     <T>  Socket/request timeout
       -v, --version          Print version details
                                                   
      Numeric arguments may include a SI unit (1k, 1M, 1G)
      Time arguments may include a time unit (2s, 2m, 2h)
    

    • wrk -t2 -c200 -d60s --latency http://$(hostname):8888/ # caddy-0.10.9-official
    • wrk -t2 -c200 -d60s --latency http://$(hostname):8888/ # caddy-0.10.9-sourcebuilt
    • wrk -t2 -c200 -d60s --latency http://$(hostname):80/ # cmm-nginx-1.13.5
    caddy-0.10.9-official
    Code (Text):
    wrk -t2 -c200 -d60s --latency http://$(hostname):8888/
    Running 1m test @ http://hostname:8888/
      2 threads and 200 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency   191.73ms  284.89ms   2.00s    86.26%
       Req/Sec     1.54k   804.48     4.25k    66.08%
      Latency Distribution
        50%   71.86ms
        75%  283.67ms
        90%  577.07ms
        99%    1.28s
      183306 requests in 1.00m, 740.23MB read
      Socket errors: connect 0, read 0, write 0, timeout 72
    Requests/sec:   3053.09
    Transfer/sec:     12.33MB
    

    caddy-0.10.9-sourcebuilt
    Code (Text):
    wrk -t2 -c200 -d60s --latency http://$(hostname):8888/
    Running 1m test @ http://hostname:8888/
      2 threads and 200 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency   131.75ms  184.25ms   1.50s    85.65%
       Req/Sec     1.87k     0.92k    4.84k    69.21%
      Latency Distribution
        50%   59.60ms
        75%  202.88ms
        90%  386.41ms
        99%  799.69ms
      222556 requests in 1.00m, 867.45MB read
    Requests/sec:   3706.13
    Transfer/sec:     14.45MB
    


    cmm-nginx-1.13.5
    Code (Text):
    wrk -t2 -c200 -d60s --latency http://$(hostname):80/
    Running 1m test @ http://hostname:80/
      2 threads and 200 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     6.91ms    4.59ms  88.18ms   84.16%
       Req/Sec     7.50k     2.56k   24.46k    75.28%
      Latency Distribution
        50%    6.36ms
        75%    8.65ms
        90%   11.01ms
        99%   23.25ms
      890755 requests in 1.00m, 3.41GB read
    Requests/sec:  14833.86
    Transfer/sec:     58.11MB
    


    Summary



    web server siege req/s siege resp time wrk req/s wrk avg latency
    caddy 0.10.9 official binary 1138 0.17s 3053 191.73ms
    caddy 0.10.9 source built binary 1136 0.18s 3706 131.75ms
    cmm nginx 1.13.5 2329 0.08s 14834 6.91ms
     
    Last edited: Sep 16, 2017
    • Winner Winner x 2
  19. eva2000

    eva2000 Administrator Staff Member

    29,743
    6,719
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,031
    Local Time:
    9:34 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1
Thread Status:
Not open for further replies.