Learn about Centmin Mod LEMP Stack today
Become a Member

Nginx Centmin Mod Nginx with Cloudflare HTTP/3 Nginx Patch !

Discussion in 'Nginx and PHP-FPM news & discussions' started by eva2000, Oct 16, 2019.

  1. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    @buik you might be happy to hear Cloudflare has released a Nginx HTTP/3 Nginx patch so that Nginx can support HTTP/3 (HTTP over QUIC) based HTTPS. Unfortunately, patch is for Nginx 1.16.1 specific stable version and Nginx mainline 1.17 is untested right now.

    The Cloudflare Nginx HTTP/3 patch is part of the Cloudflare Quiche (QUIC) + BoringSSL library which has a dependency requirement - with newer cmake 3.x and rust 1.38 versions and is outlined at cloudflare/quiche.

    Centmin Mod 123.09beta01 private branch with Cloudflare HTTP/3 Nginx patch in play via persistent config variable.

    Note the Nginx compile flags and OpenSSL version is using Google BoringSSL instead of OpenSSL 1.1.1 highlighted in bold below.
    Test Centmin Mod Nginx patched with Cloudflare HTTP/3 Nginx patch via custom built curl binary which is built against Cloudflare Quiche/BoringSSL to support HTTP/3 h3-23 draft protocol.
    Code (Text):
    curl-http3 -V
    curl 7.67.0-DEV (x86_64-pc-linux-gnu) libcurl/7.67.0-DEV BoringSSL zlib/1.2.11 brotli/1.0.7 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 quiche/0.1.0
    Release-Date: [unreleased]
    Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
    Features: alt-svc AsynchDNS brotli HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL UnixSockets
    

    HTTP/3 check
    Code (Text):
    curl-http3 --http3 -4Ik https://domain.com/
    HTTP/3 200
    date: Wed, 16 Oct 2019 06:43:40 GMT
    content-type: text/html; charset=utf-8
    content-length: 3801
    vary: Accept-Encoding
    etag: "58e34c93-ed9"
    server: nginx centminmod
    x-powered-by: centminmod
    expires: Thu, 17 Oct 2019 06:43:40 GMT
    cache-control: max-age=86400
    x-frame-options: SAMEORIGIN
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    accept-ranges: bytes
    

    Verbose HTTP/3 header check for ALPN h3-23 draft
    Code (Text):
    curl-http3 --http3 -4Ikv https://domain.com/
    *   Trying ip:443...
    * Sent QUIC client Initial, ALPN: h3-23
    * h3 [:method: HEAD]
    * h3 [:path: /]
    * h3 [:scheme: https]
    * h3 [:authority: domain.com]
    * h3 [user-agent: curl/7.67.0-DEV]
    * h3 [accept: */*]
    * Using HTTP/3 Stream ID: 0 (easy handle 0x5647a6df4f10)
    > HEAD / HTTP/3
    > Host: domain.com
    > user-agent: curl/7.67.0-DEV
    > accept: */*
    >
    < HTTP/3 200
    HTTP/3 200
    < date: Wed, 16 Oct 2019 06:44:23 GMT
    date: Wed, 16 Oct 2019 06:44:23 GMT
    < content-type: text/html; charset=utf-8
    content-type: text/html; charset=utf-8
    < content-length: 3801
    content-length: 3801
    < vary: Accept-Encoding
    vary: Accept-Encoding
    < etag: "58e34c93-ed9"
    etag: "58e34c93-ed9"
    < server: nginx centminmod
    server: nginx centminmod
    < x-powered-by: centminmod
    x-powered-by: centminmod
    < expires: Thu, 17 Oct 2019 06:44:23 GMT
    expires: Thu, 17 Oct 2019 06:44:23 GMT
    < cache-control: max-age=86400
    cache-control: max-age=86400
    < x-frame-options: SAMEORIGIN
    x-frame-options: SAMEORIGIN
    < x-xss-protection: 1; mode=block
    x-xss-protection: 1; mode=block
    < x-content-type-options: nosniff
    x-content-type-options: nosniff
    < accept-ranges: bytes
    accept-ranges: bytes
    
    <
    * Excess found: excess = 3801 url = / (zero-length body)
    * Connection #0 to host domain.com left intact
    

    Testing HTTP/2
    Code (Text):
    curl-http3 --http2 -4Ik https://domain.com/
    HTTP/2 200
    date: Wed, 16 Oct 2019 06:38:49 GMT
    content-type: text/html; charset=utf-8
    content-length: 6040
    last-modified: Tue, 04 Apr 2017 10:14:33 GMT
    vary: Accept-Encoding
    etag: "58e37209-1798"
    server: nginx centminmod
    x-powered-by: centminmod
    expires: Thu, 17 Oct 2019 06:38:49 GMT
    cache-control: max-age=86400
    x-frame-options: SAMEORIGIN
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    accept-ranges: bytes
    

    Testing HTTP/1.1
    Code (Text):
    curl-http3 --http1.1 -4Ik https://domain.com/
    HTTP/1.1 200 OK
    Date: Wed, 16 Oct 2019 06:54:14 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 3801
    Last-Modified: Tue, 04 Apr 2017 07:34:43 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "58e34c93-ed9"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Thu, 17 Oct 2019 06:54:14 GMT
    Cache-Control: max-age=86400
    X-Frame-Options: SAMEORIGIN
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Accept-Ranges: bytes
    

    Still along way to automating everything in Centmin Mod Nginx compile routines i.e. proper listen directive setup requires 2 listen directives now, one for quic directive
    Code (Text):
           # Enable QUIC and HTTP/3.
           listen 443 quic reuseport;
    
           # Enable HTTP/2 (optional).
           listen 443 ssl http2;
    

    Does not work if you try quic and http2 on same listen directive for me
    Code (Text):
           listen 443 ssl quic http2;

    you get following errors when trying to do curl HTTP/3 requests to test headers if quic and http2 are on a single listen directive
    Code (Text):
    curl-http3 --http3 -4Ik https://domain.com/
    curl: (7) quiche_conn_recv() == -10

    Code (Text):
    curl-http3 --http3 -4Ikv https://domain.com/
    *   Trying ip:443...
    * Sent QUIC client Initial, ALPN: h3-23
    * quiche_conn_recv() == -10
    * connect to ip port 443 failed: No such file or directory
    * Failed to connect to domain.com port 443: No such file or directory
    * Closing connection 0
    curl: (7) quiche_conn_recv() == -10

    Of course still waiting on Nginx's official HTTP/3 QUIC support which is being developed in Nginx mainline 1.17 which Centmin Mod Nginx usually uses. So Cloudflare HTTP/3 Nginx patch might not last the distance once official Nginx HTTP/3 QUIC support lands according to Nginx roadmap Roadmap – nginx

    Note: as Nginx is built against Google's BoringSSL crypto library, you don't get OCSP Stapling support like you do if you build Nginx against LibreSSL or OpenSSL crypto libraries. So if you used default Nginx OpenSSL built server with OCSP stapling options, now you will get not supported messages when you test nginx configure.
    Code (Text):
    nginx -t
    nginx: [warn] "ssl_stapling" ignored, not supported
    nginx: [warn] "ssl_stapling" ignored, not supported
    nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
    
     
    • Like Like x 2
  2. BamaStangGuy

    BamaStangGuy Active Member

    608
    179
    43
    May 25, 2014
    Ratings:
    +244
    Local Time:
    3:40 PM
    This isn't something we need to worry about if we are actually using cloudflare as a proxy right?
     
  3. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah not right now unless Cloudflare enable support for their edge servers to speak with origin servers like Centmin Mod Nginx over HTTP/3 and you have Cloudflare Full or Full Strict SSL enabled. Right now Cloudflare edge servers peak with origins over HTTP/1.1.
     
  4. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    For Google Canary browser enabled with HTTP/3 h3-23 draft flags --enable-quic --quic-version=h3-23, also need alt-svc header added
    Code (Text):
    add_header alt-svc 'h3-23=":443"; ma=86400';

    Code (Text):
    curl-http3 --http3 -4Ik https://domain.com/
    HTTP/3 200
    date: Wed, 16 Oct 2019 08:04:53 GMT
    content-type: text/html; charset=utf-8
    content-length: 3801
    vary: Accept-Encoding
    etag: "58e34c93-ed9"
    server: nginx centminmod
    x-powered-by: centminmod
    expires: Thu, 17 Oct 2019 08:04:53 GMT
    cache-control: max-age=86400
    x-frame-options: SAMEORIGIN
    x-xss-protection: 1; mode=block
    x-content-type-options: nosniff
    alt-svc: h3-23=":443"; ma=86400
    accept-ranges: bytes
    

    In Google Canary web browser :)

    where protocol http/2+quic/99 = HTTP/3

    centminmod-nginx-http3-patched-canary-01.png
     
    • Like Like x 1
    • Informative Informative x 1
  5. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Build with GCC 8.3.1 compiler :)

    Seems to be a bug with GCC 9.2.1 compiler builds with HTTP/3 patch while without HTTP/3 patch, GCC 9.2.1 compiler builds Nginx fine.
     
    • Informative Informative x 1
  6. buik

    buik Well-Known Member

    1,209
    321
    83
    Apr 29, 2016
    Ratings:
    +948
    Local Time:
    10:40 PM
    @eva2000 Thanks for reporting. You are the one!
     
    • Like Like x 1
  7. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Let the fun begin, fixed GCC 9.2.1 compilations - issue was on my end in how I built GCC 9.2.1. So rebuilt GCC 9.2.1 :)

     
  8. buik

    buik Well-Known Member

    1,209
    321
    83
    Apr 29, 2016
    Ratings:
    +948
    Local Time:
    10:40 PM
    To be honest, I don't know if I will board this Cloudflare boat.
    There are a number of reasons for this.

    1. HTTP/3 is not yet standard enabled on the production versions of Chrome, Firefox etc. Testing on a concrete, large scale is difficult.

    2. HTTP/3 is not yet an IETF internet standard. The same hassle with TLS 1.3 Many versions and many patches later still no results. This would already be finalized in summer 2019. I read here and there. and postponed for the umpteenth time to
    Latest draft with expiration date at March 2020. I can spend my time better.
    To much politics. nowadays it is often too many meetings and make far too few decisions.

    3. I no longer use BoringSSL.

    4. Looking at the Nginx roadmap, it shouldn't take long before the official Nginx team comes with a version of HTTP/3. without the requirement of quiche, BoringSSL.

    5. Time. My biggest problem is time. In the past I used to use customized, optimized software myself or for non-profit. Business wise (reason is known to some) never.
     
  9. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah true to some extent. That always nice to have options to compare and test - Nginx official HTTP/3 vs Cloudflare's HTTP/3 patched Nginx. Being a Cloudflare MVP has me in a unique position to also pass on feedback and suggestions to folks developing/releasing the Cloudflare HTTP/3 patch as well :) My first suggestion is re-incorporating the old Cloudflare HPACK full encoding HTTP/2 patch so I don't need to maintain a separate Cloudflare HPACK full encoding HTTP/2 patch Beta Branch - nginx 1.17.3 dynamtic tls & hpack patch support in 123.09beta01 :) Pretty sure Nginx official won't be touching that for HTTP/2. Though you can argue, once HTTP/3 goes mainstream, HTTP/2 usage may lessen eventually.

    Will be interesting how Nginx implement it officially or they're waiting on OpenSSL folks to officially support it. Again for me it will be interesting to test and benchmark both implementations eventually for Nginx official HTTP/3 vs Cloudflare's HTTP/3 patched Nginx :D
     
  10. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    FYI, for folks wanting to know more about HTTP/3 HTTP over QUIC, nice video from curl's author, Daniel Stenberg below



    FYI, right now HTTP/3 HTTP over QUIC uses UDP which can have 2-4x times more cpu usage according to Google and video. So there is price to pay in terms of cpu utilisation for faster page speed :)
     
    • Like Like x 1
  11. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    To address these 2 additional steps quoted above for Centmin Mod Nginx enabled with Cloudflare HTTP/3 patch, I am doing the following:

    1. add /usr/local/nginx/conf/altsvc_http3.conf in 123.09beta01. For Cloudflare HTTP/3 patch for Nginx when enabled, requires alt-svc header added specifying the HTTP/3 protocol draft version which is currently h3-23 but will increment as Cloudflare Quiche library updates to newer HTTP/3 protocol drafts i.e h3-24, h3-25 and so on. So need for a way to automate the updating of alt-svc header's mentioned HTTP/3 draft protocol version. The way currently I have done this is when NGINX_HTTP3='y' is enabled via persistent config file, is populate a new include file at /usr/local/nginx/conf/altsvc_http3.conf with the required alt-svc header.

    Example
    Code (Text):
    add_header alt-svc 'h3-23=":443"; ma=86400';

    and then each time centmin.sh menu option 4 is run to compile Nginx with NGINX_HTTP3='y' enabled for Cloudflare HTTP/3 patch + BoringSSL + Quiche builds is to query Cloudflare Quiche's supported HTTP/3 protocol draft and assign it to variable QUICHE_H3_VER. The QUICHE_H3_VER variable is used to populate the new include file at /usr/local/nginx/conf/altsvc_http3.conf. So when you want to add alt-svc header to your Nginx vhosts use the include file instead
    Code (Text):
    include /usr/local/nginx/conf/altsvc_http3.conf;

    This way HTTP/3 protocol version is continually updated with Cloudflare Quiche updates. If you set NGINX_HTTP3='n' to disable HTTP/3 Nginx patch, then centmin.sh menu option 4 will empty out /usr/local/nginx/conf/altsvc_http3.conf include file so nothing is used so as to remove alt-svc header.

    2. add /usr/local/nginx/conf/listener_http3.conf. For Cloudflare HTTP/3 patch for Nginx when enabled, requires a 2nd separate listen directive added to existing listen http2 directive in order for HTTP/3 HTTP over QUIC to work.
    Code (Text):
    # Enable QUIC and HTTP/3.
    listen 443 quic reuseport;
    
    # Enable HTTP/2.
    listen 443 ssl http2;

    The way currently I have done this is when NGINX_HTTP3='y' is enabled via persistent config file, is populate a new include file at /usr/local/nginx/conf/listener_http3.conf with the additional listen directive.

    Example
    Code (Text):
    listen 443 quic reuseport;

    and then each time centmin.sh menu option 4 is run to compile Nginx with NGINX_HTTP3='y' enabled for Cloudflare HTTP/3 patch + BoringSSL + Quiche builds is populate the new include file at /usr/local/nginx/conf/listener_http3.conf. Then include this include file in your Nginx vhosts:
    Code (Text):
    listen 443 ssl http2;
    include /usr/local/nginx/conf/listener_http3.conf;

    This way HTTP/3 required listen directive is automatically added with centmin.sh menu option 4 runs. If you set NGINX_HTTP3='n' to disable HTTP/3 Nginx patch, then centmin.sh menu option 4 will empty out /usr/local/nginx/conf/listener_http3.conf include file so nothing is used so as to remove this second listen directive.

    Progress :D

    centminmod-http3-branch-commits-171019-01.png
     
  12. buik

    buik Well-Known Member

    1,209
    321
    83
    Apr 29, 2016
    Ratings:
    +948
    Local Time:
    10:40 PM
    Both victims of the success.
    Cloudflare has open-sourced various functions via patches on the Nginx development mailing list.

    The Nginx team is very conservative and accepts almost no code from outside.
    As a result, both Nginx and Cloudflare pulls its hands off the code and does not keep the patches up-to-date.

    Exactly the same with OpenSSL.
    For example equal-preference groups of cipher suites presented by BoringSSL on a silver platter.
    And what does the OpenSSL team do. Refuse it and reinvent the wheel with its own Oompa Loompa solution:
    SSL_OP_PRIORITIZE_CHACHA.

    Wondering what OpenSSL and Nginx are coming with.
     
    • Like Like x 1
  13. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah true but understandable with Nginx's stance - they have to be picky with what code merges they make to upstream as it does affect alot of Nginx users. Great thing about open source = forks, patches etc allowing Cloudflare to share their alternative code. Same process which allows Centmin Mod LEMP stack to share it's version of how Nginx and PHP-FPM are built compared to official Nginx and PHP-FPM versions :D

    Yeah curious too, I think Nginx might still be waiting on OpenSSL to add/merge support from the OpenSSL forked versions with have HTTP/3 in development?
     
  14. buik

    buik Well-Known Member

    1,209
    321
    83
    Apr 29, 2016
    Ratings:
    +948
    Local Time:
    10:40 PM
    The easiest way for Cloudflare is to source their Nginx varaint on Github or related service. Problem.
    Too much competition will then get away with it.
    So it is not going to happen.

    Open source or no open source supporter as Cloudflare indicates to be.
    You want to remain number one and not just give everything away to the competitor.

    Same goes for their Cloudflare's BoringSSL fork with stapling support et all etc etc.

    OpenSSL is perhaps much more conservative than Nginx.
    I simply don't know what exactly is going on.

    Time issue over here. Too busy.
     
  15. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Yeah everyone is busy when you're in demand :)
    Yeah Cloudflare's OCSP Stapling support was built outside of BoringSSL so nothing to share there see High-reliability OCSP stapling and why it matters

     
  16. BamaStangGuy

    BamaStangGuy Active Member

    608
    179
    43
    May 25, 2014
    Ratings:
    +244
    Local Time:
    3:40 PM
    Does CloudFlare support OCSP stapling to orgin servers?
     
  17. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    I am going to have to ask Cloudflare folks about that one, usually OCSP stapling is for web browsers and revocation.
     
  18. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    As Cloudflare's Nginx HTTP/3 patch relies on building Nginx against BoringSSL/Quiche for crypto library for HTTPS support, was curious how BoringSSL performance compares with OpenSSL 1.1.1d (Centmin Mod Nginx's usual defaults) and also OpenSSL 1.0.2k CentOS 7 system defaults.

    Tests done on my Intel Core i7 4790K Haswell dedicated server with 4 cores/8 threads. Tests however are done against 1 cpu core due to BoringSSL's bssl binary not supporting -multi flag like OpenSSL's binary.
    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    2
    Core(s) per socket:    4
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 60
    Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
    Stepping:              3
    CPU MHz:               4200.439
    CPU max MHz:           4400.0000
    CPU min MHz:           800.0000
    BogoMIPS:              7981.91
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              8192K
    NUMA node0 CPU(s):     0-7
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
    

    OpenSSL 1.1.1d RSA 2048bit and ECDSA 256bit
    Code (Text):
    /opt/openssl/bin/openssl speed -multi 1 rsa2048 ecdsap256
    
    OpenSSL 1.1.1d  10 Sep 2019
    built on: Fri Oct 18 18:23:43 2019 UTC
    options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr)
    compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG
                     sign    verify    sign/s verify/s
    rsa 2048 bits 0.000487s 0.000022s   2051.4  45029.3
                                 sign    verify    sign/s verify/s
     256 bits ecdsa (nistp256)   0.0000s   0.0001s  43212.3  13410.0

    OpenSSL 1.0.2k RSA 2048bit and ECDSA 256bit
    Code (Text):
    /usr/bin/openssl speed -multi 1 rsa2048 ecdsap256
    
    OpenSSL 1.0.2k-fips  26 Jan 2017
    built on: reproducible build, date unspecified
    options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
                     sign    verify    sign/s verify/s
    rsa 2048 bits 0.000490s 0.000023s   2040.8  43478.3
                                 sign    verify    sign/s verify/s
     256 bit ecdsa (nistp256)   0.0000s   0.0001s  32258.1  12500.0

    BoringSSL RSA 2048bit
    Code (Text):
    /usr/local/bin/bssl speed -filter RSA
    
    Did 1935 RSA 2048 signing operations in 1015302us (1905.8 ops/sec)
    Did 37000 RSA 2048 verify (same key) operations in 1000820us (36969.7 ops/sec)
    Did 30000 RSA 2048 verify (fresh key) operations in 1022743us (29332.9 ops/sec)
    Did 180 RSA 4096 signing operations in 1000514us (179.9 ops/sec)
    Did 11000 RSA 4096 verify (same key) operations in 1022200us (10761.1 ops/sec)
    Did 9482 RSA 4096 verify (fresh key) operations in 1053081us (9004.1 ops/sec)

    BoringSSL ECDSA 256bit
    Code (Text):
    /usr/local/bin/bssl speed -filter ECDSA
    Did 8820 ECDSA P-224 signing operations in 1009853us (8733.9 ops/sec)
    Did 4290 ECDSA P-224 verify operations in 1062476us (4037.7 ops/sec)
    Did 45000 ECDSA P-256 signing operations in 1015598us (44308.9 ops/sec)
    Did 16000 ECDSA P-256 verify operations in 1036399us (15438.1 ops/sec)
    Did 924 ECDSA P-384 signing operations in 1092780us (845.5 ops/sec)
    Did 1166 ECDSA P-384 verify operations in 1086584us (1073.1 ops/sec)
    Did 440 ECDSA P-521 signing operations in 1083263us (406.2 ops/sec)
    Did 539 ECDSA P-521 verify operations in 1086361us (496.2 ops/sec)

    As expected from previous tests, BoringSSL is faster than OpenSSL 1.1.1 and OpenSSL 1.0.2 for ECDSA ciphers and OpenSSL 1.1.1 is faster than OpenSSL 1.0.2 for ECDSA ciphers and sign/s for ECDSA ciphers are faster than RSA 2048bit. Which is why for best HTTPS SSL performance you'd want to be using ECC 256bit ECDSA SSL certificates and not standard RSA 2048bit SSL certificates where possible i.e. SSL - Cloudflare - Improving Cloudflare Connections To Origin Server - Use ECDSA SSL Certs.

    However, for RSA 2048bit ciphers, BoringSSL is slower than OpenSSL 1.1.1d and OpenSSL 1.0.2k.

    These are only 1 cpu core tests as bssl binary doesn't support -multi flag like OpenSSL does so OpenSSL tests were done with -multi 1
    • Cloudflare Quiche's BoringSSL RSA 2048bit sign/op per second = 1905.8 ops/sec vs OpenSSL 1.1.1d = 2051.4 sign/s vs OpenSSL 1.0.2k = 2040.8 sign/s
    • Cloudflare Quiche's BoringSSL RSA 2048bit (same) sign/op per second = 36969.7 ops/sec vs OpenSSL 1.1.1d = 45029.3 verify/s vs OpenSSL 1.0.2k = 43478.3 verify/s
    • Cloudflare Quiche's BoringSSL ECDSA 256bit sign/op per second = 44308.9 ops/sec vs OpenSSL 1.1.1d = 43212.3 sign/s vs OpenSSL 1.0.2k = 32258.1 sign/s
    • Cloudflare Quiche's BoringSSL ECDSA 256bit verify/op per second = 15438.1 ops/sec vs OpenSSL 1.1.1d = 13410.0 sign/s vs OpenSSL 1.0.2k = 12500.0 verify/s
     
    • Informative Informative x 1
  19. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Some Centmin Mod Nginx HTTP/2 HTTPS benchmarks using h2load HTTP/2 tester on same server. No tools available right now for HTTP/3 benchmarking as yet. FYI, Nginx HTTP/3 patch's BoringSSL doesn't support Nginx dual RSA + ECDSA ssl certificates as that is only supported with OpenSSL. So tests only test ECDSA SSL certificates below.

    Nginx HTTP/2 HTTPS tests both gzip and brotli encoded requests and are an average of 5x h2load test runs each.

    Centmin Mod Nginx 1.17.4 + OpenSSL 1.1.1d without HTTP/3 patch



    requests/sec
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 13697.94 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 13023.80 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 13796.50 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 14282.42 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 13118.83 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 11132.57 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 9945.27 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 9036.14 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10326.26 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10247.48 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    -------------------------------------------------------------------------------------------
    h2load result summary
    min:      avg:       max:       stddev:   requests-succeeded:
    9036.140  11860.721  14282.420  1916.518  100.00
    -------------------------------------------------------------------------------------------
    

    Centmin Mod Nginx 1.16.1 + BoringSSL/Quiche with HTTP/3 patch



    requests/sec
    Code (Text):
    users requests req/s encoding cipher protocol started succeeded
    300 6000 13507.09 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 13661.51 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 12878.71 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 12835.46 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 14197.08 gzip ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10007.82 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10899.76 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 9894.54 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10782.19 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    300 6000 10341.12 br  ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 6000 6000
    -------------------------------------------------------------------------------------------
    h2load result summary
    min:      avg:       max:       stddev:   requests-succeeded:
    9894.540  11900.528  14197.080  1669.248  100.00
    -------------------------------------------------------------------------------------------
    


    I'd say it's fairly close for HTTP/2 benchmarks under h2load, favouring Nginx 1.17.4 with OpenSSL 1.1.1d without HTTP/3 patch for gzip encoded requests - requests/sec though seems close though there's a few dips in gzip encoded tests with Nginx 1.16.1 with Cloudflare HTTP/3 patch and BoringSSL/Quiche. But Nginx 1.16.1 with HTTP/3 patch and BoringSSL/Quiche makes up for it in brotli encodded HTTPS requests tests which end up making the average requests/seconds slightly higher.

    Added chart for 5x run h2load HTTP/2 benchmark averages for gzip and brotli encoded requests separately

    centminmod-nginx-cloudflare-http3-vs-cmm-nginx-h2load-01.png
     
  20. eva2000

    eva2000 Administrator Staff Member

    42,079
    9,499
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,615
    Local Time:
    7:40 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Another HTTP/3 video for folks interested in learning