Join the community today
Register Now

Nginx Security HTTP/2 Rapid Reset DDOS Attack Vulnerability CVE-2023-44487

Discussion in 'Nginx and PHP-FPM news & discussions' started by eva2000, Oct 11, 2023.

  1. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Cloudflare, Amazon AWS and Google announced a HTTP/2 Rapid Reset DDOS Attack vulnerability (CVE-2023-44487) that affects all web servers. Cloudflare CDN has already rolled out mitigation and protection for this vulnerability so if your Apache, Nginx, OpenLiteSpeed/Litespeed web server is behind Cloudflare with orange cloud proxy enabled, you should be protected.

    HTTP/2 Rapid Reset DDOS Attack Vulnerability Fix/Mitigation



    For Centmin Mod Nginx users that are not using Cloudflare orange cloud enabled proxying in front, Nginx has committed a fix but it's in their master branch (HTTP/2: per-iteration stream handling limit) and hasn't been released in Nginx 1.24 or 1.25 branches yet. They also mention how this patch fix will work in their blog post. I suspect Nginx will have versions 1.24.1 and 1.25.3 with the eventual committed fix.

    Luckily, Centmin Mod Nginx supports builds from bleeding edge Nginx master branch for testing and emergencies like this. Centmin Mod centmin.sh menu option 4 can build Nginx using master branch by just inputting the Nginx version when prompted as = master

    There maybe more commits for this vulnerability, so check the master branch link and use centmin.sh menu option 4 = master Nginx rebuilds as needed until official Nginx versions are released.

    Code (Text):
    --------------------------------------------------------
         Centmin Mod Menu 130.00beta01 centminmod.com
    --------------------------------------------------------
    1).  Centmin Install
    2).  Add Nginx vhost domain
    3).  NSD setup domain name DNS
    4).  Nginx Upgrade / Downgrade
    5).  PHP Upgrade / Downgrade
    6).  Option Being Revised (TBA)
    7).  Option Being Revised (TBA)
    8).  Option Being Revised (TBA)
    9).  Option Being Revised (TBA)
    10). Memcached Server Re-install
    11). MariaDB MySQL Upgrade & Management
    12). Zend OpCache Install/Re-install
    13). Install/Reinstall Redis PHP Extension
    14). SELinux disable
    15). Install/Reinstall ImagicK PHP Extension
    16). Change SSHD Port Number
    17). Multi-thread compression: zstd,pigz,pbzip2,lbzip2
    18). Suhosin PHP Extension install
    19). Install FFMPEG and FFMPEG PHP Extension
    20). NSD Install/Re-Install
    21). Data Transfer (TBA)
    22). Add Wordpress Nginx vhost + Cache Plugin
    23). Update Centmin Mod Code Base
    24). Exit
    --------------------------------------------------------
    Enter option [ 1 - 24 ] 4
    --------------------------------------------------------
    

    Code (Text):
    Nginx Upgrade - Would you like to continue? [y/n] y
    
    Current Nginx Version: 1.25.2 (041023-060220-almalinux9-384e3e2-br-659b4b3)
    
    Install which version of Nginx? (version i.e. type 1.25.2): master
    
    Do you still want to continue? [y/n] y
    

    End result is Centmin Mod Nginx built on master branch 1.25.3 version with the mitigation fix for HTTP/2 Rapid Reset DDOS Attack vulnerability just ~2hrs after Nginx committed the patch for CVE-2023-44487 :)
    Nginx folks have a blog for mitigation via Nginx configuration too https://www.nginx.com/blog/http-2-rapid-reset-attack-impacting-f5-nginx-products/

    From Cloudflare blog https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/
    From https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/

    More HTTP/2 Rapid Reset DDOS Attack Vulnerability coverage:

     
  2. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Found more response/commentary for CVE-2023-44487 by Maxim Dounin from Nginx for this at https://mailman.nginx.org/pipermail/nginx-devel/2023-October/S36Q5HBXR7CAIMPLLPRSSSYR4PCMWILK.html

    Seems some miscommunication from F5 folks https://mailman.nginx.org/pipermail/nginx-devel/2023-October/2YGF3BG7S47VPUNQJ3D5S6FODPMSAX4A.html

     
  3. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Added Centmin Mod Nginx master branch build automated testing for EL8/EL9 as well for AlmaLinux vs Rocky Linux vs Oracle Linux, so now for Nginx tests, I automate h2load HTTP/2 testing against Nginx OpenSSL 1.1.1 vs Nginx OpenSSL 3.0 vs Nginx QuicTLS vs Nginx BoringSSL vs Nginx OpenSSL 1.1.1 with official zlib instead of Cloudflare zlib vs Nginx master branch :D

    cmm-github-workflow-compare-almalinux8-rockylinux8-oraclelinux8-nginx-master-tests-01.png

    It looks like the Nginx 1.25.3 master branch patch for HTTP/2 Rapid Reset and other updates, improved h2load HTTP/2 benchmark performance over Nginx 1.25.2 latest public mainline version on 2x CPU Azure server running an Intel Xeon Platinum 8370C CPU @ 2.80GHz running AlmaLinux 8.8 with custom 5.15 Linux Kernel

    Code (Text):
    5.15.0-1047-azure
    AlmaLinux release 8.8 (Sapphire Caracal)
    
    Architecture:        x86_64
    CPU op-mode(s):      32-bit, 64-bit
    Byte Order:          Little Endian
    CPU(s):              2
    On-line CPU(s) list: 0,1
    Thread(s) per core:  1
    Core(s) per socket:  2
    Socket(s):           1
    NUMA node(s):        1
    Vendor ID:           GenuineIntel
    CPU family:          6
    Model:               106
    Model name:          Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
    Stepping:            6
    CPU MHz:             2793.438
    BogoMIPS:            5586.87
    Hypervisor vendor:   Microsoft
    Virtualization type: full
    L1d cache:           48K
    L1i cache:           32K
    L2 cache:            1280K
    L3 cache:            49152K
    NUMA node0 CPU(s):   0,1
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
    
    CPU Flags
     fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
     
    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE
    0   0    0      0    0:0:0:0       yes
    1   0    0      1    1:1:1:0       yes
    
                  total        used        free      shared  buff/cache   available
    Mem:           6932        1377         443          43        5111        5201
    Low:           6932        6488         443
    High:             0           0           0
    Swap:          4095          22        4073
    
    Filesystem      Size  Used Avail Use% Mounted on
    overlay          84G   73G   11G  88% /
    tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
    tmpfs            64M     0   64M   0% /dev
    shm              64M     0   64M   0% /dev/shm
    tmpfs            64M  6.5M   58M  11% /run
    tmpfs           4.0M     0  4.0M   0% /run/lock
    devtmpfs        3.4G     0  3.4G   0% /dev/tty
    /dev/root        84G   73G   11G  88% /etc/hosts
    tmpfs           3.4G     0  3.4G   0% /proc/acpi
    tmpfs           3.4G     0  3.4G   0% /proc/scsi
    tmpfs           3.4G     0  3.4G   0% /sys/firmware
    tmpfs           3.4G  8.0K  3.4G   1% /tmp


    FYI, Centmin Mod Nginx defaults to OpenSSL 1.1.1 with Cloudflare zlib performance forked library unless otherwise stated. You can see how official zlib is much slower than Cloudflare zlib performance forked Nginx build. Below are h2load HTTP/2 benchmark tests with Centmin Mod 130.00beta01's Nginx

    cmm-github-workflow-compare-almalinux8-rockylinux8-oraclelinux8-nginx-master-tests2-table-01.png
     
  4. Jon Snow

    Jon Snow Active Member

    727
    150
    43
    Jun 30, 2017
    Ratings:
    +215
    Local Time:
    11:44 PM
    Nginx 1.13.9
    MariaDB 10.1.31
    So I just type:
    Code (Text):
    master

    When prompted to enter the nginx version and not a specific nginx version followed by "master"?

    Do you know anyone who has been affected by this?
     
  5. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Yup. Don't know of any Centmin Mod users impacted but there's up to 100,000 web sites that use Centmin Mod AFAIK, so who knows.
     
  6. Itworx4me

    Itworx4me Active Member

    285
    26
    28
    Mar 14, 2017
    Ratings:
    +45
    Local Time:
    7:44 PM
    Nginx 1.25.X
    MariaDB 10.6.X
    I tried running the master version of nginx and it crashed nginx. Had to revert back to 1.25.2 in order to access my site. Not sure why....
     
  7. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Crashed is a vague term, describe the specific symptoms and error messages i.e. check nginx config check command's output
    Code (Text):
    nginx -t
    

    But might be unrelated to Cloudflare zlib library breaking change for Nginx compiles at https://community.centminmod.com/threads/24139/

    So always run cmupdate command BEFORE running centmin.sh menu options to ensure you have latest Centmin Mod code updates first.
     
  8. Andy

    Andy Active Member

    523
    85
    28
    Aug 6, 2014
    Ratings:
    +124
    Local Time:
    10:44 PM
    Same issue with @Itworx4me

    # nginx -t
    nginx: [emerg] unknown directive "pagespeed" in /usr/local/nginx/conf/pagespeedadmin.conf:9
    nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed

    I have to comment out this line in the nginx.conf file and do nprestart to restart nginx.
    #include /usr/local/nginx/conf/pagespeedadmin.conf;
     
    Last edited: Oct 20, 2023
  9. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    ngx_pagespeed should be disabled in latest 124.00stable and 130.00beta01 beta versions of Centmin Mod by default already. What version of Centmin Mod are you using.
     
  10. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    Just a strange CVE. First, a programmer at Nginx wrote that their product is not subject to, only to release a patch for the paid Nginx Plus edition. But not for the open-source version, as of today (same source code).

    https://www.nginx.com/blog/http-2-rapid-reset-attack-impacting-f5-nginx-products/

    Even third parties have already released a patch. And even Red Hat.
    Who tend to be relatively conservative in releasing patches and prefer to test it a bit longer
    (better safe than sorry principle).

    https://gitlab.com/redhat/centos-stream/rpms/nginx/-/commit/ba15c0f4a408e728b64b92d6d478ec8afac68732

    https://access.redhat.com/errata/RHSA-2023:5713

    Now in all honesty, I have to admit that I didn't put super much time into this case, I am simply far too busy. But what is truth? Remarkable, to say the least.
     
  11. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    @eva2000 Something is going wrong on your forum apparently? My response is published simultaneously, multiple times.
     
  12. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Yeah that sometimes happens heh

    The patch nginx has in master branch is to further enhance mitigation and not to fix it according to them, they consider nginx default settings out of box enough to mitigate.

    But yeah I would consider nginx impacted myself.
     
  13. Andy

    Andy Active Member

    523
    85
    28
    Aug 6, 2014
    Ratings:
    +124
    Local Time:
    10:44 PM
    Centmin Mod Menu 130.00beta01 centminmod.com
    Did I fix that issue correctly or is there the right way to do it?
     
  14. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    Exactly this. The default settings that should provide enough resistance. Only to then do release an update to their paid service. And the free edition doesn't. Nice that the patch is in git though, that is of no use to the average user.

    We've talked about this before. It just goes to show where the priority is at Nginx, these days.
     
  15. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    Money talks as expected as that money keeps Nginx free open source running. I have paid clients hire me to extend and customise Centmin Mod for the specific needs :D

    But after developing Centmin Mod, I understand something that I didn't before, as your project gets wider adoption, you have to be more cautious releasing new features or code as the impact is greater once you have more users. A controlled release to smaller segments can be more beneficial to tackling issues that may arise. Just so happens for Nginx that smaller group is also their paid Nginx Plus customers

    Yup that fixes it.
     
  16. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    Clients asking for customization and/ or custom-made features, and you ask those customers for a fee is obvious.

    I see a substantial difference here, positively in your favor.
    You did release a fix, almost instantaneously (also) on the forum.

    Nginx does the exact opposite. You could get their beta (mainline) bits. But should there be a CVE, Nginx plus takes priority. Exactly the same with CentOS Stream and RHEL.
     
  17. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
  18. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    QUIC is a complete disaster.

    Not so much at the hands of team Nginx. But those Goof's from OpenSSL. Who despite much opposition from the community. Still stubbornly reinvent the wheel with their own QUIC implementation.

    While there was a commit ready to insert the full feature right away.
    The only logical reason I can think of to reject that: OpenSSL is largely made up of freelancers. And they can write hours now and with a ready-made commit so they can't.

    And yes QUIC on Nginx can be solved with quictls, but I don't believe the gross of users, know about that.
     
  19. eva2000

    eva2000 Administrator Staff Member

    51,209
    11,897
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,370
    Local Time:
    1:44 PM
    Nginx 1.25.x
    MariaDB 10.x
    +1. Not to mention the OpenSSL 3.0 and 3.1 performance regressions compared to OpenSSL 1.1.1 https://github.com/openssl/openssl/issues/17064. Though for Nginx OpenSSL usage, there difference for HTTP/2 at least isn't that bad as my testing is single threaded. But still OpenSSL 1.1.1 > OpenSSL 3.1 > OpenSSL 3.0 in terms of best to worst performers.

    Though these days, I am not as concerned. Just put site behind Cloudflare and let it do the HTTP/3 QUIC on their CDN edge servers :D
     
  20. buik

    buik “The best traveler is one without a camera.”

    1,944
    507
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,620
    Local Time:
    4:44 AM
    I'm a little subscription tired these days. They plop everything behind subscription form these days. That you have to depend on a far too large party just to play a decent movie, delivering a simple package without unnecessary costs, or in the website area that you're paying through the nose; Because a software party is arguing over Quic. While the code is already there. Too much politics. They are just pussyfooting around.

    Source: https://nitter.net/jdevalk/status/1633028813880799233#m