/ Register

Join the community today
Register Now

Wordpress Suspected bug with Redis Caching when Accept-Encoding is "gzip, deflate, br"

Discussion in 'Blogs & CMS usage' started by ct_roy, Feb 11, 2024.

Previous Thread Next Thread
Loading...
  1. Feb 11, 2024 #1
    ct_roy

    ct_roy Premium Member Premium Member

    55
    9
    8
    Jun 21, 2020
    Ratings:
    +17
    Local Time:
    3:38 AM
    1.17.10
    10.3.22
    This is a VERY peculiar issue which took me forever to figure out...

    CentOS Linux release 7.9.2009 (Core)
    Centmin Mod version: 130.00beta01
    nginx version: nginx/1.25.3 (091123-121301-centos7-kvm-48ad4d7)
    WordPress installed with option 2: Redis Nginx Level Caching
    nginx helper plugin configured and working correctly.

    I have Cloudflare in front of my website with the proxy enabled (but no APO, caching rules etc.)

    Everything works perfectly fine with the caching config when I access the website via Cloudflare.

    Redis cache keys look like:

    Code:
    1707585329.721277 [0 127.0.0.1:41916] "get" "nginx-cache:httpsGETwww.mydomain.com/mypage/gzip"
    and X-Cache: returns a HIT for $srcache_fetch_status as expected.

    I had to briefly set a host header in /etc/hosts on my local workstation to bypass Cloudflare to do some testing before I moved the site from one server to another, when I noticed that X-Cache: persistently returned a MISS status which I thought was unusual.

    Running redis-cli->MONITOR I then noticed that the cache key was storing a different value for http_accept_encoding as follows:

    Code:
    1707585546.242559 [0 127.0.0.1:41916] "get" "nginx-cache:httpsGETwww.mydomain.com/mypage/gzip,%20deflate,%"
    this then triggered a srcache_store event with the following key:

    Code:
    1707585546.942840 [0 127.0.0.1:41916] "set" "nginx-cache:httpsGETwww.mydomain.com/mypage/gzip, deflate, br"
    Note the difference in the values for http_accept_encoding ?

    I wonder is this due to the differences in how the keys are set when there are multiple accepted encoding values?

    Code:
       
srcache_fetch GET /redis-fetch $key;
srcache_store PUT /redis-store key=$escaped_key;
    CloudFlare seems to only send "gzip" as the Accept-Encoding value, whereas when I bypass Cloudflare completey, Chrome is sending "gzip, deflate, br" as the Accept-Encoding value.
     
  2. Feb 11, 2024 #2
    eva2000

    eva2000 Administrator Staff Member

    53,461
    12,128
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,668
    Local Time:
    12:38 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    From my understanding that's how latest Centmin Mod Wordpress provided guest full HTML page caching is setup i.e. for Redis the cache key is now adding accept encoding request as part of the key to ensure caches return the properly encoded response i.e. gzip or brotli

    Centmin Mod Wordpress redis cached PHP include file cache key set as below in /usr/local/nginx/conf/php-rediscache.conf
    Code (Text):
    set $key nginx-cache:$scheme$request_method$host$request_uri$http_accept_encoding;

    This was added ~6 months ago via commit update centmin.sh menu option 22 cache keys in 130.00beta01 · centminmod/centminmod@6bbb4dd

    But is subsequent Cloudflare bypassed requests returning a HIT with cached value of
    Code (Text):
    "nginx-cache:httpsGETwww.mydomain.com/mypage/gzip, deflate, br"

    ? Or is it always a cache miss?

    That is the default Cloudflare behaviour as traditionally it only supported native gzip encoded requests connecting back to origin server to request gzip encoded content from the origin. But Cloudflare now optionally supports brotli native requests too All the way up to 11: Serve Brotli from origin and Introducing Compression Rules and you can have finer grain control via Cloudflare Compression Rules Cloudflare Rules · Cloudflare Rules docs and Compression Rules · Cloudflare Rules docs
     
  3. Feb 11, 2024 #3
    ct_roy

    ct_roy Premium Member Premium Member

    55
    9
    8
    Jun 21, 2020
    Ratings:
    +17
    Local Time:
    3:38 AM
    1.17.10
    10.3.22
    it's always a miss.
    likely because of the difference in set/get rules

    if you look again the SET rule ends with "gzip,%20deflate," and GET rule ends with "gzip, deflate, br"
     
  4. Feb 11, 2024 #4
    eva2000

    eva2000 Administrator Staff Member

    53,461
    12,128
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,668
    Local Time:
    12:38 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    I suspected as much. Hmm going to have to think on this - maybe a way to normalize the http_accept_encoding variable's value if you're not using Cloudflare in front of Centmin Mod Nginx.

    Maybe try adding new nginx include file named /usr/local/nginx/conf/normalize_encoding.conf with contents
    Code (Text):
    map $http_accept_encoding $normalized_encoding {
    ~*br     br;
    ~*gzip   gzip;
    default  "";
}

    and in /usr/local/nginx/conf/nginx.conf http{} context add
    Code (Text):
    include /usr/local/nginx/conf/normalize_encoding.conf;

    after existing include file /usr/local/nginx/conf/default_phpupstream.conf
    Code (Text):
    include /usr/local/nginx/conf/default_phpupstream.conf;
include /usr/local/nginx/conf/normalize_encoding.conf;

    add to nginx vhost location of your choosing a new debug header
    Code (Text):
            location / {
            add_header X-Normalized-Encoding $normalized_encoding;
        }

    restart nginx server
    Code (Text):
    ngxrestart

    or
    Code (Text):
    systemctl restart nginx

    use curl to check assigned value for $normalized_encoding variable ensuring location you're testing exists on server and ensure the custom response header set shows up for X-Normalized-Encoding
    Code (Text):
    curl -I -H "Accept-Encoding: gzip, br" http://localhost/
HTTP/1.1 200 OK
Date: Sat, 10 Feb 2024 21:05:06 GMT
Content-Type: text/html; charset=utf-8
Last-Modified: Sat, 11 Nov 2023 17:24:02 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"654fb8b2-11a3"
Server: nginx centminmod
X-Powered-By: centminmod
X-Normalized-Encoding: br
Content-Encoding: br

    Code (Text):
    curl -I -H "Accept-Encoding: gzip" http://localhost/
HTTP/1.1 200 OK
Date: Sat, 10 Feb 2024 21:05:16 GMT
Content-Type: text/html; charset=utf-8
Last-Modified: Sat, 11 Nov 2023 17:24:02 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"654fb8b2-11a3"
Server: nginx centminmod
X-Powered-By: centminmod
X-Normalized-Encoding: gzip
Content-Encoding: gzip


    If that works, then modify the redis php include file redis cache key

    from
    Code (Text):
    set $key nginx-cache:$scheme$request_method$host$request_uri$http_accept_encoding;

    to use $normalized_encoding normalized version of $http_accept_encoding
    Code (Text):
    set $key nginx-cache:$scheme$request_method$host$request_uri$normalized_encoding;
     
    Last edited: Feb 11, 2024
  5. Feb 11, 2024 #5
    ct_roy

    ct_roy Premium Member Premium Member

    55
    9
    8
    Jun 21, 2020
    Ratings:
    +17
    Local Time:
    3:38 AM
    1.17.10
    10.3.22
    @eva2000 you are an absolute legend!
    that works perfectly!
    thanks so much for solving that so quickly AND on a weekend.
    do you ever sleep? :)
     
  6. Feb 12, 2024 #6
    eva2000

    eva2000 Administrator Staff Member

    53,461
    12,128
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,668
    Local Time:
    12:38 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    So it works in terms of cache hit right? The thing I see as an issue is if browser request supprts Accept-Encoding gzip, deflate, br values and Centmin Mod Nginx isn't built with optional Brotli Nginx module (default isn't installed), so there's a chance that while web browser client supports Brotli advertised by Accept-Encoding request header and Centmin Mod Nginx doesn't support Brotli, it would return a gzip compressed encoded response but have a redis cache key tagged by br from $normalized_encoding value. Do you have Nginx brotli enabled on Centmin Mod https://community.centminmod.com/th...sion-for-centmin-mod-nginx-web-servers.10688/

    Not sure how you ended up with gzip,%20deflate,% value for original $http_accept_encoding value in redis cache key as pretty sure most web browsers now support brotli so would expect maybe value of gzip,%20deflate,%20br

    What web browser did you test with when overridden DNS?
     
  7. Feb 12, 2024 #7
    ct_roy

    ct_roy Premium Member Premium Member

    55
    9
    8
    Jun 21, 2020
    Ratings:
    +17
    Local Time:
    3:38 AM
    1.17.10
    10.3.22
    Yes correct.
    No - I didn't follow the steps to enable Brotli compression, but... interestingly, checking this on my 2 most recent Centmin Mod machines, looks like Brotli is in place on the AlmaxLinux version.

    Machine 1:
    AlmaLinux release 8.9 (Midnight Oncilla)
    Centmin Mod 130.00beta01

    Brotli looks to be in place
    Code:
    nginx version: nginx/1.25.3 (140124-161036-almalinux8-kvm-a6feed4-br-a71f931)
built with OpenSSL 3.1.4 24 Oct 2023
TLS SNI support enabled
configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/nginx-dep/lib -lpcre2-8 -ljemalloc -Wl,-z,relro,-z,now -Wl,-rpath,/usr/local/nginx-dep/lib:/usr/local/zlib-cf/lib:/usr/local/nginx-dep/lib -pie -B/usr/local/libexec/mold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/nginx-dep/include -m64 -march=x86-64-v3 -mavx -mavx2 -mpclmul -msse4 -msse4.1 -msse4.2 -fPIC -g -O3 -Wno-strict-aliasing -fstack-protector-strong -B/usr/local/libexec/mold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wno-pointer-sign -Wimplicit-fallthrough=0 -Wno-implicit-function-declaration -Wno-int-conversion -Wno-error=unused-result -Wno-unused-result -Wno-error=vla-parameter -fcode-hoisting -Wno-cast-function-type -Wno-format-extra-args -Wp,-D_FORTIFY_SOURCE=2' --prefix=/usr/local/nginx --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=140124-161036-almalinux8-kvm-a6feed4-br-a71f931 --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-http_flv_module --with-http_mp4_module --add-module=../nginx-rtmp-module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --add-dynamic-module=../njs/nginx --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5.1 --add-dynamic-module=../ngx_devel_kit-0.3.2 --add-dynamic-module=../set-misc-nginx-module-0.33 --add-dynamic-module=../echo-nginx-module-0.63 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.4.0-cmm --add-module=../memc-nginx-module-0.19 --add-module=../srcache-nginx-module-0.33 --add-dynamic-module=../headers-more-nginx-module-0.34 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.3 --with-zlib-opt=-fPIC --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-3.1.4 --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold'

    Machine 2:
    CentOS Linux release 7.9.2009 (Core)
    Centmin Mod 130.00beta01

    Code:
    nginx version: nginx/1.25.3 (091123-121301-centos7-kvm-48ad4d7)
built by gcc 10.2.1 20210130 (Red Hat 10.2.1-11) (GCC)
built with OpenSSL 1.1.1w  11 Sep 2023
TLS SNI support enabled
configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/nginx-dep/lib -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/nginx-dep/lib -flto=4 -fuse-ld=gold' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/nginx-dep/include -m64 -march=x86-64 -g -O3 -fstack-protector-strong -flto=4 -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wno-pointer-sign -Wimplicit-fallthrough=0 -Wno-missing-profile -Wno-implicit-function-declaration -Wno-int-conversion -Wno-unused-result -Wno-unused-result -Wno-stringop-overflow -fcode-hoisting -Wno-cast-function-type -Wno-format-extra-args -Wno-vla-parameter -Wno-stringop-overflow -Wp,-D_FORTIFY_SOURCE=2' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=091123-121301-centos7-kvm-48ad4d7 --with-compat --without-pcre2 --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5.1 --add-dynamic-module=../ngx_devel_kit-0.3.2 --add-dynamic-module=../set-misc-nginx-module-0.33 --add-dynamic-module=../echo-nginx-module-0.63 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.4.0-cmm --add-module=../memc-nginx-module-0.19 --add-module=../srcache-nginx-module-0.33 --add-dynamic-module=../headers-more-nginx-module-0.34 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.3 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1w --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3 -fuse-ld=gold'

    Chrome 121.0.6167.85 (macOS 14.2.1)
     
  8. Feb 12, 2024 #8
    eva2000

    eva2000 Administrator Staff Member

    53,461
    12,128
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,668
    Local Time:
    12:38 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    If you used EL8+ edge installers, they have Nginx Brotli module enabled by default. The default EL8+ installers don't have Nginx Brotli enabled.
    Cheers, interesting. Anyway, I implemented the fix outlined at https://community.centminmod.com/th...encoding-is-gzip-deflate-br.24435/#post-98553 in 130.00beta01 to see how it goes. Existing users can just run cmupdate, and run and exit centmin.sh once to apply the updates.
     
  9. Feb 12, 2024 #9
    ct_roy

    ct_roy Premium Member Premium Member

    55
    9
    8
    Jun 21, 2020
    Ratings:
    +17
    Local Time:
    3:38 AM
    1.17.10
    10.3.22
    Ah yip I did indeed use the EL8+ edge installer :)

    Fantastic! Will let you know if I spot any other side effects. thanks again!
     
  10. Sep 25, 2024 #10
    Konstantin

    Konstantin New Member

    2
    0
    1
    Sep 25, 2024
    Ratings:
    +0
    Local Time:
    7:38 AM
    Are there any reasons to store different versions of the cache depending on the encoding? It seems like the values stored in Redis are the same for Gzip, Brotli and not encoded.

    Is it OK to replace
    Code:
    set $key "nginx-cache:$scheme$request_method$host$request_uri$normalized_encoding";
    with
    Code:
    set $key "nginx-cache:$scheme$request_method$host$request_uri";
    ?
     
  11. Sep 25, 2024 #11
    eva2000

    eva2000 Administrator Staff Member

    53,461
    12,128
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,668
    Local Time:
    12:38 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    The reason was to fix the original poster's issue and the solution is what Centmin Mod defaults to now https://community.centminmod.com/th...encoding-is-gzip-deflate-br.24435/#post-98553

    You could try changing it and see if it works for you. I guess you just want to improve cache hit rate? Not sure changing it back to what you propose will make any dramatic performance improvements
     
Previous Thread Next Thread
Loading...

.