Wordpress Cache-Enabler Plugin Ensuring urls with utm params, fbclid, adwords params are served from cache

ct_roy · Aug 26, 2020

How do I ensure that urls with lots of utm params and strings from FB ads, Google ads etc. are served from cache?

A site I run just got hammered yesterday with traffic from a few different places - this traffic was going to WP pages that would have been served from cache if it were not for the utms etc.

I was using the nginx/redis cache option. In rediscache_mysite.com.conf I can see:
Code:
if ($query_string != "") {
  set $skip_cache 1;
}
which is a bit too general imho.

I then switched to using Cache Enabler which actually has an option to ensure urls with params are served from cache via Cache Inclusions:

wpcacheenabler_mysite.com I also had to comment out this
Code:
    #if ($query_string != "") {
    #    set $cache_uri 'nullcache';
    #}
I'd prefer to go back to using nginx/redis so as to avoid going near the php worker processes for these requests at all. Can anyone think of a simple modification to the nginx redis cache config that will allow these kind of urls to be served from cache?
Thanks.

eva2000 · Aug 26, 2020

Cache Enabler used in centmin.sh menu option 22 bypasses PHP with advanced cache until default Cache Enabler stock out of the box. So if you switch to centmin.sh menu option 22's Cache Enabler you get to serve those query strings via cache + bypass PHP

ct_roy · Aug 26, 2020

eva2000 said: ↑

Cache Enabler used in centmin.sh menu option 22 bypasses PHP with advanced cache until default Cache Enabler stock out of the box. So if you switch to centmin.sh menu option 22's Cache Enabler you get to serve those query strings via cache + bypass PHP
Click to expand...

thanks for the quick reply @eva2000 much appreciated.
That is indeed what I'm using.
Correct me if I'm wrong, but doesn't this:
Code:
    
if ($query_string != "") {
set $cache_uri 'nullcache';
}
mean that nginx itself isn't serving the cached file if there's a query string present? (which is what I'd like to have happen).
and if nginx sets a nullcache if there's a query string then how is php still bypassed?

eva2000 · Aug 26, 2020

You can also in Cache Enabler include file below existing $query_string if match add the following in /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf

Code (Text):

    if ($query_string != "") {
        set $cache_uri 'nullcache';
    }
    if ($q_ignorearg) {
      rewrite ^ $uri? redirect;
    }

then in /usr/local/nginx/conf/nginx.conf within http{} context add a mapping for instead add it to already generated /usr/local/nginx/conf/wpcacheenabler_map.conf the below map to existing map

Code (Text):

map $args $q_ignorearg {
  default             0;
  ~*fbclid            1;
  ~*gclid             1;
  ~*utm               1;
}

so /usr/local/nginx/conf/wpcacheenabler_map.conf looks like

Code (Text):

map $http_user_agent $cmwpcache_device {
    default                                     'desktop';
    ~*(iPad|iPhone|Android|IEMobile|Blackberry) 'mobile';
    "~*Firefox.*Mobile"                         'mobile';
    "~*ipod.*mobile"                            'mobile';
    "~*Opera\ Mini"                             'mobile';
    "~*Opera\ Mobile"                           'mobile';
    "~*Mobile"                                  'mobile';
    "~*Tablet"                                  'mobile';
    "~*Kindle"                                  'mobile';
    "~*Windows\ Phone"                          'mobile';
}

map $args $q_ignorearg {
  default             0;
  ~*fbclid            1;
  ~*gclid             1;
  ~*utm               1;
}

Then it will strip query strings defined in map $args to $q_ignorearg via 302 redirect

Code (Text):

curl -I http://cache-enabler.domain.com/?fbclid
HTTP/1.1 302 Moved Temporarily
Date: Wed, 26 Aug 2020 12:59:44 GMT
Content-Type: text/html
Content-Length: 138
Location: http://cache-enabler.domain.com/
Connection: keep-alive
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Set-Uri: nullcache

Code (Text):

curl -I http://cache-enabler.domain.com/?gclid
HTTP/1.1 302 Moved Temporarily
Date: Wed, 26 Aug 2020 12:50:07 GMT
Content-Type: text/html
Content-Length: 138
Location: http://cache-enabler.domain.com/
Connection: keep-alive
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Set-Uri: nullcache

Code (Text):

curl -I http://cache-enabler.domain.com/?utm
HTTP/1.1 302 Moved Temporarily
Date: Wed, 26 Aug 2020 12:50:12 GMT
Content-Type: text/html
Content-Length: 138
Location: http://cache-enabler.domain.com/
Connection: keep-alive
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Set-Uri: nullcache

wrk load test

Code (Text):

wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?gclid
Running 5s test @ http://cache-enabler.domain.com/?gclid
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   610.19us    1.61ms  37.39ms   93.47%
    Connect   176.25us   84.89us 337.00us   62.50%
    TTFB      607.10us    1.61ms  37.39ms   93.47%
    TTLB        1.90us   13.49us   5.01ms   99.99%
    Req/Sec    44.14k    13.16k   81.15k    65.67%
  Latency Distribution
     50%  182.00us
     75%  341.00us
     90%    1.13ms
     95%    3.04ms
     99%    7.80ms
  883043 requests in 5.10s, 388.22MB read
Requests/sec: 173190.48
Transfer/sec:     76.14MB

Code (Text):

wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?fbclid
Running 5s test @ http://cache-enabler.domain.com/?fbclid
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.17ms    5.16ms  90.48ms   96.32%
    Connect     1.13ms    0.96ms   3.45ms   63.39%
    TTFB        1.17ms    5.16ms  90.48ms   96.32%
    TTLB        1.97us   26.28us   7.88ms   99.99%
    Req/Sec    42.86k    13.12k   66.46k    58.50%
  Latency Distribution
     50%  179.00us
     75%  381.00us
     90%    1.44ms
     95%    4.59ms
     99%   24.79ms
  854923 requests in 5.03s, 375.86MB read
Requests/sec: 170129.87
Transfer/sec:     74.80MB

Code (Text):

wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?utm                             
Running 5s test @ http://cache-enabler.domain.com/?utm
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   838.50us    3.99ms  83.77ms   96.80%
    Connect   186.48us   89.73us 348.00us   62.50%
    TTFB      835.26us    3.99ms  83.76ms   96.80%
    TTLB        2.01us   31.77us   9.01ms   99.99%
    Req/Sec    43.97k    14.27k   84.01k    63.18%
  Latency Distribution
     50%  177.00us
     75%  357.00us
     90%  753.00us
     95%    2.57ms
     99%   15.19ms
  880017 requests in 5.10s, 386.89MB read
Requests/sec: 172567.75
Transfer/sec:     75.87MB

Going to update centmin.sh menu option 22's Cache Enabler configurations for new nginx vhost creations to have this supported too by end of today. So a cmupdate will update your local code too. But won't update existing Wordpress installs, so you'd have to do it manually as outlined above in this post for existing Wordpress vhosts and include files.

eva2000 · Aug 26, 2020

eva2000 said: ↑

Going to update centmin.sh menu option 22's Cache Enabler configurations for new nginx vhost creations to have this supported too by end of today.
Click to expand...

done Beta Branch - update centmin.sh menu option 22 cache enabler cache config in 123.09beta01

ct_roy · Aug 27, 2020

@eva2000 AMAZING! thank you SO much! that works brilliantly!

I'm wondering if the 302 redirect will cause issues w/ the likes of Google Analytics no longer recognizing the utm's as they'll probably never see them now?

eva2000 · Aug 27, 2020

ct_roy said: ↑

I'm wondering if the 302 redirect will cause issues w/ the likes of Google Analytics no longer recognizing the utm's as they'll probably never see them now?
Click to expand...

Not sure. But yeah seems like the price you pay for such.

ct_roy · Aug 27, 2020

eva2000 said: ↑

Not sure. But yeah seems like the price you pay for such.
Click to expand...

yip fair enough - let me run some tests and I'll report back my findings. Thanks again for your rapid help on this - I really appreciate it.

ct_roy · Aug 27, 2020

@eva2000 I've not had a chance to test and report back on this yet. But an unrelated but VERY important thing I just noticed re: Cache Enabler's default nginx rules is reported here
Wrong nginx recommendations in docs | WordPress.org

In summary, Cache Enabler has changed the naming conventions for their file caches (and they've not updated their docs yet ?!?).

Minor tweak that's needed for https hosts in
/usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf

is:

set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}index.html';

becomes
set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}https-index.html';

This caused a more than a few minutes of head scratching today

eva2000 · Aug 27, 2020

ct_roy said: ↑

https-index.html
Click to expand...

Thanks for the heads up. You should able to replace https part with nginx variable $scheme to support both https and non-https http://nginx.org/en/docs/http/ngx_http_core_module.html#var_scheme

eva2000 · Aug 27, 2020

ct_roy said: ↑

This caused a more than a few minutes of head scratching today
Click to expand...

might be a bug if you're testing a non-https wordpress site schema based cache file generation bug in v1.4.6 ? · Issue #108 · keycdn/cache-enabler ?

I posted a pull request fix at fix HTTPS scheme prefix detection for nginx/php-fpm servers by centminmod · Pull Request #109 · keycdn/cache-enabler so will see if that gets accepted and merged but bug only would apply if you're using non-https wordpress site versus https based wordpress site.

ct_roy · Aug 27, 2020

eva2000 said: ↑

might be a bug if you're testing a non-https wordpress site schema based cache file generation bug in v1.4.6 ? · Issue #108 · keycdn/cache-enabler ?

I posted a pull request fix at fix HTTPS scheme prefix detection for nginx/php-fpm servers by centminmod · Pull Request #109 · keycdn/cache-enabler so will see if that gets accepted and merged but bug only would apply if you're using non-https wordpress site versus https based wordpress site.
Click to expand...

Thankfully I didn't have that bug. I just replied to you on the Cache Enabler support forum So I won't repeat myself here. Other than to say my tests conclude that your solution is great as long as you're not worried about attribution or any other events that the param might be needed for.

Here's another example. Klaviyo adds a _ke param to urls being sent from their platform. We have one site with over 500k subscribers on it. If we send them an email, we'll generate 500k requests all with the _ke param in it. If we use the current solution we lose the features that Klaviyo provide for us via the _ke param. If we don't remove the param, we'll DDOS ourselves

I'd ideally like to find a way to preserve the params in the final url sent back to the client, but pluck from cache the file without the params.

Aside: all the big tech co's are now adding these params to their links as a way to try and workaround ITP/privacy regulations and in effect are destroying the kind of fundamentals us webdevs have relied on for years (i.e. don't cache urls with params).
This is becoming a big problem for lots of high volume websites.

eva2000 · Aug 28, 2020

ct_roy said: ↑

Thankfully I didn't have that bug. I just replied to you on the Cache Enabler support forum
Click to expand...

Ah interesting, I believe this may work

edit /usr/local/nginx/conf/wpcacheenabler_map.conf and change the previous $q_ignorearg nginx map to below one
Code (Text):
map $http_user_agent $cmwpcache_device {
    default                                     'desktop';
    ~*(iPad|iPhone|Android|IEMobile|Blackberry) 'mobile';
    "~*Firefox.*Mobile"                         'mobile';
    "~*ipod.*mobile"                            'mobile';
    "~*Opera\ Mini"                             'mobile';
    "~*Opera\ Mobile"                           'mobile';
    "~*Mobile"                                  'mobile';
    "~*Tablet"                                  'mobile';
    "~*Kindle"                                  'mobile';
    "~*Windows\ Phone"                          'mobile';
}

map $args $q_ignorearg {
  default                 0;
  "~*fbclid"            1;
  "~*gclid"             1;
  "~*utm"               1;
  "~*fb_action_ids"     1;
  "~*fb_action_types"   1;
  "~*fb_source"         1;
  "~*age-verified"      1;
  "~*ao_noptimize"      1;
  "~*usqp"              1;
  "~*cn-reloaded"       1;
  "~*_ga"               1;
  "~*_ke"               1;
}
then in your generated /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf include file change to
Code (Text):
    if ($query_string != "") {
        set $cache_uri 'nullcache';
    }

    if ($q_ignorearg) {
      set $check_qurl $request_uri;
      set $check_surl $request_uri;
      set $cache_uri $uri;
     #rewrite ^ $uri? redirect;
    }
    add_header Check-Querystring-Uri "$check_qurl";
restart nginx and php-fpm
Code (Text):
nprestart
then in WP Admin cache enabler settings add to cache inclusions, the regex
Code (Text):
/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/
Then verify by first visiting the wordpress blog page in web browser as guest to populate the cache and then inspect the source code footer for cache enabler timestamp
Code (Text):
curl -sL "http://cache-enabler.domain.com/?fbclid" 2>&1 | tail -2

Code (Text):
curl -sL "http://cache-enabler.domain.com/?gclid" 2>&1 | tail -2

Code (Text):
curl -sL "http://cache-enabler.domain.com/?fb_action_ids" 2>&1 | tail -2

Code (Text):
curl -sL "http://cache-enabler.domain.com/?fb_source" 2>&1 | tail -2

Code (Text):
curl -sL "http://cache-enabler.domain.com/?f_ke" 2>&1 | tail -2

Code (Text):
curl -sL "http://cache-enabler.domain.com/1/hello-world/?_ke" 2>&1 | tail -2

Edit: looks like while the cache inclusion defined query strings are cached, they are cached and processed by PHP itself and not at nginx level, so there's more cpu load than the previous 302 redirect + stripe query string method as that is served via nginx cache and not php. I think I might have an idea what's going on so will see. Ok figured it out needed to change in your generated /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf include file change to
Code (Text):
    if ($query_string != "") {
        set $cache_uri 'nullcache';
    }

    if ($q_ignorearg) {
      set $check_qurl $request_uri;
      set $check_surl $request_uri;
      set $cache_uri $uri;
     #rewrite ^ $uri? redirect;
    }
    add_header Check-Querystring-Uri "$check_qurl";
key being the
Code (Text):
set $cache_uri $uri;
Code (Text):
curl -IL "http://cache-enabler.domain.com/?fbclid"
HTTP/1.1 200 OK
Date: Thu, 27 Aug 2020 19:03:08 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 24120
Last-Modified: Thu, 27 Aug 2020 18:10:31 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5f47f717-5e38"
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Check-Querystring-Uri: /?fbclid
Check-Uri: /?fbclid
Set-Uri: /
Accept-Ranges: bytes

eva2000 · Aug 28, 2020

Made an update to previous post so nginx serves the cached query string urls and not PHP

eva2000 · Aug 28, 2020

eva2000 said: ↑

might be a bug if you're testing a non-https wordpress site schema based cache file generation bug in v1.4.6 ? · Issue #108 · keycdn/cache-enabler ?

I posted a pull request fix at fix HTTPS scheme prefix detection for nginx/php-fpm servers by centminmod · Pull Request #109 · keycdn/cache-enabler so will see if that gets accepted and merged but bug only would apply if you're using non-https wordpress site versus https based wordpress site.
Click to expand...

Auto update fix for cache enabler 1.4.0+ changed configuration code has been released, run cmupdate first and then follow instructions at Beta Branch - update centmin.sh menu 22 cache enabler cache config in 123.09beta01

ct_roy · Aug 29, 2020

@eva2000 thanks for your efforts on this!
I've updated my config files to match your most recent updates (I had actually spotted the same issue you did re: php being the cache handler before you added the additional `
set $cache_uri $uri;
`
However, even after adding that I'm still seeing the cache handler being the advanced cache rather than nginx

I've double checked my configs - but I'm assuming if nginx was handling this I wouldn't see that x-cache-handler header?

eva2000 · Aug 29, 2020

Easiest way to troubleshoot is on updated 123.09beta01 via cmupdate, create a 2nd dummy wordpress site domain via centmin.sh menu option 22 with cache enabler selected and test it there. If it works on dummy wordpress site, then compare your live and test config file and include files and see what differs.

ct_roy · Aug 29, 2020

eva2000 said: ↑

Easiest way to troubleshoot is on updated 123.09beta01 via cmupdate, create a 2nd dummy wordpress site domain via centmin.sh menu option 22 with cache enabler selected and test it there. If it works on dummy wordpress site, then compare your live and test config file and include files and see what differs.
Click to expand...

Hey @eva2000
Just tried that. Still showing x-cache-handler as WP.
Example.

ct_roy · Aug 29, 2020

eva2000 said: ↑

Easiest way to troubleshoot is on updated 123.09beta01 via cmupdate, create a 2nd dummy wordpress site domain via centmin.sh menu option 22 with cache enabler selected and test it there. If it works on dummy wordpress site, then compare your live and test config file and include files and see what differs.
Click to expand...

Here's my relevant config files:
wpcacheenabler_wpcacheenabler2.xeroshoeslabs.com.conf
wpcacheenabler_map.conf
wpcacheenabler2.xeroshoeslabs.com.ssl.conf

eva2000 · Aug 29, 2020

eva2000 said: ↑
then in WP Admin cache enabler settings add to cache inclusions, the regex
Code (Text):
/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/
Click to expand...
Did you do this part?

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Wordpress Cache-Enabler Plugin Ensuring urls with utm params, fbclid, adwords params are served from cache

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Wordpress Cache-Enabler Plugin Ensuring urls with utm params, fbclid, adwords params are served from cache

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

ct_roy Premium Member Premium Member

ct_roy Premium Member Premium Member

eva2000 Administrator Staff Member

.