Learn about Centmin Mod LEMP Stack today
Become a Member

Wordpress Cache-Enabler Plugin Ensuring urls with utm params, fbclid, adwords params are served from cache

Discussion in 'Blogs & CMS usage' started by ct_roy, Aug 26, 2020.

  1. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    How do I ensure that urls with lots of utm params and strings from FB ads, Google ads etc. are served from cache?

    A site I run just got hammered yesterday with traffic from a few different places - this traffic was going to WP pages that would have been served from cache if it were not for the utms etc.

    I was using the nginx/redis cache option. In rediscache_mysite.com.conf I can see:
    Code:
    if ($query_string != "") {
      set $skip_cache 1;
    }
    which is a bit too general imho.

    I then switched to using Cache Enabler which actually has an option to ensure urls with params are served from cache via Cache Inclusions:
    [​IMG]
    wpcacheenabler_mysite.com I also had to comment out this
    Code:
        #if ($query_string != "") {
        #    set $cache_uri 'nullcache';
        #}
    
    I'd prefer to go back to using nginx/redis so as to avoid going near the php worker processes for these requests at all. Can anyone think of a simple modification to the nginx redis cache config that will allow these kind of urls to be served from cache?
    Thanks.
     
  2. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Cache Enabler used in centmin.sh menu option 22 bypasses PHP with advanced cache until default Cache Enabler stock out of the box. So if you switch to centmin.sh menu option 22's Cache Enabler you get to serve those query strings via cache + bypass PHP
     
  3. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    thanks for the quick reply @eva2000 much appreciated.
    That is indeed what I'm using.
    Correct me if I'm wrong, but doesn't this:
    Code:
        
    if ($query_string != "") {
    set $cache_uri 'nullcache';
    }
    
    mean that nginx itself isn't serving the cached file if there's a query string present? (which is what I'd like to have happen).
    and if nginx sets a nullcache if there's a query string then how is php still bypassed?
     
  4. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    You can also in Cache Enabler include file below existing $query_string if match add the following in /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf

    Code (Text):
        if ($query_string != "") {
            set $cache_uri 'nullcache';
        }
        if ($q_ignorearg) {
          rewrite ^ $uri? redirect;
        }
    

    then in /usr/local/nginx/conf/nginx.conf within http{} context add a mapping for instead add it to already generated /usr/local/nginx/conf/wpcacheenabler_map.conf the below map to existing map
    Code (Text):
    map $args $q_ignorearg {
      default             0;
      ~*fbclid            1;
      ~*gclid             1;
      ~*utm               1;
    }
    

    so /usr/local/nginx/conf/wpcacheenabler_map.conf looks like
    Code (Text):
    map $http_user_agent $cmwpcache_device {
        default                                     'desktop';
        ~*(iPad|iPhone|Android|IEMobile|Blackberry) 'mobile';
        "~*Firefox.*Mobile"                         'mobile';
        "~*ipod.*mobile"                            'mobile';
        "~*Opera\ Mini"                             'mobile';
        "~*Opera\ Mobile"                           'mobile';
        "~*Mobile"                                  'mobile';
        "~*Tablet"                                  'mobile';
        "~*Kindle"                                  'mobile';
        "~*Windows\ Phone"                          'mobile';
    }
    
    map $args $q_ignorearg {
      default             0;
      ~*fbclid            1;
      ~*gclid             1;
      ~*utm               1;
    }
    

    Then it will strip query strings defined in map $args to $q_ignorearg via 302 redirect
    Code (Text):
    curl -I http://cache-enabler.domain.com/?fbclid
    HTTP/1.1 302 Moved Temporarily
    Date: Wed, 26 Aug 2020 12:59:44 GMT
    Content-Type: text/html
    Content-Length: 138
    Location: http://cache-enabler.domain.com/
    Connection: keep-alive
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Set-Uri: nullcache
    

    Code (Text):
    curl -I http://cache-enabler.domain.com/?gclid
    HTTP/1.1 302 Moved Temporarily
    Date: Wed, 26 Aug 2020 12:50:07 GMT
    Content-Type: text/html
    Content-Length: 138
    Location: http://cache-enabler.domain.com/
    Connection: keep-alive
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Set-Uri: nullcache
    

    Code (Text):
    curl -I http://cache-enabler.domain.com/?utm
    HTTP/1.1 302 Moved Temporarily
    Date: Wed, 26 Aug 2020 12:50:12 GMT
    Content-Type: text/html
    Content-Length: 138
    Location: http://cache-enabler.domain.com/
    Connection: keep-alive
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Set-Uri: nullcache
    

    wrk load test
    Code (Text):
    wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?gclid
    Running 5s test @ http://cache-enabler.domain.com/?gclid
      4 threads and 50 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency   610.19us    1.61ms  37.39ms   93.47%
        Connect   176.25us   84.89us 337.00us   62.50%
        TTFB      607.10us    1.61ms  37.39ms   93.47%
        TTLB        1.90us   13.49us   5.01ms   99.99%
        Req/Sec    44.14k    13.16k   81.15k    65.67%
      Latency Distribution
         50%  182.00us
         75%  341.00us
         90%    1.13ms
         95%    3.04ms
         99%    7.80ms
      883043 requests in 5.10s, 388.22MB read
    Requests/sec: 173190.48
    Transfer/sec:     76.14MB
    

    Code (Text):
    wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?fbclid
    Running 5s test @ http://cache-enabler.domain.com/?fbclid
      4 threads and 50 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency     1.17ms    5.16ms  90.48ms   96.32%
        Connect     1.13ms    0.96ms   3.45ms   63.39%
        TTFB        1.17ms    5.16ms  90.48ms   96.32%
        TTLB        1.97us   26.28us   7.88ms   99.99%
        Req/Sec    42.86k    13.12k   66.46k    58.50%
      Latency Distribution
         50%  179.00us
         75%  381.00us
         90%    1.44ms
         95%    4.59ms
         99%   24.79ms
      854923 requests in 5.03s, 375.86MB read
    Requests/sec: 170129.87
    Transfer/sec:     74.80MB
    

    Code (Text):
    wrk-cmm -t4 -c50 -d5s --latency --breakout http://cache-enabler.domain.com/?utm                             
    Running 5s test @ http://cache-enabler.domain.com/?utm
      4 threads and 50 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency   838.50us    3.99ms  83.77ms   96.80%
        Connect   186.48us   89.73us 348.00us   62.50%
        TTFB      835.26us    3.99ms  83.76ms   96.80%
        TTLB        2.01us   31.77us   9.01ms   99.99%
        Req/Sec    43.97k    14.27k   84.01k    63.18%
      Latency Distribution
         50%  177.00us
         75%  357.00us
         90%  753.00us
         95%    2.57ms
         99%   15.19ms
      880017 requests in 5.10s, 386.89MB read
    Requests/sec: 172567.75
    Transfer/sec:     75.87MB
    


    Going to update centmin.sh menu option 22's Cache Enabler configurations for new nginx vhost creations to have this supported too by end of today. So a cmupdate will update your local code too. But won't update existing Wordpress installs, so you'd have to do it manually as outlined above in this post for existing Wordpress vhosts and include files.
     
    Last edited: Aug 26, 2020
  5. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  6. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    @eva2000 AMAZING! thank you SO much! that works brilliantly!

    I'm wondering if the 302 redirect will cause issues w/ the likes of Google Analytics no longer recognizing the utm's as they'll probably never see them now?
     
  7. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Not sure. But yeah seems like the price you pay for such.
     
  8. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    yip fair enough - let me run some tests and I'll report back my findings. Thanks again for your rapid help on this - I really appreciate it.
     
  9. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    @eva2000 I've not had a chance to test and report back on this yet. But an unrelated but VERY important thing I just noticed re: Cache Enabler's default nginx rules is reported here
    Wrong nginx recommendations in docs | WordPress.org

    In summary, Cache Enabler has changed the naming conventions for their file caches (and they've not updated their docs yet ?!?).

    Minor tweak that's needed for https hosts in
    /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf

    is:

    set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}index.html';

    becomes
    set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}https-index.html';

    This caused a more than a few minutes of head scratching today :)
     
  10. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  11. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    might be a bug if you're testing a non-https wordpress site schema based cache file generation bug in v1.4.6 ? · Issue #108 · keycdn/cache-enabler ?

    I posted a pull request fix at fix HTTPS scheme prefix detection for nginx/php-fpm servers by centminmod · Pull Request #109 · keycdn/cache-enabler so will see if that gets accepted and merged but bug only would apply if you're using non-https wordpress site versus https based wordpress site.
     
    Last edited: Aug 27, 2020
  12. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    Thankfully I didn't have that bug. I just replied to you on the Cache Enabler support forum :) So I won't repeat myself here. Other than to say my tests conclude that your solution is great as long as you're not worried about attribution or any other events that the param might be needed for.

    Here's another example. Klaviyo adds a _ke param to urls being sent from their platform. We have one site with over 500k subscribers on it. If we send them an email, we'll generate 500k requests all with the _ke param in it. If we use the current solution we lose the features that Klaviyo provide for us via the _ke param. If we don't remove the param, we'll DDOS ourselves :)

    I'd ideally like to find a way to preserve the params in the final url sent back to the client, but pluck from cache the file without the params.

    Aside: all the big tech co's are now adding these params to their links as a way to try and workaround ITP/privacy regulations and in effect are destroying the kind of fundamentals us webdevs have relied on for years (i.e. don't cache urls with params).
    This is becoming a big problem for lots of high volume websites.
     
  13. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Ah interesting, I believe this may work

    edit /usr/local/nginx/conf/wpcacheenabler_map.conf and change the previous $q_ignorearg nginx map to below one
    Code (Text):
    map $http_user_agent $cmwpcache_device {
        default                                     'desktop';
        ~*(iPad|iPhone|Android|IEMobile|Blackberry) 'mobile';
        "~*Firefox.*Mobile"                         'mobile';
        "~*ipod.*mobile"                            'mobile';
        "~*Opera\ Mini"                             'mobile';
        "~*Opera\ Mobile"                           'mobile';
        "~*Mobile"                                  'mobile';
        "~*Tablet"                                  'mobile';
        "~*Kindle"                                  'mobile';
        "~*Windows\ Phone"                          'mobile';
    }
    
    map $args $q_ignorearg {
      default                 0;
      "~*fbclid"            1;
      "~*gclid"             1;
      "~*utm"               1;
      "~*fb_action_ids"     1;
      "~*fb_action_types"   1;
      "~*fb_source"         1;
      "~*age-verified"      1;
      "~*ao_noptimize"      1;
      "~*usqp"              1;
      "~*cn-reloaded"       1;
      "~*_ga"               1;
      "~*_ke"               1;
    }
    

    then in your generated /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf include file change to
    Code (Text):
        if ($query_string != "") {
            set $cache_uri 'nullcache';
        }
    
        if ($q_ignorearg) {
          set $check_qurl $request_uri;
          set $check_surl $request_uri;
          set $cache_uri $uri;
         #rewrite ^ $uri? redirect;
        }
        add_header Check-Querystring-Uri "$check_qurl";
    

    restart nginx and php-fpm
    Code (Text):
    nprestart

    then in WP Admin cache enabler settings add to cache inclusions, the regex
    Code (Text):
    /^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/

    cache-enabler-query-string-inclusions-01.png

    Then verify by first visiting the wordpress blog page in web browser as guest to populate the cache and then inspect the source code footer for cache enabler timestamp

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/?fbclid" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:10:31 (http html) -->
    

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/?gclid" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:10:31 (http html) -->
    

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/?fb_action_ids" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:10:31 (http html) -->
    

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/?fb_source" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:10:31 (http html) -->
    

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/?f_ke" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:10:31 (http html) -->
    

    Code (Text):
    curl -sL "http://cache-enabler.domain.com/1/hello-world/?_ke" 2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 27.08.2020 18:27:41 (http html) -->
    


    Edit: looks like while the cache inclusion defined query strings are cached, they are cached and processed by PHP itself and not at nginx level, so there's more cpu load than the previous 302 redirect + stripe query string method as that is served via nginx cache and not php. I think I might have an idea what's going on so will see. Ok figured it out needed to change in your generated /usr/local/nginx/conf/wpincludes/yourdomain.com/wpcacheenabler_yourdomain.com.conf include file change to
    Code (Text):
        if ($query_string != "") {
            set $cache_uri 'nullcache';
        }
    
        if ($q_ignorearg) {
          set $check_qurl $request_uri;
          set $check_surl $request_uri;
          set $cache_uri $uri;
         #rewrite ^ $uri? redirect;
        }
        add_header Check-Querystring-Uri "$check_qurl";
    

    key being the
    Code (Text):
    set $cache_uri $uri;

    Code (Text):
    curl -IL "http://cache-enabler.domain.com/?fbclid"
    HTTP/1.1 200 OK
    Date: Thu, 27 Aug 2020 19:03:08 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 24120
    Last-Modified: Thu, 27 Aug 2020 18:10:31 GMT
    Connection: keep-alive
    Vary: Accept-Encoding
    ETag: "5f47f717-5e38"
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    Check-Querystring-Uri: /?fbclid
    Check-Uri: /?fbclid
    Set-Uri: /
    Accept-Ranges: bytes
    
     
    Last edited: Aug 30, 2020
  14. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Made an update to previous post so nginx serves the cached query string urls and not PHP :)
     
  15. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Auto update fix for cache enabler 1.4.0+ changed configuration code has been released, run cmupdate first and then follow instructions at Beta Branch - update centmin.sh menu 22 cache enabler cache config in 123.09beta01
     
  16. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    @eva2000 thanks for your efforts on this!
    I've updated my config files to match your most recent updates (I had actually spotted the same issue you did re: php being the cache handler before you added the additional `
    set $cache_uri $uri;
    `
    However, even after adding that I'm still seeing the cache handler being the advanced cache rather than nginx
    [​IMG]

    I've double checked my configs - but I'm assuming if nginx was handling this I wouldn't see that x-cache-handler header?
     
  17. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Easiest way to troubleshoot is on updated 123.09beta01 via cmupdate, create a 2nd dummy wordpress site domain via centmin.sh menu option 22 with cache enabler selected and test it there. If it works on dummy wordpress site, then compare your live and test config file and include files and see what differs.
     
  18. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    Hey @eva2000
    Just tried that. Still showing x-cache-handler as WP.
    Example.
     
  19. ct_roy

    ct_roy Premium Member Premium Member

    19
    4
    3
    Jun 21, 2020
    Ratings:
    +7
    Local Time:
    12:59 PM
    1.17.10
    10.3.22
    Here's my relevant config files:
    wpcacheenabler_wpcacheenabler2.xeroshoeslabs.com.conf
    wpcacheenabler_map.conf
    wpcacheenabler2.xeroshoeslabs.com.ssl.conf
     
  20. eva2000

    eva2000 Administrator Staff Member

    45,201
    10,280
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,934
    Local Time:
    9:59 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Did you do this part?