Learn about Centmin Mod LEMP Stack today
Register Now

Wordpress Cache-Enabler Plugin Ensuring urls with utm params, fbclid, adwords params are served from cache

Discussion in 'Blogs & CMS usage' started by ct_roy, Aug 26, 2020.

  1. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    To see if it's working check source code footer and load test while running top in ssh session to see if only nginx is loaded or if php-fpm gets hit
     
  2. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    Yip I sure did.
     
  3. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    FYI, presence of X-Cache-Handler: wp is fine as long as source code's footer shows cache enabler timestamp and load testing shows only nginx and not php-fpm being loaded

    I installed a fresh test cache-enabler2.domain.com test domain with self-signed ssl cert to test
    Code (Text):
    curl -sk "https://cache-enabler2.domain.com/?fbclid"  2>&1 | tail -2
    <!-- Cache Enabler by KeyCDN @ 29.08.2020 15:28:36 (https html) -->
    
    curl -sk "https://cache-enabler2.domain.com/"  2>&1 | tail -2    
    <!-- Cache Enabler by KeyCDN @ 29.08.2020 15:28:36 (https html) -->
    

    load test
    Code (Text):
    wrk-cmm -t4 -c50 -d20s --latency --breakout https://cache-enabler2.domain.com/?fbclid  
    Running 20s test @ https://cache-enabler2.domain.com/?fbclid
      4 threads and 50 connections
      Thread Stats   Avg      Stdev     Max   +/- Stdev
        Latency     2.79ms    2.48ms  28.26ms   87.47%
        Connect     6.28ms    2.65ms  14.09ms   60.47%
        TTFB        1.80ms    2.40ms  27.10ms   87.58%
        TTLB        0.97ms  570.55us  23.65ms   92.91%
        Req/Sec     5.03k     1.02k    6.55k    72.75%
      Latency Distribution
         50%    1.93ms
         75%    3.16ms
         90%    6.11ms
         95%    8.66ms
         99%   11.79ms
      401610 requests in 20.07s, 10.48GB read
    Requests/sec:  20007.75
    Transfer/sec:    534.78MB
    

    in 2nd ssh session window had top running and see only wrk-cmm and nginx loaded and not hitting php-fpm
    Code (Text):
    top -c
    top - 15:29:15 up 18 days, 17:20,  2 users,  load average: 1.50, 0.82, 0.57
    Tasks: 239 total,   7 running, 232 sleeping,   0 stopped,   0 zombie
    %Cpu0  : 36.5 us, 50.8 sy,  0.0 ni,  5.6 id,  0.0 wa,  0.0 hi,  7.0 si,  0.0 st
    %Cpu1  : 36.8 us, 49.2 sy,  0.0 ni,  6.7 id,  0.0 wa,  0.0 hi,  7.4 si,  0.0 st
    %Cpu2  : 38.9 us, 46.2 sy,  0.0 ni,  4.3 id,  0.0 wa,  0.0 hi, 10.6 si,  0.0 st
    %Cpu3  : 36.9 us, 42.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 20.6 si,  0.0 st
    %Cpu4  : 39.1 us, 43.4 sy,  0.0 ni,  6.0 id,  0.0 wa,  0.0 hi, 11.6 si,  0.0 st
    %Cpu5  : 35.2 us, 45.6 sy,  0.0 ni,  8.4 id,  0.0 wa,  0.0 hi, 10.7 si,  0.0 st
    %Cpu6  : 39.0 us, 44.3 sy,  0.0 ni,  3.9 id,  0.0 wa,  0.0 hi, 12.8 si,  0.0 st
    %Cpu7  : 38.1 us, 48.0 sy,  0.0 ni,  5.0 id,  0.0 wa,  0.0 hi,  8.9 si,  0.0 st
    KiB Mem : 32741024 total,   629204 free,  5995700 used, 26116120 buff/cache
    KiB Swap:  2095096 total,  1698040 free,   397056 used. 25647112 avail Mem
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  
    30251 root      20   0  455796   6568   2860 S 292.7  0.0   0:30.03 wrk-cmm                                                  
    29585 nginx     10 -10  816064 283976   3304 R  96.0  0.9   0:10.26 nginx                                                    
    29591 nginx     10 -10  811968 279616   2680 R  89.7  0.9   0:09.49 nginx                                                    
    29588 nginx     10 -10  820160 283708   3248 S  81.5  0.9   0:08.46 nginx                                                    
    29583 nginx     10 -10  811968 279628   2656 R  59.3  0.9   0:06.68 nginx                                                    
    29590 nginx     10 -10  811968 279624   2720 R  58.9  0.9   0:06.23 nginx                                                    
    29592 nginx     10 -10  811968 279640   2676 S  46.0  0.9   0:04.93 nginx                                                    
    29580 nginx     10 -10  811968 279400   2660 R  20.2  0.9   0:02.08 nginx                                                    
    29582 nginx     10 -10  811968 279412   2652 R  10.9  0.9   0:01.44 nginx
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    @ct_roy found the mapping had syntax errors, you need to edit /usr/local/nginx/conf/wpcacheenabler_map.conf and change the previous $q_ignorearg nginx map to below one
    Code (Text):
    map $http_user_agent $cmwpcache_device {
        default                                     'desktop';
        ~*(iPad|iPhone|Android|IEMobile|Blackberry) 'mobile';
        "~*Firefox.*Mobile"                         'mobile';
        "~*ipod.*mobile"                            'mobile';
        "~*Opera\ Mini"                             'mobile';
        "~*Opera\ Mobile"                           'mobile';
        "~*Mobile"                                  'mobile';
        "~*Tablet"                                  'mobile';
        "~*Kindle"                                  'mobile';
        "~*Windows\ Phone"                          'mobile';
    }
    
    map $args $q_ignorearg {
      default                 0;
      "~*fbclid"            1;
      "~*gclid"             1;
      "~*utm"               1;
      "~*fb_action_ids"     1;
      "~*fb_action_types"   1;
      "~*fb_source"         1;
      "~*age-verified"      1;
      "~*ao_noptimize"      1;
      "~*usqp"              1;
      "~*cn-reloaded"       1;
      "~*_ga"               1;
      "~*_ke"               1;
    }
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    hmm I can even setup these query string regexes at initial wordpress install time via wp-cli command line too

    Update cache inclusion field = incl_paramters
    Code (Text):
    domain=cache-enabler2.domain.com
    
    wp option patch update cache-enabler incl_parameters '/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/' --path=/home/nginx/domains/$domain/public/
    

    example output running wp option patch
    Code (Text):
    domain=cache-enabler2.domain.com
    
    wp option patch update cache-enabler incl_parameters '/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/' --path=/home/nginx/domains/$domain/public/
    Success: Updated 'cache-enabler' option.
    

    check the value set
    Code (Text):
    domain=cache-enabler2.domain.com
    wp option pluck cache-enabler incl_parameters --format=json --path=/home/nginx/domains/$domain/public/
    

    example output
    Code (Text):
    domain=cache-enabler2.domain.com
    wp option pluck cache-enabler incl_parameters --format=json --path=/home/nginx/domains/$domain/public/
    
    "\/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$\/"
    

    list all cache enabler options
    Code (Text):
    domain=cache-enabler2.domain.com
    wp option get cache-enabler --format=json --path=/home/nginx/domains/$domain/public/ | jq
    

    example output
    Code (Text):
    domain=cache-enabler2.domain.com
    wp option get cache-enabler --format=json --path=/home/nginx/domains/$domain/public/ | jq
    {
      "expires": 6,
      "clear_on_upgrade": 1,
      "new_post": 1,
      "new_comment": 1,
      "update_product_stock": 0,
      "webp": 0,
      "compress": 1,
      "excl_ids": "",
      "excl_paths": "",
      "excl_cookies": "",
      "incl_parameters": "/^fbclid|utm_(source|medium|campaign|term|content)|gclid|fb_(action_ids|action_types|source)|age-verified|ao_noptimize|usqp|cn-reloaded|_ga|_ke$/",
      "minify_html": 0
    }
    
     
  6. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    That did the job - thanks @eva2000 !
    I had spotted that syntax change earlier but assumed it was intentional - I should have said something :)
    This will dramatically improve webperf for a number of sites I run where they receive a ton of incoming requests with utms and all the other fb/google params so thank you so much again.
     
  7. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    I always thought that the presence of the "X-Cache-Handler: wp" header meant that advanced-cache.php was handling the request? And that nginx was only truly serving the cache if no handler X-Cache-Handler header was present.
    As per the docs:
    In my case, after the last syntax change (which is performing amazingly!), I now see no X-Cache-Handler (which is good) and I now also see "check-querystring-uri" header which I wasn't seeing previously (which again would reaffirm nginx is now solely serving the cache).
     
  8. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Great to hear :)
    Indeed the header usually does mean advanced-cache.php but have found cases it also meant nginx handling but not always
    Yup added "check-querystring-uri" header for verification purposes :)
     
  9. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Let me know how this fairs for you and if there's other query strings I need to account for as I am thinking of enabling this config for caching select query strings by default during 123.09beta01's centmin.sh menu option 22 wordpress cache enabler initial installs via wp-cli as outlined at here

    FYI, for now I added a persistent config variable which can enable Cache Enabler query string inclusions at centmin.sh menu option 22 install time Beta Branch - centmin.sh menu 22 add WPCLI_CE_QUERYSTRING_INCLUDED='n' in 123.09beta01
     
    Last edited: Sep 1, 2020
  10. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    It's working fantastically well @eva2000 (y)
    The current set of query strings looks to be catching pretty much 99% of what we see coming in. I'll grep our access logs later in the week to see if there's anything else worthy of being added to the defaults - but I think that current cohort catches all the major offenders :)

    You're now in a very select few WP stacks that deal with this problem properly :)
    Might be good to consider it the Redis/nginx config too at some point.
     
  11. David Schargel

    David Schargel New Member

    18
    6
    3
    Feb 2, 2020
    Ratings:
    +15
    Local Time:
    2:25 PM
    @eva2000...
    • mc_cid (Mailchimp campaign ID)
    • mc_eid (Mailchimp member email's unique_id)
     
  12. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yes once got Cache Enabler query string cache inclusions working well, will add it to Redis Nginx level cache and fastcgi_cache (4th upcoming cache mode) routines too :)
    cheers !
     
  13. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    Ooh nice :)
    Meant to ask you how you were getting on with the fastcgi_cache mode - I'm somewhat of a traditionalist in that it's never let me down in the past :)
     
  14. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    FYI, current Cache Enabler config in Centmin Mod will end up faster that fastcgi_cache mode due to Cache Enabler also doing pre-compression of cached html pages and Centmin Mod Nginx enabling gzip static directive by default. So Nginx will serve the pre-compressed cached html response which is much faster than serving cached html response which isn't pre-compressed. fastcgi_cache can't do that as the cached response is in memory and not a html cache file.
     
  15. ct_roy

    ct_roy Premium Member Premium Member

    33
    6
    8
    Jun 21, 2020
    Ratings:
    +11
    Local Time:
    10:25 PM
    1.17.10
    10.3.22
    aha! good to know!
     
  16. Benjamin74

    Benjamin74 Premium Member Premium Member

    58
    5
    8
    May 2, 2016
    Ratings:
    +12
    Local Time:
    11:25 PM
    Hummm...

    Looks like I have an issue with a freshly installed WP that is likely related to the recent changes mentioned in this thread:

    In my case utm_source should NOT be cached.

    In the past I could just tell Cache Enabler to NOT cache some URLs (by setting the post IDs in Cache Enabler config) and it would work just fine.

    Now with a newly created WP site, utm_source disappears from URL when going to :

    Code:
    https://mydomain.com/?utm_source=ABCD
    Despite cache enabler having a rule to exclude this post ID from cache.

    Any idea?

    Thanks,

    ------
    EDIT: FOUND THE SOLUTION
    ------
    It was indeed related to these recent changes.

    The vhost had this line here, uncommented:

    include /usr/local/nginx/conf/wpincludes/mydomain.com/wpcacheenabler_mydomain.com.conf;

    I've had to comment this line to have utm_source to NOT go to the cache.
     
    Last edited: Sep 3, 2020
  17. Benjamin74

    Benjamin74 Premium Member Premium Member

    58
    5
    8
    May 2, 2016
    Ratings:
    +12
    Local Time:
    11:25 PM
    OK so there is a bug (or maybe there "was" a temporary bug, maybe @eva2000 knows if he corrected something very recently) because on another site (installed maybe 7 days ago), commenting the line:

    #include /usr/local/nginx/conf/wpincludes/mydomain.com/wpcacheenabler_mydomain.com.conf;

    Makes it impossible to restart nginx:

    sept. 02 16:56:39 ultraserv-prod nginx[39286]: Starting nginx: nginx: [emerg] unknown "cache_enabler_uri_webp" variable
    sept. 02 16:56:39 ultraserv-prod nginx[39286]: [FAILED]

    whereas on the other site mentioned above (newer install, maybe 4 days old), nginx can be restarted with that line commented.

    I'm wondering what's the best way to have a clean vhost: maybe I should just delete and re-create this site?
     
  18. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Run update and then create a separate dummy centmin.sh menu 22 WordPress site with cache enabler and compare its /usr/local/nginx/conf/wpincludes/mydomain.com/wpcacheenabler_mydomain.com.conf include file and nginx vhost's try_files syntax with your sites' with issues and migrate the changes from new version to your existing ones replacing the domain name mentions with your existing domain names and then adjust cache enabler's query string cache inclusion settings in admin site WordPress.
     
  19. eva2000

    eva2000 Administrator Staff Member

    46,238
    10,510
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,293
    Local Time:
    8:25 AM
    Nginx 1.19.x
    MariaDB 5.5/10.x
  20. ahmed

    ahmed Active Member

    360
    48
    28
    Feb 21, 2017
    Ratings:
    +62
    Local Time:
    12:25 AM
    each time I add this the nginx restart fails, am I doing wrong?