Want to subscribe to topics you're interested in?
Become a Member

Redis Coding issue with redis at nginx level caching

Discussion in 'Other Centmin Mod Installed software' started by Saumya Majumder, Aug 22, 2016.

  1. Saumya Majumder

    Saumya Majumder Member

    60
    3
    8
    Mar 16, 2016
    Ratings:
    +12
    Local Time:
    9:09 AM
    1.9.12
    10.0.24
    I know this forum is mostly for system admins and not sure if there is any coders here but still I'm sharing my findings here in hope it will help others one way or another.

    Some Background:
    I've been using centminmod for years now and initially I used to use ngx_pagespeed. But after using it for almost 7-8 months I saw many problems with it. One of the major problem was it increases overall server response time and TTFB. Also implementing CDN with pagespeed or any other custom things was not easy at all.

    So, after 7-8 months I thought to switch from ngx_pagespeed to redis object caching + redis server level caching + opcache along with PHP 7 & CentOS 7 (I've been using them from the beginning).

    Now it is very true that the percentage of breaking your website with ngx_pagespeed is very low (unless you do some insane config) but after I started using redis at nginx level I've started seeing some very weird issues. Like often time some part of my site doesn't load at all whereas the rest part loads fine. But again the speed with redis at nginx level is far better than using ngx_pagespeed.

    Now after seeing those wired issues I was 200% sure that it was happening due to caching (as it gets automatically fixed when I purge) but I was unable to find the reason behind it. So I dig even deeper until I find the reason.

    Things you should not do in your coding if you are using Redis at nginx level caching:
    One thing I understand after digging too much is that though redis is a super fast caching system but by default it keeps one single cache and not provide different cache for PC, Tablet and smartphones. Now this is a huge problem for some themes plugins or custom codes.

    In short if you are using wp_is_mobile() function in any part of your code or using
    mobble plugin to stuffs like:
    is_iphone();
    is_ipad();
    is_ipod();
    is_android();
    is_blackberry();
    is_opera_mobile();
    is_palm();
    is_symbian();
    is_windows_mobile();
    is_motorola();
    is_samsung();
    is_samsung_tablet();
    is_sony_ericsson();
    is_nintendo();

    then you are going to be in very big problem. Because when these pages gets hit too many times via tablet or phones, redis will cache that version only and then when the same page will be called from a PC, redis will return the cache page with those function outputs.

    How should you code then?
    If you are using redis at nginx level caching then the only option you have is to do your device specific things with the help of CSS3 media queries no matter how complicated it is. Thats the only way. Don't use php level device detection or you will be screwed. If your project must need php level device detection, then don't use redis at nginx level caching.


    Hope this helps one way or another.
     
  2. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    This would be the case for any sort of caching that attempts to bypass php in favour of pageload speeds.

    In a recent test, i found that cache enabler outperformed redis front page caching. This is especially true on php7. On php 5.6 redis cache peforms better.

    For mobile views if you have a subdomain for mobile views or cache that creates a cached html version for mobile maybe can explore using http_user_agent to redirect at the nginx level.

    if ($http_user_agent "(device|device)" {
    Rewrite ^ $url;
    }
     
  3. Saumya Majumder

    Saumya Majumder Member

    60
    3
    8
    Mar 16, 2016
    Ratings:
    +12
    Local Time:
    9:09 AM
    1.9.12
    10.0.24
    That's interesting, can you share your test result between cache enabler and redis at nginx level caching
     
  4. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    This is before mounting the cache directory to tmpfs. If you use autoptimize or other js/css minifying and combining plugins that also saves file to cache , i would expect performance increase as well when mounting the /wp-content/cache directory to tmpfs.

    Also redis cache is configured as per centminmod defaults/recommendation including redis object cache and front page caching. No redis or memcached object cache for the cache enabler plugin so lots of room for improvement.

    As you can see at 100 concurrent connections difference is marginal. At 200 there is a more obvious difference. Looking at the pattern, the difference will be greater the more number of concurrent connections.
     
    Last edited: Aug 23, 2016
  5. Saumya Majumder

    Saumya Majumder Member

    60
    3
    8
    Mar 16, 2016
    Ratings:
    +12
    Local Time:
    9:09 AM
    1.9.12
    10.0.24
    So you recommend me to disable redis at front end and use redis object cache + cache enabler?

    It is very weird because redis is working at the cure nginx level whereas cache enabler is working on WordPress level.

    What do you recommend me?
     
  6. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    WordPress Cache Enabler Plugin - KeyCDN Support

    Cache enabler with appropriate nginx configuration is also able to route subsequent hits to static files once the cache has been generated. Fortunately, centminmod set up with menu #22 also has appropriately set up nginx vhost configurations for webmasters who wish to use cache enabler. It is very nicely commented out

    Code:
      # for wordpress super cache plugin
      #try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;
    
      # for wp cache enabler plugin
      try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args; 
    
      # Wordpress Permalinks
      #try_files $uri $uri/ /index.php?q=$uri&$args;
    
      # Nginx level redis Wordpress
      # https://community.centminmod.com/posts/18828/
      #try_files $uri $uri/ /index.php?$args;
    
    It would look for the html cached page, and only pass it to php if it does not exist.

    Code:
        # default html file
        set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}index.html';
    
        # webp html file
        if ($http_accept ~* "image/webp") {
            set $cache_enabler_uri '${custom_subdir}/wp-content/cache/cache-enabler/${http_host}${cache_uri}index-webp.html';
    
    "Best" is often subjective as there are too many variables and normally it depends on each individual site the solution to adopt.

    I switched over to Cache Enabler because most of my sites are WooCommerce and I found that it plays better with a native wordpress cache plugin. Also cache enabler is the only cache plugin that allows creating separate cache for webp images (which I found to reduce the page payload significantly - images are often the cause of slow page loading speeds). I pair it with EWWW image optimizer since cache enabler added support for it a month or so ago, you do not need to use the premium Optimus plugin to take advantage of webp images.

    Perhaps you could set up 2 staging links and compare it side by side and share your findings? Might need to use separate php-fpm pools as I run into an issue having both on the same one.
     
  7. eva2000

    eva2000 Administrator Staff Member

    54,859
    12,239
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,810
    Local Time:
    1:39 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah @Saumya Majumder only way to know for your specific setup is to test and see

    for redis cache include file /usr/local/nginx/conf/wpincludes/${vhostname}/rediscache_${vhostname}.conf where vhostname is your domain in 123.09beta01, i can probably extend it to account for mobile and tablet devices and probably can do the same for other wp cache options. Will see
     
  8. eva2000

    eva2000 Administrator Staff Member

    54,859
    12,239
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,810
    Local Time:
    1:39 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    for folks using centmin mod 123.09beta01 and centmin.sh menu option 22 created wordpress site and switched to redis cache, try excluding mobile devices from redis cache using these steps

    1. edit /usr/local/nginx/conf/redisupstream.conf and add an map for mobile device user agents to top of file
    Code (Text):
    map $http_user_agent $redis_device {
        default                               'desktop';
        ~*(iPhone|Android|IEMobile|Blackberry) 'mobile';
        "~*Firefox.*Mobile"                    'mobile';
        "~*ipod.*mobile"                       'mobile';
    }
    

    /usr/local/nginx/conf/redisupstream.conf is that automatically added as an include in /usr/local/nginx/conf/nginx.conf if you use centmin.sh menu option 22
    Code (Text):
    include /usr/local/nginx/conf/redisupstream.conf;

    example contents after additions
    Code (Text):
    map $http_user_agent $redis_device {
        default                               'desktop';
        ~*(iPhone|Android|IEMobile|Blackberry) 'mobile';
        "~*Firefox.*Mobile"                    'mobile';
        "~*ipod.*mobile"                       'mobile';
    }
    
    upstream redisbackend {
      zone upstream_redis 64k;
      server 127.0.0.1:6379 weight=1 max_fails=3 fail_timeout=30s;
      #server 127.0.0.1:6380 weight=1 max_fails=3 fail_timeout=30s;
      #server 127.0.0.1:6381 weight=1 max_fails=3 fail_timeout=30s;
      #server 127.0.0.1:6382 weight=1 max_fails=3 fail_timeout=30s;
      #server 127.0.0.1:6383 weight=1 max_fails=3 fail_timeout=30s;
      #server 127.0.0.1:6384 weight=1 max_fails=3 fail_timeout=30s;
    
      #server 127.0.0.1:6380 backup;
      keepalive 4096;
    }
    

    2. edit /usr/local/nginx/conf/wpincludes/${vhostname}/rediscache_${vhostname}.conf where vhostname is yourdomain.com name and add an entry to exclude mobile devices
    Code (Text):
    # exclude mobile devices from redis caching
    if ($redis_device = mobile) {
      set $skip_cache 1;
    }
     
    Last edited: Aug 23, 2016
  9. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    Interesting. Why not allow mobile to have a separate cache key?
     
    Last edited by a moderator: Aug 24, 2016
  10. eva2000

    eva2000 Administrator Staff Member

    54,859
    12,239
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,810
    Local Time:
    1:39 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    probably due to devices having different screen resolutions and responsive web themes - probably need a much finer grain definition of mobile user agents
     
  11. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    Good point. Guess the way forward is either to have another theme for mobile or to have media query based responsive themes (just makes it look nice on mobile - doesnt actually has smaller file size :()
     
  12. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:39 AM
    Mainline
    10.2
    I have this config for several months now, using WP Super Cache and separate cache files for mobile visitors.
    PHP:
    #### Mobile Detect
        
    set $mobile_rewrite index.html;
        if (
    $http_user_agent ~* "(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge |maemo|midp|mmp|mobile.+firefox|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows ce|xda|xiino") {
        
    set $mobile_rewrite index-mobile.html;
        }
        
    #### Mobile Detect

        
    location / {
         
    try_files /wp-content/cache/supercache/$http_host/$cache_uri/$mobile_rewrite $uri $uri/ /index.php?$args;
        }
     
  13. eva2000

    eva2000 Administrator Staff Member

    54,859
    12,239
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,810
    Local Time:
    1:39 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    nice so what's creating the actual index-mobile.html version ? a wp plugin ?
     
  14. rdan

    rdan Well-Known Member

    5,447
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:39 AM
    Mainline
    10.2
    I think the owner of the site use "Jetpack's Mobile Theme Module".
     
  15. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    I spent a bit of time with Saumya few days back and I think he decided that redis cache was for him - granted he didn't have a server to siege with so I did it from my VPS - which I personally felt was not ideal due to geographical latency fluctuations.

    For my particular setup I ran again on a blogpost with 2 images, one jpg and one webp, this time with webp on redis full cache too, as well as Better Wordpress Minify (I prefer this over autoptimize due to its using wp enqueue way of concatenation, and able to easily exclude scripts from the wp-admin without tearing my eye viewing source).

    Code:
    Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
    **** BWP Minify + Redis FP Cache + Redis Object Cache ****
    2016-08-26 00:24:35,  26000,     426.02,        2888,       1.61,       61.03,        6.78,       98.52,   26000,       0
    
    **** BWP Minify + Redis FP Cache + Redis Object Cache + Webp image format ****
    2016-08-26 01:23:54,  21997,     365.22,        2821,       1.64,       60.23,        7.72,       98.80,   21997,       3
    
    **** BWP Minify + Cache Enabler (On TMPFS) + Redis Object Cache + Webp image format ****
    2016-08-26 02:16:29,  25998,     364.00,        2872,       1.38,       71.42,        7.89,       98.65,   25998,       2
    
    Actually, until redis sockets are supported for nginx helper, at this point performance difference is minimal. Testing and testing again, I cannot reiterate that the biggest performance gain is from tweaking the wordpress theme itself.

    Plugin Organizer — WordPress Plugins

    Above plugin (free) should be useful for many who are into theme optimization.

    Speed up WordPress | Gonzales WordPress plugin

    Another premium paid plugin that is more dummy-proof and integrates nicely into admin bar.
     
  16. eva2000

    eva2000 Administrator Staff Member

    54,859
    12,239
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,810
    Local Time:
    1:39 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  17. JarylW

    JarylW Active Member

    216
    41
    28
    Jun 19, 2014
    Singapore
    Ratings:
    +103
    Local Time:
    11:39 AM
    Yea it is quite useful especially in backend. Many people often do not exclude the correct scripts and css when using Autoptimize, this causes subsequent page visits for first time visitors to be slow because you need to download another huge concatenated css/js that is not the same as the one on homepage. If exclusion is done correctly, only need to download the 'extra css/js' files that are much smaller, and fat concatenated css/js is served from browser cache.

    I think with HTTP2 parallelisation, the enqueue way of BWP minify seems to be easier to use and works better perf wise.

    [​IMG]