Want more timely Centmin Mod News Updates?
Become a Member

Wordpress Configure Redis page cache to bypass specific urls

Discussion in 'Blogs & CMS usage' started by savvasha, Nov 30, 2018.

  1. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Well, I am using a download manager plugin to share some pdf files from my site.
    The problem is that when someone tries to download a file the url used is something like domain.com/download/?down=23 which is served through Redis cache and the Nginx Timestamp is added at the source of the pdf!

    Is there a way to bypass all urls with ?down= variable from redis cache?

    I am using:
    123.09beta01 (latest)
    PHP 7.2.12
    nginx 1.15.6


    Sorry for my bad english :)

    Thanks,
    Savvas
     
  2. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    That's an interesting problem to have :)

    Is /download page itself a valid page ? if not you can exclude cache from /download/* ???

    If so you could do similar custom bypass as outlined at Wordpress - WP Super Cache, KeyCDN Cache Enabler, Redis Cache User Agent Bypass for Wordpress just for url and not user agent

    For Redis caching of Wordpress at Nginx level outlined at community.centminmod.com/posts/18828/, in /usr/local/nginx/conf/wpincludes/newdomain.com/rediscache_newdomain.com.conf include file add an extra entry to bypass cache on url containing /download or ?/down=
    Code (Text):
    if ($request_uri ~* "/download/|\?down=") {
      set $skip_cache 1;
    }
    

    or if just ?down=
    Code (Text):
    if ($request_uri ~* "\?down=") {
      set $skip_cache 1;
    }
    

    restart nginx
    Code (Text):
    ngxrestart
    
     
  3. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Hi eva2000,

    Unfortunately your solution is not working :(

    My actual url format is something like this:
    domain.com/download/file-name/?wpdmdl=xxxxxxx

    So I added the following code :
    Code:
    # Don't use the cache for download links
    if ($request_uri ~* "\?wpdmdl=") {
      set $skip_cache 1;
    }
    but it keeps adding
    Code:
    <!--Cached using Nginx-Helper on 2018-11-30 07:05:48. It took 55 queries executed in 0,109 seconds.-->
    <!--Visit http://wordpress.org/extend/plugins/nginx-helper/faq/ for more details-->
    at the end of pdf files
     
  4. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    oh might need to restart php-fpm + nginx services
    Code (Text):
    nprestart
    
     
  5. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Unfortunately no joy...
    Code:
    <!--Cached using Nginx-Helper on 2018-11-30 10:28:11. It took 54 queries executed in 0,124 seconds.-->
    <!--Visit http://wordpress.org/extend/plugins/nginx-helper/faq/ for more details-->
     
  6. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    what if you exclude /download urls too
    Code (Text):
    if ($request_uri ~* "/download/|\?wpdmdl=") {
      set $skip_cache 1;
    }
    
     
  7. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Still not working. Maybe the problem is in Nginx-Helper plugin?
     
  8. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    did you add any other wordpress plugins after install ? i.e. like a redis object cache plugin ? that could maybe mess with saving nginx-helper settings.
     
  9. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Hi there eva2000,

    Sorry for the long delay on reply :)

    After some more in depth investigation, it seems that when someone wants to download something from my website and the user is using an Apple Device (Mac or iPhone or iPad), the file is not dowloading completely. It seems to stop downloading at 99% and gives an error "The network connection was lost".

    I try to figure out what the exact problem was and I did the following:
    I create a new VPS at digitalocean ($5) and install Centmimod 123.09beta. I then create a website through menu 22 and select the 2nd option (Redis Nginx Level Caching). I then migrate my live site, to this new testing host. So I have the same plugins in both sites! In my Original Site, the files are not downloaded correctly in Apple Devices, but in my Testing Site everything is working just perfect!

    So the problem doesnt seem to be at the cache mechanism after all :)

    The question is, how can I check what is the difference between the configuration of these sites? What files should I check?

    Thanks,
    Savvas
     
  10. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    check the nginx site domain vhost config file differences first for /usr/local/nginx/conf/conf.d/yourdomain.com.conf or yourdomain.com.ssl.conf. You can use Diff Checker to compare 2 files or a local file compare too i.e. I use beyondcompare

    Or you can just move everything to the working VPS :)
     
  11. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Hi there eva2000,

    Well I follow your second suggestion, move the site to the new VPS instance and guess what. It is not working again :(

    It is hard to investigate what exactly is going wrong because I don't have an Apple device. I try to install a MacOs on Virtual Box and discover that if you try to download a file from Safari, the downloading failed with "The network connection was lost". Then I try to install Chrome and use that one. Everything worked like magic! :) The problem is that I still don't know what is the cause of the problem, and I am getting everyday complaints from Apple users that they cannot download any file from my site.

    I will propose them to try a different browser, but still is a very strange behaviour...

    Thanks,
    Savvas
     
  12. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    instead of redis cache selection, have you tried using keycdn cache enabler instead ?
     
  13. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    The truth is that I tried it and the problem was insisting. Thats why I return to redis cache, because I think it is more optimized.
    I am not sure that the problem is with the cache after all. Is it possible some http headers to make this whole mess?
    My first page headers are the following:
    Code:
    HTTP/1.1 200 OK
    Date: Mon, 28 Jan 2019 07:18:27 GMT
    Content-Type: text/html; charset=UTF-8
    Connection: keep-alive
    Vary: Accept-Encoding
    Set-Cookie: PHPSESSID=vmhu0enmjm8cl6ghjoon0llsou; path=/
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Cache-Control: no-store, no-cache, must-revalidate
    Pragma: no-cache
    Link: https://www.themata4all.com/wp-json/; rel=https://api.w.org/
    Link: https://wp.me/6LYH3; rel=shortlink
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Cache: MISS
    X-Cache-2: BYPASS
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    
    Thanks
     
  14. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    that means redis cache was bypassed so it isn't related to redis caching for that request.

    how large are download sizes and what's average speed of the users download speed/ISP. Could be downloads timing out but that would explain it if same user can download using Chrome.

    When you ran centmin.sh menu option 22, at the prompt to exclude mobile devices from redis cache, did you answer yes or no ?

    If you still have access to both Centmin Mod servers with the working and not working site, can you post the nginx vhost configs

    When you create a new nginx vhost domain via centmin.sh menu option 2 or menu option 22 or via /usr/bin/nv cli command line, you will create the Nginx vhost files and directories. You will get an outputted the path location where it will create the domain name's vhost conf file named newdomain.com.conf (and newdomain.com.ssl.conf if you selected yes to self signed SSL)
    • Nginx vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.conf
    • Nginx HTTP/2 SSL vhost conf path will be at /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf
    • Nginx Self-Signed SSL Certificate Directory at /usr/local/nginx/conf/ssl/newdomain.com
    • Vhost public web root will be at /home/nginx/domains/newdomain.com/public
    • Vhost log directory will be at /home/nginx/domains/newdomain.com/log
    Please post the contents of /usr/local/nginx/conf/conf.d/newdomain.com.conf and if applicable /usr/local/nginx/conf/conf.d/newdomain.com.ssl.conf wrapped in CODE tags (outlined at How to use forum BBCODE code tags)

    For current site, what is output of these commands in ssh
    Code (Text):
    curl -I https://domain.com
    

    Code (Text):
    curl -I https://www.domain.com
    

    Code (Text):
    curl -I http://domain.com
    

    Code (Text):
    curl -I http://www.domain.com
    

    wrap output in CODE tags
     
  15. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    Well,

    My files are mostly pdf files with average size of 200-300kb, nothing special here.

    Well I am not 100% sure but I think that I answered NO :)

    Unfortunately I deleted my old VPS :(

    Below are my themata4all.com.ssl.conf contents. I dont have themata4all.com.conf (probably because I selected to use ONLY HTTPS?)
    Code:
    #x# HTTPS-DEFAULT
     server {
      
       server_name themata4all.com www.themata4all.com;
       return 302 https://www.themata4all.com$request_uri;
       include /usr/local/nginx/conf/staticfiles.conf;
     }
    
    server {
      listen 443 ssl http2 reuseport;
      server_name themata4all.com www.themata4all.com;
    
      include /usr/local/nginx/conf/ssl/themata4all.com/themata4all.com.crt.key.conf;
      include /usr/local/nginx/conf/ssl_include.conf;
    
      # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
      #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/themata4all.com/origin.crt;
      #ssl_verify_client on;
      http2_max_field_size 16k;
      http2_max_header_size 32k;
      # mozilla recommended
      ssl_ciphers XXXXXXXX;
      ssl_prefer_server_ciphers   on;
      #add_header Alternate-Protocol  443:npn-spdy/3;
    
      # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
      #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
      #add_header X-Frame-Options SAMEORIGIN;
      add_header X-Xss-Protection "1; mode=block" always;
      add_header X-Content-Type-Options "nosniff" always;
      #add_header Referrer-Policy "strict-origin-when-cross-origin";
      #spdy_headers_comp 5;
      ssl_buffer_size 1369;
      ssl_session_tickets on;
     
      # enable ocsp stapling
      resolver 8.8.8.8 8.8.4.4 valid=10m;
      resolver_timeout 10s;
      ssl_stapling on;
      ssl_stapling_verify on;
    
    # ngx_pagespeed & ngx_pagespeed handler
    #include /usr/local/nginx/conf/pagespeed.conf;
    #include /usr/local/nginx/conf/pagespeedhandler.conf;
    #include /usr/local/nginx/conf/pagespeedstatslog.conf;
    
      # limit_conn limit_per_ip 16;
      # ssi  on;
    
      access_log /home/nginx/domains/themata4all.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/themata4all.com/log/error.log;
    
      include /usr/local/nginx/conf/autoprotect/themata4all.com/autoprotect-themata4all.com.conf;
      root /home/nginx/domains/themata4all.com/public;
      # uncomment cloudflare.conf include if using cloudflare for
      # server and/or vhost site
      #include /usr/local/nginx/conf/cloudflare.conf;
      include /usr/local/nginx/conf/503include-main.conf;
    
      #include /usr/local/nginx/conf/wpincludes/themata4all.com/wpcacheenabler_themata4all.com.conf;
      #include /usr/local/nginx/conf/wpincludes/themata4all.com/wpsupercache_themata4all.com.conf;
      # https://community.centminmod.com/posts/18828/
      include /usr/local/nginx/conf/wpincludes/themata4all.com/rediscache_themata4all.com.conf; 
    
      location / {
      include /usr/local/nginx/conf/503include-only.conf;
     
    
      # Enables directory listings when index file not found
      #autoindex  on;
    
      # for wordpress super cache plugin
      #try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;
    
      # for wp cache enabler plugin
      #try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args; 
    
      # Wordpress Permalinks
      #try_files $uri $uri/ /index.php?q=$uri&$args;
    
      # Nginx level redis Wordpress
      # https://community.centminmod.com/posts/18828/
      try_files $uri $uri/ /index.php?$args;
    
      }
    
    location ~* /(wp-login\.php) {
        limit_req zone=xwplogin burst=1 nodelay;
        #limit_conn xwpconlimit 30;
        #auth_basic "Private";
        #auth_basic_user_file /home/nginx/domains/themata4all.com/htpasswd_wplogin;   
        #include /usr/local/nginx/conf/php-wpsc.conf;
        
        # https://community.centminmod.com/posts/18828/
        include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /(xmlrpc\.php) {
        limit_req zone=xwprpc burst=45 nodelay;
        #limit_conn xwpconlimit 30;
        #include /usr/local/nginx/conf/php-wpsc.conf;
        
        # https://community.centminmod.com/posts/18828/
        include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /wp-admin/(load-scripts\.php) {
        limit_req zone=xwprpc burst=5 nodelay;
        #limit_conn xwpconlimit 30;
        #include /usr/local/nginx/conf/php-wpsc.conf;
        
        # https://community.centminmod.com/posts/18828/
        include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
    location ~* /wp-admin/(load-styles\.php) {
        limit_req zone=xwprpc burst=5 nodelay;
        #limit_conn xwpconlimit 30;
        #include /usr/local/nginx/conf/php-wpsc.conf;
        
        # https://community.centminmod.com/posts/18828/
        include /usr/local/nginx/conf/php-rediscache.conf;
    }
    
      include /usr/local/nginx/conf/wpincludes/themata4all.com/wpsecure_themata4all.com.conf;
      #include /usr/local/nginx/conf/php-wpsc.conf;
     
      # https://community.centminmod.com/posts/18828/
      include /usr/local/nginx/conf/php-rediscache.conf;
      include /usr/local/nginx/conf/pre-staticfiles-local-themata4all.com.conf;
      include /usr/local/nginx/conf/pre-staticfiles-global.conf;
      include /usr/local/nginx/conf/staticfiles.conf;
      include /usr/local/nginx/conf/drop.conf;
      #include /usr/local/nginx/conf/errorpage.conf;
      include /usr/local/nginx/conf/vts_server.conf;
    }
    Code:
    [07:42][[email protected] ~]# curl -I https://themata4all.com
    HTTP/1.1 301 Moved Permanently
    Date: Tue, 29 Jan 2019 05:42:56 GMT
    Content-Type: text/html; charset=UTF-8
    Connection: keep-alive
    Set-Cookie: PHPSESSID=2033ok4g4rreq7oo586jdlf94f; path=/
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Cache-Control: no-store, no-cache, must-revalidate
    Pragma: no-cache
    Location: https://www.themata4all.com/
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Cache: MISS
    X-Cache-2: BYPASS
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    
    [07:42][[email protected] ~]# curl -I https://www.themata4all.com
    HTTP/1.1 200 OK
    Date: Tue, 29 Jan 2019 05:43:08 GMT
    Content-Type: text/html; charset=UTF-8
    Connection: keep-alive
    Vary: Accept-Encoding
    Set-Cookie: PHPSESSID=9uq1lghjdel7gq687ql0pt19ec; path=/
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Cache-Control: no-store, no-cache, must-revalidate
    Pragma: no-cache
    Link: <https://www.themata4all.com/wp-json/>; rel="https://api.w.org/"
    Link: <https://wp.me/6LYH3>; rel=shortlink
    Server: nginx centminmod
    X-Powered-By: centminmod
    X-Cache: MISS
    X-Cache-2: BYPASS
    X-Xss-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    
    [07:43][[email protected] ~]# curl -I http://themata4all.com
    HTTP/1.1 302 Moved Temporarily
    Date: Tue, 29 Jan 2019 05:43:27 GMT
    Content-Type: text/html
    Content-Length: 138
    Connection: keep-alive
    Location: https://www.themata4all.com/
    Server: nginx centminmod
    X-Powered-By: centminmod
    
    [07:43][[email protected] ~]# curl -I http://www.themata4all.com
    HTTP/1.1 302 Moved Temporarily
    Date: Tue, 29 Jan 2019 05:43:40 GMT
    Content-Type: text/html
    Content-Length: 138
    Connection: keep-alive
    Location: https://www.themata4all.com/
    Server: nginx centminmod
    X-Powered-By: centminmod
     
  16. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Code (Text):
      ssl_ciphers XXXXXXXX;
    

    i assume you cut short the output here ?
    you can verify your redis cache nginx level settings by inspecting your include file at
    /usr/local/nginx/conf/wpincludes/themata4all.com/rediscache_themata4all.com.conf

    output for cat display of contents of /usr/local/nginx/conf/wpincludes/themata4all.com/rediscache_themata4all.com.conf
    Code (Text):
    cat /usr/local/nginx/conf/wpincludes/themata4all.com/rediscache_themata4all.com.conf
    

    also wonder if the issue is safari itself "The Network Connection Was Lost" Safari - Apple Community ?

    or due to why the download is served to safari Safari network connection lost iOS 7... O… - Apple Community
    these are headers i get in chrome webpagetest.org for one of your pdf download urls
    Code (Text):
    status: 200
    content-length: 249103
    x-xss-protection: 1; mode=block
    x-robots-tag: noindex, nofollow
    content-disposition: attachment;filename="5kefalaio_luseis_extra.pdf"
    x-powered-by: centminmod
    content-transfer-encoding: binary
    expires: Wed, 11 Jan 1984 05:00:00 GMT
    x-cache: BYPASS
    robots: none
    server: nginx centminmod
    x-cache-2: BYPASS
    content-description: File Transfer
    cache-control: no-cache, must-revalidate, max-age=0
    date: Tue, 29 Jan 2019 07:22:12 GMT
    x-content-type-options: nosniff
    content-type: application/octet-stream
    :status: 200
    

    is it only pdf downloads having issues ? what about other download extension/file types ?

    versus direct pdf download headers
    Code (Text):
    curl -Ik https://domain.com/d700.pdf
    HTTP/1.1 200 OK
    Date: Tue, 29 Jan 2019 08:47:38 GMT
    Content-Type: application/pdf
    Content-Length: 16615306
    Last-Modified: Thu, 16 Jul 2015 15:27:02 GMT
    Connection: keep-alive
    ETag: "55a7cd46-fd878a"
    Server: nginx centminmod
    X-Powered-By: centminmod
    Expires: Thu, 28 Feb 2019 08:47:38 GMT
    Cache-Control: max-age=2592000
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate, proxy-revalidate, immutable, stale-while-revalidate=86400, stale-if-error=604800
    Accept-Ranges: bytes
    

    maybe your wordpress download plugin needs update to pass proper content type on downloads Does Safari on iOS refuse to download `application/octet-stream` content? for pdfs ?

    You can try to upload a pdf download file directly to web root of your site at
    /home/nginx/domains/themata4all.com/public/your.pdf and then try getting Apple device users to download it from themata4all.com/your.pdf and see if they can. If they can, then it confirms problem maybe with how your wordpress download manager plugin is serving the download files. But that also doesn't rule out redis caching interference though header for download pdf url with x-cache: BYPASS, means it bypassed and wasn't cached by redis cache.
     
  17. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  18. savvasha

    savvasha New Member

    10
    0
    1
    Nov 30, 2018
    Ratings:
    +1
    Local Time:
    9:08 PM
    Nginx 1.15.x
    MariaDB 10.1.x
    I finally identify the issue (I hope at least) :D I manage to install a MacOS on a VirtualBox and with the trial and error I manage to find it :)

    I had to disable the "Resumable Downloads" and "Output Buffering" options from Download Manager options (https://snag.gy/SJhLgo.jpg)

    Somehow it was confusing Safari browsers on MacOS (and iOS hopefully) browsers.

    I will wait for any user feedback :)

    Thanks a lot eva2000 for your help!
     
    • Informative Informative x 1
  19. eva2000

    eva2000 Administrator Staff Member

    41,070
    9,175
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,083
    Local Time:
    4:08 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    awesome investigative work.. (y):cool: