Discover Centmin Mod today
Register Now

WebPerf Cloudflare PageSpeed Looking into uncached requests

Discussion in 'Web Development & Web Performance' started by Brad393, Nov 7, 2020.

  1. Brad393

    Brad393 New Member

    11
    3
    3
    Oct 6, 2020
    Ratings:
    +7
    Local Time:
    4:09 AM
    @eva2000

    We have talked before about Cloudflare workers and APO. I am tracking down some uncached requests. I am seeing that Pinterest Mobile Apps seem to be sending an Accept:*/* header only without text/html quite a number of times. So these are not being cached by my Cloudflare worker and would not be by APO.

    So right now my worker (mostly default from pmeenan repository) only caches when it sees a text/html Accept header and I am thinking of modifying that with an OR condition to also cache if it sees a "*/*" Accept header.

    Does anyone see a problem with doing that? I am running some extra PHP to dump request headers to a file so I can examine them further first.

    Edit: I have now gone through several hundred dumped request headers and the majority of the uncached requests that look like they should have been cached have an agent string with [Pinterest/iOS] or [Pinterest/Android] and they weren't cached because Accept header was indeed only */* and no other reason it appears. So maybe I will just opt in to cache those partial agent strings.

    Interestingly Google-Mediapartners, Googlebot-Image and Bingbot also used an Accept: */* only header while the regular Googlebot used a longer Accept header that includes text/html and several other things including */*. (Assuming these are the real thing and not someone else's crawler spoofing the real bots).

    Thoughts anyone?

    Do you cache beyond Accept text/html with your worker Eva2000?
     
    Last edited: Nov 7, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    45,633
    10,356
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,070
    Local Time:
    10:09 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Yeah I don't differentiate on Accept header text/html and don't have problems that I am aware of on my forums or WordPress sites like https://blog.centminmod.com/ and https://servermanager.guide/

    So try and see
     
  3. Brad393

    Brad393 New Member

    11
    3
    3
    Oct 6, 2020
    Ratings:
    +7
    Local Time:
    4:09 AM
    Deployed worker that also caches a plain */* Accept header and seems to work fine. Looks like an additional 4-5% of total requests is now cached.
     
  4. eva2000

    eva2000 Administrator Staff Member

    45,633
    10,356
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +16,070
    Local Time:
    10:09 PM
    Nginx 1.19.x
    MariaDB 5.5/10.x
    Nice. Guess remaining difference is due to GET vs non-GET requests ?
     
  5. Brad393

    Brad393 New Member

    11
    3
    3
    Oct 6, 2020
    Ratings:
    +7
    Local Time:
    4:09 AM
    Looks like mostly bots that send a "Cache-Control: no-cache" which I am respecting for now. I might change that to send the cached response anyway but refresh for next load. I don't like to make more than one change close in time on a valuable site so maybe next week.

    Edit: According to CF worker subrequest analytics only 5.8% of requests were uncached last hour.
     
    Last edited: Nov 8, 2020