WebPerf Cloudflare PageSpeed Looking into uncached requests

@eva2000

We have talked before about Cloudflare workers and APO. I am tracking down some uncached requests. I am seeing that Pinterest Mobile Apps seem to be sending an Accept:*/* header only without text/html quite a number of times. So these are not being cached by my Cloudflare worker and would not be by APO.

So right now my worker (mostly default from pmeenan repository) only caches when it sees a text/html Accept header and I am thinking of modifying that with an OR condition to also cache if it sees a "*/*" Accept header.

Does anyone see a problem with doing that? I am running some extra PHP to dump request headers to a file so I can examine them further first.

Edit: I have now gone through several hundred dumped request headers and the majority of the uncached requests that look like they should have been cached have an agent string with [Pinterest/iOS] or [Pinterest/Android] and they weren't cached because Accept header was indeed only */* and no other reason it appears. So maybe I will just opt in to cache those partial agent strings.

Interestingly Google-Mediapartners, Googlebot-Image and Bingbot also used an Accept: */* only header while the regular Googlebot used a longer Accept header that includes text/html and several other things including */*. (Assuming these are the real thing and not someone else's crawler spoofing the real bots).

---

Thoughts anyone?

Do you cache beyond Accept text/html with your worker Eva2000?

 

Deployed worker that also caches a plain */* Accept header and seems to work fine. Looks like an additional 4-5% of total requests is now cached.

 

Looks like mostly bots that send a "Cache-Control: no-cache" which I am respecting for now. I might change that to send the cached response anyway but refresh for next load. I don't like to make more than one change close in time on a valuable site so maybe next week.

Edit: According to CF worker subrequest analytics only 5.8% of requests were uncached last hour.