Caching headers are surprisingly complex and often misconfigured. Here we look at some key cache scenarios and recommend the ideal headers to set.
Caching headers are one of those deceptively complex web technologies which are so often overlooked or misconfigured. The fastest request is the one that is not made, and caching headers allow us to tell browsers when they can reuse an asset that they have already downloaded. The reason that these headers are often misconfigured, or at least configured suboptimally, is often through a descent to lowest risk. Allowing a browser to use a cached asset can be considered risky - JS which falls out of sync with HTML, CSS which persists an old campaign style, personalised assets accidentally being shared between visitors.
In this post we will review what caching headers are available and when they should be used. We’ll also talk about invalidating caches and ensuring browsers use the correct assets at the correct time. Note that the focus of this post is on client-side (or downstream) caching - in the client device. Caching in proxies, load balancers and Content Delivery Networks (CDNs) adds some more complexity, and is not covered here.
Use versioned assets wherever possible (e.g.
main.min.css?v=123) and set a single caching header allowing the maximum cache duration of one year:
Cache-Control: max-age=31536000, immutable
For non-versioned assets which may change, combine the Cache-Control header with an ETag for asynchronous revalidation in the client:
Cache-Control: max-age=604800, stale-while-revalidate=86400 ETag: "<file-hash-generated-by-server>"
For HTML files, set a low TTL and private cache flags:
Cache-Control: max-age:300, private
Do not emit unnecessary caching headers (including
Last-Modified) to prevent unexpected client and server behaviours.
There are a number of response headers set by a web server or CDN which manage client-side caching. Some are more obvious than others!
Expires - a date (in GMT) after which this asset may no longer be used from the browsers cache and must be re-fetched (docs)
Cache-Control - a combination of features in one header, including how long the resource can be cached by the client (in seconds) as well as whether proxies can cache it, whether to force revalidation and more (docs)
ETag - a string that uniquely identifies an asset version, generally a server-generated hash of the file (docs)
Last-Modified - a timestamp which allows browsers to validate the freshness of cached assets (docs)
Pragma - a hangover from HTTP/1.0, this should generally not be used in preference for
Cache-Control except where HTTP/1.0 clients must be supported (docs)
In general I recommend to not emit an
Expires header and rely instead on the more comprehensive
Cache-Control header. I also recommend to not emit a
Last-Modified header and use
ETag instead for asset revalidation, this avoids edge cases such as newer files with identical content or clock mismatches between web servers which would cause unnecessary bandwidth consumption.
For revalidation (aka conditional requests) to work, responses must be served with one or both of the
Last-Modified headers. The server must also understand conditional get requests and respond with a
304 Not Modified in the case that the cached asset matches that on origin. Note that validation of weak ETags (prefixed by
W/ in the header) are unsupported in some scenarios — including if using Akamai as a CDN — so you may want to use strong ETags where possible.
ETags are simply strings which identify a specific version of an asset. Weak ETags (prefixed with
W/) should match assets which are semantically the same (e.g. metadata has been updated but the content is the same), whereas strong ETags should change whenever the asset is changed in any way. (See RFC 7232 for more info!)
By default, Apache 2.3.14 and earlier included
INodein the ETag - meaning that the ETag for an identical asset would change between servers! This is no longer included by default, and you can configure what is used to produce the ETag in Apache. The default is to use the last-modified time and the file size, but you can also choose to use a file digest, and manually include the INode back into the calculation.
If you have your own versioning implemented on the web server you could generate ETags yourself. E.g.
ETag: core-js-es6-v13.1234-gzip. But then you might as well rename the file to break the cache on the front-end.
There are generally four types of caching behaviour we may want a browser to use when it has downloaded a static asset:
1. Not Cacheable
For assets that are dynamically generated, that are unique or that are only valid once.
This directive tells the client that it can cache the asset, but it cannot use the cached asset without revalidating with the server. If the asset cannot be revalidated (i.e. there were no
Last-Modified response headers on the asset) then the cached asset will never be used.
This directive tells the client that the asset may not be stored in cache at all. Any further requests for this asset will be full requests back to the server.
These assets that can be stored by the browser indefinitely because they never change. Use this for versioned assets (e.g.
Cache-Control: max-age=31536000, immutable
immutable directive explicitly tells the browser (where supported) that the asset can be stored for a year and never needs to be revalidated.
For assets which should be stored for the duration of a session (e.g. one day or one week) but should be refreshed or revalidated if the visitor returns later.
When combined with #3, this allows browsers to use a cached asset for a period of time (from zero seconds up to a year) and then revalidate the object with origin once that period has expired.
stale-while-revalidate causes the revalidation request to happen asynchronously, improving performance at the potential risk of using stale content with a second time to live (TTL) value for how long the stale asset may be used.
Cache-Control: max-age=604800, stale-while-revalidate=86400
Optimal Caching Strategy
In general we want browsers to cache everything forever. This can be achieved quite simply by setting a
Cache-Control: max-age=31536000 response header - using the maximum TTL value of one year. The issue is that your web applications likely change more frequently than yearly. This is where the most important feature of your web build pipeline comes in - versioned assets!
With versioned assets, the browser will automatically ignore stale cached assets as the references will be updated. Instead of the HTML document requesting
main.v123.min.css, the reference will be to
main.v124.min.css (note the incremented version number, this will be automatically generated at build time and may be an asset hash).
If assets will always be versioned, we can consider them
immutable. That means that if the content of the file changes we guarantee that the filename is changed. Some browsers support this concept natively in the
cache-control header, preventing revalidation requests ever being made for these assets:
Cache-Control: max-age=31536000, immutable
Query Strings and Caching
If an asset filename cannot be updated automatically, a query string parameter can be added to the URL to break the cache. For example
main.min.css?v=124. This method should be robust in most cases, ensuring that your CDN configuration treats query strings as part of the asset cache key for caching at the CDN level:
- Cloudflare includes query strings by default
- Akamai excludes query strings by default
- Fastly includes query strings by default
Irregular Updates to Unversioned Assets
One of the trickiest caching scenarios to manage is where an asset is unversioned (no unique identifier in the URL) and it is updated on an irregular schedule. This could be, for example, a document containing stock information, transport schedules or feature flags.
In this case, we still want the browser to be able to use the cached asset but also ensure that it remains relatively fresh. This is (in my opinion) the only scenario where entity tags (ETags) make sense. ETags should be used in combination with a valid
Cache-Control header to ensure you have control over how the browser manages its cache state:
Cache-Control: max-age=86400, must-revalidate ETag: "a-unique-hash-generated-by-the-server"
This example will allow the browser to use the cached asset for up to 24 hours (86,400 seconds), after which it must revalidate with the server. A revalidation request (also known as a conditional GET request) acts just like a normal request for the asset, with the addition of one or two request headers:
If-None-Match where an ETag was present on the original response,
If-Modified-Since in the case that a Last-Modified header was present:
GET /main.min.css Accept: */* Accept-Encoding: gzip,br If-None-Match: "a-unique-hash-generated-by-the-server"
If-None-Match header is a message to the server that the client has a version of the asset in cache. The server can then check to see whether this is still a valid version of the asset - if so, we will receive an empty
304 response with another ETag which will match the original:
304 Not Modified ETag: "a-unique-hash-generated-by-the-server"
If the asset has changed since the client cached it, we will get a full response with the new ETag:
200 Found Content-Length: 100000 ETag: "a-NEW-unique-hash-generated-by-the-server"
This approach is valuable, but does have drawbacks:
- Generating & maintaining asset hashes has a small compute cost on the server
- Some web servers have bugs with ETag generation and validation, especially with multiple servers behind a load balancer
304response will add a small front-end delay for each request and incurs a small amount of compute on the web server - the client will not use the (valid) cached asset until the
304response is received
A similar process occurs when the
Last-Modified header is present on responses. You may see conditional requests sent with the
GET /main.min.css If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT
If-None-Match request headers are present then the server must only return a
304 if both conditional fields match the origin content. The
If-Modified-Since header allows the server to check the last modified time of the requested asset and again return a
304 Not Modified if it is the equal to or earlier than the timestamp in the request.
Last-Modified may cause more cache-misses than
ETag if releases touch all files on the web server, even if they have no updates (updating the last modified time on all files). There is a benefit to using
ETag in the edge case where servers may hold older versions of an asset - the
ETag will not match and result in a full
200 response, whereas the
Last-Modified date will be later than the origin asset, resulting in an empty
What about HTML?
HTML assets are critical to performance in traditional (non-SPA) web applications. A 100ms delay downloading the HTML for a page will make everything else 100ms slower.
Folks are normally very nervous about allowing browsers to cache HTML though, for a number of good reasons:
- Personalisation in the page (e.g. user name, basket contents, geolocation logic)
- Asset versions (e.g.
main.v123.min.css) are updated in the HTML to purge the client cache on release
- Rapid updates (e.g. a news publication with breaking stories)
Even with all of these considerations, caching HTML can still be achieved and performance can be significantly improved. We just need to be careful!
Short TTLs & private caches
Allowing an HTML asset to be cached for a short period can improve user experience with minimal risk. Setting a TTL of five minutes (300s) for example should be fine, you can also add
must-revalidate to ensure that browsers do not use a stale version of the asset after the TTL has expired. This will benefit visitors who click links returning them to previously visited pages, this does not affect the back/forward cache.
Cache-Control header offers the
private attribute, indicating that assets should not be stored by any proxies or CDNs, but may be cached by the client. This attribute enables you to allow browsers to cache personalised content - as the cached asset will not be shared across multiple visitors.
Remove dynamic elements from the HTML document
This concept can be taken further in some scenarios, such as when anonymous visitors are presented generic / un-personalised pages. In this case, you can detect the anonymous visitor (e.g. by presence of a user cookie) and serve them a cached page from the CDN, whereas logged-in / recognised visitors will be served by origin to allow personalisation code to run. The possibilities here are quite exciting!
Explicitly set the
Cache-Control header on all responses. Do not set the following response headers in most scenarios:
Always set the
Cache-Control header, preferably with the value
max-age=31536000,immutable alongside unique asset filenames (or
no-cache for non-cacheable assets).
In some scenarios, use ETags to allow browsers to revalidated cached content using the following headers (e.g. cache for one week, allow stale assets with async revalidation for up to one day after cache expiry):
Cache-Control: max-age=604800,stale-while-revalidate=86400 ETag: "<generated-by-server>"
The lack of
ETag headers on a response prevents the browser from making conditional GET requests, potentially reducing back-end load.
Ensure that your web application server / load-balancer / CDN handles conditional GET requests correctly and returns a
304 Not Modified response when ETags match. Use
cURL or redbot.org to test this.
How to check your headers
Throughout this post I have detailed headers that you might see when requesting assets - of course these aren’t visible on the web application directly! There are two key methods to check the response headers of your assets: using the browser and using command-line tools.
Browser developer tools
All browsers have developer tools to help analyse network traffic, in Google Chrome you can press ⌘ + ⌥ + i ( ctrl + shift + i on Windows) to bring up Chrome Developer Tools, it will default to the last open tab so you may need to select the network tab. The network tab only records traffic whilst open, so reload the page to view the requests if none are present.
Once you have some traffic, click on a request to see more details:
If this is a task that you will repeat often, I recommend adding the relevant response headers to the network table. Right-click on the column headers and select
Cache-Control should be there by default and you can also add custom headers here.
cURL is one of the most widely deployed command line tools and is ideal for checking response headers. A simple command such as
curl 'https://simonhearne.com/css/main.css?v=4.1' -Is will show you all response headers for the object, although we can build on this to focus just on the headers that we care about for caching (note that I have added some required request headers
accept-language so that a valid response is provided):
curl 'https://phoenixnap.com/kb/wp-includes/css/dist/block-library/style.min.css?ver=5.9' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36' \ -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \ --compressed -Is | grep -E 'cache-control|etag|last-modified|expires|pragma' last-modified: Wed, 26 Jan 2022 05:24:24 GMT etag: W/"61f0db08-1357b" expires: Thu, 26 Jan 2023 11:29:02 GMT cache-control: max-age=31536000 pragma: public cache-control: max-age=31536000, public
For this example, I would recommend that the website owner investigates the various response headers here which may conflict with each other. This set of six response headers could be replaced with a single header:
Cache-Control: max-age=31536000! Redbot also shows a number of warnings for this implementation:
How to change response headers
Once you have determined your caching header strategy and reviewed the headers that are currently being emitted - it’s time to make some changes!
Response headers can be set at multiple stages depending on your application architecture - for example the web server, load balancer and CDN can all set and modify response headers. Generally we should expect response headers to be forwarded transparently through proxies such as load balancers and CDNs unless explicit changes have been made, so the web server is the best place to start. You should be able to find what you need with a search for
<web server name> set response headers or
<web server name> set caching headers on your search engine of choice. For convenience, here are the relevant docs pages for some major web servers, PaaS / SaaS and CDN solutions:
PaaS / SaaS
- Netlify - note that there is an open issue regarding setting headers by content-type.
- Wix - note that Wix does not provide fine-grained control over caching headers
- Webflow - it doesn’t look like you can control headers and the recommendation is to use a CDN in front of Webflow
- Cloudflare - lets you set a TTL but no fine-grained control over response headers
- Cloudflare Pages
- Cloudflare Workers - I use a worker in front of Netlify to overcome limitations in netlify.toml. Provides absolute control over response headers.
A note on Cache-Control: public
A lot of resources (e.g. on web.dev) recommend setting the
public attribute on
Cache-Control headers if resources can be shared between visitors. This is potentially misleading as
public is the default behaviour where
private is not set.
public attribute should not be set in almost all cases, as it can lead to an issue with authorized requests. Setting the
public attribute in a
Cache-Control header for an authorized request (i.e. one which is made with an
Authorization header) will explicitly allow intermediary servers to store and serve the response to other visitors. In most cases this is not the expected behaviour and could lead to data leakage.
Client-side caching is a key technique to improving front-end speed and user experience. Whilst it may appear complex and risky, investing the time to review your content and setting the correct response headers will reduce bandwidth utilisation and improve speed for return visitors as well as mid-session.
There are multiple methods to manage caching, in this article I have presented a preference for
ETag where appropriate). Exact implementation is not as important as correctness and consistency, though. Use what works for your application architecture, environment and processes.
Work with me!
I'm currrently available to consult - from web performance workshops to reviewing new site designs, third-party audits to global performance assessments.
Head over to my Consultancy page to see the kind of work I help my clients with and for details on how to get started.
Join the conversation
The comments on this page are fed by tweets, using brid.gy, webmention.io and a modified version of jekyll-webmention_io. To comment, just send a webmention or use a link below to interact via twitter.
This is great timing. Right now I’m investigating Cloudflare and the “cache everything” with regard to query strings. I get cache everything for static assets for the reasons you provide. Should text/html requests ignore query strings? #webperf
Generally yes, as query strings on pages don’t impact the content. Especially true of SEM tags like
utm_***. It can be tricky, best approach is probably to include qs in cache then exclude known ones. E.g. from my friend @TimVereecke
Or better, you could use “SimonHearneIsCool-v1” and then when you make a change, “SimonHearneIsCool-v1.1” and so on. IMO, ETag + INM is much better than LM + IMS conceptually. Of course, you’re right that most servers generate an ETag as a hash, which SUCKS
Good points, I’ve just pushed an update with a call-out to reflect that. Also referenced the infamous INode-in-ETag issue in old versions of Apache. I agree with you ETag concept as great, but if folks just leave server defaults then it’s easy to run a sub-optimal config.
Just to underscore what I suspected when I look at the cache HIT ratio for URLs that have query strings that contain utm_* (for 24 hours) it is obvious that there is a potential improvement! #webperf
Most web server defaults (or at least the popular ones) do not to use a hash (as you note) and that Apache inode issue is so old it can be forgotten IMHO. If anyone is running two instance of Apache that old in load balanced way with defaults then they have bigger issues!
What’s more annoying is Apache Etag is STILL broken when you sue gzip or br- check out their home page and you’ll see you can only get 304s on images but not text responses. Crazy that they won’t fix that until 2.5/2.6 (which shows no sign of ever reaching release status).
1/2: IMO, when caching in a CDN, it’s much better to NOT include all query parameters - just include the ones you know make a difference. That avoids issues with your company’s Marketing team adding additional parameters without you knowing about it.
2/2: This happens A LOT when the Marketing folks do something like add an (obfuscated? maybe!) email address as a query parameter to every link included in a promotional email - all of a sudden your cache is utterly fragmented. I’ve seen it, and it’s not pretty.
Yeah, as @TimVereecke says. those utm_* query parameters are just BAD - at least when it comes to caching. But rather than “blacklist” the ‘bad’ parameters, “whitelist” the ‘good’ parameters that SHOULD affect the cache and disallow all the others.
I think it’s stupid that Apache won’t let you either specify either a hard-coded string or a separate mod which can be used to calculate the ETag based on hard-coded info. Because the options they give you (INode, SIze, Digest, Time etc.) all make the ETag absolutely unique.
Thank you, good read. Why do you recommend private caches for HTML pages? For example, if the page doesn’t include user-specific or authenticated data, such as your blog post.
mattzeunert bookmarked a post https://simonhearne.com/2022/caching-header-best-practices/
CKsTechnologyNews bookmarked a post https://simonhearne.com/2022/caching-header-best-practices/