Simon Hearne

Web Performance Consultant

Email Twitter LinkedIn Github RSS

Caching Header Best Practices

Published , last modified on

Caching headers are surprisingly complex and often misconfigured. Here we look at some key cache scenarios and recommend the ideal headers to set.

❤️ 156 likes 🔁 48 reposts 💬 19 comments 🔗 55 shares 🔖 5 bookmarks as of 13:00, April 5


Contents


Introduction

Caching headers are one of those deceptively complex web technologies which are so often overlooked or misconfigured. The fastest request is the one that is not made, and caching headers allow us to tell browsers when they can reuse an asset that they have already downloaded. The reason that these headers are often misconfigured, or at least configured suboptimally, is often through a descent to lowest risk. Allowing a browser to use a cached asset can be considered risky - JS which falls out of sync with HTML, CSS which persists an old campaign style, personalised assets accidentally being shared between visitors.

In this post we will review what caching headers are available and when they should be used. We’ll also talk about invalidating caches and ensuring browsers use the correct assets at the correct time. Note that the focus of this post is on client-side (or downstream) caching - in the client device. Caching in proxies, load balancers and Content Delivery Networks (CDNs) adds some more complexity, and is not covered here.

The Solution

Use versioned assets wherever possible (e.g. main.v123.min.css or main.min.css?v=123) and set a single caching header allowing the maximum cache duration of one year:

Cache-Control: max-age=31536000, immutable

For non-versioned assets which may change, combine the Cache-Control header with an ETag for asynchronous revalidation in the client:

Cache-Control: max-age=604800, stale-while-revalidate=86400
ETag: "<file-hash-generated-by-server>"

For HTML files, set a low TTL and private cache flags:

Cache-Control: max-age:300, private

Do not emit unnecessary caching headers (including ETag and Last-Modified) to prevent unexpected client and server behaviours.

Flowchart of decisions to determine which caching headers are best in different scenarios. The logic is described fully in the document.
A summary of the logic to determine which caching headers are best in different scenarios.

Caching Headers

There are a number of response headers set by a web server or CDN which manage client-side caching. Some are more obvious than others!

Expires - a date (in GMT) after which this asset may no longer be used from the browsers cache and must be re-fetched (docs)

Cache-Control - a combination of features in one header, including how long the resource can be cached by the client (in seconds) as well as whether proxies can cache it, whether to force revalidation and more (docs)

ETag - a string that uniquely identifies an asset version, generally a server-generated hash of the file (docs)

Last-Modified - a timestamp which allows browsers to validate the freshness of cached assets (docs)

Pragma - a hangover from HTTP/1.0, this should generally not be used in preference for Cache-Control except where HTTP/1.0 clients must be supported (docs)

In general I recommend to not emit an Expires header and rely instead on the more comprehensive Cache-Control header. I also recommend to not emit a Last-Modified header and use ETag instead for asset revalidation, this avoids edge cases such as newer files with identical content or clock mismatches between web servers which would cause unnecessary bandwidth consumption.

For revalidation (aka conditional requests) to work, responses must be served with one or both of the ETag or Last-Modified headers. The server must also understand conditional get requests and respond with a 304 Not Modified in the case that the cached asset matches that on origin. Note that validation of weak ETags (prefixed by W/ in the header) are unsupported in some scenarios — including if using Akamai as a CDN — so you may want to use strong ETags where possible.

ETags are simply strings which identify a specific version of an asset. Weak ETags (prefixed with W/) should match assets which are semantically the same (e.g. metadata has been updated but the content is the same), whereas strong ETags should change whenever the asset is changed in any way. (See RFC 7232 for more info!)

By default, Apache 2.3.14 and earlier included INode in the ETag - meaning that the ETag for an identical asset would change between servers! This is no longer included by default, and you can configure what is used to produce the ETag in Apache. The default is to use the last-modified time and the file size, but you can also choose to use a file digest, and manually include the INode back into the calculation.

If you have your own versioning implemented on the web server you could generate ETags yourself. E.g. ETag: core-js-es6-v13.1234-gzip. But then you might as well rename the file to break the cache on the front-end.

Caching Behaviours

There are generally four types of caching behaviour we may want a browser to use when it has downloaded a static asset:

1. Not Cacheable

For assets that are dynamically generated, that are unique or that are only valid once.

Cache-Control: no-cache

This directive tells the client that it can cache the asset, but it cannot use the cached asset without revalidating with the server. If the asset cannot be revalidated (i.e. there were no ETag or Last-Modified response headers on the asset) then the cached asset will never be used.

Cache-Control: no-store

This directive tells the client that the asset may not be stored in cache at all. Any further requests for this asset will be full requests back to the server.

2. Immutable

These assets that can be stored by the browser indefinitely because they never change. Use this for versioned assets (e.g. main.v123.min.css or main.min.css?v=123).

Cache-Control: max-age=31536000, immutable

The immutable directive explicitly tells the browser (where supported) that the asset can be stored for a year and never needs to be revalidated.

3. Time-restricted

For assets which should be stored for the duration of a session (e.g. one day or one week) but should be refreshed or revalidated if the visitor returns later.

Cache-Control: max-age=86400

4. Revalidated

When combined with #3, this allows browsers to use a cached asset for a period of time (from zero seconds up to a year) and then revalidate the object with origin once that period has expired. stale-while-revalidate causes the revalidation request to happen asynchronously, improving performance at the potential risk of using stale content with a second time to live (TTL) value for how long the stale asset may be used.

Cache-Control: max-age=604800, stale-while-revalidate=86400

Optimal Caching Strategy

In general we want browsers to cache everything forever. This can be achieved quite simply by setting a Cache-Control: max-age=31536000 response header - using the maximum TTL value of one year. The issue is that your web applications likely change more frequently than yearly. This is where the most important feature of your web build pipeline comes in - versioned assets!

With versioned assets, the browser will automatically ignore stale cached assets as the references will be updated. Instead of the HTML document requesting main.v123.min.css, the reference will be to main.v124.min.css (note the incremented version number, this will be automatically generated at build time and may be an asset hash).

If assets will always be versioned, we can consider them immutable. That means that if the content of the file changes we guarantee that the filename is changed. Some browsers support this concept natively in the cache-control header, preventing revalidation requests ever being made for these assets:

Cache-Control: max-age=31536000, immutable

Query Strings and Caching

If an asset filename cannot be updated automatically, a query string parameter can be added to the URL to break the cache. For example main.min.css?v=124. This method should be robust in most cases, ensuring that your CDN configuration treats query strings as part of the asset cache key for caching at the CDN level:

Irregular Updates to Unversioned Assets

One of the trickiest caching scenarios to manage is where an asset is unversioned (no unique identifier in the URL) and it is updated on an irregular schedule. This could be, for example, a document containing stock information, transport schedules or feature flags.

In this case, we still want the browser to be able to use the cached asset but also ensure that it remains relatively fresh. This is (in my opinion) the only scenario where entity tags (ETags) make sense. ETags should be used in combination with a valid Cache-Control header to ensure you have control over how the browser manages its cache state:

Cache-Control: max-age=86400, must-revalidate
ETag: "a-unique-hash-generated-by-the-server"

This example will allow the browser to use the cached asset for up to 24 hours (86,400 seconds), after which it must revalidate with the server. A revalidation request (also known as a conditional GET request) acts just like a normal request for the asset, with the addition of one or two request headers: If-None-Match where an ETag was present on the original response, If-Modified-Since in the case that a Last-Modified header was present:

GET /main.min.css

Accept: */*
Accept-Encoding: gzip,br
If-None-Match: "a-unique-hash-generated-by-the-server"

This If-None-Match header is a message to the server that the client has a version of the asset in cache. The server can then check to see whether this is still a valid version of the asset - if so, we will receive an empty 304 response with another ETag which will match the original:

304 Not Modified

ETag: "a-unique-hash-generated-by-the-server"

If the asset has changed since the client cached it, we will get a full response with the new ETag:

200 Found

Content-Length: 100000
ETag: "a-NEW-unique-hash-generated-by-the-server"

This approach is valuable, but does have drawbacks:

  • Generating & maintaining asset hashes has a small compute cost on the server
  • Some web servers have bugs with ETag generation and validation, especially with multiple servers behind a load balancer
  • A 304 response will add a small front-end delay for each request and incurs a small amount of compute on the web server - the client will not use the (valid) cached asset until the 304 response is received

A similar process occurs when the Last-Modified header is present on responses. You may see conditional requests sent with the If-Modified-Since header:

GET /main.min.css

If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT

If both If-Modified-Since and If-None-Match request headers are present then the server must only return a 304 if both conditional fields match the origin content. The If-Modified-Since header allows the server to check the last modified time of the requested asset and again return a 304 Not Modified if it is the equal to or earlier than the timestamp in the request.

Using Last-Modified may cause more cache-misses than ETag if releases touch all files on the web server, even if they have no updates (updating the last modified time on all files). There is a benefit to using Last-Modified over ETag in the edge case where servers may hold older versions of an asset - the ETag will not match and result in a full 200 response, whereas the Last-Modified date will be later than the origin asset, resulting in an empty 304 response.

What about HTML?

HTML assets are critical to performance in traditional (non-SPA) web applications. A 100ms delay downloading the HTML for a page will make everything else 100ms slower.

Folks are normally very nervous about allowing browsers to cache HTML though, for a number of good reasons:

  • Personalisation in the page (e.g. user name, basket contents, geolocation logic)
  • Asset versions (e.g. main.v123.min.css) are updated in the HTML to purge the client cache on release
  • Rapid updates (e.g. a news publication with breaking stories)

Even with all of these considerations, caching HTML can still be achieved and performance can be significantly improved. We just need to be careful!

Short TTLs & private caches

Allowing an HTML asset to be cached for a short period can improve user experience with minimal risk. Setting a TTL of five minutes (300s) for example should be fine, you can also add must-revalidate to ensure that browsers do not use a stale version of the asset after the TTL has expired. This will benefit visitors who click links returning them to previously visited pages, this does not affect the back/forward cache.

The Cache-Control header offers the private attribute, indicating that assets should not be stored by any proxies or CDNs, but may be cached by the client. This attribute enables you to allow browsers to cache personalised content - as the cached asset will not be shared across multiple visitors.

Remove dynamic elements from the HTML document

Dynamic elements such as a basket count, user name, logged in / out status indicator all make caching HTML more tricky. Externalising these out to API calls, JavaScript and localStorage will mean that the base HTML template can be cached in the browser, with dynamic elements updated as the page is building. This will also allow pages to be cached by CDNs - greatly reducing Time to First Byte (TTFB) and thus improving experience.

This concept can be taken further in some scenarios, such as when anonymous visitors are presented generic / un-personalised pages. In this case, you can detect the anonymous visitor (e.g. by presence of a user cookie) and serve them a cached page from the CDN, whereas logged-in / recognised visitors will be served by origin to allow personalisation code to run. The possibilities here are quite exciting!

General Recommendations

Explicitly set the Cache-Control header on all responses. Do not set the following response headers in most scenarios:

  • Last-Modified
  • Expires
  • ETag
  • Pragma

Always set the Cache-Control header, preferably with the value max-age=31536000,immutable alongside unique asset filenames (or no-cache for non-cacheable assets).

In some scenarios, use ETags to allow browsers to revalidated cached content using the following headers (e.g. cache for one week, allow stale assets with async revalidation for up to one day after cache expiry):

Cache-Control: max-age=604800,stale-while-revalidate=86400
ETag: "<generated-by-server>"

The lack of Last-Modified and ETag headers on a response prevents the browser from making conditional GET requests, potentially reducing back-end load.

Ensure that your web application server / load-balancer / CDN handles conditional GET requests correctly and returns a 304 Not Modified response when ETags match. Use cURL or redbot.org to test this.

How to check your headers

Throughout this post I have detailed headers that you might see when requesting assets - of course these aren’t visible on the web application directly! There are two key methods to check the response headers of your assets: using the browser and using command-line tools.

Browser developer tools

All browsers have developer tools to help analyse network traffic, in Google Chrome you can press + + ictrl + shift + i on Windows) to bring up Chrome Developer Tools, it will default to the last open tab so you may need to select the network tab. The network tab only records traffic whilst open, so reload the page to view the requests if none are present.

Once you have some traffic, click on a request to see more details:

screenshot of chrome developer tools showing response headers
Chrome Developer Tools showing response headers

If this is a task that you will repeat often, I recommend adding the relevant response headers to the network table. Right-click on the column headers and select Response Headers, Cache-Control should be there by default and you can also add custom headers here.

screenshot of chrome developer tools showing response headers in a column
Chrome Developer Tools showing a column for Cache-Control headers

Command-line tools

cURL is one of the most widely deployed command line tools and is ideal for checking response headers. A simple command such as curl 'https://simonhearne.com/css/main.css?v=4.1' -Is will show you all response headers for the object, although we can build on this to focus just on the headers that we care about for caching (note that I have added some required request headers user-agent and accept-language so that a valid response is provided):

curl 'https://phoenixnap.com/kb/wp-includes/css/dist/block-library/style.min.css?ver=5.9' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  --compressed -Is | grep -E 'cache-control|etag|last-modified|expires|pragma'

last-modified: Wed, 26 Jan 2022 05:24:24 GMT
etag: W/"61f0db08-1357b"
expires: Thu, 26 Jan 2023 11:29:02 GMT
cache-control: max-age=31536000
pragma: public
cache-control: max-age=31536000, public

For this example, I would recommend that the website owner investigates the various response headers here which may conflict with each other. This set of six response headers could be replaced with a single header: Cache-Control: max-age=31536000! Redbot also shows a number of warnings for this implementation:

screenshot of redbot showing warnings about response headers
Redbot showing potential issues with response headers (view on redbot.org)

How to change response headers

Once you have determined your caching header strategy and reviewed the headers that are currently being emitted - it’s time to make some changes!

Response headers can be set at multiple stages depending on your application architecture - for example the web server, load balancer and CDN can all set and modify response headers. Generally we should expect response headers to be forwarded transparently through proxies such as load balancers and CDNs unless explicit changes have been made, so the web server is the best place to start. You should be able to find what you need with a search for <web server name> set response headers or <web server name> set caching headers on your search engine of choice. For convenience, here are the relevant docs pages for some major web servers, PaaS / SaaS and CDN solutions:

Web Servers

PaaS / SaaS

  • Netlify - note that there is an open issue regarding setting headers by content-type.
  • Wix - note that Wix does not provide fine-grained control over caching headers
  • WPengine
  • Webflow - it doesn’t look like you can control headers and the recommendation is to use a CDN in front of Webflow

CDNs

A note on Cache-Control: public

A lot of resources (e.g. on web.dev) recommend setting the public attribute on Cache-Control headers if resources can be shared between visitors. This is potentially misleading as public is the default behaviour where private is not set.

The public attribute should not be set in almost all cases, as it can lead to an issue with authorized requests. Setting the public attribute in a Cache-Control header for an authorized request (i.e. one which is made with an Authorization header) will explicitly allow intermediary servers to store and serve the response to other visitors. In most cases this is not the expected behaviour and could lead to data leakage.

In closing…

Client-side caching is a key technique to improving front-end speed and user experience. Whilst it may appear complex and risky, investing the time to review your content and setting the correct response headers will reduce bandwidth utilisation and improve speed for return visitors as well as mid-session.

There are multiple methods to manage caching, in this article I have presented a preference for Cache-Control (and ETag where appropriate). Exact implementation is not as important as correctness and consistency, though. Use what works for your application architecture, environment and processes.


Work with me!

I'm currrently available to consult - from web performance workshops to reviewing new site designs, third-party audits to global performance assessments.
Head over to my Consultancy page to see the kind of work I help my clients with and for details on how to get started.


Join the conversation

The comments on this page are fed by tweets, using brid.gy, webmention.io and a modified version of jekyll-webmention_io. To comment, just send a webmention or use a link below to interact via twitter.




Recent comments

  1. Daniel Schildt

    Really good to see detailed overview of different options to consider. Thank you for writing the blog post.

    view
  2. Tim Bednar

    This is great timing. Right now I’m investigating Cloudflare and the “cache everything” with regard to query strings. I get cache everything for static assets for the reasons you provide. Should text/html requests ignore query strings? #webperf

    view
  3. Simon Hearne

    Generally yes, as query strings on pages don’t impact the content. Especially true of SEM tags like utm_***. It can be tricky, best approach is probably to include qs in cache then exclude known ones. E.g. from my friend @TimVereecke

    twitter.com/timvereecke/st…

    view
  4. Rory Hewitt ≠ Hewitt Rory

    Or better, you could use “SimonHearneIsCool-v1” and then when you make a change, “SimonHearneIsCool-v1.1” and so on. IMO, ETag + INM is much better than LM + IMS conceptually. Of course, you’re right that most servers generate an ETag as a hash, which SUCKS

    view
  5. Simon Hearne

    Good points, I’ve just pushed an update with a call-out to reflect that. Also referenced the infamous INode-in-ETag issue in old versions of Apache. I agree with you ETag concept as great, but if folks just leave server defaults then it’s easy to run a sub-optimal config.

    view
  6. Tim Bednar

    Just to underscore what I suspected when I look at the cache HIT ratio for URLs that have query strings that contain utm_* (for 24 hours) it is obvious that there is a potential improvement! #webperf

    view
  7. Simon Hearne

    Great find!

    view
  8. Tim Bednar

    Now the problem is sifting through all the page rules in Cloudflare to figure out how to add or edit a rule to fix this without breaking other stuff. :)

    view
  9. Barry Pollard

    Most web server defaults (or at least the popular ones) do not to use a hash (as you note) and that Apache inode issue is so old it can be forgotten IMHO. If anyone is running two instance of Apache that old in load balanced way with defaults then they have bigger issues!

    view
  10. Barry Pollard

    What’s more annoying is Apache Etag is STILL broken when you sue gzip or br- check out their home page and you’ll see you can only get 304s on images but not text responses. Crazy that they won’t fix that until 2.5/2.6 (which shows no sign of ever reaching release status).

    view
  11. Barry Pollard

    And, given the Etag in Apache is basically the same as the date header, you should just turn it off with the exact same functionality - but gzip and br support! Etag is pointless for Apache.

    view
  12. Rory Hewitt ≠ Hewitt Rory

    1/2: IMO, when caching in a CDN, it’s much better to NOT include all query parameters - just include the ones you know make a difference. That avoids issues with your company’s Marketing team adding additional parameters without you knowing about it.

    view
  13. Rory Hewitt ≠ Hewitt Rory

    2/2: This happens A LOT when the Marketing folks do something like add an (obfuscated? maybe!) email address as a query parameter to every link included in a promotional email - all of a sudden your cache is utterly fragmented. I’ve seen it, and it’s not pretty.

    view
  14. Rory Hewitt ≠ Hewitt Rory

    Yeah, as @TimVereecke says. those utm_* query parameters are just BAD - at least when it comes to caching. But rather than “blacklist” the ‘bad’ parameters, “whitelist” the ‘good’ parameters that SHOULD affect the cache and disallow all the others.

    view
  15. Rory Hewitt ≠ Hewitt Rory

    I think it’s stupid that Apache won’t let you either specify either a hard-coded string or a separate mod which can be used to calculate the ETag based on hard-coded info. Because the options they give you (INode, SIze, Digest, Time etc.) all make the ETag absolutely unique.

    view
  16. Tim Bednar

    Thanks for the confirmation – I’m working with legacy Cloudflare page rules and am trying to figure out how to do exactly that without breaking Rules that are there for a reason.

    view
  17. Stefan Judis

    That was a great read! 👏 Thanks, Simon! 🙇‍♂️

    view
  18. Jesse Jafa

    Wouldn’t you need to also do server-caching for the stale-while-revalidate/must-revalidate to work effectively?

    view
  19. Kevin Farrugia

    Thank you, good read. Why do you recommend private caches for HTML pages? For example, if the page doesn’t include user-specific or authenticated data, such as your blog post.

    simonhearne.com/2022/caching-h…

    view

Recent shares