Simon Hearne

Web Performance Consultant

Email Twitter LinkedIn Github RSS

Measuring Performance behind consent popups

Published

Opting in to cookies can load multiple megabytes of additional content. Here are tips on how to measure web performance both with and without cookie consent.

❤️ 77 likes 🔁 24 reposts 💬 12 comments 🔗 9 shares 🔖 1 bookmark as of 22:00, August 3


Contents


Introduction

Cookie consent has given users control over the data that is shared with websites and third-parties, but it has made measuring performance difficult. Cookie consent is managed outside of standard APIs, so it can be implemented any way a website wants. Within the EU (and generally for websites hosted in the EU) consent is not assumed, so a visitor must opt-in to third-party tracking etc.

This makes testing performance difficult as standard synthetic test tools such as WebPageTest will only measure the opted-out experience. Conversely, real user measurement (RUM) tools such as mPulse will only measure the opted-in experience. See the problem? This also means that large-scale datasets such as HTTP Archive and Google’s CRUX present the performance of the opted-out and opted-in experience, respectively.

The problem with RUM data is generally unavoidable, as performance measurement is not a functional requirement, although few websites have so far implemented a true opt-in solution for performance tracking. For reference, mPulse provides the inline consent plugin as a simple method to implement this correctly.

The problem of synthetic tests is thankfully more simple to solve: we simply need to have the test ‘opt-in’ as a user would. Once we have the ability to opt-in, there will be multiple scenarios we can explore:

  • Opted-out first view - this is the easy one
  • Opted-out repeat view - cached experience
  • Opted-in first view - consent is given, page not cached
  • Opt-in after page load - measure just the opt-in activity
  • Opted-in repeat view - cached experience, with cookies

The results can be… eye opening. In the following sections we will walk through how to succesfully test these states, how to compare the results and present a small case study based on a corporate website.

Opt-in Techniques

There are generally two methods to test the opt-in process in synthetic tests: setting the opt-in cookie manually or effectively clicking on the “accept” or “I agree” elements. I’ll show you how to do both using WebPageTest scripts, this is one of the many surprisingly powerful features of WPT and I recommend reading up on the documentation. To learn even more about WPT, I recommend you grab a copy of the book Using WebPageTest

The Script UI is shown below for reference.

screenshot of adding a custom script to web page test
Adding a script to WebPageTest

This can actually be quite tricky and requires some detective work. Consent is normally stored in a cookie, which means we can simulate the opt-in state by setting that cookie value manually. First, though, we need to find out what cookie to set!

Open a new private / incognito tab and crack open the developer tools on your browser of choice. Head to the section which shows cookies (Application in Chrome) and load the page under test.

Observe the cookies that are set on the initial page load (and screenshot them if you wish). Then accept the cookie banner and watch what changes! Unfortunately, there are likely to be a lot of new cookies, so spotting the one you care about can be difficult. Here, for example, is an enticingly named CONSENT cookie, but it is not the cookie we are looking for. We can tell this as the cookie domain is google.com, not that of the page under test:

Developer tools showing multiple cookies

You can also try filtering the cookies by the term consent to see if anything jumps out, and if all else fails? View-source!

If you view-source and search for “cookie” or “consent”, you are likely to find the script responsible for the cookie banner. It might look something like this:

<!-- OneTrust Cookies Consent Notice start -->
<script src="https://cdn.cookielaw.org/scripttemplates/otSDKStub.js" type="text/javascript"></script>
<script type="text/javascript">
function OptanonWrapper() {window.dataLayer.push({event:'OneTrustGroupsUpdated'});}
</script>
<!-- OneTrust Cookies Consent Notice end -->

This in turn points me at https://cdn.cookielaw.org/scripttemplates/otSDKStub.js. Loading that script shows me right at the top of the file: this.optanonCookieName="OptanonConsent". Sleuthing complete! Back in developer tools we can filter cookies to OptanonConsent and see if any have been set.

Developer tools showing optnanon cookie is set

Once you have the cookie, copy the value that has been set. It may be something simple like true or 1, or it could be more obtuse and include a unique ID, timestamp and any number of other variables. We can add the cookie to our test scripts using a WebPageTest script and the setCookie command. Note that the domain should have http:// as the protocol and should not have a trailing slash. Double check this if you see results like (Test Error: Unhandled exception in test run: coercing to Unicode: need string or buffer, int found)

setCookie http://www.akamai.com OptanonConsent=consentId=1f6c1653-871e-468d-913c-378d05a8539e&datestamp=Tue+May+12+2020+21%3A42%3A07+GMT%2B0100+(British+Summer+Time)&version=5.15.0&interactionCount=2&isIABGlobal=false&landingPath=NotLandingPage&groups=&hosts=&legInt=
setCookie http://www.akamai.com OptanonAlertBoxClosed=2020-05-13T06:20:21.745Z
navigate %URL%

If the cookie value has a timestamp it may become invalid after a while, so if you are running these tests in an automated environment (e.g. CI testing or monitoring) then you may need to make this dynamic at the time of the test.

You may also notice that I’ve set two cookies in the script above. The first is the actual consent, allowing Google Tag Manager to inject third-parties. The second is necessary to prevent the consent banner re-appearing. Why are they separate? Who knows.

Clicking on the Element

This is more simple, conceptually, than setting the cookie in a script. Here we emulate a user actively opting in by triggering the event in the website’s code. To do this in WPT, we just need a valid selector for the accept action. Right-click on the element in your browser of choice and hit “Inspect”.

Right-click context menu in browser showing inspect element option

Once you have the element highlighted in the developer tools window, you can quickly observe if there is an easy selector such as an element ID (present in this example). If not, you can right-click on the element here and copy a selector. This can often end up being pretty long if the element does not have an ID, but it should at least give you a good starting point:

Right-click context menu in developer tools showing copy selector option

Once you have your selector, check it works by heading to the browser console and attempting to select it:

document.querySelector('#onetrust-accept-btn-handler')

If the result is the element then you are on to a winner! We now just need to add this to our WPT script. There are a number of ways to do this, including WPT functions like click and clickAndWait. I find that this can get frustrating at times when switching between WPT and the browser console, so I tend do use exec or execAndWait which just execute the JavaScript you provide. If you need to chain multiple actions such as opening a modal then clicking a button, make sure your last action is execAndWait and preceding actions are exec. Assuming a simple interation, we will want a script that looks like:

setEventName loadPage
navigate %URL%
setEventName acceptCookies
execAndWait document.QuerySelector('#onetrust-accept-btn-handler').click()

This will load the page at the URL you have entered in the URL field as the first step, then accept cookies and record any further activity as a separate step / page load. If you’d like to combine these you can use combineSteps, although note that there may be a small gap in the waterfall between the page loading and the script executing:

combineSteps
navigate %URL%
execAndWait document.QuerySelector('#onetrust-accept-btn-handler').click()

In this case, the button will be clicked shortly after the initial page load, close to the expected user behaviour. Both test options are valid, and you should try both!

Bonus Tips

Here are a few handy tips for working with WPT and cookie consent prompts.

Clearing up cookies

It is quite likely that you will need to emulate a first-time viewer multiple times in this process. The most simple method to reset state is to clear site data. In Chrome, this is under the “Applications” tab. Once cleared, hit Cmd+R / Ctrl+F5 to perform a hard reload.

Screenshot of developer tools showing clear site data option

Debugging WPT Scripts

WPT scripts do not provide debugging information, but we can debug the output. In this case we are expecting to see either a cookie being sent with the page request, or an element being clicked in the page load. Use the filmstrip view or page screenshot to check that the page looks as you expect for an opt-in user, i.e. the cookie banner is not present. If the banner is not present, you need to verify that your JavaScript actually works in your browser or check that the cookie value is being set.

To check the cookie value, click on the test waterfall to get to the Details view, then click on the first request to open the modal request viewer. Click on the request tab and check that your cookie header is being set:

Screenshot of web page test showing cookie data in request header

If not, you might need to try experimenting with syntax until it works 🙂

Some cookie consent services may load the banner in an iframe, this makes selecting content a little more tricky. First you need to select the iframe document and then select the button / confirmation element within it:

execAndWait document.querySelector('iframe[name=consentFrame]').contentWindow.document.querySelector('#confirmBtn').click()

Geographic Variations

Some sites may have different policies based on the location of the visitor, for example respecting CCPA in California vs. Cookie Law in EU. I would recommend testing from at least one North American location and one European location. Countries within Europe may have slightly different laws, so testing from the Netherlands (known as relatively strict) is a good choice.

Results

We need a test subject to use as an example for the following sections. I wanted to use a website which implements cookie consent, but is not heavily driven by third-party content (i.e. not a digital publisher). Corporate websites are great for this, so that’s what we’ll look at.

The headlines are:

  • Opted out experiences are ~35% faster
  • Opting in downloads 2.5MB of additional JavaScript
  • Opted in repeat views are twice as slow as opted out

The following sections go into a bit more detail about each test. Below are links to the test results:

First View Comparison

This is the most common type of comparison you are likely to perform, what does a full page load look like with and without consent? I’ve used the handy wpt-compare.app by Matt Hobbs to make this a little easier 🙂

Filmstrip images showing cookie opt-in is slower
Filmstrip comparison showing similar visual performance. Source

The visual performance comparison shows relatively little difference between the initial render performance as we would expect, except for the visual addition of the cookie banner. The total load time was somewhat higher for the opted-in test, at 5s vs 4.2s for opted-out.

Filmstrip images showing cookie opt-in is slower
Timing comparison showing significant increases for opt-in on first view. Source

Looking at more of the timers we can see some significant degradations. Notably, CPU Busy Time increases 2.5x from 4.9s to 12s, indicating that the third-party content has a significant impact on the browser main thread. This could cause a laggy experience, especially on mobile devices. The visual performance of the page is impacted as well, with the visual complete time increasing by 35% from 5.4s to 7.3s.

The request maps below show the significance of the opt-in on the diversity of domains on the webpage. The number of requests increases 2.4x from 60 to 144, with an additional 30 TCP connections established.

Request map of opted-out experience showing few domains
Requestmap for opted-out experience. Source
Request map of opted-out experience showing few domains
Requestmap for opted-in experience. Source

Repeat View Comparison

Whereas first-party content is likely to be cached for a repeat view, third-party content may be downloaded again. Third-party trackers and analytics scripts will also execute on every page view, potentially increasing the relative impact of these on cached experiences.

Filmstrip images showing cookie opt-in is slower
Filmstrip comparison showing opt-in performance is slower. Source

The visual performance comparison above shows a close race, with the opted-out experience winning by a small margin of 200ms. Note that the opted-in experience filmstrip continues for another 2.5s. Again, the timers tell a more interesting story:

Filmstrip images showing cookie opt-in is slower
Timing comparison showing significant increases for opt-in on repeat view. Source

CPU Busy Time again shows a significant jump, as do the visual metrics. These show that the relative impact of the opted-in experience is much greater on cached views. Page load increases over 160% from 1.2s to 3.2s and visual complete more than doubles from 1.9s to 4.9s!

Opt-in after page load

The sections above have shown the impact of cookie consent on page load performance, but what about the actual action of consenting? Clicking the button immediately triggers a cascade of requests from third-parties, each hoping to collect as much data as possible. We can measure this performance impact by splitting the navigate and action steps, then we can analyse just the activity which occurs after the opt-in event.

Filmstrip images showing cookie opt-in is slower
Content breakdown for the opt-in interaction. Source

Almost 2.5MB of JavaScript content is loaded across 33 objects, at a cost of 680kB of bandwidth and 1.8s of download time. Knowing this, would you opt-in to cookies? What’s the performance cost for your opt-in experience?

As for the UX of cookie consent, Vitaly Friedman has written about the varied (and extensive) issues with the current state of cooke prompts. Vitaly covers this and more in his excellent closing keynote at performance.now() conference 2019.


Join the conversation

The comments on this page are fed by tweets, using brid.gy, webmention.io and a modified version of jekyll-webmention_io. To comment, just send a webmention or use a link below to interact via twitter (comments are fetched if the full URL is referenced in the tweet). Webmentions are pulled once per day.




Recent comments

  1. Magnus Dahl

    I think I need to gather the available data and see where we end up!

    reply
  2. Simon Hearne

    I look forward to seeing the data!

    reply
  3. Michael Gooding

    I always accept purely because of FOMO. They get me every time with talk of a reduced experience if I opt out. That and I know it’s annoying if they have RUM!

    reply
  4. as a total aside Simon, are you dropping #perfmatters for #siteSpeed? I thought - for some reason - that #webperf would have been the victim of the new hashtag.

    reply
  5. Simon Hearne

    I’m keeping an open mind 🙂 What are your thoughts Henri?

    reply
  6. Simon Hearne

    My wife said something similar, that she doesn’t know if the website will break later on if she doesn’t accept cookies straight away.

    reply
  7. I def had a few, and in fact have the tweet bookmarked. But I do mostly agree. Been meaning to add to the thread. Will ping you when I do. ✍🏾

    reply
  8. Silvermouse

    I’m curious if there’s any metrics on people leaving the site because a cookie consent interrupted them reading some content or similar

    reply
  9. Silvermouse

    Also it’s a pain to reject. It’s almost always either “accept” or click “manage” and then another 3 or 4 clicks on an interface that already got in your way

    reply
  10. Simon Hearne

    The sad irony is that we cannot measure that in a way that complies with cookie law.

    reply
  11. Emil Rasmussen

    Thanks. Absolutely love “navigate %URL%” trick! 👍

    reply
  12. Simon Hearne

    @AndyDavies taught me that one years ago, super handy!

    reply

Recent shares


Recent bookmarks

  1. Jeremy Keith
    view