Do you use Google or Adobe Analytics for reporting? How about tools like Doubleclick or Facebook for advertising? Do you have a large amount of iOS traffic or high usage of Safari in your analytics? Does your optimization program hinge on a client-side testing tool? If any of these scenarios describe you, you should be concerned about the impact of browser privacy changes.
Across a range of industries and clients, we’ve seen these tools (and more) commonly impacted by browser privacy changes. Cookies don’t last as long as they used to, and that’s if they even get sent in the first place. These changes to how the internet works can affect how your Optimization program operates today. Solutions may be available, but only if you act to implement them.
This post is a Q & A born from our TLC discussion with Cory Underwood, privacy expert and Platform Engineer at Search Discovery.
First, I can’t overemphasize the importance of these changes. They will impact you! Act now to be prepared!
Second, cookies used to connect the dots for marketers between an individual’s behavior in the past and their behavior in the present, linking distinguishing features to help us identify someone over time. Now that link is often broken.
When this happens we can’t go back, we can only go forward. So, for marketers, an individual’s timeline starts much more recently than we’re used to. A person’s historic details are lost and they are, in effect, a new/different person each time they show up on a website
2. What change will be the biggest problem for optimizers? What we can measure or how long we can retain the information?
The biggest problem is the constantly shifting technical landscape that’s forcing changes in infrastructure or test design. This is because not all brands maintain in-house staff, and, depending on the solutions and how they are deployed, there’s an increased chance that brands may end up having to pay for work twice.
We saw this play out in early 2019. When ITP capped cookies to 7 days, many vendors shifted to using Local Storage as a storage mechanic, only to have that also be rendered ineffective a few months later. Continued investment was required just to keep the optimization platform (and, by extension, the experimentation program) operational.
To stay ahead of this problem, optimizers need to 1.) evaluate their browser mix breakdown and 2.) understand the changes and how they relate to their site and testing solution. With this information, they can estimate the impact that changes might be having on their testing program.
As far as data retention, for the client, it’s going to depend on how they set state (see below). The testing platform will set state and the browser may cap it at 24 hours, 7 days, or a time period specified/desired. Which of these it ends up being will depend on the platforms and scenarios at play.
Data Retention on the server would be governed by corporate policy or legal terms, depending on the jurisdiction and type of data involved. A system or platform could be developed or purchased that meet all the legal and technical requirements.
3. So, what could go wrong? That is, what things might now be incorrect in the shifting technical landscape?
- Someone who visits multiple times during a test may receive different experiences
- The same activity may behave differently on each browser due to different privacy protection policies
- For long conversion windows, conversions might not be correctly attributed to the experiences delivered
- Your optimization program will see more “inconclusive” or “flat” results as clean separation of variation impact becomes more difficult.
- Experiments are likely reaching pre-calculated minimum sample size faster than predicted as return visitors are mislabeled as new visitors.
- Tests of functionality that require state persistence may not function correctly
- Users won’t be identified across visits. This implies:
- – Marketing Channel attribution issues
- – Inflated number of unique users
- – Inflated % of new users
- – Deflated number of visits per user
- – Deflated conversion rate
- – Deflated Frequency
- – Wrong “recency”
4. What % of audiences will this impact in the short-term? Longer-term?
Short Term: All Safari traffic across the board. More recently, every browser on iOS 14 or iPadOS 14, as well as Safari on MacOS Big Sur.
Longer-Term: The latter will continue to increase until it eventually covers all currently active devices over the next several years.
This does not touch upon any challenges from a legal standpoint.
5. Are there workarounds potential in experimentation technology to address some of these concerns?
State (the mechanic that says who you are now is who you were 5 minutes ago) is vital to experimentation. Without state persistence, you get placed into the test multiple times, possibly seeing multiple experiences. Once primarily an issue with advanced omnichannel, cross-device testing, the recent changes make state persistence difficult even in test scenarios on a single-device.
What you’ll need to do varies depending on your particular test platform. SiteSpect and Conductics work this way out of the box. Optimizely and VWO can be made to work this way via enhancements the brands have to do manually. Adobe accomplishes this via the ECID service for Target, but that’s likely next on the chopping block.
Here’s a table to help you check in with various platforms to see their statements on ITP changes.
Google has not addressed the problem directly. Here’s what we know.
6. Would server-side testing become the norm? Will that address at least some of the changes?
Most browser changes impact what can be seen client-side, but the client has no concept of what happens on the server. Because of this, the client can’t impact to varying degrees what the server does.
So the current best long-term solution would be to have at least a hybrid, or a full server-side solution that operates either in the current domain namespace, or is shielded via a proxy server and not a CNAME record. CNAME records ceased to be viable for this purpose with the release of iOS 14.2 and Mac OS Big Sur.
Assuming such a solution sets state in a compatible way to the current browser protections, it’s the best long-term approach I could recommend based on the current and short-term foreseen state of the industry.
7. Do we need to re-think the types of tests we’re running if a portion of our traffic will no longer persist in the experience they’re first shown?
You might want to introduce a slight bias (prioritization) towards session-based tests (tests that are about “this visit to the site” rather than “this AND future visits to the site.” The kicker is that the driver for these types of tests tends to be more the overall business model and site purpose, so it generally only applies at the margins.
Also, consider longitudinal tests such as causal impact or diff-in-diff based designs. These are inherently messier than an A/B/n test, IMHO, but they might provide good solutions.
8. Is there a way to assess the potential impact to any particular company or program of these pending changes?
YES! Search Discovery is offering a free audit of websites to determine ITP impacts to the site. Beyond the experimentation program impacts, the free audit also reviews the state of browser impacts covering the areas of:
- Possible Functionality related features (this will vary by the website of course, but here I’m referring to features that have been built around client-side storage, including tagging, consent management, surveys, etc.)
9. What resources would you send readers to if they want to learn more about these pending changes?
Well, first, they can always find me on the TLC, Measure, and cunderwood.dev. In addition, check out these blog posts: The New Reality: Data Collection in iOS14 and Intelligent Tracking Prevention in iOS14, iPadOS 14, and Safari 14 by Simo Ahava.