Browser Changes Impact on Testing, with Cory Underwood

Due to browser privacy laws and regulations, the underlying conditions for how the internet works is constantly changing. You might be thinking, “Okay, how is this impacting my testing and experimentation program, and what do I need to do to stay up to date?”

Recently, the Test and Learn Community met with Search Discovery’s Platform Engineer and browser expert, Cory Underwood, discusses the current landscape of browser privacy regulations and the implications they have for marketing analysts within the optimization space.

Since this conversation has been one of our most-viewed videos, we decided to bottom-line it here to provide the testing community with fast/historical access to this conversation on the edge of tremendous industry change.

Learn more about and join the Test & Learn Community!

Current landscape

“Not keeping up with the browser privacy-changes risks experiment breakage, inaccurate data, and velocity delays.”

Before browser privacy regulations, visitors were responsible for proactively deleting cookies from their browser. Without the deletion of cookies, it was fairly easy to accurately track a visitor’s new and returning visits. Cookies are important for gathering accurate information about the visitor, such as what pages they viewed, the products they purchased, and, in the case of testing and experimentation, what experience the visitor received.

To comply with browser privacy regulatory requirements, browsers, including Brave and Safari, have started to put technology in place that makes it increasingly more difficult to track visitors. Instead of users deleting cookies from their browser, nowadays the user just needs to close their computer or wait 24 hours for cookies to be automatically deleted. (In most cases Safari will keep data for 7 days, but it can go as low as 24 hours.)

This is an issue for testing because a visitor who visits your site unauthenticated may receive a different experience each time they return, resulting in poor user experience and muddied analytics data. To make this issue a bit more complex, each browser is complying with browser privacy regulations in different ways, making it an even greater challenge to work around the restrictions in place.

Another issue that impacts testing and analytics reporting is that devices can be deceiving. For example, back in September 2019 Apple changed how iPads were being identified. The result of this type of action—especially when there is no release statement from the browser and no awareness on behalf of the public—is resegmentation. This is because the user agent has changed: instead of the iPad being identified as an iPad, it was being identified as a desktop. Similar to cookies being erased on browsers, this resegmentation may lead to inaccurate data reporting, dependent on the test design.

Screen Shot 2020 08 04 at 10.26.06 AM

You can see from this graph, it appears that iPad traffic is decreasing, but that’s not really what’s happening. The change in user agent caused everything that relies on it, such as Analytics segmentation and Optimization targeting, to no longer be able to detect the platform properly. Without the additional context, it’s easy to assume that you’re losing iPad traffic, since that is what the charts reflect.

Steps analysts can take to brace for browser change impact

1. Evaluate your organization’s testing platform and implementation
First, you’ll need to gain an understanding of the testing platform your organization uses for testing. How compatible is your testing platform? Is it currently being impacted, and, if so, is there an easy solution without switching vendors? Tools may be deployed in several ways, so it’s important to understand how these tools are currently being deployed within your organization and if it impacts your data collection. If the testing platform your organization currently leverages for tests is on client-side, there is a greater chance that your data will be impacted by browser changes, because client-side utilizes Javascript to track cookie state.

2. Work with your vendors
Next, Cory suggests working with your existing vendors for guidance. Review your current setup to ensure it’s working as well as it could be. It’s important to stay up to date with the testing tool’s instructions to help you to navigate all the changes occurring on each browser type. For organizations utilizing client-side testing, some vendors such as VWO, Adobe, and Optimizely are quick to update their documentation to enable client-side testing to continue to work.

3. Meet periodically with your developers and IT team
Meet regularly with your developers and IT team to resolve issues with data collection. Your IT department could be able to find a work-around to deploy the tool in a way that is not impacted by browser changes. To obtain buy-in from IT, highlight the business impact that is resulting from the way your tool is currently being deployed. It’s important to understand and vocalize to stakeholders that business requirements have completely changed over the past year because the scenarios are different. This should help you obtain buy-in to get the tools deployed in the correct way.

For tools that have server-side capabilities but that are currently being deployed on client-side, Cory advises that organizations work through the process changes required to switch to the alternative method for data collection. Shifting to the more native server-side tool is recommended because of the way it delivers the content within the browser. This will help prevent data loss and ensure accurate data collection.

What if your organization does not currently have the budget to pay for a new testing tool and the resources required for tool implementation? Cory suggests to first get an understanding of what testing platform you utilize and how compatible it is with browser changes. If you find there is not a straightforward or easy resolution, then consider switching vendors.

4. Identify individuals within your organization who understand the technical side and analytics implications
To avoid jumping into a rabbit hole of data quality investigations, read Cory’s blog and follow him on LinkedIn. it is (and will continue to become) increasingly important to find individuals within your organization who can read release notes and understand how this impacts both the technical side and the impact to the analytics. This will help comply with browser privacy changes and with legal requirements like CCPA and GDPR.

Possible future

Cory’s prediction is that we will continue to see more regulatory compliance. Browsers will be required to comply with a patchwork of laws at the federal level, consequently requiring organizations to invest in processes to regularly validate data quality. From the perspective of your organization, continuous investment in this space not only retains accuracy but also allows you to keep the data you already have, instead of losing it entirely.

Google and Safari have issued proposals to change the way ad attribution works. These proposed changes could affect how ad attribution is measured, including the amount of time to receive information, making data collection and accuracy more difficult. Additionally, the amount of data communicated back to the system can change. According to Cory, “We’re likely to see some sizeable change in how ad attribution is measured, and so [for example] if you’re trying to figure out A/B testing ad creative or the effect on the website based off interactions with ad, then this will be more difficult to do moving forward if this occurs.” This also impacts upper-funnel test analysis.

Safari proposes to delete all client-side storage when the user is not logged in ( but keep in mind this is a proposal and subject to change.) Because testing requires some sort of state on the client-side browser to detect the variation a user received, then it would become impossible to ensure a user receives the same experience when A/B testing. If this scenario were to occur, then you would need to have the user self-identify by signing in and also figure out what variation they had on the server and recreate this experience on the client-side.

We will see a continued loss of data. According to Cory, “Data may become off-limits.” Browsers are putting into place different things that could be used as a tracking alternative to cookies. Referral data is one way that data will be lost. Instead of knowing the domain name and path for a specific page, browsers are putting into changes how that works. You will only be able to see the domain name but will not be able to know where they came from on the site.

While client-side workaround might be sufficient for upcoming changes, Cory predicts that there will be a rise in use for server-side platforms, including Conductrics and Sitespect. Consequently, A/B testing and tagging will move off of the client-side. This is good for performance reasons and eliminates flicker occurrences when testing. According to Cory, “Assuming that you comply with the legal landscape then I think this will be the best long-term approach.”

Screen Shot 2020 08 04 at 3.45.28 PM

Struggling to understand how recent changes are affecting your experimentation?

Our team can set you up for success.

Leave a Comment

Your email address will not be published. Required fields are marked *

Contact Us

Related Posts

Join the Conversation

Check out Kelly Wortham’s Optimization based YouTube channel: Test & Learn Community.

Search Discovery
Education Community

Join Search Discovery’s new education community and keep up with the latest tools, technologies, and trends in analytics.

Follow Us


Share on facebook
Share on twitter
Share on linkedin
Scroll to Top