A/B Testing During & After COVID-19

On April 24, 2020, the Test and Learn Community met to discuss testing in the time of COVID with expert panelists Ton Wesseling (Founder of Conversion Hotel, Owner ABTtestguide.com, CXL Teacher-A/B testing master course), Cassandra Campbell (Testing + Experimentation Lead, Shopify), and Ben Labay (Research Director, CXL). Since this conversation has been one of our most-searched videos, we decided to bottom-line it here to provide the testing community with fast/historical access to this conversation on the edge of tremendous industry change.

Learn more about and join the Test & Learn Community!

Why we should be testing right now

There have been large rapid shifts in customer behavior, leaving businesses to figure out how to pivot and adapt as quickly as possible. That is why now, more than ever, we need to capture data, learn from it, and make a lot of informed decisions. Testing can help with all three, and it is the best way to help us understand what the new normal means for our business and our strategies.

Businesses have either needed to move rapidly to digital or improve their current online business. Testing is a fast and direct path to insights that can inform these quick moves.

Shifts in types of testing

Imagine every day is Black Friday-type traffic but your customers are almost unrecognizable! How can you get to know them and serve them fast? The first step is to get data however you can!

Companies are racing to understand and adapt to the new COVID landscape. Many organizations have found that standard ways of testing or collecting data has too long of a feedback loop to be useful in making quick decisions. They have had to look to other ways to get information.

Three useful tips in getting data:

  • Stay close to the customers with polls, surveys, interviews, chat logs, etc.
  • Focus on how to serve people new to online rather than incremental channel impact.
  • Mine archives of learning, read, go dig in the data that already exists. Utilize this time to learn more about your customers and business.

Effect of physical limitations

Thanks to COVID, there has been a dramatic shift away from physical business to the web. While many companies have been waiting for the opportunity to shift/grow their business in the digital space, they never expected it would happen this fast or be paired with a drastic drop in their brick-and-mortar sales. So, many companies who had not previously competed in the digital space are now having to learn to do so as their opportunity balloons.

The goal of testing, going forward, should be to help clients retain that huge growth in online sales, even as the face-to-face part of their business starts to come back.

Changes across industries

These, obviously, are numerous and widespread. Here are a few the panelists have seen:

  • Since the supply chain has been greatly affected, delivery times are much longer. However, e-commerce rates are not down and people have an understanding for the lack of speed-convenience they were used to. There has also been a behavioral shift in customers caring about supporting local businesses. How long this trend will last remains to be seen.
  • There is a shift to focusing on the usability of a site rather than on the long term motivation of users. In order to accommodate quick (myriad) changes, companies have shifted from focusing on long-term value add to mitigating short-term loss. For example, Shopify offered an extended 90-day free trial simply because their focus is on “how can we help the customer” instead of “how long will this change bring us increased revenue in the future.”
  • Customers are moving away from interacting with a site simply for “I want to do” and seem to have the intent of “I want to know.” Data shows an increase in conversational engagement. Now is the time to produce deeper, higher-quality content, because this lower-funnel information is not only creating the same (self-selected) quality of leads, but also, increasingly, more leads, too.
  • Some companies are shifting strategies based on need for customer retention vs. acquisition.
  • New-to-digital companies, that are being forced to pivot with new efforts/products, seem to be testing more than some digital native companies, who have paused.

The Ethical Dilemma

Optimization experts truly want to help. We don’t just want to help our clients succeed, we also want to help the people who rely on our clients. But there are limitations to how much we can help, and important considerations about how much we should help.

While optimization experts are not the people able to help businesses create new apps or build workarounds to the lack of physical interaction, we can help companies understand how to persuade users to interact with them during periods of high uncertainty and change. We’re also able to help local businesses serve customers in a more timely manner than the big names, who face supply chain limitations and extended delivery times. For example, we’re able to leverage the message, “We’re from here! Support local.”

These scenarios (which sometimes oppose one another) necessitate the question of how ethical it is, during a crisis, to use the power of suggestion and messaging for companies’ gains. It’s worth taking time to understand a company’s goals and craft a nuanced, thoughtful message: Don’t craft a message playing on customer’s fears (i.e. “buy it now before it’s gone!”), and instead look to help a client helping themselves by also helping their customers. We hope to see this customer-focused strategy stick around in the new normal.

How to get buy-in for testing

Having a difficult time getting your clients or organization to buy in to the idea that now is the time to be testing? Or how this is the opportunity to really build out a new and improved internal process? Here are a couple of strategies to persuade executives/decision-makers:

  1. Ask them to LISTEN: Start with customer research. Get their stories and have executives listen to them. Let them hear customer pain points first-hand. Compile this information into a list and create test ideas that directly seek to help solve customers’ “true” problems. Learnings from those tests will directly impact the company’s ability to help customers now and moving forward.
  2. Explain how this is an opportunity to improve the entire testing process: Use a team’s open time now to create the most efficient process possible. The focus on structure will improve a company’s ability to get ideas into tests, which will increase the insights necessary to make decisions faster and more efficiently than before. This is a big opportunity for change, so invest!

Looking for more insights around A/B testing now and in the coming months?

Fill out this form and our experts will set you up for success.

Related Posts

Join the Conversation

Check out Kelly Wortham’s Optimization based YouTube channel: Test & Learn Community.

Search Discovery
Education Community

Join Search Discovery’s new education community and keep up with the latest tools, technologies, and trends in analytics.

Follow Us


Share on facebook
Share on twitter
Share on linkedin
Scroll to Top