How to A/B Test Statistical Intuition via Simulation. Bonus! Free A/B Test Simulation Tool Included!

Developing a solid intuition about the statistical concepts that drive the interpretation of a basic A/B test is a powerful thing. Explore an A/B test simulator to lock in that intuition!

One of the challenges of successfully running an experimentation program is that experimentation relies on statistics, and statistics can be confusing! Marketers are motivated to make data-informed decisions, and the idea of an A/B test—split the universe in two and deliver experience A to half of the universe and experience B to the other half and see which experience performs better—is an excellent means of doing just that!



Except…we can’t actually split the universe. We split “the people that came to our site (or use our mobile app, or to whom we sent an email)” into two (or more) groups. Then, we use the data we collect from that splitting to make inferences (#fancystatisticsword) about the performance and impact of the different experiences we tested.

Some of the misperceptions that we run into with our clients (and even with some of our industry colleagues!) include:

  • “The confidence level is how sure we are that the results we observed are accurate.” Well…not really.
  • “The run-time for the test is designed to collect enough data to quantify how much of a “lift” one variation delivers over another.” Mmmm… rarely!
  • “A well-designed test will definitively show which variation performs better.” Not…quite.
  • “With a well-designed test, it’s both straightforward and safe to estimate an annualized revenue impact.” Oooh… danger! Sharpen up your favorite Caveat Pencil if you’re going to do that!

At the root of many of these misperceptions is a misunderstanding of the fundamental role of statistics. At its core, statistics is (emphasis added):

"the practice or science of collecting and analyzing numerical data in large quantities, especially for the purpose of inferring proportions in a whole from those in a representative sample." [Source: Oxford Languages]

It turns out that the “whole” is a bit of a sticky wicket. What is “the population” when we’re conducting an A/B test? Arguably, it is “all users from now to some point in the medium-to-distant future.” The “true” conversion rate of a somewhat-vaguely-defined population gets tricky!

In a Simulated World, We Can Know the Truth

One way to develop some intuition about statistics is to explore them in a world where the population is known, even if that world is one entirely of our own making!

That was the inspiration behind Search Discovery’s  A/B test simulator:

A/B Test Simulation Tool

The simulator is purely an educational tool. We use it as we’re establishing optimization programs with our clients, but our hope is that analysts can use it to improve their own intuition and, perhaps, to improve the statistical intuition of their own stakeholders.

The tool has three educational goals:

  • Help the user develop an understanding of distributions of sample means.
  • Help the user develop intuition around statistical significance when it comes to detecting an effect.
  • Help the user develop intuition around the challenges of estimating the size of an effect.

The simulator has documentation and explanations built into it, but we also have put together the following video that explains and demos the tool:

What do you think?

We’re always looking for new and useful ways to help explain the statistics of optimization, so we would love to hear what you think of this simulation tool on that front.

Or, of course, if you are looking for support in building your organization's optimization program, including educating stakeholders effectively, we're here to help!

Related Posts

Join the Conversation

Check out Kelly Wortham’s Optimization based YouTube channel: Test & Learn Community.

Search Discovery
Education Community

Join Search Discovery’s new education community and keep up with the latest tools, technologies, and trends in analytics.




Scroll to Top


Catch the latest industry trends we’re watching and get new insights from our thought leaders delivered directly to your inbox each month.