What is A/B testing in the Personalization Renaissance?

What is a/b testing? AB testing for websites can improve marketing results, define and solve problems, and benefit a company’s bottom line.

In the age of niche internet micro-influencers, curated subscription boxes, and all things do-it-yourself, it would be far from heresy to peg the 2020s as the Personalization Renaissance.

With only a simple questionnaire, Curology will send me a skincare routine tailored to correct all of my dermatological ailments. On Letterboxd, I can chat with the world’s most persnickety film lovers about whether film noir peaked with Hitchcock’s Shadow of a Doubt. An iPhone automation notifies members of my work carpool of my arrival without me lifting a finger. The world is each of our oysters, and by God, aren’t we just the pearls?

Earth’s increasing malleability to each of our needs seemingly presents an impasse for members of the Optimization community. In a world we’ve turned to silly putty in our palms, how can the tests we run to improve our outcomes help define and solve our often nebulous problems? A/B testing offers an enticing solution.

What is A/B testing?

A/B Testing isn’t novel. Testers split a sample size— the control group receives the standard-issue commodity, and the treatment group receives the updated treatment. Rudimentary as testing an existing procedure against a “2.0” may seem, A/B testing’s near-universal applicability and its position within the larger optimization ecosystem makes it instrumental in operational progress.

What are the benefits of A/B testing?

A/B testing leverages a site’s existing user base, makes low-risk adjustments, and produces actionable insights regardless of a treatment’s success.

In practice, this method saves users time. The “set it and forget it” nature of A/B testing means that implementation scientists need not forage for the perfect audience for their tests. Testers choose a control page against which they will test a new variant, and they divide their users into receiving either that control or any variant(s). There is no need to hunt for that which exists.

It also helps prioritize goals. Website optimization doesn’t need to be sweeping. By looking to multiple forms of ideation for evidence, test planners can ascertain what really needs fixing to drive their respective KPIs. Why waste time and money on a full-page redesign when adding a dropdown link would substantially increase clicks just as well? A/B testing saves resources–which then, in turn, can further the implementation of more A/B tests.

A/B tests are categorically actionable. No matter the test results, there is never a bad outcome for an A/B test. If a variant or challenger succeeds, the website developers queue it for immediate implementation. If the control outperforms the variant, the test has served the purpose of “insurance” by protecting the company from unknowingly publishing lower-performing content, but it also provides the optimization specialist the opportunity to further iterate in their hypothesis research for the next test idea.

What are the types of A/B testing?

Since A/B testing is so modular, it manifests itself in multiple forms. In the case of A/B testing for digital optimization, we’ll look at split testing, multivariate testing, and multipage testing.

Split testing might be what a psychologist calls the “prototype” of A/B testing. It’s what most people have in mind when they hear of an A/B test. Some users will receive the control version of the site and others will receive the site with the test.

Multivariate testing (or MVT) is a little more complex in its execution, but its guiding principles remain the same. Also called Factorial Testing, multivariate testing allows testers to look at the results of multiple variants at the same time. There are more numbers for analysts to track, but when the chips are down, implementation scientists will determine if a challenger outperformed a control—just like in a split test.

Multipage testing is when experimenters want to measure the impact of changing a site-wide element like the navigation or a button that appears across a site. The analytics tracking on a site has to be seamless for this test to be effective, but when it works, the outcomes this test produces have unparalleled actionability.

A/B testing in the learning flywheel

A/B testing is a cornerstone within the continuous learning flywheel. Evidence leads to ideation. Ideation produces ideas for experimentation—specifically, A/B tests.

The results from those tests, in turn, produce a set of recommended actions that correct a problem. Actions yield more evidence. Each step of the continuous learning flywheel informs the one before it, and the process itself never exists at equilibrium.Within an optimization program, AB testing is a cornerstone of the continuous learning flywheel, whether you're doing ab testing for websites or ab testing for marketing.

Ideation to experimentation

Data-fortified evidence can present itself in many different forms. As such, different forms of ideation can leverage these varying data types. From leveraging the voice of the customer to measuring the website or application against industry titans to turning to maxims and guiding principles, ideation transforms data into relevant questions that A/B testing answers.

Experiment design to results

Though ideation methods differ across data types, the result of these sessions is a step toward running an A/B test. In the case of optimizing digital services (e.g. websites, mobile applications, and emails), A/B testing, once initiated, is largely a turn-key operation. Since users randomly receive assignments to either control or treatment groups, analysts need not manually sift through the sample and number of users.

A/B test design mitigates an aforementioned woe of testing new ideas in the age of plasticity and personalization. Math is math. Math applied to silly putty is still math.

When it seems difficult to clearly delineate the population into two groups (or more, in the case of A/B/n or MVT), testing can both capitalize on the opportunities our interconnected world offers us and simulate the results of A/B tests to ascertain the existence and the size of an effect the treatment process caused.

Results to recommendations and action

Testing the performance of a new process against the status quo is a breeze. The difficulty arises in establishing procedures that manage to advance learning. A/B testing does just that.

The culmination of the modular ideation process, A/B testing is the clutch that maneuvers through the gears of the continuous learning flywheel—always prompting further action, yet generating opportunities to make progress beyond one learning cycle.

Since A/B test results can inform company practice regardless of their outcomes, companies employing ideation-driven A/B testing never become complacent. They strive to optimize their services for their customers’ benefit.

Turning to A/B testing is just good business. When a test fails, a company doesn’t rush to throw money at leveraging the negative outcomes, and when a test produces positive outcomes and remains in the learning flywheel, a company can know that the test’s outcomes are actionable and valuable.

What can A/B tests test? (A/B testing examples)

A/B testing enables test designers to propose changes both big and small. A challenger (the hypothetical change or addition) can be as comprehensive as a full-checkout funnel redesign or as slight as tweaking a single title.

A/B testing and the decision matrix

Still, A/B testing is just one element within the continuous learning flywheel. A decision matrix can provide context and understanding for a team advancing a problem through the learning flywheel, independent of an A/B test’s results.

We recommend that a decision matrix accompany each A/B test plan to explain the test’s potential outcomes and what those outcomes mean for the organization. Let’s pretend Company XYZ wanted to test the impact that placing a chatbot on their website could have on their overall lead generation. Their decision matrix could look like this:

A decision matrix helps answer the question, what is a/b testing. AB testing accompanied by a decision matrix can help improve a website regardless of test results.

If the chatbot variant yielded more lead generation in the runtime of the A/B test, Company XYZ would likely keep the chatbot as a more permanent fixture of the site. They could then draw the conclusion that their users previously did not have a clear path to get in contact with a Company XYZ representative, and the firm could consider other potential barriers their users faced in their quest to patronize their company.

However, if the version of the site without a chatbot yielded more lead generation, the test is not all for naught. Maybe Company XYZ’s users would rather do business with a ‘hands-off’ firm rather than make direct contact with a representative. Company XYZ could, in turn, flesh out their FAQ page with any issues their users could face with instructions on how to cure those ailments—all without contacting a soul.

Chatbots succeed and fail, but with A/B testing at the helm of Company XYZ’s experimentation journey, they’ll be able to improve their site regardless of their test results.

A/B testing as common ground

Once, when hiking, I came upon a small metal sign spiked into an oak: Bearing Tree. A physical marker of a spot on a map to orient folks in the actual woods.

What is a/b testing? AB testing is an analyst’s bearing marker in the uncertain terrain of our world’s operational landscape. It names where we are and gives our progress the common ground (the bearings) necessary to communicate productively.

A/B testing is an analyst’s bearing marker in the uncertain terrain of our world’s operational landscape. It names where we are and gives our progress the common ground (the bearings) necessary to communicate productively.

As our proclivities tend toward personalization, the semblance of standardization A/B testing provides is welcome and necessary. In A/B tests, testers randomly clump participants together—regardless of their TikTok algorithm, subscription boxes, and film noir opinions. Folks get what they get. And because they do, brands can, certainly, offer something measurably better. 

Search Discovery can help you advance your A/B testing program, whether you’re just getting started or looking for advanced solutions. Reach out today to talk with our optimization experts.

Contact Us

Related Posts

Join the Conversation

Check out Kelly Wortham’s Optimization based YouTube channel: Test & Learn Community.

Search Discovery
Education Community

Join Search Discovery’s new education community and keep up with the latest tools, technologies, and trends in analytics.




Scroll to Top


Catch the latest industry trends we’re watching and get new insights from our thought leaders delivered directly to your inbox each month.