A/B testing is a way to evaluate our ideas for improving customer experience before we invest heavily in a fully-fledged implementation. But where do test ideas come from?
At Search Discovery, we like to focus on six main ideation methods. These are tried and true methods to build hypothesis backlogs supported by evidence—qualitative, quantitative, or both! We use each of these methods with clients, depending on their business needs. This post details ways to identify which ideation method is best for you based on your resources, types of evidence, and business goals.
Comparative Analysis
In a comparative analysis, analysts use related sites to understand what practices customers are used to seeing and using to leverage industry best practices for their site’s optimization.
Timeline: This method has a variable timeline. The more sites you compare, the more information you’ll get, but the longer it will take. Comparative analyses could take up to two or more weeks (or less than one), depending on the number of sites you’ve identified for comparison.
What resources will you need: You don’t necessarily need an analytics platform—you only need access to competitors or best-in-industry public-facing sites. Ideally, this method requires more than one person to review each competitor’s site to ensure the reliability of any conclusion. Record findings in a central location like our SDI comparative analysis insight template.
What kind of data should you use, and where should you source it: The comparative analysis looks at site design, content, navigation ability, functionality, etc. for companies that compete with or complement your site. The objective is to document these other companies’ use of UX themes related to your business goals. Services like Owler or Comparably are an excellent way to identify competitors.
Ex. Financial Institution
What are the potential risks to outcome validity: This method naturally assumes more risk than others because it bases your insights on competitors’ practices, and you can’t be sure of their performance. It’s best to have multiple analysts participate so that you can have the most diverse range of perspectives to evaluate external companies’ digital strategies.
Which digital environments are you looking to test in: The ideas derived from this method can be applied anywhere on a site, mobile or otherwise. However, you can only view competitor site pages that you can access. You might have difficulty if the page you’re trying to assess is behind a login screen.
Heuristic Analysis
During a heuristic analysis, analysts use well-known heuristic frameworks to assess the effectiveness or quality of certain elements of their site pages.
Timeline: A heuristic analysis takes about 3-4 weeks from conception to final presentation. It requires a planning stage where you must select the relevant frameworks to assess the sites, which can take some time.
What resources will you need: A heuristic analysis works with qualitative data, so you aren’t required to use analytics reporting tools. However, a heuristic analysis does need several analysts to review the site or digital environment against the selected frameworks to ensure insight reliability.
What kind of data should you use, and where should you source it: You can focus on any of your site pages for a heuristic analysis, but you want to ensure that the heuristic frameworks you’ve selected are best practices. You can use any number of heuristic frameworks based on the research objective. Search Discovery offers a great heuristic library resource to help you get started, conveniently available here.
Ex. Heuristic Frameworks
Additionally, you will want to collect recommendations that align with your business goals, so consider assessing the site traffic and conversion rates before beginning the analysis to ensure you’re starting in the most effective direction.
What are the potential risks to outcome validity: Heuristic analyses tend to be speculative if you don’t check to make sure users also encounter the problems identified by analysts. Because of this, with heuristic analysis, you run the risk of overanalyzing and then potentially over-testing.
Which digital environments are you looking to test in: Because the heuristic analysis looks at your own site using a best-practice-lens, you can test its results in any environment you can access.
Voice of Customer Analysis
Voice of Customer analysis (VoC) requires analysts to look at a combination of quantitative and qualitative data from their users. This information is typically sourced through ratings, emails, reviews, and more. While it can be used to identify usability issues, VoC is best used to expound on the severity of those issues and how they affect the user’s journey.
Timeline: The timeline for this analysis varies by the type of data you’re collecting, whether it’s already collected, and what usability elements you’re exploring. Depending on what information you require, it can take anywhere from a couple of days to a few weeks to complete this process.
What resources will you need: To evaluate the customer feedback data, you’d need access to your company’s feedback platforms. This could be in-house, like form completes, or through a third-party system, like SurveyMonkey or Qualtrics. Or through your experimentation platform if you use our partner Conductrics, which has added an industry-first survey capability with their already industry-leading experimentation and prediction engine.
What kind of data should you use, and where should you source it: For a VoC, it is best to wait until you have multiple forms of user responses like surveys, interviews, and customer service call logs. This response data can be either solicited—a customer was prompted to respond—or unsolicited—a customer sought to give feedback to the company. 
What are the potential risks to outcome validity: Unsolicited feedback can come with many ideas or suggestions that aren’t particularly relevant to the business question you’re attempting to assess and can obfuscate the purpose of the ideation session. Additionally, there can be costs associated with this type of data collection, like mailing and survey software fees, which you may not recoup if the test isn’t fruitful. Which digital environments are you looking to test in: For VoC analysis, you can attempt to use a pre-defined testing environment. But, given that your data is based on customer feedback (both solicited and not), the testing environment will likely be determined by the customer’s environment when they encountered an issue and submitted feedback.
Usability Testing
Usability, or “user testing,” involves measuring user actions to determine user pathing, conversion timelines, task error rates and more. This requires that you first select customers via a screening process, then ask them to complete specific actions on your site. Users’ actions are recorded and tracked through a usability research-related technology and then analyzed for trends.
Timeline: You can implement usability testing quickly since it only requires a walkthrough of a user’s path on a page. But, you’ll want to take time in the planning phase to accurately identify your business goals, user panel criteria, usage script, page elements for analysis, and a pilot test to ensure everything is optimally set up. This planning process alone can take up to a week.
What resources will you need: You need a user testing platform like UserTesting, Userlytics, or WEVO if you want to conduct a usability test. You may need to gather a user panel that will be included in the experience, or you can rely on the vendor’s panel, depending on your desired audience. This process can be expensive, so we recommend conducting user testing sparingly.
What kind of data should you use, and where should you source it: Usability testing returns qualitative data that can explain why users are having issues with your page elements and/or task completions. It can be helpful to identify areas of concern before user testing so that you have a scope for your project, and you should select your users based on the scope of the test. It’s wise to pre-screen them to gather additional data on your sample.
What are the potential risks to outcome validity: While this is an excellent method for understanding the reasoning behind usability issues on your page, results likely won’t be statistically significant, especially at scale, because of the bespoke nature of this method.The exception to this can be with the use of WEVO due to their statistically significant sample sizes, which start at a minimum of 120 users for businesses trying to assess usability at scale.
Which digital environments are you looking to test in: You’ll need to decide on the user testing environment before ideation begins. These environments can be public-facing, mobile, or even behind a login, but users should be able to access each environment through a regular browsing session via the platform.
Ideation Workshops
This method aims to engage teams in a collaborative setting to create a diverse ideation pool. Ideation workshops, which require a group to work together, are generally used to promote empathy or customer-centricity.
Timeline: Ideation workshops are typically several-hour intensive sessions, preceded by a day or two of planning.
What resources will you need: You don’t explicitly need any resources, but it is beneficial to come to the session with an idea or business objective. Know what kinds of goals and problems you’re attempting to address. In order to encourage diversity of ideas, it is best to have a diverse team present for the session.
What kind of data should you use, and where should you source it: Likely, you’ll source your problems from a primary analysis. That could include VoC analysis, heuristics, comparative analysis, or user testing. Your sources should showcase challenges faced by different groups of customers.
What are the potential risks to outcome validity: In an interactive workshop, analysts might give more weight to ideas from, say, executives or extroverts. That’s why it’s important to maintain a diverse analyst group for this method and to employ strategies that complement the group’s creativity.
Which digital environments are you looking to test in: You can apply ideation workshop outcomes in several environments, including public-facing, in an app, or on multiple pages. It’s more challenging to apply ideation workshop outcomes to secure pages, since not everyone in the session may have access.
Digital Behavior Analysis
Digital behavioral analysis is a hypothesis-driven analysis that requires analysts to research onsite behavior of customers.
Timeline: This is a quicker method as the data is already stored in your analytics system, like Adobe Analytics or GA4. You just need to create a measurement plan to determine how you will use that data to derive insights.
What resources will you need: You’ll need access to the site data for whatever page you’re assessing, which requires a pre-established tagging system and analytics platform. This method is executable by an individual, but ideally, you should collaborate on the planning phase to better understand the saliency of the issues and align on business goals to be addressed by this exercise.
What kind of data should you use, and where should you source it: Digital Behavior Analysis looks at audience information and behavioral traits for users and is generally collected through a combination of tagging and analytics platforms.
What are the potential risks to outcome validity: The validity of ideas that stem from this analysis is based on the analyst’s knowledge and setup of the test. You can adjust the level of risk you’re willing to accept by carefully planning what variables to measure and being transparent about how they might interact with your desired business goal.
Which digital environments are you looking to test in: For Digital Behavior Analysis, you can run ideation and testing methods in any environment you have permissions and analytics access to; in-app, public-facing, or behind a login screen.
For a more detailed description of these ideation methods and how you can leverage them for business improvements, request a copy of our guide, Structured Ideation Methods to Drive High-Value Test Ideas.
