When I first started A/B testing 15 years ago, our focus was purely conversion rate optimization with no concern about scalable learning and little to no understanding of how to use data to provide supporting evidence for test ideation and prioritization. It was very common for me to schedule a brainstorming session with a few key stakeholders where we would literally sit in a room and throw out ideas we could test. We would debate the ideas with a focus on feasibility and I would leave the meeting with a list of tests to go run. Unfortunately, we didn’t get great or even ok results most of the time. Of course, there would always be that one big win that would keep us coming back for more, but we never seemed to have wins large enough to outweigh the negative impact of all those losses.
It took a few years and a job at a new data-driven company before I started applying data to my brainstorming process. Even so, it was still the same basic principle: get everyone in a room and throw out test ideas. The only difference was that, this time, we used data to identify problems and the ideas we brainstormed were about solutions. Now we were using data to make sure we were solving real problems. Unsurprisingly, we saw much better results and could drive real improvement. And because we were using data to identify customer pain points, we were also addressing problems our customers were having. Much better!
But something was still missing. Using analysis to identify problems is great, but it will never make you better. It will only help you fix what’s broken. What happens when you’ve run out of things that need fixing? I know, I know … never should happen, but at some point, the juice just might not be worth the squeeze. And if you use data to identify something broken, why test it when you could just fix it? I was thinking these Jack Handey-esque deep thoughts about five years ago when it hit me: what if we flipped things around and used brainstorming to identify problems and data to suggest solutions?
In this new world scenario I had created, it would be easy for us to utilize existing use cases or create new ones, and then run through the site experience from the perspective of those who would fit in those use cases. We could then identify areas of opportunity to make the site experience better for the customer (and more lucrative for the business). Once we identified areas to evaluate, we could dive through the available data (CRM, web behavioral, VoC, etc.) to develop potential solutions to test. We were shifting our focus from fixing things that were broken to truly optimizing the customer relationship with the brand.
It took just a few trials through this new process to understand its potential and realize its benefit. But we’re in the business of continuous optimization, right? So we can’t stop there! On a whim, I decided to see what would happen if I combined elements of all of these methodologies and I struck gold.
Let me break down what this looks like in practice. On a regular cadence, we meet for optimization brainstorming. Analysts are asked to prepare data identifying potential friction points, business owners are asked to do a site run through using various use cases to identify opportunities or issues. All come together to present their identified problems and/or proposed solutions. Analysts take all ideas and/or problems currently without data support as takeaways to conduct the requisite analysis to back up the solution and/or propose ideas for identified opportunities or problems.
At the next regularly scheduled brainstorming session, the analysts come with what they learned about the identified ideas brought up in the prior meeting and business owners come with new areas to assess . . . and a cycle is born! Now, we get all the best ideas from all possible areas and we use data both to identify problems and opportunities! Plus, there’s the added benefit of teaching the entire organization to be data-driven in their decision-making. And the outcome? The most successful optimization program I’ve ever been a part of to this day.
Want help with your optimization ideation process?
Search Discovery can help develop a new process or improve an existing process from idea intake to prioritization. Hit us up!
