In May of last year, the Test & Learn Community and Chad Sanderson, Head of Product at Convoy, walked through marketing optimization vs. product validation experimentation methods, including when, why, and how they should play nice and work together, and why doing either one without the other might be shortsighted. Before we dive into the differences between marketing optimization and product validation experiments, let’s discuss the roles that the two teams play in a business:
Marketing teams are focused on selling a product. They identify selling strategies for their target audiences and ensure that the value proposition aligns with customer demand. Common key metrics include conversions, revenue, and ROI.
Product teams are focused on developing the product that marketers then sell. They identify a way to build the product in a sustainable, replicable way, then continue to design new features to better address customer needs. Their primary key metrics are related to stability and satisfaction.
Learn more about and join the Test & Learn Community!
Types of Businesses and Products
The product is the most central aspect of a business. The closer your team is to the product, the more leverage you have in making business decisions; the more leverage you have, the more freedom you have to experiment. So if the product is most important, doesn’t that mean the product team is responsible for experimentation? The short answer is that it depends on what the product is. Chad defines three types of companies based on the role that technology plays in the product:
- Tech-First Companies: Technology is the product—it is what the customer is consuming. Examples of tech-first companies are social media platforms, email hubs, and search engines.
- Tech-Second Companies: Technology serves as a bridge between the online and offline product— there is a physical product the customer will consume, but these companies internally operate as if they are tech-first. Examples of tech-second companies are hotel booking websites, food delivery services, and ticket sale websites.
- Tech-Third Companies: The product is offline and physical. Until recently, technology tended to be an afterthought. The digital experience is just a portal to find where you can buy the product or order the product online. Examples of tech-third companies are restaurants and brick and mortar stores that have websites and apps.
In tech-first and tech-second companies, there are primarily product validation experiments run by the product team. In tech-third companies, it is more likely for marketing teams to run optimization experiments.
Marketing Optimization vs. Product Validation ExperimentationThe goal of marketing optimization is to sell more product, drive more revenue, and increase ROI. These experiments test marketing tactics and do not modify the actual product. A common example is A/B testing a hero image on an ad landing page or the copy of a call-to-action button. Product validation occurs as a part of the product development cycle. The ultimate goal is safety. Validation experimentation is less familiar amongst marketers who are used to optimizing ROI. With validation, the ROI is almost always negative when we calculate the cost of running the experiment. Common examples of product experimentation are testing out a new feature or re-organizing the information architecture to make the user flows throughout the product more intuitive. So why should we validate if it’s not making us money? Author Nassim Taleb says that it’s just as important to guard against rare but catastrophic events than to make incremental, positive improvements. Chad makes the comparison between validation and insurance:
Bringing Different Types of Experimentation Together
Regardless of what your product is, both validation and optimization are important and necessary. Chad highlights 5 steps to success in bringing teams closer together in experimentation:
- Understand the current structure of your organization. What is most valuable and which team holds the most leverage?
- Figure out where the coverage is lacking. Are you monitoring ROI of all experiments? Do you need to be?
- Bridge the gap! Start a conversation between teams to open perspectives on future experimentation.
- Establish a way to share results and work on joint projects across teams and products.
- Create a meeting cadence to review program metric goals and give insight to the overall state of experimentation.
Each type of experimentation has different levels of ROI, risk, and learnings. It is important for marketing and product teams to understand the value that the other brings to the company.