Building an experimentation roadmap at scale

A fundamental component of any great product development process which leads to successful innovations is experimentation. Although more than 80% of controlled experiments fail to achieve their desired outcomes, a healthy culture of experimentation is how companies discover and deliver value for customers and grow their businesses.

Traditional experiment approaches go like this: you come up with a hypothesis, you test it, you launch the winner or rollback the loser. While this practical process can lead to successful outcomes, it considers each individual experiment on its own, overlooking how every experiment can produce valuable findings that can be applied elsewhere.

A more thorough approach involves thinking about each experiment as having multiple stages in a lifecycle, that, when combined with other experiments, can create a flywheel effect. Each experiment and finding can propel your business forward. The experimentation lifecycle spans across four stages: Discover, Hypothesise, Execute, and Learn.

Discovery is something every product professional should be familiar with. In the context of experimentation, discovery is about building an understanding of the problem you are trying to solve. Before jumping to metrics, you need a crisp problem statement. You should understand what happens if you were to do nothing. Outline what you know about your customers and their behaviour, as well as what you don’t know. Know what you have control over. And finally, be clear about the outcome you seek to deliver—a success metric that you will measure each experiment against.

In all organisations, this discovery process should be inclusive of anyone who has a stake in the outcome you are trying to deliver. While at Amazon, I was charged with coming up with an experiment roadmap to help address a sales target we were on track to miss. The discovery work I led involved ideation with designers, engineers, marketing managers, as well as Directors and Vice Presidents.

Throughout discovery, you will have gathered enough information that will naturally lead to ideas and hypotheses.

Equipped with insights from your discovery efforts, formulate your hypotheses to test. Good ideas can come from anywhere. The job of the product manager is to gather those ideas, from a perspective on how those ideas can solve the problem, and then leverage various functions to deliver the outcome defined in the discover stage.

When working in larger teams with an extensive list of experiment ideas, grouping ideas into themes is a step you can take to organise the start of your experimentation roadmap. Consider what’s similar about your different hypotheses and bucket them together. In the experience I had at Amazon, we had over 40 different hypotheses we planned to test—grouping them into themes helped organise ideas and determine which themes were worth exploring further.

Next, estimate the impact you expect each experiment will have on your success metric. It’s less important to be precise with this estimation so you’re not wasting cycles doing a deep analysis, forecasting results that you won’t have high confidence in (if you did have high confidence, you should consider whether to just push such a change forward without experimenting).

Equally as important to estimating impact is understanding the level of effort each experiment will take to execute. There will be manual effort, such as engineering or design work, and knowing how much will serve you when building your experimentation roadmap.

With groups of hypotheses to test, a rough idea of their expected impact, and a high-fidelity level of effort estimate you can put together an experiment roadmap and begin work on the earliest of ideas.

When working with a large number of hypotheses to test, you can map your ideas across the dimensions of expected impact and level of effort.

You should seek to prioritise ideas that you expect to have a high level of impact but require a lower amount of effort to execute. These are hard to come by, and it’s most likely you’ll need to make instinctual decisions for experiments that will have a medium impact and skew on the lower level of effort.

As experiments conclude you will be able to assess the results of the experiment and then share those findings with the wider group invested in the outcomes. Outside of whether the experiment was a success or failure, it’s important to capture additional findings along the experiment journey. 

At this point, you should also revisit your plan—does it still make sense based on what you’ve learned so far? An experiment roadmap should change just like any other product roadmap as you learn.

At larger organisations, sharing outcomes with other teams who may benefit from your findings is another way to demonstrate value as a product manager, as well as ownership in helping the broader company drive better results. As my large experimentation roadmap continued at Amazon, I would share findings with other teams who would then adapt their experimentation plans based on our findings.

Here are some practical suggestions to help improve the experiment culture in your organisation:

Embracing experimentation is representative of an organisation that understands the importance of a healthy product operating model. Running experiments at scale relies mostly on thinking about each experiment as playing a bigger role in your journey to drive better value for your customers and the business. Embracing the discovery process, applying learnings from each experiment to the next iteration of your experiment roadmap, as well as sharing your findings with stakeholders and other teams will improve the culture of experimentation at your company.

Explore more great product management content by exploring our Content A-Z