post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Louron serves as the Editor at Mind the Product, bringing nearly a decade of experience in editorial positions across business and technology publications. For any editorial inquiries, you can connect with him on LinkedIn or Twitter.

Louron Pratt

In this spotlight session at #mtpcon London 2023, <a href="https://www.linkedin.com/in/dmcarlile/">David Carlile</a>, Sr Director, Product Strategy at Optimizely shared insights on how we can reduce risk and fail faster by building experimentation into the product development lifecycle. Watch the video in full, or read on for his key points.
<h2>Product development and experimentation</h2>
David walks us through the product development process to fail effectively and consequently build better features. Here are some examples of questions to ask at each stage:
<ul>
<li>Discover: Which features should be added to our roadmap and why?</li>
<li>Design: What should these features look like, and how should they be built?</li>
<li>Build: What is the most efficient and cost-effective way to build this feature?</li>
<li>Validate: How can we validate that this feature works?</li>
<li>Roll-out: How can we roll out this feature in the most efficient way, whilst reducing risk?</li>
<li>Iterate: How can we optimise this existing feature?</li>
</ul>
This is a very similar process to the experimentation side of product management, he explains:
<ul>
<li>Discover: Test to learn: Painted door tests to validate demand</li>
<li>Design: Test to decide: User studies and rapid prototyping to validate feature design</li>
<li>Build: Use feature flags to limit blast radius</li>
<li>Validate: Test to measure: roll out test to prove behavioural change and business impact</li>
<li>Roll-out: Staged rollouts to monitor the quality and performance of the experience</li>
<li>Iterate: Test to learn: Experiments on iterations of features to optimise user experience and business impact</li>
</ul>
<h2>Common challenges</h2>
When going through this process for both the product development process and experimentation, David shares a few common challenges we may face.
Ship-first mentality: This would mean not asking enough questions about whether we’re shipping the right thing. David explains that we can use testing to validate our assumptions before we invest in building products.
Productivity threat: Often, when we’re looking to build a rigid case for a product or feature, it can be seen as a blocker to productivity. He encourages us to push back on the productivity threat to ensure the validation of new products and features.
Org-wide buy-in: It can be difficult to influence the whole business to invest in this new process, especially if a business has a built-in product, feature, or way of working.
Validation only: David adds how this development cycle can be perceived as the final stage to simply validate the product or feature that we were already going to proceed with. It’s important to integrate this validation stage earlier into the product development lifecycle, he explains.
<h2>Build the right ‘thing’</h2>
To ensure that we’re building the right products, David breaks down three product test types. Also, he includes examples to showcase what successful companies have been doing to experiment and validate well, such as media platform, Medium.
<ul>
<li>Test to learn: Learn about user demand and interest in new features. Learn how users respond to your feature and why they respond the way they do.</li>
<li>Test to decide: Decide on feature design, build and roll-out strategy.</li>
<li>Test to measure: Validate the impact of the feature on user experience and/or key business metrics.</li>
</ul>
<img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2023/12/b589532b-f793-4d77-87fe-c6ab13d2880a_Screenshot202023-11-1420at2010.34.15.jpg" />
In terms of collaboration, David breaks down the key responsibilities of each team.
<ul>
<li>Product teams &#8211; Testing to learn: Exploratory testing, validates the demand.</li>
<li>Engineering &#8211; Testing to decide: Feature flagging, easy rollouts, decouples code release from deployment.</li>
<li>Marketing &#8211; Testing to measure: Message validation, personalisation, quantifying the impact.</li>
</ul>
This showcases how businesses can optimise experiences, and consistently test across various teams and channels from end-to-end.
<img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2023/12/7f2f2e09-75ac-4856-a6e4-da2b8227e8a3_Screenshot202023-11-1520at2009.50.22.jpg" />
<h2>Measuring continuous optimisation</h2>
David says that constant iteration and repetition are key. He lists some metrics we can consider to validate program performance:
<h3>Program metrics</h3>
<ul>
<li>Test velocity</li>
<li>Conclusive rate</li>
<li>Win rate / Lose rate</li>
<li>Roll-out and roll-back rate</li>
<li>Test set-up time</li>
<li>Tests per initiative</li>
<li>% product discovery tests</li>
<li>Test duration</li>
</ul>
Although these metrics are important, David mentions that value metrics are equally critical to provide an estimation of what effect a new product or feature is having.
<h3>Value metrics</h3>
<ul>
<li>Positive impact</li>
<li>Negative impact</li>
<li>Annual estimated impact</li>
<li>ROI</li>
</ul>
<h3>Key takeaways</h3>
Wrapping up his talk, David shared some key takeaways from using metrics to drive value and validate business ideas
<ul>
<li>Program metrics should evolve over time: they should reflect cultural challenges and evolve with your experimentation maturity</li>
<li>Program metrics can be harder to collect: Extra work may be required to capture and catalogue project data and operational metrics</li>
<li>Program metrics serve as a directional compass for culture change: Establish baselines, implement cultural practices, and give time to remeasure</li>
<li>Experimentation value includes achieved gains and avoided losses: Experimentation should lead to improvement and catch potential mistakes.</li>
</ul>
<hr />
Try Optimizely Feature Experimentation for free and start shipping confidently with
Rollouts &#8211; Optimizely’s free feature flagging and experimentation solution. <a href="https://www.optimizely.com/free-feature-flagging?utm_source=mtp&amp;utm_medium=referral&amp;utm_campaign=free-rollouts" target="_blank" rel="noopener">Create a free account here</a>

core/freeform

Reduce risk, fail faster: Experimenting across the product development lifecycle