post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Emily brings nearly 15 years of product management experience to her role as Managing Director for Mind the Product. While she has worked across a variety of industries, she will always have a soft spot for the travel industry where she spent most of her early career building software for airlines and developing a leading consumer travel app, TripCase. Emily is passionate about the craft of product management and loves talking about new ways to make products people love.

Managing Director

Emily Tate

<p>Everyone wants to be in an evidence-driven team, using research and data to drive their product decisions. We all know it’s the best way to work, but what does it mean in practice? How can you tell if your team is truly evidence-driven? And if it’s not, how do you change things for the better?</p>
<p>Teams may think they’re evidence-driven because they track <a href="https://www.mindtheproduct.com/category/product-management-metrics-and-kpis/" target="_blank" rel="noopener noreferrer">metrics</a>, run experiments, or state goals in terms of outcomes rather than outputs. These are great practices, but it takes a more cohesive approach and culture to really unlock value. Knowing where your team starts from can help you to identify weaker areas, and to start the conversations that help to move your team towards being evidence-driven.</p>
<h2>The metrics maturity landscape</h2>
<p>After working with and talking to many product teams, I started to realise that the teams who used metrics most successfully to transform their business have two characteristics: they set <span style="font-weight: bold;">clear desired outcomes and goals</span> upfront, and they <span style="font-weight: bold;">follow up on the results</span> of their builds to see if they have worked and achieve the desired outcomes. This may seem like common sense, but in reality, most teams struggle in one or both of these practices.</p>
<figure id="attachment_23563" aria-describedby="caption-attachment-23563" style="width: 1024px" class="wp-caption aligncenter"><img loading="lazy" decoding="async" class="wp-image-23563 size-large" src="https://www.mindtheproduct.com/wp-content/uploads/2021/03/2-by-2-1024x664.png" alt="Grid representing the metrics landscape" width="1024" height="664" srcset="https://www.mindtheproduct.com/wp-content/uploads/2021/03/2-by-2-1024x664.png 1024w, https://www.mindtheproduct.com/wp-content/uploads/2021/03/2-by-2-300x195.png 300w, https://www.mindtheproduct.com/wp-content/uploads/2021/03/2-by-2-768x498.png 768w, https://www.mindtheproduct.com/wp-content/uploads/2021/03/2-by-2.png 1600w" sizes="(max-width: 1024px) 100vw, 1024px" /><figcaption id="caption-attachment-23563" class="wp-caption-text">Grid representing the metrics landscape (Image: Emily Tate)</figcaption></figure>
<p>When you look at how teams behave in these areas, you start to see patterns in the way they operate. Let’s take a look at four rough groupings, and see where their use (or lack) of goals and follow-up affects their success, and where they might start if they want to make a change for the better.</p>
<h3>Feature factories</h3>
<p>John Cutler coined the term <a href="https://cutle.fish/blog/12-signs-youre-working-in-a-feature-factory" target="_blank" rel="”noopener” noopener noreferrer">“feature factory”</a> to describe teams who prioritize building above all else. These teams may have some goals, but they are more likely to describe a feature to be built rather than a measure that will drive value. Their roadmaps are filled with lists of features to be completed, and everyone knows at the beginning of an effort what the end result will look like. There is little space for testing or iteration, and once a feature is delivered, it is considered “done” and the team moves on.</p>
<p><a href="https://www.mindtheproduct.com/bets-boards-missions-and-north-stars-by-john-cutler/" target="_blank" rel="noopener noreferrer">In this ProductTank Barcelona talk, John Cutler showcases some simple hacks he’s used as an advisor to help nudge organizations forward and get them thinking in new ways.</a></p>
<p>Because the goals are generally more delivery-focused (on time and on budget), there is little reason to go back and evaluate past projects to see how they performed. The measure of success is “Did we ship the features?” rather than “Did the features achieve the outcomes we hoped for?”.</p>
<h3>Combating the feature factory</h3>
<p>A feature factory is usually set up by people in other parts of the organisation who have little incentive to change the structure, and this makes it difficult to fight. Things often work well enough that the lack of connection between building and results can be glossed over, and the people who manage the budget get to choose what gets built. The features chosen regularly come from conversations that the factory gatekeepers have had with customers, and moving to a model where the team doesn’t simply build what the customer has asked for will (at least temporarily) make their jobs harder. So what can you do?</p>
<p>As with many challenges in product, <span style="font-weight: bold;">start with the goals. </span>Work with your stakeholders to define your product vision, strategy, and goals (or if they’re already defined, ensure everyone is aligned around them) and make sure the full<a href="https://www.mindtheproduct.com/the-product-decision-stack-martin-eriksson/" target="_blank" rel="”noopener” noopener noreferrer"> decision stack</a> is in place. Start to push conversations about the features that make it onto “the list” and ask why each one is there. Dig deeper into the real problem that is being solved by each feature request, and make it a regular practice to try to identify other ways to achieve the same outcome. This won’t be easy, and you will likely get pushback. But once you can get people into the habit of articulating what they’re trying to achieve in building a feature, you can start to guide them towards the practices of measuring results and iterating before moving on to the next feature.</p>
<h3>Aimless explorers</h3>
<p>Aimless explorers look like they have things figured out. They have bought into the “build-measure-learn” cycle, and test constantly. Every aspect of their product is set up for analytics and data gathering, and you will see numbers and metrics presented in every conversation. These teams may even choose their experiments based on good qualitative research, identifying pain points from user research and working toward solutions to add value for their customers.</p>
<p>This can give the illusion of being evidence-driven, but the focus of the testing doesn’t really align with a solid product goal. The measure of success is “Did we move the numbers?” rather than “Did the numbers we moved matter?”</p>
<h3>Bringing direction to the aimless explorers</h3>
<p>The good news for aimless explorers is you have already tackled the concepts that most teams struggle with &#8211; that you shouldn’t rely on your gut, and there are better ways to learn than building a full product or feature and seeing what happens. But you won’t truly unlock the value of being evidence-driven until you connect your experimentation with clear, defined product goals that focus the team’s energy in an intentional direction. Saying “Our goal is to grow” isn’t enough &#8211; try to get more specific with the segments you want to grow first, or the areas of the product that you want to improve to build a competitive advantage, or any other set of aims that will help you to narrow down the opportunity space. This will help your exploration to have more impact.</p>
<p>It may seem like common sense, but aimless explorers need to<span style="font-weight: bold;"> find focus and direction. </span>A good place to start for aimless explorers is <a href="https://www.mindtheproduct.com/critical-thinking-product-teams-teresa-torres/" target="_blank" rel="”noopener” noopener noreferrer">Teresa Torres’ Opportunity Solution Tree</a>. This provides a framework to help your team to evaluate the goals you’re trying to achieve, and work through opportunities before you spend a lot of effort in testing.</p>
<h3>Hopeful visionaries</h3>
<p>Take a feature factory, add a business plan, and you’ve got a hopeful visionary. These teams create well-defined goals and outcomes to drive towards, and often are great at the “business case” for a feature they want to build. They create hypotheses on how to achieve their outcomes, and define features to build based on their goals. Hopeful visionaries thrive in an environment with long planning cycles and budgets that get approved by executives (typically on an annual basis), because by the next time they request funding, no one remembers what they promised the year before.</p>
<p>But shortly after the launch of a feature, they lose sight of the metrics and don’t follow through to see if the outcomes are being delivered, and move on to the next thing. The measure of success is “Did we focus on the goals at kick-off?” rather than “Are we still focused on goals after launch?”.</p>
<h3>A dose of reality for the hopeful visionaries</h3>
<p>It can be difficult to break the hopeful visionary cycle, because often it arises as a result of their environment. Their funding is dependent on bigger visions, and wild promises that this feature will bring in millions, if only they’re given the chance to build it. And because there is no follow-up of any consequence, there is no incentive to make sure the plans are based in reality.</p>
<p>The simplest way to change this is from the top:<span style="font-weight: bold;"> change the way you fund development.</span> Fund teams and give them outcomes to achieve, rather than funding projects and checking if the features have been built. But even if you aren’t an executive who can change the way the company operates, you can make progress within your team. <span style="font-weight: bold;">Start presenting metrics on everything you build</span>, even if (or maybe especially if) the numbers don’t show a wild success. Place those numbers next to what was presented in the business case. It doesn’t take many cycles of “We said this feature would bring in $250,000 in new revenue&#8230;it has brought in $10,000” or “We were promised a partnership with this company would drive 500,000 new users&#8230;they’ve brought us 20” before stakeholders start to temper their claims when trying to get a feature built. (Are those real examples from my career? I can neither confirm nor deny. 😉)</p>
<p>The scary thing here is, you also have to hold <span style="font-style: italic;">yourself</span> accountable. It can be fun to be a product manager in a hopeful visionary environment, because you <span style="font-style: italic;">also</span> get to build things you want with a very light justification. But in doing this, you’re not driving your product to success. So be prepared to admit something that you thought was going to be great wasn’t great in the end. Then you can start to introduce modern discovery practices to find a better path.</p>
<h3>Evidence-driven teams</h3>
<p>These are the teams that we desire to be. Evidence-driven teams at their core are very focused on outcomes. They use their business goals to drive their metrics, and use their metrics to drive their priorities. They live in a productive “build-measure-learn” cycle, and don’t stop measuring once the “v1” or “MVP” of a feature is released. These teams also understand that every idea they test will not be a winner, and they are prepared to learn from failure.</p>
<p>Alignment across the organisation is key to building an evidence-driven team. A shared understanding of goals and purpose truly allows the leaders to empower their teams, knowing they’re making decisions based on solid information, not wild guesses. Their measures of success are both “Are our priorities in line with our goals?” AND “Are we seeing results with what we’re building?”</p>
<p><a href="https://www.mindtheproduct.com/metrics-that-matter-live-panel/" target="_blank" rel="noopener noreferrer">Watch this panel session for Prioritised members on how product leaders can create metrics that matter. </a></p>
<h3>The responsibilities of evidence-driven teams</h3>
<p>While they may have figured out the use of metrics to drive business value, it can be dangerous if an evidence-driven team becomes myopic, and loses sight of the broader implications of their products. Whether we like it or not, our products exist in an ecosystem, and we have a responsibility to society to consider the potential impact of the things we build. Regardless of where your organisation sits in the Metrics Maturity Landscape, it is important to <span style="font-weight: bold;">consider the unintended consequences</span> or potential for harm of the things we build. This is especially important for evidence-driven teams as they have figured out how to get the most traction out of the things they build.</p>
<h2>Where does your team sit today?</h2>
<p>It is important to note that while these categories fit into boxes, most teams and people don’t. It is completely possible for a team to be in multiple categories across different projects or areas of the product. This is especially true in <a href="https://www.mindtheproduct.com/reboot-your-team-by-christina-wodtke/" target="_blank" rel="”noopener” noopener noreferrer">learning teams</a> where you’re trying different ways of working, and in teams that are evolving their practices. But it can be helpful to really evaluate where your team sits <span style="font-style: italic;">most of the time</span>. Don’t just cherry-pick the one effort that you ran well and think you’ve got it all figured out. In the same vein, if you’ve made real progress, don’t let that one feature  &#8211; that you know you just built on gut instinct because you needed to move forward &#8211; make you think your team is back at the beginning. Think about your team’s default position when using metrics, and use that as a catalyst for a conversation about what you can try next to move to a more evidence-driven world.</p>
<p style="text-align: left;">Discover more content on <a href="https://www.mindtheproduct.com/category/product-management-metrics-and-kpis/" target="_blank" rel="noopener noreferrer">Product Management Metrics and KPIs</a> or use our <a class="c-link" href="https://www.mindtheproduct.com/content-a-z/" target="_blank" rel="noopener noreferrer" data-stringify-link="https://www.mindtheproduct.com/content-a-z/" data-sk="tooltip_parent">Content A-Z</a> to find even more product management content.</p>

core/freeform

Product Management Metrics and KPIs

Measuring the user experience by Tomer Sharon

Why product managers should not be data-driven by Jens-Fabian Goetzmann

Metrics don’t have all the answers: a human-centered approach to business strategy by Samihah Azim

The importance of metrics maturity for evidence-driven teams

Emily Tate

Read more

Comments