If you’re a product manager, it’s not enough simply to understand metrics, you also need to know how to pick the right ones. And it’s not easy – Scott Graffius summed up this problem very well in his book Agile Scrum, saying: “If you don’t collect any metrics, you’re flying blind. If you collect and focus on too many, they may be obstructing your field of view.”
We review advice from product leaders to examine the factors you need to consider when evaluating your product metrics.
Are they actionable?
As product leader John Cutler points out in this MTP panel discussion, How to define the metrics that matter, metrics help you to reduce uncertainty in an area where this is valuable.
Metrics should reflect a company’s strategy, so clear metrics are a good indicator that a company knows what it’s doing. By the same token if the metric doesn’t solve a problem or drive a strategy forward then it’s a vanity metric. As John Cutler also observes, vanity metrics can look great superficially but they don’t inform strategy and help anyone to make decisions. They may give the impression of improvements but don’t help you understand performance. For example, knowing the number of new users gained per day might be encouraging but it doesn’t help with decision-making.
This ProductPlan article gives a few pointers on what to look for in an actionable metric: look at the business decision the metric helps you to make, whether you can intentionally reproduce the results and whether the data is a real reflection of the truth.
Are you collecting everything?
A classic mistake is to collect so much data that a team becomes overwhelmed by the noise of it all and doesn’t focus on the things that matter. And as Mind the Product Managing Director Emily Tate observes, “the data will rarely just magically surface some insight that you never would have thought to look for”.
It’s better practice to work backwards from the outcome you expect from any change you make, and to know what action you will take when a metric moves a certain way. Then you can limit what you measure. It’s also easier to make decisions and correlate data when you limit what you measure, so you need a strategy to query your data for actionable insights.
Can you use a framework?
There are several frameworks to give you a sensible plan on how to interrogate your data. This Mind the Product post from 2018, why do you need to rethink your analytics strategy?, offers guidance on how to ask the right questions of your data rather than look for the right answers.
This post from vendor Userpilot, Product managers’ guide for selecting the right product metrics framework gives a handy run-through of the many frameworks available, including Pirate, North Star and OKRs. This post from vendor Amplitude, every product needs a north star metric: here’s how to find yours, is also a great guide to how to define your north star metric and use it to drive long-term strategy and growth. And this MTP post, Measuring the right North Star metric, from product management and discovery coach Sebastian Straube, gives some great examples of exemplary north star metrics.
What types of metrics do you need to track?
A product team needs to track different types of metrics. Operational metrics, such as the number of active users, serve to show that the business and product are not going in the wrong direction and need to be tracked even if no one actively makes decisions about them.
You should track a mix of leading and lagging indicators to help predict future performance. This MTP article from earlier this year, Are you tracking the right product metrics? gives plenty of pointers on choosing metrics, and talks to some experienced product leaders about their journeys to find the right metrics to track.
Mind the Product’s training workshop Metrics for Product Managers also focuses on how you define a North Star metric and then craft an appropriate metrics plan.
Other considerations
Remember there won’t always be a right or a wrong answer, and metrics don’t always provide clarity. Because metrics are numeric, this is something we often forget. What you should also remember is that metrics must ultimately serve or be reflective of your users and their behaviour.
Crafting metrics is a practice. It’s not something that is necessarily perfected easily, and it’s an ongoing process, not a one-time effort. A product manager needs buy-in from stakeholders, and the time and dedication to review and renew metrics with a regular cadence.
Further resources
If you’re completely new to product management and analytics, then this evergreen article from Mind the Product, Everything a product manager needs to know about analytics, is a good place to start. It drills down into the concepts behind analytics and offers advice on how to plan their implementation.
You’ll also find a comprehensive range of talks, articles and advice on all things analytics from Mind the Product here: Product Management Metrics and KPIs.
Comments
Join the community
Sign up for free to share your thoughts