In this exclusive Prioritised member session, our panellists, Ashley Fidler, John Cutler, and Adam Thomas, discussed how to create those hard-to-define ‘quality’ metrics.
The panel, moderated by Emily Tate, Managing Director at Mind the Product welcomed:
- Ashley Fidler, VP Product at Pure Group
- John Cutler, Head of Product Research and Education at Amplitude
- Adam Thomas, Lead Product Manager at SmartRecruiters
Watch the entire session in full or read on for a few of our panellists’ key points and tips:
Creating metrics that matter
Ashley kicks off the session by explaining that clear and concise metrics is an indicator of whether a company is on track to create effective change. Having conversations about metrics early on in a company’s development means you can intertwine those data sets with the overall company strategy. You should start from the user and keep things simple. “Think, what would a person using this system think quality is,” Ashley says, before describing an approach from her time at Microsoft, working on Cortana. “We built a simulator that would run simulations of Cortana answers and got a panel of core users to rate them. They were asked ‘does this make sense to you?'”. The team took this exploratory approach until they could build the results into a metric. “That qualitative exploratory piece is critical. With that type of metric you can’t just calculate your way to quality. It’s subjective by definition.”
Additionally, John discusses how, although vanity metrics can be used to create a massive amount of convergence in the company, they often fail to help differentiate the various personas that use products. “Sometimes, metrics aren’t related to customer lifetime value or growth. You have to design principles and communicate to people the metrics that you need. Sometimes you need to experiment to find the right metrics. There’s an exploratory nature to get it right,” he says.
John adds that it’s crucial to clarify company strategy and expectations: “What do people expect when they invest in a product? Rank those in terms of certainty. Your job as a product leader is to put a value on reducing uncertainty around those customer expectations. Differentiate ideas and expectations. You then have to connect those two ideas and give yourself a framework of what you need to work on.”
Ashley explains that tracking new metrics can provide new insights into a product as well as driving a strategy forward. John adds that it reduces uncertainty in essential areas.
Building own tools for metrics
Adam explains that product managers don’t start at the core of what’s most important when they look to build their own tools for metrics. He says: “Businesses are always looking at something that has an immediate impact. It’s important to go in knowing what you’re trying to achieve with a metric before thinking about the tools to assist your goals.” John adds that companies should embrace the ecosystem of metrics and reject the “silver bullet” that may offer instant satisfaction. “Start slow and count the details of the product that you initially care about. If you can measure those things, there’s a universe of tools that can help you drive value,” John says.
It’s important to make sure that you don’t just look at tools to solve your problems. You must also recognise that they can be a valuable asset to move forward beyond measuring the things that you care about.
John encourages teams to start making lists of decisions, assumptions, and enquiries that customers submit about a product. Doing so will create a framework to build on and progress with. Adam adds that product leaders should ask questions about what’s important, such as: where are we going, what’s the vision, where are we headed? “Finding out what your product will look like long-term will create a solid foundation,” he says.
As a product manager, you know what levers are being pushed to boost success. “Figure out what those levers are and how you want them to change,” Ashley explains, “you can make subtle changes even if you don’t have access to the full strategy.”
Using metrics to accomplish decision making
When asked how to use metrics to push decisions forward, Ashley explains how product leaders mustn’t attempt to criticise product ideas or business decisions as this ultimately creates unneeded conflict. Instead, let teams come to decisions on their own by providing them with the right data. Stakeholders may be open to change if they are presented with certain metrics. Adam adds that collaborating with other teams to work your way through a decision-making process can smooth out potential conflict. They may have quantitative data that product leaders can do to further rationalise decision making.
There is a mix of quantitative and qualitative metrics that can be used to influence decisions. John believes that it’s important to understand the sources of complexity in people’s heads, then use the metrics to come to a final decision. That relationship between complexity and long-term sustainable revenue is key. Overall, it’s crucial to understand what customers care about before you start using data to make a business case.
Defining different product metrics
Asked how to define different types of metrics — for example, business metrics and product metrics — John explains that you must understand what goal you’re working towards. Is it a goal with an end date, or an ongoing task? Secondly, find out what teams are going to be affected by these metrics. Finally, he says you should differentiate between learning metrics and evaluative metrics. There are different levels of uncertainty between the two and both require different approaches.
Creating valuable metrics
Ashley believes that there are some instances where there aren’t any clear valuable metrics to rationalise business decisions. Fundamentally, it’s important to always go back to stage one of understanding the business strategy and then figure out how a product aids that overarching goal. “If there isn’t a good metric, it’s better not to try and make one up,” she says. Figure out what’s driving the business forward and focus on helping with that.
Adam adds that metrics is about problem determination. He says: “If you can’t get to the base of a problem with metrics, then you have more work to do. If there’s always context into why you’re tracking certain metrics then it reduces the risk of being dormant.” It’s also important to understand why you’re tracking something, he adds. Product managers should always be asking how valuable their metrics are and how they’re driving their product forward.
More on Metrics
- Product Management Metrics and KPIs
- The importance of metrics maturity for evidence-driven teams
- The Only Product Metric that Matters by Josh Elman
Discover even more content on Product Management Metrics and KPIs.
