The wrong ways to be data-driven

Data is a critical cornerstone of informing every single product decision we make today. However, this means that there is room for error with significant adverse consequences if we leverage data incorrectly. Let’s look at all the wrong ways to be data-driven and how we can change that with a few concrete examples below.

You are leading a high priority area within your organization. The product your team is building and the work your team is doing is expected to directly impact and influence the Daily Active User (or other) northstar metric for the organization. Your team sets a goal to move the Daily Active User metric from X at the start of the year to Y at the end of the year. Your team builds and tests different product ideas to validate/invalidate your hypotheses.

Some of these ideas move northstar metrics and are shipped while others do not and are deprecated. The team ends the year exceeding the goal by 5%. Your team is expected to get three more engineers to continue the work going into the next year – that’s a 30% increase in capacity. You’re on track for your promotion. Things are looking great…

Not so fast…here’s another view of the story above.

Your team picked the metric before defining who your users are, understanding what they need and determining what value you wanted to deliver to them. As a result, you either ended up building some generic optimizations which grew Daily Active Users.

Your team had a bunch of ideas that could improve the overall product-market fit of your product. But, you were looking for bronze coins in a diamond mine. As a result, your team decided to de-prioritize product ideas which would solve real problems for your users because they wouldn’t immediately move northstar metrics.

A few ideas your team had, drove real value to end-users. But, the success and failure of these product ideas were evaluated by A/B tests and judged for their merit solely based on the impact on northstar metrics. As a result, your team decided to deprecate all of these ideas.

And while you’re on track for your promotion, you are having a harder and harder time shutting out the voice in your head that your job is no longer fulfilling and it might be time for your to move on.

Let’s break down how to use data correctly and incorrectly across the following three key aspects of the product development process:

• How we prioritize
• What we measure
• How we measure

How we prioritize

As Product Managers, our goal is to improve the value that a product delivers for our end users – our goal is not to move a metric. The most common mistake a Product Manager can make for the long term success of a product is to let the metric define the product versus letting the user needs define the product and the product define the metric.

Discover more insights on creating metrics that matter.

How often have you made a prioritization decision based on the impact on a northstar metric? It’s rare and difficult for value-added product features to lead to big gains in the northstar metrics in a short period of time.

For instance, let’s take an example of an app that helps people discover local businesses. If a product manager decided that success was to grow Daily Active Users it’s easy for a new notification to lead to immediate and short term gains in Daily Active Users. However, if all this product manager did is optimize notification systems, over a longer period of time, we haven’t moved the needle on the value the product delivers to end-users at all and the efficacy of these notifications would decrease. Instead, this same product manager could take an alternate approach where success was to help users find the best plumbers for them in addition to restaurants.

The goal is not to discount the use of data. Data would be essential in making a good prioritization decision. But the right data questions to ask would not be by how much the feature could improve Daily Active Users.

The right questions instead are:

• How many plumbers do we have on the platform?
• How many searches do we have for plumbers on the platform?
• How long / how many outreaches does it take for a user to find a good plumber?

A reason why this problem arises in the first place is organizational culture. It’s easy to justify your prioritization rationale based on impact to northstar metrics — especially with leadership teams who are well versed with these metrics. A solution to this problem is to always create a goal map and drive alignment on that goal map. A goal map in this example could look something like:

Read more about prioritisation for product managers

What we measure

Given the aforementioned familiarity with and ease of communicating northstar metrics, we often tend to measure the success of a product based on how it impacts northstar metrics. The most common mistake a product manager can make is to put the cart before the horse.

When working on a new product, there are a few key ingredients to get right before we can start focusing on northstar metrics:

• Quality: Do we see a high number of crashes? Do pages take over 10 seconds to load? Is the product easy to use?
• Value: Are we delivering value to our end users?
• Product Market Fit: Are we delivering that value to a significant number of people consistently i.e. are adoption and retention high among our target audience?

Again, data is a vital part of the entire journey of figuring out whether a product is successful or not. But, using the right data before we jump to northstar metrics is an important step we often tend to omit.

Organizational culture is difficult to combat here again if the entire organization is speaking the language of north star metrics. A solution to the problem is to proactively land a testing and launch strategy with clear phases (e.g. Phase 1A: Quality, Phase 1B, Usability, etc), timelines for each phase and exit criteria to move from one phase to another.

Read Data-Driven Product Management by Matt LeMay

How we measure

This one might be controversial but I would strongly recommend against using A/B tests that look at northstar metrics to make decisions on moving between phases of testing or making launch / deprecate decisions) for a few reasons:

• They typically don’t help us get a good signal on key metrics like Quality since your control group does not have access to the product feature at all.

• They generally take very long to provide a statistically significant signal on metrics like retention.
• They are rarely clean with all the different ML-based products running in our products.

An alternate approach could be to use the phase-based approach described above with clearly defined benchmarks of exit criteria. For instance, with every product you work on, define benchmarks for clear success, signs of success and clear failure in each phase.

In summary

Building product continues to be a combination of art and science. Data is important in each step of the product development process. But to harness the power of data to yield long term success of your product, remember that:

• The value you deliver to end-users should define the product and hence the metric – the metric should not define the product.
• Pick the right set of metrics to first understand whether you are delivering value to end-users before focusing on northstar metrics.
• Leverage goal maps to showcase how your product will ladder up to northstar metrics over time.

Discover more insights on Data-Driven Product Management