Why product managers should not be data-driven by Jens-Fabian Goetzmann

In his digital breakout session at #mtpcon London 2022, Jens-Fabian Goetzmann explained why product managers should make data-informed decisions and balance quantitative and qualitative research.

Watch this video or read on for key highlights from the talk.

Jens briefly highlights Microsoft office 97 as a prime example of how software was delivered in yesteryears on floppy disks. Back then, we couldn’t change the product once it was delivered or shipped, which led to delays in the feedback process.

Jens continues describing the modern software shipping processes and how it is making a change

Modern methods of software shipping

Jens explains that modern shipping software systems benefit from the following:

1. Rapid Shipping: Softwares are available on cloud-based systems, and developers continuously work on the software code to improve it.
2. Agile & Learn Process:  Using agile and scrum to boost the product and software development process.
3. Analytics Tooling: Using data from tools like Google Analytics and Mixpanel to collect user data and make rapid improvements.

Data-driven decisions are crucial

Following the above details, Jens enlightens us on how data-driven decisions are crucial in the business world. He described the following properties of data:

1. Data is unbiased
2. Data is neutral
3. Generating data is fast
4. Data is unambiguous

But they can be a trap for managers

Jens explains the possible issues with data-driven decisions in the following four points:

1. Goodhart’s law and the difficulty of measuring: According to Goodhart’s law, it is difficult to find the right measurement metrics.
2. Local maximum problem and the problem of incrementality: The A/B tests can provide data to an extent. Once you have reached the threshold,  the results will be pretty much the same even if you make thousands of small iterations, and we don’t know what the higher point is. As a result, teams will stop suggesting incremental solutions.
3. Thinking opportunistically instead of strategically: Opportunistically thinking means acting to improve KPIs. There is no hypothesis and no strategy. Moreover, KPIs are more of a proxy for progress toward a plan. In contrast, thinking strategically means setting up a hypothesis and working on it even after setbacks.
4. Analysis paralysis: Product development has risk and uncertainty. Managers often have to make quick decisions based on the best probability and information. Data-driven managers are dependent on data to make every decision.

It’s always better to be data-informed, not data-driven

Jens explained how to reduce dependency on data and make better product development decisions: You only need to take the following steps:

1. Create a product strategy (threats, opportunities, capital, methods, and policies) and vision (how a product will look) by maintaining a balance between qualitative research and quantitative analysis.
2. Do qualitative and quantitative research. This helps generate a hypothesis by understanding the audience (goals and problems) and customer behavior.
3. Take advantage of new opportunities. You can build prototypes without shipping and coding when you make data-informed decisions.

Choose the right tool to find relevant data

Choosing the right tool to fulfil and deliver the above requirements can be difficult, but Jens tells us how to find the right resource. We must choose a product with equal innovation (trying new things) and optimization (improving current methods) levels.

Innovation comes with uncertainty, investment, and risks but with high rewards, while optimization requires small steps to improve current processes.

How do qualitative and quantitate insights work together?

Quantitative data allows you to find potential product issues, while qualitative insights help to understand those issues. Qualitative insights help discover new opportunities, and quantitative test and analyze their impact.

In conclusion

Jens emphasizes understanding the difference between data-informed and data-driven decisions, and we should take the data-informed approach. Moreover, product managers should base their decisions on product vision and strategy and balance the quantitative and qualitative data analysis.

There’s more where that came from!

Explore more #mtpcon London 2022 content or use our Content A–Z to find even more product management insights.