post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Timo Hilhorst is an intrapreneur who heads up MondayLabs. MondayLabs is an innovation lab part of <a href="http://www.reed.co.uk" rel="nofollow">reed.co.uk</a>, the UK's leading job site. In that role, his focus is on validating new ideas in the digital recruitment space. He also founded Startup Startup, the company's first corporate startup. Timo loves hearing and reading about anything product related, especially if it involves disruptive innovation (yes, he's a big Clayton Christensen fan). You can find him on Twitter <a href="http://twitter.com/timohilhorst">@timohilhorst</a>.

Glenn Block runs product management for <a href="https://auth0.com/extend/" rel="nofollow">Auth0 Extend</a>, a new service for customization of SaaS products. Glenn has 20 years+ of industry experience, previously working at Microsoft and Splunk. Aside from building products he loves to engage with the product community as a teacher and learner. He is an organizer of ProductTank Bellevue, a frequent speaker and published author. You can find him on Twitter at <a href="http://twitter.com/gblock">@gblock</a>.

Glenn Block and Timo Hilhorst

<p><strong>Qualitative</strong> versus <strong>quantitative</strong> data: we’ve all been involved in a conversation debating their respective merits at some point in our careers. We’re often flipping backwards and forwards between letting feedback from a handful of customers drive all our product decisions or requiring everything to be backed up by statistically significant data. So which type of data is better?</p>
<p>The answer is neither.</p>
<h2>What’s the Difference?</h2>
<p>Let’s first get on the same page &#8211; what is qualitative data and what is quantitative date? In a product management context, these terms refer to data gathered from customers about their experience with our products and, if we take the most <a href="https://www.shmoop.com/probability-statistics/qualitative-quantitative-data.html" target="_blank" rel="nofollow noopener noreferrer">literal definitions, </a>they look like this:</p>
<h5>What is Quantitative Data?</h5>
<p>This is information that <span style="text-decoration: underline;">can be measured</span> and written down with numbers. For example, the number of times a certain feature was used, or the number of times a page was visited. This data is gathered through:</p>
<ul>
<li>Surveys</li>
<li>Application metrics</li>
<li>A/B tests</li>
</ul>
<h5>What is Qualitative Data?</h5>
<p>This information that <span style="text-decoration: underline;">can&#8217;t be measured</span> and is subjective. An example might be how intuitive a feature is to use.</p>
<p>This type of data is typically what customers say:</p>
<ul>
<li>during interviews</li>
<li>on phone calls</li>
<li>in email exchanges</li>
<li>in tickets</li>
<li>during casual conversation</li>
</ul>
<p>Another angle is the <a href="http://www.objectivity.com/hard-data-vs-soft-data/" target="_blank" rel="nofollow noopener noreferrer">softness and hardness</a> of the data. <span style="font-size: 1rem;">Quantitative data is </span><em style="font-size: 1rem;">hard</em><span style="font-size: 1rem;">: it’s measurable and concrete. Qualitative data is </span><em style="font-size: 1rem;">soft</em><span style="font-size: 1rem;">: it is subjective information that is filled with opinion.</span></p>
<h2>What Does the Data Tell Us?</h2>
<p>In order to make balanced product decisions, we need both types of data. They both tell us about customer behavior, but each gives a different perspective and level of detail. Hard data is more precise as it is measured &#8211; “How many customers are doing X?”. It can also give insights into our customers’ experience, such as the average latency customers are experiencing when they visit your site.  Soft data gives us a lot more detail and context behind the customer behavior: “This feature makes it very difficult to do my job”, “This is important to me”, “Your site is slow”, “That screen doesn’t work for me because it is too cluttered”.</p>
<p>On the subjectivity of soft data, <a href="https://twitter.com/sicarul" target="_blank" rel="noopener noreferrer">Pablo Seibelt</a>, a senior data engineer once said:</p>
<blockquote><p>Humans are subjective! We are not robots, but being aware of our subjectivity leads to better results.</p></blockquote>
<p>Here’s an example of how quantitative and qualitative data complement each other. We’re using an A/B test to figure out whether the call to action in our user interface should be orange or green. If we can reach significance in this test, we’ll have identified the color using hard data. What we don’t know is the context, why one color is preferred over another. This is something we could get from studying soft data, like what users are saying during user testing sessions (“Oh, I hadn’t spotted that button there, it kind of blends into the background”).</p>
<h2>Why do you Need Both Quantitative AND Qualitative Data?</h2>
<p>Taking the example above, we might say after getting the hard data “Who cares?”. We’ve found out which of the two buttons works better so we can just act on that. In this isolated case, that might be true. However, without going deeper we don’t understand why one is preferred over the other. Once we talk to customers, we can ask them, and then understand their preferences, which can then be applied in other contexts as well. If the green button won out in the test, does that mean we should change all of our buttons to green? Or is it the context of this specific test which makes it better? We’ll need more soft data to find the answer.</p>
<p>On the topic of balancing both types of data, <a href="https://twitter.com/sshevlyagin" target="_blank" rel="noopener noreferrer">Sergei Shevlyagin</a>, <span style="font-size: 1rem;">Software Development Engineer </span><span style="font-size: 1rem;">at Zillow says: </span></p>
<blockquote><p><span style="font-size: 1rem;">Qualitative helps you narrow down on the right questions by understanding what problems the customer is having, what actually matters to them and why. Hard data gets your precise answers to precise questions once you did the narrowing down.</span></p></blockquote>
<h2>Where do you Start?</h2>
<p>Wouldn’t it be great if there was a hard and fast rule? Unfortunately, as is often the case in product management, the answer is “it depends”. Where to start depends on the situation, what we’re trying to measure, and which data we have available.</p>
<h2>Starting with Quantitative Data</h2>
<p>When making decisions where there’s a lot of uncertainty, it’s common to use hard data. Imagine we are planning to add a new product to our portfolio. We might put ideas of products to build into a survey which we send to our customers. This might help us to identify how well these ideas resonate with our customers (or at least with those customers engaged enough to complete the survey), but it doesn’t tell us why they prefer one over another or whether there are more important problems we could be solving. We need soft data to make sense of the survey information.</p>
<figure id="attachment_19891" aria-describedby="caption-attachment-19891" style="width: 702px" class="wp-caption aligncenter"><a href="https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop.jpg"><img loading="lazy" decoding="async" class="wp-image-19891" src="https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop.jpg" alt="Checking data on a laptop" width="702" height="468" srcset="https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop.jpg 5260w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop-300x200.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop-768x512.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/People-looking-at-data-on-a-laptop-1024x683.jpg 1024w" sizes="(max-width: 702px) 100vw, 702px" /></a><figcaption id="caption-attachment-19891" class="wp-caption-text">Hard data is great, but we need soft data to make sense of the survey information</figcaption></figure>
<p>Another place where hard data is useful is showing patterns of customer behavior. Let’s say behavioral data from our e-commerce platform tells us that users aren’t engaging with our special offers. By taking the data at face value, we may conclude that people don’t find the special offers compelling, and we need to work on either improving them or de-prioritize the feature. This assumption may be wrong. Digging deeper and by talking to a bunch of customers, we may discover why the special offers are not obvious or discoverable. Thus, if we act simply on the hard data, we’ll go down the wrong path. Starting only with hard data is starting with only half the picture.</p>
<h2>Starting With Qualitative Data</h2>
<p>An alternative approach to figuring out which product to build would be to start with the soft data. If we spend some time with our customers (or people we think could become our customers), we’ll probably learn an awful lot about their problems, challenges and needs. We can use these insights to form hypotheses around the products or services that will deliver the most value to this group.</p>
<figure id="attachment_19889" aria-describedby="caption-attachment-19889" style="width: 700px" class="wp-caption aligncenter"><a href="https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview.jpg"><img loading="lazy" decoding="async" class="wp-image-19889" src="https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview.jpg" alt="Man discussing problems with a customer" width="700" height="466" srcset="https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview.jpg 6000w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview-300x200.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview-768x512.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/two-people-in-an-in-person-interview-1024x683.jpg 1024w" sizes="(max-width: 700px) 100vw, 700px" /></a><figcaption id="caption-attachment-19889" class="wp-caption-text">By spending time with our customers we can learn a lot about their problems, challenges and needs</figcaption></figure>
<p>We can then aim to gather hard data to validate these hypotheses. By starting with the soft data, we can figure out what questions to ask in a survey or better design experiments to validate the RIGHT hypotheses with hard data.<a href="https://www.linkedin.com/in/brucemccarthy/" target="_blank" rel="noopener noreferrer"> Bruce McCarthy</a>, the founder of UpUp Labs and Product Culture, explains this way of working in more detail in <a href="https://foldingburritos.com/customer-feedback/bruce-mccarthy/" target="_blank" rel="nofollow noopener noreferrer">this interview</a>.</p>
<p>&nbsp;</p>
<p>If there are more “knowns” and hard data already available, it is common to start with soft. For example, we’ve been given the task to identify the best way to improve the conversion rate on an e-commerce platform that’s already getting a decent amount of traffic. At that point, we have access to hard, behavioral data to show us where the main pain points might be. However, as <a href="https://www.linkedin.com/in/lulucheng21/" target="_blank" rel="noopener noreferrer">Lulu Cheng</a>, Product Lead at Pinterest explains in <a href="https://foldingburritos.com/customer-feedback/lulu-cheng/" target="_blank" rel="nofollow noopener noreferrer">this interview</a>:</p>
<blockquote><p>If you just see a number go up or down week to week, that tells you what’s happening, but it doesn’t tell you why users are doing that particular action.</p></blockquote>
<p>Soft data will uncover the reason. Without soft data, we could very well end up solving the wrong problem or solving the wrong part of the right problem.</p>
<h2>The Customer Data Cycle</h2>
<p>Both hard and soft data feed into each other. Starting with hard data, we can uncover significant patterns in our product usage that lead us to do interviews to understand more context. We may uncover new clues while interviewing, and we can then go back to our hard data to measure their significance. In some cases getting some hard data will lead us to other related hard data to make more sense of what we are seeing. The same applies to starting with soft data.</p>
<figure id="attachment_19888" aria-describedby="caption-attachment-19888" style="width: 202px" class="wp-caption aligncenter"><a href="https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm.png"><img loading="lazy" decoding="async" class="wp-image-19888" src="https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm.png" alt="Screenshot of the Calm mobile app" width="202" height="403" srcset="https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm.png 801w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm-150x300.png 150w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm-768x1534.png 768w, https://www.mindtheproduct.com/wp-content/uploads/2018/01/7-Days-of-Calm-513x1024.png 513w" sizes="(max-width: 202px) 100vw, 202px" /></a><figcaption id="caption-attachment-19888" class="wp-caption-text">The Calm mobile app</figcaption></figure>
<p>&nbsp;</p>
<p>A great illustration of this cycle is <a href="https://www.calm.com/" target="_blank" rel="nofollow noopener noreferrer">Calm</a>, a mobile app for meditation. After launching a meditation reminders feature, the team at Calm <a href="https://amplitude.com/blog/tag/calm/" target="_blank" rel="nofollow noopener noreferrer">looked at their data</a> and noticed only 1% of their users took advantage of reminders. Further investigation showed that those 1% had a retention rate three times as high as the rest of their users. Why weren’t reminders used by more customers? They looked at their settings page and realised reminders were deeply buried.</p>
<p>Based on the hypothesis that this was the reason for low adoption, they experimented with increasing the feature’s discoverability, by prompting new users to create reminders. Through testing, they found 40% of users who saw the new prompt then set a reminder with the same three times retention rate! As you can see, Calm’s continual delving into soft and hard data was critical in helping to arrive at the right decision.</p>
<h2>Be Careful of Your Assumptions</h2>
<p>Both hard and soft data need to be analyzed, and <strong>the results can be wrong</strong>. We have a bunch of metrics about feature usage, but what is the meaning behind those metrics, and what action do we take? We heard a bunch of requests from different customers in interviews, how common are they? Are they really valid? Did we really hear what we really thought we heard? In the same way, we should question the data, we should question our assumptions.</p>
<h2>Don’t Choose &#8211; Use Both</h2>
<p>Ultimately, when it comes to qualitative and quantitative data, it’s key to understand that <strong>we need both</strong>. Without knowing how significant the feedback of a handful of users is against our entire user base, we might overemphasize its importance. Without knowing why we’re seeing a certain pattern in behavioral data, we might try to solve the wrong problem, or solve the right one incorrectly.  Without validating gut feelings and intuition, we might go completely down the wrong path.</p>
<p>As product managers, our job is to learn as much as possible about our customers to help us understand where to invest.  Balancing qualitative and quantitative data is critical for driving product decisions.</p>

core/freeform

How to lead data-centric product teams: Jon Mora (Chief AI Officer, Zefr)

How to be a better data-driven product manager

User insights for product decisions by Cris Valerio

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

Why you Need Quantitative AND Qualitative Data

Glenn Block and Timo Hilhorst

Read more

Comments