post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Benn is an accomplished data analyst with deep expertise in collaborative Business Intelligence and Interactive Data Science. Stancil is Co-founder, President, and Chief Analytics Officer of Mode, an award-winning SaaS company that combines the best elements of Business Intelligence (ABI), Data Science (DS) and Machine Learning (ML) to empower data teams to answer impactful questions and collaborate on analysis across a range of business functions. Under Stancil’s leadership, the Mode platform has evolved to enable data teams to explore, visualize, analyze and share data in a powerful end-to-end workflow. Prior to founding Mode, Stancil served in senior Analytics positions at Microsoft and Yammer, and worked as a researcher for the International Economics Program at the Carnegie Endowment for International Peace. Stancil also served as an Undergraduate Research Fellow at Wake Forest University, where he received his B.S. in Mathematics and Economics. He believes in fostering a shared sense of humility and gratitude.

Benn Stancil

<p>When we ask questions about our data, we rarely walk in a straight line to the answer. Instead, data analysis often follows a more circuitous, creative process: we ask a question, discover our first effort to answer it is incomplete, reshape our data, bring in new data, consult a domain expert to rework our hypothesis, adjust our question, and so on.<br />
We refer to this process as iterative analysis—though most practitioners will just call it analysis. This way of thinking is intuitive and natural for most people—so natural, in fact, we often don’t even recognize we’re doing it, in both our professional and personal lives. But whether we recognize it or not, it’s necessary to answer the undefined questions that sit at the heart of any big decision.</p>
<p>Take, for example, common questions that business leaders ask. Marketers might wonder what predicts high purchase intent. Customer success managers might ask which characteristics or behaviors signal that a <a href="https://www.mindtheproduct.com/12-ways-to-reduce-customer-churn/" target="_blank" rel="noopener noreferrer"><span style="text-decoration: underline;">customer is likely to succeed or to churn</span></a>. And product managers may ask if the team should spend time fixing bugs or building new features. To answer these questions, we need iterative analytics.</p>
<h2>Should the product team spend time fixing bugs or building new features?</h2>
<p>Suppose your customer support and renewal teams want your product team to invest time into fixing bugs. Customers are getting frustrated by bugs and it’s becoming a fire. Your sales team, however, says the bugs are edge cases and suggests the product team spends time on building new features, reasoning that this would let them close more new business, faster.</p>
<p>As a product leader, how do you make this decision? You likely can answer some basic questions pretty easily:</p>
<ul>
<li>How often do we record errors?</li>
<li>How many customers see them?</li>
<li>How many support tickets do we get?</li>
<li>How many customers use features that we think are particularly buggy?</li>
</ul>
<p>These questions help set context, but they don’t help make the decision you need to make. A few charts of error rates won’t help us understand the potential tradeoffs we have to make between investing in one direction compared to another. Instead, we have to analyze this decision from a variety of angles and consider all of that evidence together. To do this, we have to drill two layers deeper.</p>
<p>Our first step is to define the important considerations that are necessary for making our final decision:</p>
<ul>
<li>Are bugs actually affecting the customer’s experience?</li>
<li>Do bugs, or related measures like support tickets, correlate with retention?</li>
<li>Are some bugs much more impactful to customers than others?</li>
<li>How valuable are the new features the sales team is asking for?</li>
</ul>
<h3>To answer each of these questions, we’ve got to go another level deeper.</h3>
<p>For example, when we ask “Are support tickets correlated with retention?” we can’t just figure out if churned customers send more support tickets than retained customers.. We instead have to ask a series of more detailed question, each of which changes based on the answer to the previous question:</p>
<ul>
<li>Do we need to make adjustments for customer size (e.g. Are larger customers more likely to send more support tickets? Are they more likely to retain?)</li>
<li>Are all support tickets the same? Are there some that we should exclude, or some we should weigh more heavily?</li>
<li>Does timing matter? Should we be more concerned about tickets from new customers? From those about to renew?</li>
<li>How do we adjust for tickets with good or bad CSAT scores?</li>
<li>Do customers who churn disengage with our product and send fewer support tickets despite having a bad user experience?</li>
</ul>
<p>Each of these questions is a piece of evidence. That evidence, however, won’t necessarily all point in the same direction. To answer the question, “are support tickets correlated with retention?” we have to combine this evidence with our reasoning and intuition, and come to the answer that we think is best supported.</p>
<p>And that answer, again combined with the answers to similar questions, as well as your own reasoning and intuition, rolls up to help us decide if we should invest in fixing bugs or building new features.</p>
<h2>Iteration helps you get a more complete picture</h2>
<p>This process—answering multiple layers of questions, looping over each, rolling them up to a final set of inputs, and weighing all of them together to come to a final conclusion—is iterative <a href="https://www.mindtheproduct.com/analytics-are-the-key-to-optimizing-the-product-roadmap/" target="_blank" rel="noopener noreferrer">analytics</a>.</p>
<p>Data doesn’t change the foundational approach; it simply provides a powerful way to answer the inner set of questions. And even then, data won’t always answer those questions directly. You often have to combine different datasets together, find creative measures of seemingly immeasurable concepts, and cut and analyze results in different ways, with each new iteration offering a new perspective on your problem.</p>
<p>As an example, let’s take a step-by-step look at how we’d use iterative analysis to approach the question, “Are support tickets correlated with retention?”:</p>
<h3>Establish the context</h3>
<p>Ask high-level questions that give you a rough understanding of the problem you are trying to solve. Figure out how many support tickets are submitted per customer. Are they changing over time? Do churned customers submit more? Asking these questions will help you find that churned customers, on average, write fewer tickets.</p>
<h3>Drill deeper</h3>
<p>Ask questions that could explain your initial findings. Why do you think the trends are what they are?  Then ask a series of more detailed questions to investigate these hypotheses.</p>
<p>Say that the initial result—that <a href="https://www.mindtheproduct.com/need-product-analytics-playbook/" target="_blank" rel="noopener noreferrer">churned customers</a> submit fewer tickets—is counter to the intuition of the Customer Success team. Why might that be? Maybe there’s a hidden confounding variable, like customer size or time as a customer. Bigger or older customers might ask more questions but are also retained at a higher rate.</p>
<p>Additionally, is there any segment of customers for whom this is not true? Pull in some dimensions about the customer and create a few quick charts to see if the relationship holds across different customer cohorts.</p>
<p>Digging deeper in this scenario solidifies the point that across many segments of customers, churned customers write on average fewer tickets.</p>
<h3>Combine data with human reasoning</h3>
<p>Digest and analyze the data collected from the previous questions.  Understand that the evidence won’t necessarily point in the same direction. You’ll have to use your reasoning and intuition to arrive at the best decision.</p>
<p>Why are retained customers writing more tickets? Maybe it’s because they spend more time on the product, and as a result, find more bugs. Let’s explore this. Does our result change if we control idle session time? This might require introducing new data. This time, it’s complex enough that the best way to check the relationship is through regression in retention.</p>
<p>In this example, even after controlling for idle session time, we still see a positive relationship between retention and support ticket volume. Retained customers submit more support tickets per active minute than those who churn.</p>
<h3>Combine different datasets together</h3>
<p>Consider what’s not being measured in your current dataset. It may be that the answer is hidden in data you haven’t explored yet.</p>
<p>We have very high support <a href="https://www.mindtheproduct.com/how-humanising-your-product-can-make-all-the-difference/" target="_blank" rel="noopener noreferrer">customer satisfaction scores</a> (CSAT). Could people who write into support have such a positive experience that they forgive us for the bug? Is there a correlation between CSAT scores by customer and churn rates and support tickets? Pull in CSAT scores, and look at average CSAT scores by customer.</p>
<p>Asking these questions in this scenario finds that churned customers give much lower CSAT scores. That’s promising, but it’s hard to understand the causal direction. From the previous analysis, CSAT scores are known to be much lower for new users.</p>
<h3>Keep iterating</h3>
<p>When you find threads to pull on, keep digging to understand them better. Iterative analysis is like solving a mystery—there’s rarely a smoking gun. Instead, you find answers by diligently following clues one at a time.</p>
<p>In this example, you can look at the relationship between CSAT scores, support tickets, and churn. CSAT scores are low for new users, so perhaps their experience is particularly bad. Rather than solving all bugs, could we just focus on those that affect new users?</p>
<p>Doing so can help to understand which bugs cause new customers to write and address those bugs. And we now know we can reduce churn if our support team prioritizes questions from new customers.</p>
<p>In this scenario, should we put a few <a href="https://www.mindtheproduct.com/a-guide-to-creating-api-products/" target="_blank" rel="noopener noreferrer">engineers</a> on this core set of bugs, and the rest of the team on new features? Should everyone work on bugs? To answer those questions, we return to our original larger loop question set, and work through them much as we did this one. This process — answering multiple layers of questions, looping over each, rolling them up to a final set of inputs and weighing all of them together to come to a final conclusion — is Iterative Analytics.</p>
<h2>In conclusion</h2>
<p>Iterative analytics gives product analysts a better understanding of their users, and thus, a better way to prioritize the product team’s work. Using iterative analytics in combination with product analytics and BI tools allows product teams to make better informed decisions and deliver results more quickly, thereby driving faster product-led growth with less risk.</p>
<p>Explore more content on <a href="https://www.mindtheproduct.com/category/product-analytics/" target="_blank" rel="noopener noreferrer">Analytics</a>. For even more content on a range of product management topics, see our <a href="https://www.mindtheproduct.com/content-a-z/" target="_blank" rel="noopener noreferrer">Content A-Z.</a></p>

How product leaders can make better decisions with iterative analytics

Benn Stancil

Read more

Comments