post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

I'm Niraj Upadhyay, a seasoned professional with over 20 years of combined experience in product management and engineering. Currently, I lead the product efforts at LeadIQ, where I oversee a $20MN flagship product. My focus is on data-driven strategies and fostering a product-led culture, recognized through the annual Founder's award.

Niraj Upadhyay

<p>Conducting experiments is an essential part of product discovery for product managers. It goes hand-in-hand with user interviews, when your goal is to understand the user’s behavior patterns.</p>
<p>However, conducting in-app experiments in B2B products, <span style="font-weight: bold;">especially when targeted at enterprise persona</span>, can be quite tricky and challenging. In this article I share some of the challenges I have faced and the lessons I learned.</p>
<h3>User interviews vs. in-app experiments vs. in-app surveys</h3>
<p>Before I talk about in-app experiments, the challenges, and their potential solutions, here’s a quick run-down of three types of validation methods along with some of their advantages and drawbacks.</p>
<h4>User interviews</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>It allows us to deep-dive into a problem with the users with open-ended questions, ask counter-questions, and seek clarifications, leading to an in-depth understanding of the problem.</li>
<li>Provides first-hand feedback with full context.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>Quite a slow process — In the B2B world, especially when you target large enterprises, it is difficult to get enough users of the <span style="font-weight: bold;">desired persona type </span>to spend time with you and give you feedback. It can take weeks on end to gather enough users for interviews. Some teams follow the practice of enticing them with gift vouchers but it’s not always a viable option, especially when you are on a shoestring budget like I am and it also runs the risk of “positive response bias” in their feedback.</li>
<li>Frequently, there’s a notable gap between what users express in interviews and their actual interactions with your product. So, just depending on qualitative data from the interviews has a good chance of being mired with confirmation bias.</li>
</ul>
<h4>In-app surveys</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>Conducting in-app surveys is most useful when you just want to get the majority opinion between multiple choices.</li>
<li>Can get a large volume of responses pretty quickly.</li>
<li>Allows you to serve different questions based on different segments.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>No one-on-one interaction; therefore no follow-up questions or clarifications.</li>
<li>It also doesn’t give enough context for the users to provide answers to any open-ended questions in the surveys.</li>
</ul>
<h4>In-app experiments</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>It is the best way to put your ideas to the test as you can gather the actual usage pattern at scale and analyze it for insights.</li>
<li>Allows you to quickly validate a solution within weeks and is also a relatively low-cost option because it’s usually only a prototype being tested.</li>
<li>Fake-door tests, A/B tests, and prototype testings are some of the examples of in-app experiments, which when paired with user interviews and analytics, can help paint the complete picture.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>Biased — Any test needs a random sample within your target cohort, but if the target users are part of your enterprise customer base, there’s a high chance that account managers/CSMs of those enterprise customers would want to know beforehand and control whether or not their customers are part of the experiment. That is mainly because they don’t want any surprises for their customers and do not come across as ignorant when the customer asks about the “feature” in any meeting. They also do not want to expose customers with a churn risk to such tests. So, then if you were to hand-pick customers for tests, it can introduce bias in your result.</li>
<li>The concern also is around the look and feel of the prototype. Since it is only a test, a PM would be justified in not spending a ton of dev time polishing the solution before rolling the test out, however, the sales/account managers would be against putting anything out there that is not up to the standards. As a result, the PM might end up investing excessive resources in refining the prototype.</li>
</ul>
<h3>Potential solutions</h3>
<p>1. First, let’s address the experiment itself. Ensure the UI clearly conveys that it is only a prototype for testing and there’s no commitment that the feature will make it to the generally available feature set. Otherwise, invariably some customers will like what they see in the test and would want to have it available for them.</p>
<p>2. Run your experiment project by account managers before starting. Clearly explain why the target sample users need to be random but be open to excluding the customers who are due for renewal just to avoid serving an experience that might be a bit scrappy on purpose.</p>
<p>3. Provide clarity on what the experiment is about, how long it will run and how will the result be determined to ensure account managers are well-informed, especially if these details become topics of discussion with their customers.</p>
<p>4. Make sure the target for the experiment does not include any new customers unless the experiment is about the onboarding itself.</p>

core/freeform

Product Management Skills

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

In-app experiments in B2B: Navigating challenges in product discovery

Niraj Upadhyay

Read more

Comments