post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Building for change: When transformation and product management meet

Product Development

What are product features? Definition, examples, and 5 types

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Lessons from my first year as a Prod Ops Manager

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

5 ways to invent and simplify as an AI/ML product manager

Niraj Upadhyay

<p>Conducting experiments is an essential part of product discovery for product managers. It goes hand-in-hand with user interviews, when your goal is to understand the user’s behavior patterns.</p>
<p>However, conducting in-app experiments in B2B products, <span style="font-weight: bold;">especially when targeted at enterprise persona</span>, can be quite tricky and challenging. In this article I share some of the challenges I have faced and the lessons I learned.</p>
<h3>User interviews vs. in-app experiments vs. in-app surveys</h3>
<p>Before I talk about in-app experiments, the challenges, and their potential solutions, here’s a quick run-down of three types of validation methods along with some of their advantages and drawbacks.</p>
<h4>User interviews</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>It allows us to deep-dive into a problem with the users with open-ended questions, ask counter-questions, and seek clarifications, leading to an in-depth understanding of the problem.</li>
<li>Provides first-hand feedback with full context.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>Quite a slow process — In the B2B world, especially when you target large enterprises, it is difficult to get enough users of the <span style="font-weight: bold;">desired persona type </span>to spend time with you and give you feedback. It can take weeks on end to gather enough users for interviews. Some teams follow the practice of enticing them with gift vouchers but it’s not always a viable option, especially when you are on a shoestring budget like I am and it also runs the risk of “positive response bias” in their feedback.</li>
<li>Frequently, there’s a notable gap between what users express in interviews and their actual interactions with your product. So, just depending on qualitative data from the interviews has a good chance of being mired with confirmation bias.</li>
</ul>
<h4>In-app surveys</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>Conducting in-app surveys is most useful when you just want to get the majority opinion between multiple choices.</li>
<li>Can get a large volume of responses pretty quickly.</li>
<li>Allows you to serve different questions based on different segments.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>No one-on-one interaction; therefore no follow-up questions or clarifications.</li>
<li>It also doesn’t give enough context for the users to provide answers to any open-ended questions in the surveys.</li>
</ul>
<h4>In-app experiments</h4>
<p><strong>Pros:</strong></p>
<ul>
<li>It is the best way to put your ideas to the test as you can gather the actual usage pattern at scale and analyze it for insights.</li>
<li>Allows you to quickly validate a solution within weeks and is also a relatively low-cost option because it’s usually only a prototype being tested.</li>
<li>Fake-door tests, A/B tests, and prototype testings are some of the examples of in-app experiments, which when paired with user interviews and analytics, can help paint the complete picture.</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>Biased — Any test needs a random sample within your target cohort, but if the target users are part of your enterprise customer base, there’s a high chance that account managers/CSMs of those enterprise customers would want to know beforehand and control whether or not their customers are part of the experiment. That is mainly because they don’t want any surprises for their customers and do not come across as ignorant when the customer asks about the “feature” in any meeting. They also do not want to expose customers with a churn risk to such tests. So, then if you were to hand-pick customers for tests, it can introduce bias in your result.</li>
<li>The concern also is around the look and feel of the prototype. Since it is only a test, a PM would be justified in not spending a ton of dev time polishing the solution before rolling the test out, however, the sales/account managers would be against putting anything out there that is not up to the standards. As a result, the PM might end up investing excessive resources in refining the prototype.</li>
</ul>
<h3>Potential solutions</h3>
<p>1. First, let’s address the experiment itself. Ensure the UI clearly conveys that it is only a prototype for testing and there’s no commitment that the feature will make it to the generally available feature set. Otherwise, invariably some customers will like what they see in the test and would want to have it available for them.</p>
<p>2. Run your experiment project by account managers before starting. Clearly explain why the target sample users need to be random but be open to excluding the customers who are due for renewal just to avoid serving an experience that might be a bit scrappy on purpose.</p>
<p>3. Provide clarity on what the experiment is about, how long it will run and how will the result be determined to ensure account managers are well-informed, especially if these details become topics of discussion with their customers.</p>
<p>4. Make sure the target for the experiment does not include any new customers unless the experiment is about the onboarding itself.</p>

core/freeform

Product Management Skills

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

How to F.A.I.L. better in product by Marc Abraham

10 years Wiser: Lessons learned from scaling Wise by Nilan Peiris

Fighting product-market drift by Dave Wascha

Experiment like a boss by Merissa Silk

How to find out what your customers want by Janet Bumpas

All product management should be focused on growth by Partho Ghosh

Product, dance, and pockets of brilliance by Pippa Topp

A quick look at decision supervision by Lucy Spence

Identify and manage imposter syndrome by Phyllis Njoroge

In-app experiments in B2B: Navigating challenges in product discovery

ProductTank Madrid

ProductTank Montevideo

ProductTank Montréal

ProductTank Boston

ProductTank Ahmedabad

ProductTank Bangalore

ProductTank Berlin

ProductTank Oslo

ProductTank Des Moines

ProductTank Bangkok

ProductTank Abu Dhabi

ProductTank Helsinki

ProductTank Pune

ProductTank Kraków

ProductTank Cardiff

ProductTank Mumbai

ProductTank Johannesburg

ProductTank Tauranga

ProductTank Bielefeld

ProductTank Vilnius

ProductTank Timisoara

ProductTank Stockholm

ProductTank München

ProductTank Düsseldorf

ProductTank Birmingham

ProductTank Norwich

ProductTank Heidelberg

ProductTank Malmö

ProductTank Barcelona

ProductTank Newcastle Upon Tyne

ProductTank London

ProductTank Perth

ProductTank Málaga

ProductTank Tokyo

ProductTank Braunschweig

ProductTank Edinburgh

ProductTank Pittsburgh

ProductTank Vienna

ProductTank Bristol

ProductTank Washington

ProductTank Raleigh

ProductTank Brno

ProductTank Paris

ProductTank Orlando

ProductTank Hong Kong

ProductTank Charlotte

ProductTank Exeter

ProductTank Manchester

In-app experiments in B2B: Navigating challenges in product discovery

Niraj Upadhyay

Read more

Comments