post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

Sunday Rewind: Whose job is it anyway? The rise of Product Ops and why it deserves to be an independent function

Guest Post

Creating metrics that matter

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Dhia Eddine Bouali is a results-driven Product Manager with a strong focus on B2B products. He has played a key role in driving growth and improving key metrics in various contexts. Also, he has shared his expertise in product management as a mentor and intervener, guiding and inspiring future product generations.

Dhia Eddine Bouali

We&#8217;ll explore why using solely on data might fall short for small businesses and find the perfect balance between <a href="https://www.mindtheproduct.com/an-approach-to-a-data-driven-product-launch/" target="_blank" rel="noopener">data-driven</a> insights and examine some alternative approaches.
<h3>1. Minimum detectable effect</h3>
Finding value is tough, and it requires trying things out, making changes and waiting. The difficulty varies depending on how many people use a solution.
If a solution has high traffic, you can quickly see if changes make a difference,ven small improvements can be spotted. If a solution has low traffic, it&#8217;s much harder to run experiments. It might take a long time, possibly months, to figure out if a change is working, andnd you can only really notice big changes.
So, more users make it easier to test and spot small improvements, while fewer users mean it takes longer and you can only see big changes.
To put it another way: when lots of people use your solution, you can notice even tiny improvements.
These small changes can help you get bigger and find even tinier improvements, which then lead to more growth, and the cycle continues.
When you&#8217;re trying to test an idea, you should plan how long your test will last in advance. Imagine you toss a coin five times, and you get 4 heads and 1 tail.
Based on just this, you can&#8217;t say for sure that heads are more likely.
But if you do the same thing 100 or even 10,000 times, your results will get closer to a 50/50 split between heads and tails. This is because, statistically, there&#8217;s a 50% chance of getting either result when you roll a fair die.
The big question here is: how patient should we be? There are some math formulae that can answer this, but there&#8217;s an easier way. You can use an online tool that figures out the right sample size you need<a href="#cmnt1">[a]</a><a href="#cmnt2">[b]</a> (here are some examples: <a href="https://www.abtasty.com/sample-size-calculator/">https://www.abtasty.com/sample-size-calculator/</a>, https://www.evanmiller.org/ab-testing/sample-size.html )
If you have a website with low traffic, you&#8217;ll see that it could take many months just to do one experiment and show if there&#8217;s a connection between what you tried and the change you see.
<h3>2. Example</h3>
Let&#8217;s use an example<a href="#cmnt3">[c]</a><a href="#cmnt4">[d]</a> to understand the minimum detectable effect better. Consider a scenario where we are managing an e-commerce website aiming to boost its conversion rates: encouraging visitors to make a purchase.
<ul>
<li>We have a conversion rate of 2% on the product page</li>
<li>We want to find out if there&#8217;s a change of at least 0.25% (either less than 1.75% or more than 2.25%).</li>
<li>We want to be pretty sure, let’s say with a confidence rate of 95%, that our results are correct. <img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2024/01/5d10a197-eb42-45cc-9690-70175237e15f_Capture20d_e_cran202023-12-0520a_2014.32.02.png" />Source image: https://www.evanmiller.org/</li>
</ul>
To do this, we&#8217;d need about 167,510 visitors (83,755 multiplied by 2).
Now, if your webpage gets around 8,000 visitors per week, it would take you more than 20 weeks to finish this test and be sure about the results.
But, if you have 200,000 visitors per week, you can finish the test in less than a week and be confident in the results.
This matters a lot for <a href="https://www.mindtheproduct.com/enterprise-b2b-product-management-by-jason-knight/" target="_blank" rel="noopener">B2B solutions</a> as they need more time to run their tests and be sure about what works.
Let&#8217;s clear up a small but important thing: there&#8217;s a difference between absolute and relative changes.
<ol>
<li>Absolute change: If your conversion rate is 2% and you make an absolute improvement of 0.25%, your new conversion rate becomes 2.25% (2% + 0.25%). 
</li>
<li>Relative change: But if you make a relative improvement of 0.25%, your new conversion rate becomes 2.005% (2% + 2*0.25%).</li>
</ol>
<h2>3. Alternatives</h2>
Facing this problem, B2B websites with not many visitors can try some other ways:
3.1. Before and after testing: 
If you can&#8217;t do tests at the same time, you can compare what happened before you made a change and what happened after. But this method is simpler, and you need to be really careful to make sure that the changes you see are because of what you did, and not because of other things. You have to consider things like:
<ul>
<li>If your company was part of a big event or show.</li>
<li>If the marketing team changed their budget and tried a new way to reach people.</li>
<li>If a famous person shared a link to your site.</li>
<li>If there are certain times of the year when your business is busier or slower.</li>
</ul>
It can be tricky to be sure if the changes you see are because of what you did or because of other factors.
3.2. Data-informed decision-making: 
If you can&#8217;t be 100% <a href="https://www.mindtheproduct.com/data-driven-product-management-by-matt-lemay/">data-driven</a> due to the lack of data, you could always be data-informed by combining different insight types into the decision-making, such as data, experience, intuition, best practices etc.
To illustrate the data-informed mindset, let&#8217;s take an example:
Imagine you&#8217;re a coach of a football team, and you rely on statistics. You&#8217;ve noticed that your best player often scores in the last 10 minutes of a match. Today, in an important game, he has a terrible first 30 minutes, and you&#8217;re contemplating a decision. You have two choices:
1. On one hand, you could say, “He usually scores in the last 10 minutes, so I&#8217;ll base my decision on the data and give him a chance until the end of the match.”
2. On the other hand, you might think, “I don&#8217;t have all the data; I&#8217;m missing the ratio of matches in which he scored in the last 10 minutes, considering he had a lacklustre first 30 minutes. I will introduce other factors into my decision-making process, such as experience, and intuition.”
This latter choice is the data-informed mindset, which takes missing data into account and acknowledges that it&#8217;s not always possible to rely 100% on data in a purely data-driven process.
3.3. Configure the confidence rate: 
You can also lower the confidence rate to 90%, for example. This means you might be a little less sure about the results, but it can help you finish your test faster.
<h3>Conclusion</h3>
Based on what we talked about earlier, for B2B websites with limited traffic, relying solely on data may not be sufficient for effective experimentation and value discovery. Since the sample size is smaller, statistical significance is harder to achieve, making it imperative for these businesses to recognize the limitations of their data and explore alternative methods of experimentation.
Contrastingly, larger B2B solutions enjoy the advantage of significant data volumes. providing them with a statistical robustness that allows for more confidence in their experiments.

core/freeform

How to lead data-centric product teams: Jon Mora (Chief AI Officer, Zefr)

How to be a better data-driven product manager

User insights for product decisions by Cris Valerio

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

B2B solutions between data-driven approaches and intuition

Dhia Eddine Bouali

Read more

Comments