post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Building for change: When transformation and product management meet

Product Development

What are product features? Definition, examples, and 5 types

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Lessons from my first year as a Prod Ops Manager

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

5 ways to invent and simplify as an AI/ML product manager

Founder

Laure Parsons

<p>I spoke to Lean Startup coach Tristan Kromer at Lean Startup Week earlier this month, where he presented a workshop about a “lean experiments toolbox”. Lean Startup Week is a conference for teams and leaders interested in applying a Lean Startup methodology to their product and Tristan helped to organise the Lean Unconference that kicked off the week’s activities. He is also editor and primary author of the <a href="http://www.trikro.com/downloads/playbook">Real Startup Book</a> and runs lean consulting firm TriKro LLC.</p>
<p>The product I work on, <a href="http://usenotion.com/">Notion</a>, is designed to help teams communicate and collaborate on their data so they can align everyone around their goals and experiments. I wanted to find out how Tristan approaches the use of data to guide product process.</p>
<h2>Designing your hypothesis</h2>
<p>Tristan recently published a blog which discusses the idea of ”<a href="https://grasshopperherder.com/assumption-vs-hypothesis-to-the-death/">assumptions vs. hypotheses</a>.” He believes that when just starting, our biggest challenge is identify our own assumptions rather than to build an MVP. I’m interested in <a href="http://www.slideshare.net/LaureParsons/overcome-your-mind-the-psychology-of-better-product-decisions">how cognitive bias can interfere</a> with product decisions, so I asked Tristan how teams can determine if their ideas are being posed as real foundations for experimentation. “An assumption is something accepted as true without question. A hypothesis is a proposed explanation or prediction that is a starting point for experimentation,” he says. “Assumptions are fine if they are made consciously. For example, I have an assumption that Javascript is a thriving programming language. I don&#8217;t really feel like testing that assumption and it&#8217;s very low risk, so I&#8217;ll leave it as an assumption I take for granted.”</p>
<p>However: “If we have an assumption such as ‘Everyone will buy our product’, that&#8217;s obviously very dangerous and probably not true. You can tell the difference because an assumption is usually quite vague and goes unchallenged as the truth. A hypothesis must be very specific, should make a falsifiable prediction, and should be discussed in the context of an experiment.”</p>
<h2>The metrics that matter</h2>
<p>Experiments generate data for a team to decide whether to move forward or to change direction. What does he feel are the most important metrics to inform these decisions? Below are four of the most important metrics in his practice:-</p>
<ul>
<li><b>Iteration Velocity</b> &#8211; How quickly is the team running experiments? (see more <a href="https://grasshopperherder.com/the-complete-team/">here</a>)</li>
<li><b>Knowledge / Experiment Ratio</b> &#8211; What % of those experiments are resulting in knowledge?</li>
<li><b>Team Morale</b> &#8211; How does the team get along and work together from a qualitative standpoint?</li>
<li><b>Compass</b> &#8211; Does the team have a product dashboard with actionable metrics?</li>
</ul>
<p>The Lean mantra is &#8220;fail fast&#8221;, but this can feel uncomfortable to many product people. The principle is rooted in the assertion that it is better to know if a product hypothesis fails to meet customer needs, so you avoid making things that don’t add value. Tristan comments: “The best way to avoid failure is to stop defining an invalid hypothesis as failure. It&#8217;s not failure, it&#8217;s learning. The only way we truly fail as a startup is if we&#8217;re not learning.”</p>
<h2>Does Lean really work?</h2>
<p>Tristan cites AirBnB as a true lean success story. As detailed in <a href="https://growthhackers.com/growth-studies/airbnb">this GrowthHackers case study</a>, AirBnB endured two years of early-stage development without seeing much traction. Like many startup founders, Brian Chesky and Joe Gebbia set out to solve a problem for themselves (they were struggling to pay the rent on their San Francisco apartment, and decided to engineer a better solution). But early on, customers were reluctant, as were some high-profile investors.</p>
<p>By using lean principles and relentless hustle, the company began to run experiments to increase confidence for reluctant guests and hosts. In one of their most famous experiments, AirBnB paid professional photographers to visit hosts and make listings more enticing. Customers responded and listings with the professional photos performed 2.5 times better. Says Tristan: “AirBnB is one of the most impressive success stories I&#8217;ve seen and the way they implement lean thinking consistently and even at scale is truly impressive.”</p>
<p>What then are some of the most successful lean experiments Tristan has seen? While many of the best results are protected by NDAs, one great example he has worked with is the now-shuttered music startup <a href="http://tentune.me/">tentune.me</a>.  He says: “They invalidated their business model and iterated incredibly quickly with paper prototypes in a coffee shop, but it doesn&#8217;t make for a sexy lean startup story because they ultimately decided the business was a bad idea and shut it down. I&#8217;m more impressed with people who have the courage to shut down bad ideas than people who were lucky enough to succeed with their first experiment.”</p>
<h2>Start at Learn</h2>
<p>“Build-Measure-Learn” may be the most famous of the Lean Startup mottos, but many teams struggle with how to implement experiments and data tracking in their organisation. What is Tristan’s advice is for teams who  want to start using lean principles? “Start at Learn; we shouldn&#8217;t start at Build,” he says. “We should start by asking, ‘What do we need to learn? What&#8217;s the riskiest thing about this business?’. Once we have that learning goal, we can go backwards to measure and figure out exactly what data would help us learn or answer a specific question. They we go to Build and build an experiment to help us acquire that data. So we go through the loop backwards before we go forward.”</p>
<p>Experiments can sometimes be subject to confirmation bias by the sponsor or proposer of the project or feature, or if the proposer has already decided they’ve come up with the right solution, it can be hard to invalidate. Tristan has this advice on combatting this issue: “The most important thing with buy-in on experiments is to get agreement on the fail conditions before running the experiment.”</p>
<p>He adds that a  good lean team will run an experiment to try and (in)validate the idea. But if the sponsor hasn&#8217;t agreed to the fail condition, their ego and reality distortion field will likely get in the way of accepting any negative results. “If we work with the sponsor as part of the team to agree on the criteria that would invalidate the hypothesis before the experiment, it&#8217;s very, very hard to ignore the data. It forces the sponsor into a position where they have to disagree with themselves in order to disagree with the results. That&#8217;s a cognitive dissonance that most people will try to avoid.”</p>
<p>I’m looking forward to sharing more takeaways from Lean Startup Week. For more insights into running lean experiments, Tristan’s recent article, “<a href="https://grasshopperherder.com/what-type-of-lean-startup-experiment-should-i-run/">What type of Lean Startup Experiment Should I Run?</a>” is worth a look.</p>

core/freeform

Lean Product Management

Hooked! Connect, engage and inspire with storytelling by Yamini Naidu

Lean Product Management by Itamar Gilad

Evidence-Based Product Backlogs, by John Pagonis

How Design Thinking, Lean, and Agile Work Together by Jonny Schneider

How Medicine and Research Translates to Patients by Shree Patel

How to Influence the Business by David J Bland

Lean, Agile, & Design Thinking by Jeff Gothelf

Lucia Adams – How to Innovate in Organisations That Don’t Like Failure

Tom-Tom Erik Isaksen – Disrupt Yourself or Die

Debbie Wren – Scaling Autonomous Teams

Lean experiments: Tristan Kromer on better product process

ProductTank Madrid

ProductTank Montevideo

ProductTank Montréal

ProductTank Boston

ProductTank Ahmedabad

ProductTank Bangalore

ProductTank Berlin

ProductTank Oslo

ProductTank Des Moines

ProductTank Bangkok

ProductTank Abu Dhabi

ProductTank Helsinki

ProductTank Pune

ProductTank Kraków

ProductTank Cardiff

ProductTank Mumbai

ProductTank Johannesburg

ProductTank Tauranga

ProductTank Bielefeld

ProductTank Vilnius

ProductTank Timisoara

ProductTank Stockholm

ProductTank München

ProductTank Düsseldorf

ProductTank Birmingham

ProductTank Norwich

ProductTank Heidelberg

ProductTank Malmö

ProductTank Barcelona

ProductTank Newcastle Upon Tyne

ProductTank London

ProductTank Perth

ProductTank Málaga

ProductTank Tokyo

ProductTank Braunschweig

ProductTank Edinburgh

ProductTank Pittsburgh

ProductTank Vienna

ProductTank Bristol

ProductTank Washington

ProductTank Raleigh

ProductTank Brno

ProductTank Paris

ProductTank Orlando

ProductTank Hong Kong

ProductTank Charlotte

ProductTank Exeter

ProductTank Manchester

Lean experiments: Tristan Kromer on better product process

Laure Parsons

Read more

Comments