In his keynote session at #mtpcon London 2022, Itamar Gilad, veteran product manager, coach and speaker with over 20 years’ experience at Google and Microsoft, examines how adopting Evidence-Guided Development can result in more reliable and repeatable decisions that empower teams to deliver greater value.
Watch this video or read on for key highlights from the talk.
- Evidence Guided Development allows teams to make more consistently better, data-informed decisions over time
- The Confidence Meter gives teams a way to quantify the 2nd metric of the ICE scoring model, assessing the amount of analysis or discovery done to give them security in their impact assessment
- By changing how we think about delivery tasks and projects to include data collection and learning, we can build empowered teams that are part of the strategy and planning process
Itamar begins by reflecting on the changes he’s seen in software development since working as an engineer in 2001, shortly before transitioning to a position in product. While today agile teams may launch multiple times a day, back then Itamar says, waterfall development resulted in ‘launching maybe three times a year’. Now, design and user data research are a part of the product and common across companies of all sizes but 20 years ago, these assets were exclusive to the richest companies. What hasn’t changed, Itamar explains, are the fundamental product decision-making practices, specifically deciding what teams should build. While the tech industry may have rebranded these decisions with new terminology like ‘quarterly planning’ stages and short-term roadmaps, the practice of talking to customers, looking at the market, competitors, stakeholders or managers remains the same, as does the inevitable meeting when ‘the most powerful people in the organisation will decide what to build next’.
‘In meetings of this kind, Itamar says, people debate ideas about new, expensive project processes, which are often fundamentally flawed. However, data shows the greatest revenue and usage come from existing company-owned technology supported with incremental improvements, rather than new initiatives. Part of the problem, he explains, is that we’re limited by our judgement, which is unable to compute and fairly assess 40 or 50 possible ideas, each with complex impacts, unknowns and ambiguities during a time-pressured planning session. Often in such cases, we select the one idea that we believe is a must based on cognitive or confirmation bias, risk-aversion mindset, narrowed framing or sunk cost fallacy, while ‘of course the reality is different’.
Itamar introduces Evidence Guided Development, one of the advancements of the last 20 years, which has given teams the ability to do research with ‘powerful tools for data analytics and data science’ resulting in ‘fact data’. Itamar discusses the GIST-Framework (Goals, Ideas, Steps, Tasks), which allows teams to support or refute that ‘image of the world’ and be empowered in their decision-making ‘to make more consistently better decisions over time’.
Value-modelling
While individuals often pursue personal, siloed goals based on their role or ambition, Itamar says, evidence-driven companies take a different approach, modelling their business on the way they create value for customers and then constructing goals. In such cases, ‘the entire organisation is about finding that product that optimises for this give and take’, delivering value while receiving revenue or data.
The value-modelling process, he explains, relies on a single key business optimisation metric that the business can focus on. Other high-priority business metrics are tracked as health metrics, while a North Star metric, one that measures the business value delivered to the market, is also tracked in parallel to the key optimisation metric.
‘Once we find these two top-level metrics the most important metrics that we have we can start breaking them into subparts because it’s really how to work on them directly’. These subparts appear as two trees, with ‘two metric hierarchies’ of submetrics. Once this work is completed, goals can be assigned to or shared across teams. In total the company should have 3-5 objectives, each with up to four key results forming their OKRs.
What teams need, Itamar says, is a GPS to guide us now ‘we know where we want to go, we can start talking about how to get there […] there are many roads, and many of them would not lead us directly there’.
Analysing ideas
‘Ideas come from everywhere and all the time’, Itamar says, advising people to judge each fairly, and avoid assuming the idea that came from the highest paid person will be the best. Evidence-Guided Development takes us through stages of evaluation, validation, re-evaluation and delivery. The evaluation and validation stages take all the ideas you have and reduce them by analysing their likelihood of delivering value. Itamar points to data scientists that follow AB experimentation in companies, such as Amazon, Microsoft and Airbnb, which found that statistically only 1 in 10 ideas were good and resulted in value-add. ‘90% of what’s right now in your roadmap and in your product backlog – you shouldn’t be doing’.
In order to evaluate ideas in an objective and fast way, Itamar recommends using the ICE scoring model, which allows teams to reduce and prioritise ideas by multiplying 3 numerical values assigned to the following criteria:
- Impact, which quantifies if a product is successful and how impactful it will be on the key metric that is being focussed on.
- Confidence is the measure of how certain you are that the project will be as impactful as you believe it should be.
- Ease, which looks at how difficult an idea or project will be to deliver.
Itmar describes in detail ‘The Confidence Meter’, a tool he developed to allow teams to assess ideas strategically and repeatedly based on the type of analysis and supporting evidence a team has. By using this repeatedly and fairly with quantifiable confidence, teams can decide whether to move forward with a decision, park it or pivot it. ‘We’re combining building with learning’ Itamar says, ‘So instead of constructing the project along a series of engineering Milestones, we’re constructing it along a series of learning milestones and then of each one of those we learned something and we can pivot the idea a bit’.
Building empowered groups
Itamar points to how in traditional development there is a ‘clear division between the planners, the people who create a strategy, and the roadmap’. He explains how with evidence-guided development, everyone, regardless of role can be grouped and tasked as part of projects to experiment and analyse data, explaining, ‘It’s not just coding and designing it’s actually making the learning happen. So we kind of convert our developers into researchers into discoverers in a sense and that step is connected to an idea and an idea that they understand I do that’.
Teams that follow evidence-guided development become empowered as they gain trust of senior leadership members, Itamar explains. Trust, he says, is earned by transparently showing how your ideas and experiments are validated and backed with confidence, in contrast with or challenge to cognitive biases, ‘it’s much more interesting and engaging for them and they’re much more likely to actually take part in this in a constructive way. Not just guiding you and trying to tell you what to do’.
The product manager role is more meaningful than just being a backlog manager or roadmap administrator. As Itamar says, product people are ‘driving this entire process […] and helping discover the product that creates the most value for the market and for the company’.
There’s more where that came from!
Explore more #mtpcon London 2022 content or use our Content A-Z to find even more product management insights.
