Booking.com is a European company with a reputation for running experiments in a sophisticated way. Lukas Vermeer is a senior product owner at Booking.com and one of the key players who established not only the tooling, but more importantly the culture behind experimentation at Booking.com. The title of his keynote at MTP Engage refers to an academic paper published by Lukas and his colleagues, but his MTP Engage talk focuses on the cultural aspects of experimentation at Booking.com.
Ideally, tricky product decisions are not just left to product managers or the HiPPOs in the room, but supported by meaningful experimentation.
The problem is that often product managers either don’t feel empowered to run experiments the way the would like to, or they find there’s difficulty in getting the rest of the organisation to understand and believe the results when they run them.
He describes how a centralised and standardised system is a necessary basis for independent, decentralised decision-making.
He emphasises the importance of formulating and testing hypotheses and how a major part of running experiments is challenging assumptions, particularly if your product team is different from the target audience.
Lukas recommends ranking assumptions in order to identify and test the most risky ones as you move forward. He shares a personal story that illustrates how this can lead to completely different solutions from those you in had in mind when setting off to optimise a particular aspect of your product.
