In this #mtpcon Digital Americas session, Cris Valerio, Product Manager at Instacart, shares her thinking on why user insights are important for making product decisions and provides some guidelines for gathering the right user insights.
Watch the session in full or read on for the highlights:
High-level reminders
“You will spend the vast majority of your product career making decisions based on user feedback,” Cris explains.
Making great decisions isn’t just about talking to users. It’s about defining the business decisions, gathering actionable insights, and understanding the evidence needed to make decisions.
To do this well, you must bring your full self to the table. Not just your analytical self, your empathetic self or your creative self, but all of them.
Defining your decision
The first step is to define your decision. User insights aren’t helpful unless you understand the problem you’re trying to solve. You need to define your user problem and ask, what is your product trying to solve for the user? What assumptions are you operating under? How confident are you in those assumptions?
It’s important also to define the business problem, even if this shouldn’t be your first area of focus. For the business problem, you might ask, what is the organisational goal, and how does your team or product contribute towards this?
User problem
Cris provides the example of vehicle fleets from her time at Uber. These were a group of accounts under one organisation or person, typically the owner. Uber noticed a big overlap in markets where there were lots of fleets and where there was a high spend on drivers. Uber also realised its incentive strategy didn’t account for the different behaviours of fleets.
The business problem was fleet drivers didn’t have a reason to follow Uber’s suggestions of where to move to in order to pick up fares because they were paid according to a different structure. Cris’ team needed to gather actionable insights to understand and then solve the problem. She provides a few ways that they went about gathering insights.
- Qualitative methods: Interviews, rapid prototyping, listening tours, forums
- Quantitative methods: Surveys, scraping and data analysis
- Systems: Driving a culture of insights across your team and company
Qualitative
Interviews
When conducting interviews, it’s important to remember it’s not just about talking. You may need to use different methods, such as games, depending on the situation. Consider what the goal is and match the method accordingly.
Some of the ways to engage users and not just ask generic questions are:
- Echoing: This helps to clarify by repeating what the user said with a question mark. For example, you can echo “this is not how I do things” with “how do you do things?”
- Boomerang: Deflecting the question by sending them a question
- Columbo: Asking partial questions, then wait for the subject to rescue you
Context is also critical when discussing interviews.
Rapid prototyping
With rapid prototyping, start by asking yourself, what are the riskiest assumptions our team is making? Then test against those. Whatever prototype you structure, ask yourself, “What will we do with this information once we get it?”. If you have a straightforward way to make that information actionable, you are on the right track.
Events (virtual and IRL)
Events are also a great way to gather user insights. Some examples of what to do are hosted forums, listening tours, on-site intercepts, and team customer calls or care listening sessions.
Cris points out that qualitative data is still data. You’re looking for directional, not comprehensive, evidence. It’s important to remind your teams and leads that extreme users and examples help to push your thinking, but the solution should not be designed for them and always involve your team in the process.
Quantitative
Cris provides some helpful hints for working with surveys.
Surveys: Helpful hints
- Be sure the information you gather is actionable
- Segmentation matters. Get good quant research
- Remember that data can be deceiving, yet more easily accepted.
- Often surveys can have self-selection bias
- What people say and what they do and believe are separate things
- When in doubt, follow up quantitative with qualitative research
- Attribution is hard, but it can be very helpful to do pre and post-launch surveys to gain directional evidence on how customers respond to products
Data analysis
When analysing data, it’s crucial to remember that quantitative data will always tell you what is happening but rarely will it tell you why and how. Different metrics impact different outcomes, so be sure of your objective function and make sure your team understands it as well.
You should also make an effort to understand the type of data you are looking at: vanity data, direct vs proxy, individual vs aggregate, magnitude vs ratio.
Systems
Cris explains that building a culture of insights is critical, and you should focus on outcomes, not outputs. Some of the ways to build this culture are:
- Empathy exercises
- Integrating methods into product development lifecycle
- Regular calls to customers
- Using the product with your team
- Guest speakers from Ops, Support and Comms
The key takeaways from this talk are that getting your team involved (operating teams AND executives/leads) when gathering user insights is important as you try fostering an evidence-driven culture.
