When Customer Feedback Leads, Positive Metrics Follow

It’s an inarguable fact: Metrics are useful. But they’re missing something—they don’t tell you the whole story. Quantitative metrics can tell you that a problem exists, but they can’t tell you what exactly the problem is or why it exists. Product managers are often tasked with increasing KPIs on their products, but they can’t do that by simply analyzing numeric data on its own.

To uncover the full story, it’s up to product managers to gather and interpret customer feedback and reveal what the metrics can’t—why the KPIs are changing (or not). In other words, you have to gather qualitative data.

Peel Back the Layers of Customer Feedback

Customer feedback has several layers. As I’ve learned over time in talking to customers, it’s often not about what people say, but rather what they don’t say. For example, when customers are struggling to accomplish their goals in your product, they will often come to you with specific solutions as how to solve the problem. It would be easy to take that one piece of feedback, add it to the roadmap and implement it directly as described. But it’s important to understand that people’s solutions are often based on their own personal preference or point of view, and that may not align with your entire customer base.

If you implement only one new tactic this year, make it this: Get out of the office and observe your customers using your product. That is the only way you will truly be able to figure out the true nature of your customers’ problems, why they exist, and how to solve them. Schedule an interview or user test with a new customer once a week (or at least every two to three weeks), and dig deep into the issues you discovered while reviewing your metrics.

It’s in these user tests that you are able to see problems occur in their natural environment. Your customers will show you their workarounds. They will tell you stories about how they “make do” using other products. You will observe their frustration as they try to do something your product doesn’t allow, or as they get confused about what to do next.

Proving That the Problem is Real

Since customers tend to start out by telling you the solutions they want, getting to the bottom of the problem is not always as straightforward as we’d like. Thankfully, there are three rules that can help you navigate the conversation to get you the answers you need.

1. Look to Answer the Four W’s

In this case, the four W’s we are referring to are Who, What, Where and Why.

• Who has the problem? Which customer group are you really focused on? Most products have more than one persona. Thus, it’s important that you understand which persona you’re speaking to.
• What is the nature of the problem? You want to get to the point where you can explain the problem simply, preferably in one to two sentences. Additionally, you need to be able to communicate to your team how you know that this particular customer is experiencing this particular problem. So, ask your customer to walk through the product and make note of where they get stuck.
• Where does the problem arise, or in what context do your customers experience the problem? When you work to come up with a solution, you’ll want to make sure you do so in the areas your customers are most negatively affected.
• Why is the problem worth solving? Is this problem affecting a large portion of your customer base? And if so, what does the problem mean for your brand, revenue and trust in your product.

2. Avoid Asking Leading Questions

Leading questions will get you the answers you want, but may not get you the answers you need. They are questions that tell the listener the answer you want them to give you.

Generally speaking, there are two ways to spot leading questions. The first indication is that the question can be answered with a yes or no. These closed questions seem pretty harmless, but they tend to be asked with a tone that puts emphasis on either the ‘yes’ or the ‘no’, thereby giving the listener the indication as to which you want them to say. The second indicator of leading questions is that they tend to have a solution in the question, e.g. “Would you use product X more if it had feature Y?”

3. Seek to Disprove Your Assumptions

When building products, we make assumptions regarding what we believe to be true about our product and our customers. It’s appealing to try and confirm that we’re right, but when going out and talking to customers, you’ll actually get to the bottom of problems (whether or not they are real and worth solving) much more easily if you try to prove yourself wrong. Ultimately, this will help you avoid trying to solve the wrong, or less important, problems.

In your interviews, the first step is to avoid bias in your questions. Particularly Confirmation Bias, which happens when you (possibly unconsciously) interpret what your customer says and does in a way that confirms your preconceptions. This also shows up when you only ask questions based on what you expect to see. Additionally, you should watch out for Diagnosis Bias, which is when you form an initial impression and are then not open to other possibilities, either based on your initial assumptions or based on something that happens early on in the user test.

Secondly, ask questions to get an opposite answer than what you think to be true. So, if you believe a feature is good, ask your question in a way that will help you to determine how it could be bad. “If feature X fails, what are your workarounds?”

Back to the Beginning—Turning Feedback Into Metrics

As I said in the beginning, quantitative metrics can tell you that there is a problem, but they won’t tell you what that problem is or why it exists; qualitative data is how we get real understanding. But when it comes to measuring whether or not we truly solved the problem we set out to solve, it’s important that we bring it back to numbers. So as you decide on product changes based on what you learned, ensure that those changes are measurable by the KPIs you set out to change.

As your customers experience your solution, your metrics will adjust to reflect the impact of your changes. If you’ve made changes that truly solve the problem, you’ll see a positive impact in your metrics. If you missed the mark, as sometimes happens, your numbers will show that too. Either way, this step is really just taking you back to the beginning, as this is an ongoing feedback loop.

As product managers, we need to constantly evaluate our product and repeat this cycle over and over: Review metrics to discover where to focus our time, talk to customers to understand the true nature of their problems, and then measure whether or not our changes had a positive or negative impact (and by how much).

Metrics are the beginning and end of the story. They indicate to you that you need to change something, and they tell you when you can move your focus to something else. But in the middle, getting quality customer feedback is essential to making decisions that will move your metrics in the direction that will positively impact your customers and business. More importantly, those conversations lead to solid understanding of your customer. And understanding your customer and their needs is what leads to increased metrics all-around, no matter what conversion metric you measure.

So while metrics are incredibly important, don’t forget that you have humans using your product. In the words of Harvard Professor Youngme Moon, “If we pay attention to things that we can measure, we will only pay attention to the things that are easily measureable. And in the process we will miss a lot.” We miss the human experience.

So, what customer will you interview this week?