Getting Value From Customer Testing

A customer testing group can provide value in a variety of situations, ranging from a gut-check to getting long term deep insight. Here’s a guide on how user testing and consultation, alongside a tiered approach to segmenting your customers, can add value to your role as a product manager.

Continuous User Testing & Consultation

When solving customer problems like growth, retention and usability, it’s important to have a group of users to poll quickly for their opinions. At a previous company we had a CEO that decided we should engage our users via push message several times a day because we saw higher engagement when we did this. Simple right?

This seemed excessive to me, but I didn’t have great data or customer sentiment on hand to push back. After speaking with only a handful of customers from different age groups and different engagement patterns I found that we didn’t need to be as conservative as I had argued. The insights were surprising and within a day I had great data to bring back to the business to use to start experimenting with an approach.

The Tiered Approach

Discussions like this come up on a daily basis and it’s hard to dig deep into every little problem you want to solve. Think about the time you spend as a product manager on a graduated scale. It’s important to keep communication with users tactical and focused. Just as you have close friends and general acquaintances, you should group the users that you reach out to. Here’s a framework that can be tailored to the needs of your organization:

Group 1 – Infrequent Contact (~2000 people)
Think of this as your catch-all group for people you’ve engaged in the past. These can also be non-customers you poll in a blind study.

This group is great for:

• Competitive research
• Early rollout of new builds and features
• Infrequent surveys, NPS, awareness

Group 2 – Advisory Committee (~200 people)
These are generally users that have opted into surveys in the past, submitted feature requests or bugs or even users culled from a Tweet, comment or review of your product.

This group is great for:

• Pool of users for testing
• Frequent surveys
• Prototypes
• Sending and auditing screenshots of near-term releases
• Feature usage data

Group 3 – Personal Group (~20 people)
This is your group. If you are on a team each of you should have your own group of 15-20 users that you have constant contact with. All of these groups should have a healthy mix of demographic, but this group should have the most variance.

This group is great for:

• Questioning the validity of new features and the removal of old ones
• Product health of patient questions
• Sharing rough prototypes
• Beta testing

Each group is not mutually exclusive, but remember that you want to limit your communication with everyone except Group 3, so that your response rates are healthy. Feeling out how often that engagement should happen comes with time.

Constant Grooming

The beauty of this tiered approach is that you can generally graduate users from one group to the next (or skip them ahead). Don’t forget that you need to constantly groom this collection of users. Ensure that these users are of varied demographics, have different use cases and are a mix of old and new customers (this is where the hard work comes in).