post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Creating metrics that matter

A case study: Launching a product process in a project-based company

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Taza’s focus for the last 10 years has been turning concepts into commercially successful products.  He’s led innovative, high profile launches across a number of the biggest names in the UK. After having worked at Computacenter and BT in product management roles across both B2B and B2C, he started his own business 4 years ago to focus on helping businesses exploit new product and service opportunities. His approach concentrates on working across the organisation to help drive pragmatic solutions…with a flair for creative solutions to balance the expectations of immediate results, perfect customer experience and minimal use of the available resources!

Taza MohammedBhai

Anyone who wants to get a new product or service to market must be confident that what they&#8217;re selling will be successful. The process of building confidence involves work  &#8211; before the concept is decided through market research to identify the opportunity, during the build with research to gauge customer reaction and again once the product is ready for use.
<img loading="lazy" decoding="async" class="aligncenter size-large wp-image-8850" src="https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3-1024x570.jpg" alt="A &quot;digital mannequin&quot;, as a placeholder for a real human customer in testing processes." width="1024" height="570" srcset="https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3-1024x570.jpg 1024w, https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3-300x167.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3-768x428.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3-285x160.jpg 285w, https://www.mindtheproduct.com/wp-content/uploads/2016/07/test3.jpg 1600w" sizes="(max-width: 1024px) 100vw, 1024px" />
I want to focus on this last element: how can you work with real customers to establish whether your product will be successful, and work out what elements need to be addressed before you launch? This effectively is beta testing, trials of a pre-released product by real users, in situations as close to real-world applications as is possible. To maximise the benefits of customer testing, consider these four fundamental questions you need to answer.
<h2>What are you hoping to learn?</h2>
Running a customer test activity is time-consuming. It’s a major undertaking to recruit an audience and then ensure that you&#8217;ve got them engaged and using the product. Once you&#8217;ve got them engaged, you then need to collect the critical information that will tell you whether the product has performed successfully&#8230;which means persuading your audience to actively participate in giving you feedback.
Before starting customer testing, it’s important to understand what you want to prove or learn, which may vary depending on your perspective. For example:
<ul>
<li style="font-weight: 400;">If you&#8217;ve invested or designed the product or service, you want feedback on its market fit &#8211; how well it meets the needs of your audience, and which user group it resonates with most.  You want to know whether your product is something that consumers can see benefit in and assign value to.</li>
<li style="font-weight: 400;">If you&#8217;re responsible for managing customer service, then you&#8217;ll want to identify any bugs that may have been missed in testing. Considering all the variables seen in real life, 100% testing simply isn&#8217;t practical or realistic.</li>
<li style="font-weight: 400;">If you&#8217;re managing the development team, you&#8217;ll want to identify any new possible features or improvements before the product ramps up to a wider audience.</li>
<li style="font-weight: 400;">If you&#8217;re in marketing or sales, you want to create a distinct group of loyal customers who will become evangelists for your service and give you onward referrals, as well as discovering the most effective messaging to use for the target audience.</li>
</ul>
So, to start with, it&#8217;s worth considering why you want to run the customer test so that you understand what your priorities are, and that you gain the maximum benefits from the activity.  If your objectives aren’t clear, or could be achieved by another route – perhaps by using internal test teams to find bugs rather than looking to customers to identify them &#8211; then customer testing may not be appropriate.
<h2>What information does your customer testing need to provide?</h2>
Understanding why you’re running customer testing is about asking critical questions before investing your time and effort. The same is true for understanding what the key information is that you want to measure. Are you exploring which parts of the service are most attractive? Testing the price points? Is it about validating how customers use the product in real life?
Perhaps the easiest way to tackle this challenge is to consider these steps:
<ul>
<li style="font-weight: 400;">Having understood the reasons for the trial, make sure you know exactly what information will inform your decision making. For example, if you’re assessing whether customers like a tweak to the service (consider the new Evernote interface) perhaps you can consider what new features they use, how new navigation affects behaviours and whether customers prefer it to the old interface.</li>
</ul>
<ul>
<li style="font-weight: 400;">For each of these pieces of information, you then need to consider how you will capture feedback. Will you monitor product usage via click-throughs and page views versus the original version? Or will you need to collect direct customer feedback from focus groups or survey tools? By asking the right questions, and guiding answers that assist your analysis, you are much more likely to get results that help you draw tangible conclusions. 
It’s worth investing time and effort to ensure that you ask the right questions of your audience. Take a look at this <a href="https://www.surveymonkey.com/blog/2015/02/11/5-common-survey-mistakes-ruin-your-data/">Survey Monkey blog</a> to understand the kind of mistakes that can be made. Teams that regularly do UX or customer research can help with balanced questions that will help you really understand the issues with your products.</li>
<li style="font-weight: 400;">It&#8217;s now time to prioritise &#8211; think through who your trial users should be and how much feedback you’re likely to get from them. Some measures may well be “nice to have”, but not at the expense of really important metrics.</li>
<li style="font-weight: 400;">Finally you need to be absolutely clear about what marks success; specifically what scores must be achieved on each of the key measures you’ve identified.  This is something that must be considered in advance; ideally you should set these targets right up front when you started setting out the vision for the new product or service. It&#8217;s much easier to remain objective before you start a trial, especially if you have multiple teams involved; put another way, it’s nigh-on impossible to remain unbiased once the trial has started and the data collection begins.</li>
</ul>
If you fail to set measures then the most likely outcome is that the service launched fails to deliver a decent return on investment. For example, I decided to add a security software subscription to a premium support offering I managed – it took time to train our field engineers to sell the product and major investment in designing systems to record and communicate the sale to the third party that provided the software. However the project failed because engineers ended up offering free security installs for the majority of customers, rather than the paid-for equivalent. We had ignored the warning sign of the poor score we received for perceived value of the software because we had thought that the positive reactions in a number of other areas would compensate for this.
In summary, a clear set of measures and targets means you&#8217;ll know what success looks like, and that you have a clear understanding of what information you need and how to get it. This should also trigger the conversation about what happens if you don&#8217;t hit the metrics. Will you allow some flexibility if a few of the success criteria aren&#8217;t met, or are these absolute minimums that must be achieved for the product to move forward into production?
<h2>Who will trial your products?</h2>
A trial clearly can&#8217;t work without the audience to test the product. Probably the biggest challenge in any trial is locating the people who will test your product for you. 
In order to identify what trial users you need, consider the following:
<ul>
<li style="font-weight: 400;">How many trial users do you need?  Is it possible to get fewer trial users who use the product more thoroughly?  Or are you interested in getting a wider base of individuals to gauge reaction and engagement? Do you need to test multiple potential target markets, or audience demographics?</li>
<li style="font-weight: 400;">What do you need the trial users to do &#8211; are you simply asking them to use the product, or to run specific trial scenarios, or even to give detailed feedback on the service?</li>
<li style="font-weight: 400;">How will customers react to the version of the product that you’re looking to test &#8211; will they be put off by any gaps in the product or any issues with the service?  Is there any risk they might publicly criticise the product you are trialling?</li>
<li style="font-weight: 400;">Do you need completely new customers or can you use existing customers to test the product?  </li>
</ul>
It&#8217;s not unusual to face a shortfall in trial users despite a creative approach to recruitment.  One option is to incentivise them, via a giveaway, competition, or discount on the product pricing.  However, you should never lose sight of the fact that the incentive is a vehicle to encourage your audience to participate in trials and give feedback on your product; it needs to be appropriate to the test and avoid unduly influencing the results you get. 
Keep in mind that you want accurate, impartial views of your product. You need the hard truth about whether your product is going to be successful or not.  If friendly customers or incentivised trial participants have sugar-coated any gaps or issues because they don&#8217;t want to offend you or risk their incentive, you may end up with a rude awakening when you take your product to the wider market.
<h2>When should you trial your products?</h2>
When is the right time to beta-test the product? Too early, and there&#8217;s a high likelihood that there will be too many bugs or issues that will affect the customer experience. Too late, and you risk burning development time and effort on the wrong areas, and needlessly delaying product launch.
Generally the pressure is to trial early, which is great as long as your trial base is keen to participate and happy to accept and give feedback on glitches in the product.  However, significant numbers of product issues and bugs could end up alienating these trial customers, so it does need to be considered carefully.  Much of the art of timing is about setting the appropriate expectations with trial users during their recruitment, and balancing what they will be prepared to forgive when they are invited to be &#8220;the first trial users who will get to see our new product&#8221;.
Clearly, there is no single right answer to this question, as it very much depends on the nature of your product and the tolerance of your audience for incomplete products. As a product manager, you need to understand your audience well enough to make this judgement, and some small early trials in the development process might help you gauge this better.
Once you’ve worked through these fundamental questions, you should have a clear idea of what you hope to learn, as well as how you will get the information you need. The next step is to work out how to run your trials.

core/freeform

Product Launch

The magic of making great products with Christian Idiodi

5 key practices to build a better product, Adam Mullett, ProductTank Perth

Keeping product creative, innovative… and fun! by Manon Dave

3 myths that get in the way of product adoption by Martina Lauchengo

Epic alignment by Nils Janse

A designer, a product manager, and an engineer walk into a bar…by Karishma Irani

The importance of a compelling mission and a user-focused strategy by Johannes Mayer

Key Questions to Answer Before Customer Testing

Taza MohammedBhai

Read more

Comments