Hypothesis-driven product management

In this guest post, Saikiran Chandha, CEO and founder of SciSpace provides an overview of hypothesis-design testing and why it is quintessential to building new product features.

---

“You know me, you know me not.”

Sincerely,

Hypothesis testing!

If you are into agile product development and design thinking, the above assertion goes without saying.

Yes, hypothesis-driven practices are ideal for building new features. Since the goal is to test the validity of each hypothesis, the uncertainty around the product development process is significantly reduced.

In a way, hypothesis testing helps you make better decisions about your product lifecycle management.

But how do we define a good hypothesis statement based on an idea or a question? What are the tests that we should even consider while defining one? How are we going to design a new feature based on this hypothesis?

Take a pause!

Let’s dive in and learn more about hypothesis-driven product management.

What is product hypothesis?

A product hypothesis is an assumption made within a limited understanding of a specific product-related situation. It further needs validation to determine if the assumption would actually deliver the predicted results or add little to no value to the product.

The crucial factors required to build a clear hypothesis statement are —

• Assumption — An assumption made based on observations (Either to fix a problem or to update the product feature).
• Condition — The condition is the reason why the assumption was made (Example — lack of user interaction with the product or dip in page traffic).
• Prediction (Impact) — Prediction is the success or failure rate of the hypothesis.

Basically, a hypothesis is a prototype where we think that an improvement in the product will give us a bump in terms of user interaction, business revenue, and other metrics.

What is the need for hypothesis-driven feature development?

Hypothesis-driven practices provide a space to brainstorm based on user behaviors, build a hypothesis, and convert them into a significant feature.

The process also gives us a thorough understanding of what, how, and why to prioritize the product’s features.

It is a cost-effective approach as the experimentation and testing is done before production.

It requires minimal effort and fosters the product team’s confidence. Overall, the impact of hypothesis testing has a direct effect on the business goals and revenue streams.

What are the steps involved in the product design hypothesis?

Untested assumptions are the biggest threats to product development. Product management teams in every organization must make it a point to highlight the importance of conducting hypothesis-driven experiments.

The product-design hypothesis is an iterative measure that defines and explores assumptions, followed by conducting suitable experiments and validating the outcome based on user feedback.

In this section, I’ve listed down six steps that we should adopt for the smooth start of the new design hypothesis.

Step 1 — Jot down the uncertainties or challenges of the product and brainstorm questions

Whether you’re adding a new feature or improving an existing one, there will always be uncertainties associated with the product.

Jot them down and come up with the possible assumptions that can solve the uncertainties of the product.

For example — Your application is not gaining users on the Google play store. The possible questions could be — How to increase the reach of your app? How to utilize current users to market your app? Would the right marketing strategy suffice? Would a user survey form help? and more.

So, brainstorming different questions to improve the product feature is the foundation of defining an effective hypothesis statement.

Step 2 — How to build a hypothesis statement?

Now that we now have ideas or questions handy, pick the ones that you firmly think add value to the process. Eliminate the least impactful questions.

The possible assumptions or solutions to the shortlisted questions are the real hypothesis for our product. The solutions should not only reflect a product manager’s thoughts. It should support the group decision structure where all the stakeholders are involved in defining a hypothesis.

There are multiple templates available to build hypothesis statements. Choosing the template that correlates well with your product development team and is most comfortable with is a good practice.

However, the simpler one could be —

We believe that {{building the feature}} for {{users}}, addresses their {{user outcome}} and help us achieve {{business outcome}}

At SciSpace, we used the template for one of our latest features, and it looked like this —

We believe that building the “Trace” feature for the researchers, addresses their literature review problem and helps us achieve an increased user engagement rate.

So, it’s crucial to work on the four factors required to frame the hypothesis statement — Problem, Solution or Feature, Users or Target Audience, Impact or outcome.

Step 3 — Segregate the Good Hypothesis from Bad Hypothesis

Practically speaking, testing and validating all the hypotheses would be challenging. We should bifurcate the hypothesis statements based on the hypothesis’s duration, research method, and outcome.

We can bucketize the statements into Good and Bad hypotheses.

An effective hypothesis is accompanied by specific and measurable outcomes. Conversely, a lousy hypothesis provides neither clear nor quantifiable outcomes.

For example, if we have a statement like, Introducing a new literature review feature will increase the user engagement rate, it is a bad hypothesis as it is not specific or a definite outcome in the statement.

An example of a good hypothesis would be, Introducing a new literature review feature that simplifies reader workflow will increase the average time a researcher spends on our site.

To be more efficient, draft strong hypothesis statements that deliver impactful outcomes with minimal effort. In addition, you can employ supportive tools like the Discovery effort worthiness (DEW) index, which helps you analyze the steps required for product development and design thinking approaches.

Step 4 — Test! Test! And Test!

Testing is vital to product lifecycle management as it decides whether the hypothesis is true or false. It gives us a conclusion on whether to accept the hypothesis and push it to the production stage or reject it and revise or tweak the assumptions.

What are the different types of hypothesis testing?

• A/B Testing — The experiment is aimed at two different user sets. Let’s say Set A and Set B, where only Set A users are introduced to the new hypothesis design feature while keeping the Set B users exposed to older features. Run the experiment for a specific period of time, analyze the data, and draw conclusions. If the result shows a spike in the new feature, then it is clear that the hypothesis functioned well and vice versa.
• Multivariate Testing — The experiment is aimed at multiple variants or user sets. It is mainly performed to track the multiple variations associated with the same product development. You can set or define variants based on your user or business outcome.

Step 5 — Evaluate the test results and define the next steps

The next step is to analyze the results of your hypothesis and create a roadmap for the new feature rollout.

The assessment can include both qualitative and quantitative approaches. Quantitative analysis includes several combinations —

• High impact + High efforts
• Low impact + Low efforts
• High impact + Low efforts
• Low impact + High efforts

While qualitative analysis majorly includes user feedback, survey records, stakeholders’ feedback, and more. By correlating all the possible conclusions drawn from the tests, you can define the next action steps.

Step 6 — Conclude the Experiments — Closing the loop

So, here comes the final verdict of the hypothesis testing. If the experiments favored your hypothesis, you can inform your stakeholders and prepare the team for the feature rollout process.

Release the feature to a wider set of audience (100% users). However, feature testing should keep continuing even after the product release. It helps you gain a deeper understanding of your product under different scenarios.

If the hypothesis is false, do not move it to the production stage and reconsider how to solve the problem. Accumulate the data or insights from the experiments, finetune the hypothesis with specific and measurable goals and build a new hypothesis.

Ensure to stay up-to-date with the latest trends, user behavior or demands, and other factors, and update your product on a timely basis to increase customer retention.

To continuously deliver value to users via products, you need to continue running experiments for various hypotheses.

Conclusion

Hypotheses-led product management clearly depicts the product’s journey from making assumptions to conclusions. The conclusion may or may not give the expected results. But the approach forms a learning cycle and helps us improve the product at scale.

Building a good hypothesis statement based on your users’ pain points, testing it, and experimenting with multiple user groups helps you build a robust roadmap for the new product development and feature release. If the hypothesis fails to meet the predicted outcome, there is always room to tweak it, rerun it, and see how the experiment pans out. Thus, to keep the continuous delivery loop running, it is imperative to execute all these steps while planning for a new feature rollout.

There’s more where that came from! Access more insights below

• The key to a great hypothesis – Mark Tsirekas on The Product Experience
• Stepping up Your A/B Tests
• Building a product that sells—and grows—itself