How Do Product Managers Validate A Product Hypothesis?

How Do Product Managers Validate A Product Hypothesis
How Do Product Managers Validate A Product Hypothesis

As we have already explained how you go about finding hypothesis to test, you might be wondering how do product managers validate a product hypothesis.

Guess what?

You’ve come to the right place to learn the process by which product managers go about validating their ideas.

 

A Warning To Product Managers About Validating Your Product Hypothesis

If you are astute enough, you’ll notice that product management has a certain academic bend to it in that you are always looking to validate your hypothesis by testing. While this is a noble calling, always bear in mind that you live in the real world, where there are always a large number of both known and unknown that you cannot control for (and guess what boys, girls and non binary folks of all shapes, sizes and colors, it’s one of the Ten Commandments of Product Management too). This little fact of life was best summed up by Donald Rumsfeld, regardless of what you think of his political leanings.

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones

Donald Rumsfeld
If this fact makes you uncomfortable, shoot me an email, I’ll send you 50 cents so you can find a payphone, call somebody who cares and maybe they can tell you the facts of life.
Call someone who cares: How Do Product Managers Validate A Product Hypothesis
Mom? Dad? It's me, Jimmy. I'm scare of unknown unknowns.
So, with that little piece of housekeeping/cathartic release out of the way, let’s push on.

Why Product Managers Validate Their Product Hypothesis

Like many things in life, the answer boils down to simple economics.  In fact,  this economic phenomenon even has a name, “Opportunity Cost”, which describes the unseen and often unaccounted for potential benefits missed when choosing one alternative over another.

Every idea decision you make in life has an opportunity cost (I need to remember to teach that to my kids); applied to your life as a product manager, working on one feature means not working on another, so you need a method to screen out the crappy ideas while not making enemies.  It won’t be a perfect method, but at least you can say you tried, right?

Say no
As a product manager, you need to get comfortable saying "NO"!

As previously described in Part 1 of this series, Product Hypothesis Testing: Generating The Hypothesis, the first step in hypothesis testing involves setting up two competing hypotheses, the null hypothesis and the alternative hypothesis.

Null hypothesis: states the “status quo”. This hypothesis is assumed to be true until there is evidence to suggest otherwise.
Alternative hypothesis: the statement that one wants to conclude.

The goal of hypothesis testing is to see if there is enough evidence to reject the null hypothesis by conducting experiment to prove or disprove the hypothesis.  But don’t be too good at it or you risk being like Facebook (Everything You Need to Know About Facebook’s Controversial Emotion Experiment)

Since we do not live in a perfect world, the reality is that you are going to be conducting these small data driven experiments on a product that a risk-taking entrepreneur build on their own intuition and savvy (Do not let the irony of this situation keep you awake at night).

There are many validation approaches you can take for a given hypothesis, but the professional approach is to use the correct combination of some of the following:

  • User analytics: Provides objective data
  • Internal stakeholders discussions:  Good for assessing the cost of the cost benefit equation
  • Customer/user interviews and surveys.  Uncovers the potential benefits of any proposed changes
  • MVP construction
  • A/B testing of different workflows: Generates more objective data

Hypothesis validation is really a puzzle that you need to solve, which each validation approach providing you with one more piece you can snap on.

Always Remember Your Dependencies

It is important to remember that because you do not operate in a vacuum, many things, usually the result of poor prior decision making (which often leads to poor future decision making) is there are many high effort technical dependencies standing between you and your desired end state. Not everything can be a quick win. Get used to that fact.

How To Validate A Product Hypothesis With Internal Data

User Analytics and Product Hypotheses Validation

Knowing how to combine product management and user analytics is a core part of the art and science of product management.

User analytics occupy a unique position with regards to product hypothesis validation as that while they sit side the company, they are generated by those external to it, so we shall be including references to user analytics in both the internal and external validation sections of this article, as I find it annoyingly inconsistent when authors gloss over these things.

Understanding The Customer Journey

The customer journey is the experience customers have when using your product or service;   In mapping out the journey,  you discover areas of  your product that need internal or external validation.  

Product Validation and The customer journey
The long and winding road of the customer journey.

Understanding how the customer defines success, their current situation, and the roadblocks they face forces you to make educated product decisions. The customer journey has a number of core components:

  • The basic user flows
  • The users’ desired states, motivations and success criteria
  • Roadblocks that hinders the user from making progress

This information is often captured in personas, metrics, and user flows.  Included in user personas should be an understanding of what motivates users to make progress towards a desired state and what holds them back.  This includes: 

  • Understanding the users’ current state.
    • What are users’ current perspective of their problem?
    • What pain does the customer feel as a result of the  problem?
    • Their journey within your application and outside of it.
    • How do they define success?
    • How does success impact them globally?
  • User Motivations:
    • What are the current solutions and where do they fall short?
    • Is new solution is uniquely appealing?
    • Will following a particular journey get them to the desired state/problem resolution?
    • Do users have the ability to do what is expected of them in order for your product to solve their problem?
  • Roadblocks
    • Are there parts of the current solution do they like?
    • What are the friction points related to switching to the new solution?

Validating A Product Hypothesis With External Data

The biggest risk to you as a product manager is coming up with fake use cases generated by internal hypothesis generation. 

Trojan Horse Use Cases
Much like a Trojan horse, be wary of a use case validated strictly by internal data.

External validation simply means getting feedback from real customers to see if your idea makes sense. Even though you might think you know what they’ll say, you don’t know for sure until you check.

The goal of external validation is to make the implicit, explicit. There are always basic feature expectations that a customer will have but won’t explicitly mention. If your product is missing these, the customer will be less than happy.

User Analytics

If you read our article on product management and user analytics that we referenced earlier in this piece, you do not need to read it again. If you haven’t stop being intellectually lazy.

Industry Research

While most product “experts” will tell you to find relevant research industry reports, census data, or even Google Trends to see what people are searching for, if you are really committed to understanding an industry you should:

  • Read annual public filings of companies in the sector. They will do a lot of the research for you.
  • Get a job in the industry. Hands on, real world experience trumps book smarts any day of the week

You can also look to industries/businesses that are analogous to your.  Focus on the problem people are trying to solve with in those niches and determine its applicability to your unique situation. 

Customer Interviews and Surveys

Knowing how to use customer surveys in product management and hypothesis validation is a profession in and of itself (I believe it’s called market research technically), so we broke it out into its own article.   Go and read it.  Now!!

Product Experiments

Real world product experiments with live data is one of the best ways to externally validate a hypothesis. Having said that, be aware that there are some ethical considerations to be had. Experiments are complementary to customer development, not a replacement, as they’ll help you see what people do when you make a change, but they won’t help you understand why people do it or what their fundamental needs are.
User experiments and external validation of a product hypothesis
Remember to validate your hypothesis with externally generated user data.

One of the most common experiments to try with an existing product is an A/B test (FYI – some people call it a split test) where two versions of the same website are presented to different users to evaluate interface changes by seeing what impact it has on your key metrics

When you implement the change, you randomly give some users the current “A” version and others the new “B” version, and then see if the metric changes in a significant way.

As the purpose of this experiment is to determine if you should invest in building what your hypothesis is testing, you often hack something quickly (and cheaply) in order to determine if this opportunity is worth pursuing in a more thorough way.
A/B tests are best used with iterative/refinement of existing features/functionality like

In start up environments, it is most common to A/B-test items like:

  • Navigation,
  • User flows,
  • Layout, and
  • Messaging, and
  • Content.

Working Prototypes

There are a few ways to go about building out working prototype:

A scaled-back version of your full product vision. These super-simple MVPs will be inexpensive to put together and not actually implemented how the real product will be implemented. But they should provide enough to a potential end customer that you can gauge if your opportunity’s worth pursuing.

Manual MVP . A manual MVP doesn’t scale, but it’s a way to validate the demand for your opportunity as well as get a better understanding the overall problem space and customer needs. . In a manual MVP, humans perform the behind the scenes tasks associated with the opportunity hypothesis you are trying to test.  You want to explicitly focus on the workflow steps users go through while attempting complete your processes.  This allows you to determine what’s involved in completing each step so that you can automate the whole.

A manual MVP helps you refine your hypothesis and create a better cost/value estimation as it allows you to determine if customers are interested in the idea and which steps:

  • Are the most useful 
  • Are unexpected but delightful 
  • You missed in the design process and
  • Customers don’t care about. 

 

Conclusion

Despite the seemingly liner nature what you read above, it will take you a lot of changes and retries to validate a product hypothesis that is worth putting on your product roadmap.  As a product manager you always need to keep three things in mind when planning a feature: 

  • You will be building it limited resources,  
  • There will be opportunity costs associated with it and,
  • You will always be waging an internal political struggles over both

 

So think strategically to make sure you work on the things that matter most in order to get the biggest bang for your bucks when doing the Product Finance Dance 🙂

Table of Contents
More From Our Blog
Understanding the importance of Software Product Roadmaps
Opportunity Identification
Software Product Roadmaps

Software Product Roadmaps: What Books Won’t Tell You. In this group of articles, The Product Manager’s Toolbox dives deep into the dark art of building

Read More »
The Product Road Map Process
Opportunity Identification
The Product Roadmap Process

To the uninitiated or junior product manager, just starting out in the field, the product roadmap process is a seemingly mechanical process.  What are we going

Read More »