As we have already explained how you go about finding hypothesis to test, you might be wondering how do product managers validate a product hypothesis.
Guess what?
You’ve come to the right place to learn the process by which product managers go about validating their ideas.
A Warning To Product Managers About Validating Your Product Hypothesis
Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones
Why Product Managers Validate Their Product Hypothesis
Like many things in life, the answer boils down to simple economics. In fact, this economic phenomenon even has a name, “Opportunity Cost”, which describes the unseen and often unaccounted for potential benefits missed when choosing one alternative over another.
Every idea decision you make in life has an opportunity cost (I need to remember to teach that to my kids); applied to your life as a product manager, working on one feature means not working on another, so you need a method to screen out the crappy ideas while not making enemies. It won’t be a perfect method, but at least you can say you tried, right?
As previously described in Part 1 of this series, Product Hypothesis Testing: Generating The Hypothesis, the first step in hypothesis testing involves setting up two competing hypotheses, the null hypothesis and the alternative hypothesis.
Null hypothesis: states the “status quo”. This hypothesis is assumed to be true until there is evidence to suggest otherwise.
Alternative hypothesis: the statement that one wants to conclude.
The goal of hypothesis testing is to see if there is enough evidence to reject the null hypothesis by conducting experiment to prove or disprove the hypothesis. But don’t be too good at it or you risk being like Facebook (Everything You Need to Know About Facebook’s Controversial Emotion Experiment)
Since we do not live in a perfect world, the reality is that you are going to be conducting these small data driven experiments on a product that a risk-taking entrepreneur build on their own intuition and savvy (Do not let the irony of this situation keep you awake at night).
There are many validation approaches you can take for a given hypothesis, but the professional approach is to use the correct combination of some of the following:
- User analytics: Provides objective data
- Internal stakeholders discussions: Good for assessing the cost of the cost benefit equation
- Customer/user interviews and surveys. Uncovers the potential benefits of any proposed changes
- MVP construction
- A/B testing of different workflows: Generates more objective data
Hypothesis validation is really a puzzle that you need to solve, which each validation approach providing you with one more piece you can snap on.
Always Remember Your Dependencies
How To Validate A Product Hypothesis With Internal Data
User Analytics and Product Hypotheses Validation
Knowing how to combine product management and user analytics is a core part of the art and science of product management.
User analytics occupy a unique position with regards to product hypothesis validation as that while they sit side the company, they are generated by those external to it, so we shall be including references to user analytics in both the internal and external validation sections of this article, as I find it annoyingly inconsistent when authors gloss over these things.
Understanding The Customer Journey
The customer journey is the experience customers have when using your product or service; In mapping out the journey, you discover areas of your product that need internal or external validation.
Understanding how the customer defines success, their current situation, and the roadblocks they face forces you to make educated product decisions. The customer journey has a number of core components:
- The basic user flows
- The users’ desired states, motivations and success criteria
- Roadblocks that hinders the user from making progress
This information is often captured in personas, metrics, and user flows. Included in user personas should be an understanding of what motivates users to make progress towards a desired state and what holds them back. This includes:
- Understanding the users’ current state.
- What are users’ current perspective of their problem?
- What pain does the customer feel as a result of the problem?
- Their journey within your application and outside of it.
- How do they define success?
- How does success impact them globally?
- User Motivations:
- What are the current solutions and where do they fall short?
- Is new solution is uniquely appealing?
- Will following a particular journey get them to the desired state/problem resolution?
- Do users have the ability to do what is expected of them in order for your product to solve their problem?
- Roadblocks
- Are there parts of the current solution do they like?
- What are the friction points related to switching to the new solution?
Validating A Product Hypothesis With External Data
The biggest risk to you as a product manager is coming up with fake use cases generated by internal hypothesis generation.
External validation simply means getting feedback from real customers to see if your idea makes sense. Even though you might think you know what they’ll say, you don’t know for sure until you check.
The goal of external validation is to make the implicit, explicit. There are always basic feature expectations that a customer will have but won’t explicitly mention. If your product is missing these, the customer will be less than happy.
User Analytics
Industry Research
While most product “experts” will tell you to find relevant research industry reports, census data, or even Google Trends to see what people are searching for, if you are really committed to understanding an industry you should:
- Read annual public filings of companies in the sector. They will do a lot of the research for you.
- Get a job in the industry. Hands on, real world experience trumps book smarts any day of the week
You can also look to industries/businesses that are analogous to your. Focus on the problem people are trying to solve with in those niches and determine its applicability to your unique situation.
Customer Interviews and Surveys
Knowing how to use customer surveys in product management and hypothesis validation is a profession in and of itself (I believe it’s called market research technically), so we broke it out into its own article. Go and read it. Now!!
Product Experiments
One of the most common experiments to try with an existing product is an A/B test (FYI – some people call it a split test) where two versions of the same website are presented to different users to evaluate interface changes by seeing what impact it has on your key metrics
When you implement the change, you randomly give some users the current “A” version and others the new “B” version, and then see if the metric changes in a significant way.
As the purpose of this experiment is to determine if you should invest in building what your hypothesis is testing, you often hack something quickly (and cheaply) in order to determine if this opportunity is worth pursuing in a more thorough way.
A/B tests are best used with iterative/refinement of existing features/functionality like
In start up environments, it is most common to A/B-test items like:
- Navigation,
- User flows,
- Layout, and
- Messaging, and
- Content.
Working Prototypes
There are a few ways to go about building out working prototype:
A scaled-back version of your full product vision. These super-simple MVPs will be inexpensive to put together and not actually implemented how the real product will be implemented. But they should provide enough to a potential end customer that you can gauge if your opportunity’s worth pursuing.
Manual MVP . A manual MVP doesn’t scale, but it’s a way to validate the demand for your opportunity as well as get a better understanding the overall problem space and customer needs. . In a manual MVP, humans perform the behind the scenes tasks associated with the opportunity hypothesis you are trying to test. You want to explicitly focus on the workflow steps users go through while attempting complete your processes. This allows you to determine what’s involved in completing each step so that you can automate the whole.
A manual MVP helps you refine your hypothesis and create a better cost/value estimation as it allows you to determine if customers are interested in the idea and which steps:
- Are the most useful
- Are unexpected but delightful
- You missed in the design process and
- Customers don’t care about.
Conclusion
Despite the seemingly liner nature what you read above, it will take you a lot of changes and retries to validate a product hypothesis that is worth putting on your product roadmap. As a product manager you always need to keep three things in mind when planning a feature:
- You will be building it limited resources,
- There will be opportunity costs associated with it and,
- You will always be waging an internal political struggles over both
So think strategically to make sure you work on the things that matter most in order to get the biggest bang for your bucks when doing the Product Finance Dance 🙂
