Ah, product hypothesis testing…..In an effort to sound all grown up, product managers borrow, bastardize and steal terms from other domains and disciplines from time to time and product hypothesis testing is just one of those terms.
Obligatory glibness aside (I like to think glibness is my value add to you 😉 ), in order to stay in business, you need to innovate your product in such a way that it:
1. Attracts customers.
2. Keeps customers loyal.
3. Differentiates your product from others to maintain your competitive edge.
This requires adding new value added benefits to your product offering. The key to do this is by product hypothesis testing, which is actually a two part process:
Part 1: Product Hypothesis Generation – Figuring out what we should be testing for
Part 2: Hypothesis validation – How Do Product Managers Validate A Product Hypothesis.
So let’s dive in a bit and learn what exactly a hypothesis is and how it applies to us.
The Basics Of Hypothesis Testing
The first step in hypothesis testing is to set up two competing hypotheses. Getting the hypotheses correct is the most important aspect of this process; if the hypotheses are incorrect, as will be your conclusion.
The two hypotheses are named the null hypothesis and the alternative hypothesis.
Null hypothesis: states the “status quo”. This hypothesis is assumed to be true until there is evidence to suggest otherwise.
Alternative hypothesis: the statement that one wants to conclude.
The goal of hypothesis testing is to see if there is enough evidence against the null hypothesis. In other words, to see if there is enough evidence to reject the null hypothesis. If there is not enough evidence, then we fail to reject the null hypothesis.
Understanding Product Hypothesis Testing
Product hypothesis testing is acknowledging it is better to be admit being wrong in order to change course than to be adamant in your rightness and continue riding the escalator to nowhere (Remember the product management commandment – Never Assume You Know Your Customer).
Many people go immediately from an idea right to building it, never once seeing if others would actually buy the product. Now some product management gurus might tell you to always do a ton of validation before building anything, I happen to disagree with this.
Due to a variety of reasons, such as a desire to respond accordingly, a lack of imagination or living in a fantasy world, you can often only elicit valid opinions as to what people will do with your product when they have access to it. Sometimes, a questionnaire or survey just won’t cut it.
Which Came First...?
Actually, this isn’t even a discussion. In order to have a product hypothesis worth testing, you need to have a relevant product based goal.
- Acquiring users?
- Lift revenue?
- Ideally anything that makes investors smile, really…..as long as it can be captured in a metric or data.
Now, next up, figure out which site feature/functionality impacts that goal What you pick to work on next will be in service of this goal, and this is the core of how PMs strategically choose what to work on next.
Figuring Out The Right Lever To Pull
So, next up you need to figure out what is going to move the needle on the metric you selected. Put in product managerese (which is really a corporatespeak dialect), you need to productize the goal….how do you modify the existing website functionality to hit the goal, bearing in mind that modification can involve tweaking existing features/functionality or adding entirely new aspects to your product.
One big caveat here is don’t spend time over optimizing existing features because you can’t see the forest for the trees as often what got you to certain level of success may not take you to the next level. Growth requires big steps so do not shy away from adding something transformational to your product if your research supports it.
Remember, business is not risk free, so do not expect product hypothesis testing to remove all elements of it. The reality is that there is no risk without reward, so you can take a metrics based approach to product management, which often results in a steady stream of base hits to invoke a sports metaphor or you can take a more qualitative approach, which has a different risk profile, but the one of the realities of life is no one ever got anywhere playing it safe, so you may either knock it out of the park or go down in a blaze of glory.
Often the initial versions of product are more qualitative in their approach to product management, because at this stage in the game, there is little need for incremental gains….., it is go big or go home.
Professional Hack: Quickly Figuring Out The Wrong Lever To Pull
Product managers are bombarded with ideas that they “could” test, but probably aren’t worth doing so, but you do need a quick and dirty (get your minds out of the gutter, this is a family show) tool to dismiss out of hand filter ideas that you know just don’t fit.
It pays to keep an open mind, but not so open your brains fall out.
In other words, you need to know how to exercise some professional judgement and say “No” (Another one of the Ten Commandments Of Product Management, but you had to have seen that coming, right?) to ideas without going through a comprehensive validation.
- Is this opportunity in line with our vision?
- Does it support the product’s core function?
- Is it within our abilities to execute on and support in the future?
- Could it move the needle on the relevant key metrics?
- Is there any data that could justify devoting resources this opportunity?
- Does the magnitude of the benefit exceed the cost?
Using Your Numbers To Find the Right Lever
Qualitative Hypothesis Testing: When Numbers Just Won't Do
If there is both an art and science to product hypothesis testing, qualitative hypothesis testing is definitely the art part. While it is a much softer skill, it is mastery of the softer skills that makes one an experienced professional
There are numerous ways to develop qualitative hypotheses that you can test:
- Because someone said so
- Bug reports
- Product backlog
Taking credit for other people’s ideasTeam SourcingTheftLook at the competition- Intuition and vision
Because Someone Said So
There are times when someone above you in the corporate food chain tells you to do something and while you might be the type to stand on principle, more than likely you are the type who likes to have a job.
Sometime your boss may be looking at a hypothesis generating opportunity based on information of which you are unaware (like a potential acquisition or investment), there may be a regulatory change to comply with, or alternatively they are making decisions based on ego.
Having said that, it is your job as a product manager to at least attempt to validate the hypothesis and if you are not allowed to, make sure you put your position in writing by sending an email to the relevant stakeholders. Remember, your goal is to be a professional product manager, not a fall guy for when things go south.
Bug Reports: The Slackers Way To Hack Hypothesis Testing
The laziest simplest most efficient method of qualitative hypothesis generation is examining how bugs impact the user experience.
Looking at user analytics for the site/product in question is going to be the usual method by which you validate the legitimacy of a bug based opportunity. You want to rank order bugs by how often they occur and their impact on the success metric. Allowing you, as a product manager, to weigh the value of a bug fix versus doing something else instead.
Feature Requests Are A Goldmine of Hypotheses
Teamsourcing
Everyone once in a while, you need to throw your team a bone and. The key to success as a product manager is engaging your team members by soliciting their input when figuring out product hypotheses to test. Since few product resources report to product managers, you can build relationships across the company by at least going through the motions of soliciting ideas and input from different departments.
While many over-entitled Generation Z workers mistake the workplace for some sort of profit-based democracy, you can earn some workplace creditability amongst the younger set by fostering the notion you are listening to their ideas. Knowing full well that at the end of the day, you bear the brunt of any boneheaded product decisions, the best way to protect your self from all things pear shaped while reaping the benefit of “work place populism” is to include cross departmental teams at the ideation stage where it causes the least harm.
OK, sardonic humor aside, teams that work closely with the customers, such as design, customer support, business development, and sales definitely have valid input to contribute to the product based on experience with customers and the product. Supporting evidence for their ideas can be found in the customer support tickets.
If you really want to be a man/woman/it/they/ze/non-binary person of the people, organize cross functional brainstorming sessions, but always remember that no good deed goes unpunished. While you will be making people feel that they work in an inclusive environment and you will get to learn from other’s perspectives, the risk is that you supply the masses with too much information and raise their expectations to the point you cause a political shit show when you don’t push any of the ideas into production. You can give yourself cover by defending your decisions using success metrics (remember the Ten Commandments of Product Management people!!!).
The Competition
If you have spent 47 minutes as a product manager, you’ve probably hear the “Good artists copy. Great artists steal” quote (I guarantee we have used it on this site), but I think Bill Gates had a more honest perspective on copying the competition.
Well, Steve, I think there's more than one way of looking at it. I think it's more like we both had this rich neighbor named Xerox and I broke into his house to steal the TV set and found out that you had already stolen it.
Having said that, there are a number of caveats to applying this principle of theft (How is that for an oxymoron):
- Never assume that anyone else knows what they are doing. From a straight up mathematical point of view, 50% of the population is of below average intelligence, so there is a very real chance you could be taking advice from a dummy.
Think of how stupid the average person is, and realize half of them are stupider than that.
- You never know how well that stolen hypothesis is working out for the person you borrowed it from.
- Make sure this feature/product fits with your product goals.
- If you are going to steal an ideal, steal it from a high volume porn site, they validate everything with data.
- Remember our own Product Manager commandments which pretty much says the same thing…test all your ideas.
Vision and Intuition
Vision and intuition can be a double edged sword. Too little and you go no where, too much and you risk going all Jim Jones.
So now that I have lost have my readership with yet another inappropriate aphorism/analogy, let’s get to the meat of the matter.
A not to be overlooked source of opportunity hypotheses should be a combination of your subject matter expertise and day to day routines. Assuming you are of at least average intelligence, you likely have a strong idea of what to build for your customers based on you experience, but please make sure to hold your own ideas to the same level of data scrutiny as you do others. Do not assume others have the same problems as you (guess, what there is a product manager commandment for this too) and besides, you spent a ton of time building user persona’s, maybe you should actually use them for product management…just a thought.
If you work in tech, you’ve likely been exposed to the ubiquitous “visionary”.
Conclusion
Now armed with the basics (or at least the jargon) of product hypothesis generation and testing, I like to think I’ve given you enough to both iterate your product to future success, while side stepping the potential product management landmines in any organization.
Now move on to part Part 2: Hypothesis validation – How Do Product Managers Validate A Product Hypothesis and learn how to do the rest of your job 😉
