How To Use Customer Surveys in Product Management

Customer Surveys and Product Management
Customer Surveys and Product Management

If it’s answers you seek…. well, then you better figure out how to use customer surveys in product management.  While user analytics capture what customers actually do, but they don’t tell you anything about what’s going on in a customer’s mind.

In product management, there are different ways you can go about surveying your customers:

  • Online customer surveys
  • Interviews

It is essential to understand the benefits and drawbacks to each method so that your firm can determine which one, or combination of the two, is best for validating your particular product hypothesis.  In this article, we are going to dive into both methods of surveying your customers.

As a matter of practicality and professional practice, when working on the validation your product hypotheses:

  • Use interviews to get real data about people’s needs after you’ve exhausted internal and less resource intensive data sources.
  • Once you feel interviews have validated your hypothesis, a customer survey help see if the hypothesis scales.
  • Never start the validation process with a survey. It requires too much work.  Additionally,  the information you get back will often be too broad and shallow to dig into user needs and pain points.
Learn how to leverage customer surveys in product management

Online customer Surveys vs In Person Interviews

Online Customer Surveys: Benefits & Drawbacks

Low Cost: Using online customer surveys in product management allow you to capture a high volume of responses for less money.
Time Savings:  Online customer surveys are flexible and accessible at the respondent’s convenience.
Scale: In using online customer surveys, product management teams sample a larger, more representative population when validating your hypothesis. Surveys  get a view inside a lot of customers’ heads.  While not as high quality as data from customer interviews, but it’s a low-cost way to see if conclusions from customer interviews scale to a larger group of people.

May require incentives:  Some respondents may want something in return for completing  your survey. 

Completion rates: To be useful in product management, customer surveys need to be completed.  People often don’t always answer every question on a survey and exit it before completing it. 
Lack of details/explanations: Most survey do not capture detailed, explanative responses. In the ideal world of product management, customer surveys would capture information via  open-ended questions, the reality is that are difficult to ask in survey format, so closed-ended “select all that apply” responses are used, preventing respondents from using their own words and phrases.

Interviews: Benefits & Drawbacks

Surprise insights and findings: Well trained interviewers can dive into specific topics and adjust their questioning depending on the information they surface. This can uncover perspectives on your product hypothesis you had previously not considered. Interviews are great because they help you figure out customers’ underlying pain points and motivations,

Higher response rates: Persistence in scheduling the interview and dealing with potential respondents individually contributes to a higher response rate compared to online surveys. 

Need experienced interviewers: Experienced interviewers know when to:

  • Probe for more detail
  • Redirect
  • Recall applicable answers from  earlier in the interview.

Takes economic resources: Hiring experienced interviewers as well as the resources for scheduling and completing interviews costs time and money.
Small sample sizes: Because of the time and economic factors, your information is going to be gathered from a smaller group of people.  They don’t help you quantify issues or measure overall attitudes.

And without any further ado, let’s tackle using customer surveys in product management

Product Management Crash Course in Customer Surveys

First things first, understand that gathering consumer research is and of itself its own field, so while you should know about how to use surveys, we are not going to be addressing things like internal/external validity and the more technical aspects of survey construction.

In fact, what you are probably going to do is use an online survey tool and send out some random questions to people, but I figure I have a responsibility to give you a little bit of an understanding on how you should really do things.  While customer surveys can be used in the different stages of the product development lifecycle, we are going to be limiting our conversation to their application to the opportunity phase in an attempt to externally validate our product hypothesis.

Stage 1: Planning A Customer Survey

While running a customer survey isn’t the product management equivalent of cross examining a witness in court, you should have some idea of the relative importance of the items upon which you seek input.  

As the saying goes, if you don’t know where you are going, you will never get there.  If you surveying customers without even a vague sense of your objectives, the gathered data will likely be useless in validating your hypothesis

When you come to a fork in the road, take it.

Yogi Berra, Amazing Baseball Coach, Probably Crappy Product Manager

In product management, customer surveys provide utility along the following lines:

  • As a mean to validate if your customers who are you think they are and
  • Finding out the problems customers want to solve,
  • Determining what customers are currently doing right to solve the problem at hand,
  • How they make product purchasing decisions.

Warning: Never Ask Your Customers What They Want

The purpose of customer surveys in product management is not to gather a laundry list of features,  bounce random ideas off of people our outsource your product vision to the masses. Doing this is a guaranteed way to generate fake use cases as customers tend to be like kids at buffet table, taking food they will never eat asking for features they will never use ’

Given that you can earn a doctorate in survey design, let’s assume right up front you will never really master that art of customer surveys in product management, but that should not preclude you from knowing about matter completely.  The tips below should empower you to have a reasonably professional conversation about the topic, or at least talk yourself out of trouble when something goes awry: 

  • Test you survey before sending it to a broad group.
  • Explicitly focus on validating your hypothesis 
  • Keep “it” short.  And by “it”, we the time required to fill out the survey.  Survey fatigue is real so you need a document that can be completed in a few minutes  
  • Keep question/answer options clear.
  • Start broad, move into specifics, and close with options to add any extra thoughts. 
  • Aim for “actual” rather than “ideal”; ask people what they do, not might do have done, not what they might do.
  • No leading/loaded questions. 
  • Know the hidden danger of comparison questions:
    • They are leading because you often attempting to evaluate a hypothetical situation It’s better to present the two options separately
    • Its an attempt to ask two or more questions in one because you’re asking respondents to make an evaluation
    • A more accurate approach is asking separate, simple questions rather than one question that really includes multiple questions. 
  • Increase accuracy by allow for  scaled responses instead simple agree/disagree options that are biased because people want to please you and are more likely to select “agree.” 
  • Ask questions that get at a customer’s goals in order to understand explicit motivation 

Stage 2: Executing A Customer Survey

Recruiting victims participants: Contact existing customers that meet your requirements to ask if they can spare a few minutes to help improve the product by taking a survey.

Additionally, there are company’s out that there that provide ready made survey panels.

Bribe your way to success: Some companies often offer  reward as an incentive for people to participate 

Get enough data: Run your survey until you’ve acquired enough data to be statistically significant.

Stage 3: Analyzing Customer Survey Data

Data Analysis of Customer Surveys In Product Management
Remember, when you analyze your data, use lots of colors so people don't fall asleep when staring at it.
When analyzing customer surveys, product management professionals need to look for information that validates their product hypothesis hypothesis. If the survey is in regards to the launch of a whole new product, you want to find out how many people have the pain point you are looking to alleviate and how significant is that pain.  So just how does one actually do this?
  • You need to validate the data and make decisions about what do do with incomplete surveys? Using data from incomplete surveys means you end up with a different sample size for each question.
  • Perform some basic data analysis with any numerical questions in order to see if the data is normally distributed or has a bimodal shape to it.
  • Consider how to segment the data or apply different cohorts in relation to the hypothesis you want to test.
  • If everyone agrees it’s a pain point but no one has spent any time or money trying to solve it, your opportunity is likely not worth building unless a solution is very cheap to build or you have reason to believe your solution will make customers realize the pain point is bigger than they thought. You could imagine what a survey about voicemail would’ve looked like before visual voicemail on the iPhone—finding, listening to, and saving a specific voicemail message was a pain, but no handset manufacturer spent money to solve it because it required service carrier cooperation.
If you used satisfaction questions to build up an importance/satisfaction graph—both in terms of how customers perceive importance and how you internally rank importance—you can see if your opportunity is the right one to pursue next or if there’s something more important. Because it’s very easy to fall victim to confirmation bias, try focusing only on data that invalidates your hypothesis before looking for data that validates it. At this point, you should have a very good sense of whether your hypothesis is valid and if it’s a valuable enough opportunity to pursue.

Become An Expert At Conducting User Interviews

If I promised that, I would be guilty of underdelivering, so just like I did with customer surveys, I am going to give you enough information so that you are not the dumbest person in the room.

Similar to customer surveys, product management uses interview to validate product hypothesis, and while if you are in a big firm, you will probably outsource this activity to someone who knows what they are doing, if you are with a small company, you may be handling this task on your own.  Beside, its good to get out and meet the common folk from time to time, it will keep you honest and humble. 

Stage 1: Planning A Interview

Some people say you should plan for a customer interview like you are planning for a job interview, but trust you me, I’ve asked enough people to leave my office half way through an interview to be suspect enough as to whether or not people actually prepare for them. A more apt statement may be to “prepare for an interview like you are going testify in your own defense at a criminal trial”.  Not for a big crime, but something like trespassing or reckless driving, something serious enough to warrant your effort and attention

Things To Avoid When Interviewing

Speaking to the wrong people: Create screening questions you to ask potential interviewees in order to ensure you’re finding the right people to talk to.

“Ideal” self situations:  Mentioned earlier when we discussed a similar issue  deploying surveys.  Humans have this innate flaw in that they want to please other people (Well except for me, just ask my ex-wife and parents, according to them, I am missing that gene), so they will often agree with you by giving you with what they think is the right answer, rather than disappointing you with the correct answer.

In an interview, avoid this by asking the customer tell you about how they currently deal with whatever hypothesis you are looking to validate, rather than what they might do.

In a similar vein, asking what someone might pay for something they currently do not pay for leads to the same issue of a fictional response.  Instead, ask what they’ve  paid in the past to solve the issue associated with the hypothesis you are attempting to validate.   This line of questioning also highlights if the problem is an issue for them, as if they have not paid anything to fix it, it’s likely not an issue for them.

Showing products or prototypes: Bringing these into a user/customer interview conversation shift the focus of the conversation to what the customer wants rather than their needs.

Loaded/leading questions:  You do not want to encourage the customer to answer in a specific way.

Yes/No Questions: Ask questions in such a way that gets customers to tell you stories in order to avoid yes/no answers (If you just wanted yes/no answers, you could have run a survey). Stories give background context on why someone users your product, their pain minimization goals, and their priorities.

Discussing features: Some interview participants think they know exactly what they want and will tell you so, but unless it is clear they are domain experts/advanced users. Most customers don’t know what’s possible or impossible, so structure questions  to figure out people’s underlying pains and see if your idea will address them, ignoring specific feature requests in the process

Interview Preparation

Prepare by writing down:

  • What you can factually state about users and the current problems they face, desired gains and what tradeoffs they’re willing to accept for these gains.
  • Your assumptions about our users, including their needs and how your product satisfies them.
  •  Your product hypothesis in such a way that it explicitly states your assumptions about segments, value propositions, channels, relationships, and  revenue streams.
  • Your success metrics
  • Your competitors.  Remember your customers might not see the same people as competitors, or they might know of existing solutions that you don’t.
  • It’s useful to look at any other persona traits relevant to your opportunity hypothesis to think about who to talk with, and if there are enough potential customers to pursue this opportunity.
Interviews and customer surveys in Product Management
Interviews and customer surveys: Product management hypothesis validation tools

Stage 2: Structuring Your Questions

With the above information, put together your first list of questions, and realize that as you start interviewing, you’ll modify the questions based on feedback, like are you getting the answers you’re hoping to hear.

A good way to do this is reverse engineer the process; develop a list of ideal responses to each question based on how you plan on using the interview results from the interviews, and formatting questions you think best elicit those responses.

Sometimes, getting useful information is like pulling teeth and it takes some heavy hardware to extract the needed information some properly structured questions to get useful information.

Extracting information in customer interviews
When you can't get information you need, try extracting some teeth, I mean using properly structured questions.

In situations where an interview subject has a situation or limitation they do not see as relevant to your discussion, make use of questions that help set the context for them.  These questions include:

  • How often do you encounter this situation?
  • What were you doing right before situation?
  • How long did it take to deal with the problem?
  • What made you buy the product relevant to this topic?
  • How often do you buy the product?
  • Where do you go to buy the product?

At the end of an interview, you should always ask an open ended question to solicit information about information you may want to probe in future interviews.

Stage 3: Interview Logistics

Now that you have your question list together, consider the following logistical issues:

Timing: Are there internal milestones beyond which new information isn’t helpful. Knowing these milestones in advance helps you craft an effective interview schedule.

Quality of Information: What trade offs do you need to make between information quality and speed of delivery? What level of quality and confidence in the results do you need?

Location:  Where are the interviews going to be conducted – in person or on the phone?

  • While in person is always best, but if you are not in the same city as your subject, you will probably use the phone.  The benefits are telephone calls are more convenient and  easier to schedule, but lack non-verbal cues.
  • Video chat solves the problem regarding being able to see non verbal cues as long your subject is familiar enough with the technology.

Duration: How much time will each interview take and what sort of buffers will you build in between sessions to account for things running late?

Note Taking: You want to record what your subject says verbatim rather than summarizing to avoid confirmation bias and improve the ease of  the analysis the conversation later on. Having a notetaker allow you to focus on the person in front of you rather than writing. 

Compensation: You should pay your interviewees between $50 and $100 per hour as you are asking people to provide their expertise to you.

Recruiting participants:  We covered this when we discussed customer surveys, but sources of participants include:

  • Anyone you know anyone matches target persona.
  • Panel recruiting/market research services
  • Your existing customer/users

Stage 4: Conducting Interviews

  • Make subjects feel comfortable and get them talking right away with simple questions that require little cognitive load.
  • You want them to open up and have a conversation with you, that follows an arc: start with small talk, go to easy questions, move to meatier questions, recap key points, and then thank them for their time.
  • Don’t jump in and start talking when the other person takes a breath. 
  • Remain neutral and encouraging.
  • Show your engagement by asking for clarification and  restate their responses in your own words from time to time

Stage 5: Analyzing Interview Data

Each interview should help you figure out if your hypothesis addresses a valid pain point the customer has by uncovering:

  • Whether the customer tried to solve that pain point on their own.
  • Whether the pain point is large enough that the customer  wants it fixed.
  •  If the are any impediments stopping the customer from using your solution.

If you follow something close to the process described below, after 15 to 20 interviews, you should have a good idea about the validity your  hypothesis:

  • Highlight 8 to 10 of the most interesting points each subject said that validated or invalidated your hypothesis
  • Pay attention to actionable words your customers use.
    • If they use words that describe how they have tried to solve the problem or clearly articulate the benefits of the solution is indicative of someone who would use your product.
  • Assemble the data from all your interviews together into an overall summary that you update as you go.
    • Categorize what the customers into two buckets “validates” and “invalidates” hypothesis .

Wrapping It All Up

Properly performed, interviews and customer surveys provide product management teams with a source of hypothesis validation and a good sense of how your product and/or changes to it, would fit into the customer’s life , why it might replace something else, or  could be ignored by the marketplace.

  • If you haven’t found anyone excited about your idea in interviews, either you’re talking to the wrong people or your problem isn’t a real problem for customers.
  • Continue your research by changing target subjects and if you still get negative results,  consider your hypothesis invalidated.
  • If few people see your opportunity as a pain point they care about, the potential market for your product might not be large enough to matter.
  • Sometimes customers will highlight a pain point that  different from your opportunity which might be worth investigating.
Table of Contents
More From Our Blog