User Research: Biases, DOs & DON’Ts

RESEARCH IS NOT…

• A demo to sell a product
• Word for word everything the user wants, sometimes known as “Requirements Gathering”. We look for pains, gaps and needs. We dig deeper to ask why they want something, when they last needed it & build a timeline for such functionality.

RESEARCH BIASES

Biases can and will interfere with gathering valuable and accurate research. Be aware of the following biases:

• Social Desirability Bias (participant)
  • Answer questions in a way that will lead to them being liked & accepted
    To counteract: positive reinforcement of criticism or negative feedback
• Sponsor Bias (participant)
  • Feelings about the sponsor, such as a company, client, or stakeholder, may bias their answers
    To counteract: Stay neutral about sponsor feedback
• Confirmation Bias
  • Using participants answer to confirm their own belief. We tend to remember points that support our hypothesis and filter out points that disprove.
    To counteract: Re-evaluate participant’s expressions, perform research synthesis as a group, challenge preexisting assumptions
• Leading question and wording Bias
  • Elaborating on a participant’s answer puts words into their mouth
    To counteract: Use participant’s own words, avoid summarizing and ask them to summarize once more instead

RESEARCH INTERVIEW DOS AND DON’TS

Below is a list of interview dos and don’ts that you can go over with your team to help them craft questions that draw insightful information from people.

Do: Start the interview with simple questions.

Don’t start with hard questions. Ease into them.

Starting with easy topics will help people get relaxed, comfortable, and talking.

Do: Ask open-ended questions with “What”, “How” and “Why”.

Don’t ask leading questions or lead users through tasks.

Don’t ask questions with “Did”, “Is”, or “Would” that provide only yes or no answers.

The goal is to hear people speak in their own words as much as possible. It’s very easy for people to say they would buy your product until they actually have to pay. Past behavior is the best predictor of future behavior.

Do: Seek concrete stories, past actions & behaviors.

Don’t seek generalizations or ask about future behavior.

The goal is to understand what happened in their past to give us clues about why they think or feel things today.

Do: Give people time to answer the question.

Don’t fill in silence with more questions.

The goal is to give people space to think because being interviewed is difficult. You can also use silence as a way to extract even more information that people may be hesitant to give. Related, also feel free to move forward if the conversation is digressing or come back to the question if someone is having a hard time answering.

Do: Nod and say less. The interviewee is always right.

Don’t correct or contradict a research participant if they do something “wrong”.

The goal is to get people speaking as honestly as possible and to encourage them to feel it’s okay when they don’t understand how to use something in an interview. Correcting interviewees makes them feel that they are being tested and they can easily become afraid to speak their mind for fear of looking dumb.

Do: Actively listen and dig deeper with “why” questions.

Don’t ignore users when they are struggling in silence, or they show surprise.

Dig deeper and ask why they are reacting a certain way.

The goal is to pay attention to the user as they are speaking and feel comfortable going off script when you see or hear something interesting. It can be easy to read straight off the script without asking follow up questions but it means that we miss out on something that may be valuable for us to know.

Do: Stay neutral about the product you are showing.

Don’t talk about or sell your product.

The goal with research is to have as unbiased feedback as possible. For example, interviewees should feel . Talking about your opinion, or even telling them what the product is supposed to do will subtly encourage interviewees to agree with you. Interviewees will feel inclined to be polite and make you feel good about your product to begin with.

Do: Wrap up with broad questions.

Don’t wrap up without having interviewees reflect on their experience.

Interviewees will have more to say. You can capture many interesting insights during the wrap-up.

INTERVIEW PRACTICE SILENT QUIZ

Here are 3 “bad” questions. Turn them into good questions.

• Was that helpful for you?
• What do you want?
• How do you like this?

INTERVIEW PRACTICE

Interview each other by only using “Tell me more about…” or “Why?”

Remember to regroup and learn from mistakes. Perfection is impossible.

RESEARCH SAMPLE STRUCTURE

Introduction

1. Introduce yourself and note taker.
2. Explain how long the interview will take and the general format.
3. State the goal is just to have a discussion about their experience with ______ .
4. No right or wrong answers to questions, the only right answer is their opinion. Let him/her know nothing they say will offend you.
5. Mention they should feel free to ask questions anytime; they’ll have time at the end for questions too.
6. Have you signed the Non Disclosure Agreement?
7. Do you mind if we record the session?
8. Any questions before we begin?

Storytelling Questions

These questions are generally asked during Exploratory Research.

Use these questions to explore these topics:

Who, What, Where, When, Why, How

• Tell me about what you do. What are your top responsibilities?
• Tell me about the last time…? And then what happened? Why (or how) did you do that?
• Tell me about a time that was difficult…?
• Your favorite moment…?
• What reasons helped you choose…?
• Anything else around phones?
• If you had a magic wand, what would you…?
• Scenarios/activities
• Real world context
• Pains and Joys
• First time, last time
• Best time, worst time
• Wish list

Reactionary Questions

These questions are asked generally during Validation and Usability Research.

• Can you describe the things you’re reading? What do you see?
• What do you think this page does? What can you do here?
• What questions are coming to your mind as you’re looking at this page?
• Before clicking on anything, what would you do next?
• What do you think would happen next?
• Why do you think that?
• When you mentioned ___, what did you mean?
• So what happened there?
• Was that what you expected? Why or why not?
• So what goes through your mind as you look at this?
• Which part of the page were you looking at?
• Did you find what you were looking for?
• What would you do next? Why?
• Is there anything else you would do at this point?
• Is there any other way to do that?
• What did you think of that?
• In what ways would you want this changed to make it better for you?
• What additional info would have helped?

Wrap Up Questions

• We’ve been talking about ___ for awhile, has this brought up anything you’ve been thinking lately?
• What were the top 2 things that worked well and didn’t work well?
• What was not clear? What would you change?
• What impact would this have given your situation?
• What does this remind you of?
• If we waved a magic wand to make this service better for you, what would that be?
• Is there anything else regarding … that you’ve been thinking of?
• What do you like/dislike about this?
• If you had 3 wishes to make this better for you, what would they be? Why?
• How would you describe this to a friend?
• Under what circumstances would you use this? Why?
• Can you describe to me what you see on this page?
• Which parts of this page are most/least important to you?
• What do you think this [point to a UI element] might do?
• What does this [point to a UI element] means?
• If you wanted to _______, how would you. . .?