Bias: the four-letter word that offends every researcher, respondent, and recruiter on the planet. Since you’ve come to SurveySparrow, let’s talk about survey bias – one of the factors that can seriously mess up the accuracy (and quotability) of your survey data.
In this article, we’ll explore these topics:
- What is survey bias?
- Types of survey bias + how to foolproof your surveys against them
- Survey bias examples from the real world
What is survey bias?
First, what is bias? Bias is defined as “the action of supporting or opposing a particular person or thing” unfairly.
Our biases shape our opinions more than we’d like to admit. That’s why it’s frequently discussed in politics, the media, education, HR, sociology, organizational psychology, consumer psychology, etc.
In survey research, survey bias is an error introduced into the survey sample or the responses – usually by encouraging or discouraging one outcome over the others. Survey bias in product, customer and employee surveys can lead to inaccurate results at best and disastrous decisions at worst.
Types of survey bias in research & how to avoid them
Let’s talk about the two Big Bads of survey bias.
- Selection bias: Biases that creep into various areas of the survey sample – like the selection method, target audience, sample size, etc.
- Response bias: Biases due to errors in the survey design; these errors lead respondents to answer in a certain way.
To reduce the second type of bias, you need to invest effort in framing the questionnaire. Survey templates can help you save a lot of time on this.
You can create a free account on SurveySparrow and get access to 600+ survey templates + our Enterprise software free for 14 days. Sign up below to start your free trial.
14-Day Free Trial • No Credit Card Required • No Strings Attached
Now let’s explore some common types of bias within each category and how you can avoid them.
Selection bias
#1. Participation or non-response bias – Getting respondents to complete a survey
Are your survey responses mainly coming from a specific section of your audience? That is non-response bias. It shows that some respondents are dropping out of the survey or skipping it entirely. Often, non-respondents will have different traits from those who respond.
How to avoid non-response bias:
Here are our top tips:
- Keep your surveys short and fun to read. Trust us; people will be more likely to complete the study this way.
- Adjust the survey type (and timings) to the feedback type you need. For example, send a patient satisfaction survey ASAP after the visit, or ideally, within 24 hours. For feedback on sensitive topics, use anonymous surveys. Go with 360 feedback for performance reviews. You get the picture.
- Experiment with different survey channels (SMS, Slack, chat, etc.) to reach as many people as possible.
For more tactics like this, check out this how-to guide for increasing survey responses.
#2. Survey selection bias – Giving only partial representation
Survey selection bias is also known as sampling bias. It happens when your sample does not represent your audience well enough to meet your survey goals.
It can happen for many reasons. For example, maybe your survey selection criteria discouraged some groups from taking the survey. Or, perhaps you already decided who the target audience should be. Still, by doing so, you miss out on perspectives you hadn’t considered.
How to avoid selection bias:
- Focus on what you would like feedback on, who is best qualified to give that feedback, and how you will reach them. Journey mapping is a great way to start because it offers insights into the ICPs that connect with your product/service at different touchpoints.
- Personalize your survey according to your ICP, and use a survey share method that makes sense for them.
- Double-check the survey design to ensure it doesn’t leave out anyone relevant.
For tips on how to create your ICP, here’s a 7-minute guide.
#3. Confirmation bias – Cherry-picking data to support your position
Confirmation bias is the Evil Overlord of survey biases because it’s everywhere. In the context of survey research, you will find three kinds of confirmation bias:
- One-sided information search: When people seek evidence that supports their views, they ignore any evidence to the contrary.
- Biased interpretation of information: People interpret the survey evidence as supporting their existing position. For example, the belief that introverts can’t be leaders.
- Selective memory: People remember information selectively according to their own internal biases. Cinema has codified this as the Rashomon effect.
How to avoid confirmation bias:
- Acknowledge your own. That won’t fix the issue, but it will help you self-regulate.
- Create a research SOP and share it with your team so everyone will work according to the same values.
- Share the survey report (with a note on the process) for review.
#4. Survivorship bias – Focusing on the silent majority
‘Survivors’ are those prospects, clients and employees who have opted to stick with your company. Therefore they are visible to you. However, it means that the feedback they have (while helpful) will not tell you why you’re failing in certain areas or how you can improve. It puts your decisions at risk of survivorship bias.
How to avoid survivorship bias:
- Customer and employee exit surveys are helpful tools for finding your weak spots.
- It’s hard to get a response from every lost prospect, churned client or silent employee. But following up with survey reminders can bring in more valuable insights.
Response bias
#1. Extreme response bias – Striving for balance
Extreme response bias is the tendency of respondents to select the extreme answer options – even when they aren’t true. That is especially common in performance reviews when employees are asked to self-assess. But it can also be caused by leading questions or lack of time.
On the other side, we have the neutral responding bias, in which the person tends to pick a neutral answer every time!
How to avoid extreme response bias:
- Ensure that your survey questions have neutral wording.
- Randomize the answer options. It will encourage the respondents to pay attention to their answers. Reversed scale questions also work well.
- Give your participants enough time to complete the survey, and follow up with gentle reminders.
#2. Recall bias – Covering gaps in memory
The timing of your survey also makes a difference in how users respond to it. Memories tend to fade with time. So chances are, the longer you wait, the less accurate the feedback becomes.
How to avoid recall bias:
- Set a time for triggering the survey. That also ensures that your survey lands in front of the users when they are most active.
- Avoid questions that ask to recall a past event – unless there are multiple perspectives to fill in the crucial details.
- Ask open-ended questions so that people can fill in their thoughts and experiences. Apart from context, this also helps with sentiment analysis.
#3. Conformity bias/social desirability bias – Craving acceptance
Our insecurities can manifest in odd ways. One is conformity bias, where the respondent deliberately exaggerates or lies about their beliefs and habits to project a positive image.
If “accepted” norms of behavior influence respondents, this can affect their answers. But more problematically, they might try to avoid racism, casteism or other negative associations that could paint them in a bad light. Unfortunately, this doesn’t leave much scope for diversity training or improvement.
How to avoid conformity bias:
- Make your survey anonymous.
- Word your questions carefully. Check out this guide to reduce conformity bias in your survey questions.
- Cross-reference the responses against existing survey data.
#4. Courtesy bias – When politeness can get in the way
You went on a vacation but had a horrible experience with your hotel. The breakfast was late; service was slow, etc. You’re ready to raise hell. But when the manager asks about your experience, you tone it down because you don’t want to cause trouble.
The example above is courtesy bias at work. Courtesy bias is the tendency to understate your unhappiness because you don’t want to offend the other person. Some cultures are more prone to courtesy bias as there is a strong emphasis on social desirability and understatement.
How to avoid courtesy bias:
- Observe your respondent’s ethnic and cultural background, and frame the questions accordingly.
- Emphasize in your survey introduction that you need both happy and unhappy feedback to improve.
- Use customer and employee NPS as a way to combat courtesy bias. People expressing understated dissatisfaction are likely to choose 6-7. In contrast, people who choose any rating below 6 are likely to be more vocal about their unhappiness.
Survey bias examples
#Example 1: In a study of patient satisfaction surveys in a healthcare organization, highly satisfied patients were more likely to respond.
However, patients dissatisfied with the quality of their care were less likely to respond.
That non-response bias led to overestimating patient satisfaction scores for that healthcare organization.
#Example 2: The next time you see a toothpaste commercial that says, “9 out of 10 doctors recommend..” it might be worth your time to tweet them and ask:
“Did nine out of ten doctors say that your brand was the best?”
‘What was the size of the sample?”
“When was the study conducted?”
“The sample size and population can be anything the advertiser wants it to be, including literally having a panel of only ten doctors asked (and cherrypicking doctors until they get the result they want).”
#Example 3: In social media, confirmation bias is amped up by the platform’s algorithmic filters.
For example, Facebook and Instagram tend to display to users the posts and polls from influencers they are likely to agree with, creating filter bubbles which exclude opposing views.
Wrapping Up
That’s all folks! In this article, we’ve gone through the eight types of survey bias that can seriously mess up the quality of your data. We’ve also shared some tips on what you can do to avoid them.
Of course, we are only human, and it’s hard to eliminate bias 100%. But with a bit of awareness, soul searching and test surveys, you can discover the most egregious biases and eliminate them before they can do some serious damage.
