Survey Bias with Examples: How To Stop Them From Creeping Into Your Survey

Bias: the four-letter word that offends every researcher, respondent, and recruiter on the planet. Since you’ve come to SurveySparrow, let’s talk about survey bias – one of the factors that can seriously mess up the accuracy (and quotability) of your survey data.

In this article, we’ll explore these topics:

• What is survey bias?
• Types of survey bias + how to foolproof your surveys against them
• Survey bias examples from the real world

What is survey bias?

First, what is bias? Bias is defined as “the action of supporting or opposing a particular person or thing” unfairly.

Our biases shape our opinions more than we’d like to admit. That’s why it’s frequently discussed in politics, the media, education, HR, sociology, organizational psychology, consumer psychology, etc.

In survey research, survey bias is an error introduced into the survey sample or the responses – usually by encouraging or discouraging one outcome over the others. Survey bias in product, customer, and employee surveys can lead to inaccurate results at best and disastrous decisions at worst.

Types of survey bias in research & how to avoid them

Let’s talk about the two Big Bads of survey bias.

• Selection bias: Biases that creep into various areas of the survey sample – like the selection method, target audience, sample size, etc.
• Response bias: Biases due to errors in the survey design; these errors lead respondents to answer in a certain way.

To reduce the second type of bias, you need to invest effort in framing the questionnaire. Survey templates can help you save a lot of time on this.

You can create a free account on SurveySparrow and get access to 600+ survey templates + our Enterprise software free for 14 days. Sign up below to start your free trial.

Now let’s explore some common types of bias within each category and how you can avoid them.

Selection bias

#1. Participation or non-response bias – Getting respondents to complete a survey

Are your survey responses mainly coming from a specific section of your audience? That is non-response bias. It shows that some respondents are dropping out of the survey or skipping it entirely. Often, non-respondents will have different traits from those who respond.

How to avoid non-response bias:

Here are our top tips:

• Keep your surveys short and fun to read. Trust us; people will be more likely to complete the study this way.
• Adjust the survey type (and timings) to the feedback type you need. For example, send a patient satisfaction survey ASAP after the visit, or ideally, within 24 hours. For feedback on sensitive topics, use anonymous surveys. Go with 360 feedback for performance reviews. You get the picture.
• Experiment with different survey channels (SMS, Slack, chat, etc.) to reach as many people as possible.

Check out this how-to guide for increasing survey responses for more tactics like this.

#2. Survey selection bias – Giving only partial representation

Survey selection bias is also known as sampling bias. It happens when your sample does not represent your audience well enough to meet your survey goals.

It can happen for many reasons. For example, maybe your survey selection criteria discouraged some groups from taking the survey. Or, perhaps you already decided who the target audience should be. Still, by doing so, you miss out on perspectives you hadn’t considered.

How to avoid selection bias:

• Focus on what you would like feedback on, who is best qualified to give that feedback, and how you will reach them. Journey mapping is a great way to start because it offers insights into the ICPs that connect with your product/service at different touchpoints.
• Personalize your survey according to your ICP, and use a survey share method that makes sense for them.
• Double-check the survey design to ensure it doesn’t leave out anyone relevant.

For tips on how to create your ICP, here’s a 7-minute guide.

#3. Confirmation bias – Cherry-picking data to support your position

Confirmation bias is the Evil Overlord of survey biases because it’s everywhere. In the context of survey research, you will find three kinds of confirmation bias:

• One-sided information search: When people seek evidence that supports their views, they ignore any evidence to the contrary.
• Biased interpretation of information: People interpret the survey evidence as supporting their existing position. For example, the belief that introverts can’t be leaders.
• Selective memory: People remember information selectively according to their own internal biases. Cinema has codified this as the Rashomon effect.

How to avoid confirmation bias:

• Acknowledge your own. That won’t fix the issue, but it will help you self-regulate.
• Create a research SOP and share it with your team so everyone will work according to the same values.
• Share the survey report (with a note on the process) for review.

#4. Survivorship bias – Focusing on the silent majority

‘Survivors’ are those prospects, clients, and employees who have opted to stick with your company. Therefore they are visible to you. However, it means that the feedback they have (while helpful) will not tell you why you’re failing in certain areas or how you can improve. It puts your decisions at risk of survivorship bias.

How to avoid survivorship bias:

• Customer and employee exit surveys are helpful tools for finding your weak spots.
• It’s hard to get a response from every lost prospect, churned client, or silent employee. However, following up with survey reminders can bring in more valuable insights.

Response bias

1. Extreme response bias – Striving for balance

Extreme response bias is the tendency of respondents to select the extreme answer options – even when they aren’t true. That is especially common in performance reviews when employees are asked to self-assess. But it can also be caused by leading questions or lack of time.

On the other side, we have the neutral responding bias, in which the person tends to pick a neutral answer every time!

How to avoid extreme response bias:

• Ensure that your survey questions have neutral wording.
• Randomize the answer options. It will encourage the respondents to pay attention to their answers. Reversed scale questions also work well.
• Give your participants enough time to complete the survey, and follow up with gentle reminders.

2. Recall bias – Covering gaps in memory

The timing of your survey also makes a difference in how users respond to it. Memories tend to fade with time. So chances are, the longer you wait, the less accurate the feedback becomes.

How to avoid recall bias:

• Set a time for triggering the survey. That also ensures that your survey lands in front of the users when they are most active.
• Avoid questions that ask to recall a past event – unless there are multiple perspectives to fill in the crucial details.
• Ask open-ended questions so that people can fill in their thoughts and experiences. Apart from context, this also helps with sentiment analysis.

3. Conformity bias/social desirability bias – Craving acceptance

Our insecurities can manifest in odd ways. One is conformity bias, where the respondent deliberately exaggerates or lies about their beliefs and habits to project a positive image.

If “accepted” norms of behavior influence respondents, this can affect their answers. But more problematically, they might try to avoid racism, casteism, or other negative associations that could paint them in a bad light. Unfortunately, this doesn’t leave much scope for diversity training or improvement.

How to avoid conformity bias:

• Make your survey anonymous.
• Word your questions carefully. Check out this guide to reduce conformity bias in your survey questions.
• Cross-reference the responses against existing survey data.

4. Courtesy bias – When politeness can get in the way

You went on a vacation but had a horrible experience with your hotel. The breakfast was late; service was slow, etc. You’re ready to raise hell. But when the manager asks about your experience, you tone it down because you don’t want to cause trouble.

The example above is courtesy bias at work. Courtesy bias is the tendency to understate your unhappiness because you don’t want to offend the other person. Some cultures are more prone to courtesy bias as there is a strong emphasis on social desirability and understatement.

How to avoid courtesy bias:

• Observe your respondent’s ethnic and cultural background, and frame the questions accordingly.
• Emphasize in your survey introduction that you need both happy and unhappy feedback to improve.
• Use customer and employee NPS as a way to combat courtesy bias. People expressing understated dissatisfaction are likely to choose 6-7. In contrast, people who choose any rating below 6 are likely to be more vocal about their unhappiness.

Survey bias examples

Example 1: In a study of patient satisfaction surveys in a healthcare organization, highly satisfied patients were more likely to respond.

However, patients dissatisfied with the quality of their care were less likely to respond.

That non-response bias led to overestimating patient satisfaction scores for that healthcare organization.

Example 2: The next time you see a toothpaste commercial that says, “9 out of 10 doctors recommend..” it might be worth your time to tweet them and ask:

“Did nine out of ten doctors say that your brand was the best?”

‘What was the size of the sample?”

“When was the study conducted?”

“The sample size and population can be anything the advertiser wants it to be, including literally having a panel of only ten doctors asked (and cherrypicking doctors until they get the result they want).”

• TVTropes

Example 3: In social media, confirmation bias is amped up by the platform’s algorithmic filters.

For example, Facebook and Instagram tend to display to users the posts and polls from influencers they are likely to agree with, creating filter bubbles that exclude opposing views.

Possible types of survey questions that cause bias

Have you ever wondered why some survey questions get you puzzled or lead you to answer in a certain way? It’s all about how the questions are asked. Some questions can be tricky, pushing you towards a specific answer or making you feel stuck.

Let’s check out six types of survey questions that can trip you up. We’ll keep it simple and fun, so you can spot these questions next time you take a survey. Ready to dive in? Let’s go!

Leading Questions

We all have a friend who always tries to make you agree with them. That’s what a leading question does. It’s not just asking for your opinion; it’s pushing you towards a specific answer. For example, “How much did you love our recent event?” already suggests that you loved the event. It doesn’t give you much room to say you didn’t like it or thought it was just okay.

Loaded Questions

Loaded questions are like being cornered by someone who asks you a question that makes you look bad no matter how you answer. Take this one: “Why do you spend so much time on your phone?” If you answer, it sounds like you admit to using your phone a lot. It’s a question that’s not just seeking information; it’s also making a judgment.

Double-Barrelled Questions

Answering a question that’s trying to do two things at once. It’s like being asked, “Do you like pizza and going to the gym?” Maybe you love pizza but don’t like the gym, or vice versa. This type of question mixes two different things, making it hard for you to give a clear answer about each one.

Absolute Questions

These are like yes-or-no questions that don’t allow for maybe. They force you to take a side. For instance, “Do you always eat healthy?” What if you mostly eat healthy but sometimes enjoy a burger? This kind of question doesn’t let you show that middle ground. It’s either all or nothing.

Ambiguous Questions

Ambiguous questions are super unclear. They’re like someone asking you, “Is our town good?” Well, good in what way? Safety? Fun places to go? Good schools? It’s too vague and doesn’t tell you what they want to know. You might be thinking about the parks in the town while they’re thinking about the schools.

Multiple Answer Questions 

These questions are like someone asking you to list everything you did on your last vacation. Where do you even start? For example, a question like, “What are all the things you like about our product?” can be overwhelming. You might forget some points or feel like it’s too much effort to answer everything, leading to incomplete or rushed responses.

How does survey bias influence search results?

Now let’s consider you’re organizing a big neighborhood party and want to decide on the music playlist. To choose the songs, you decide to survey people. However, if you only ask your close friends who love rock music, your playlist will end up being all rock. But what if half of your neighborhood loves pop or jazz? Your rock-heavy playlist won’t represent everyone’s music taste.

This is similar to how survey bias can affect search results. Search engines use data to understand what people are interested in and then show them relevant information. If this data comes from a biased survey (like only asking rock fans about their music preferences), then the search results might be skewed toward those preferences.

For example, if a biased survey suggests that most people are interested in a particular topic, like “best rock bands,” search engines might show more content related to rock bands, even though there’s also a high interest in other music genres. This means someone looking for “best jazz artists” might not get as many useful results because the data feeding the search algorithm is skewed towards rock.

Also, if the survey is biased in asking questions, like leading people to favor certain answers, this can further skew search results. For example, if the survey asks “Do you prefer rock music over other less exciting genres?” it’s pushing respondents to favor rock music. This can lead to the overrepresentation of rock in search results, even if in reality, preferences are more evenly spread across different genres.

In summary, unbiased and well-designed surveys are crucial because they provide accurate data that search engines use to reflect a wide range of interests and preferences. If the survey data is biased, it’s like having a party where only one music genre is played, even though guests have varied tastes. Surveys need to capture a broad and accurate picture so that search results can cater to everyone’s interests, not just a select few.

Wrapping Up

That’s all folks! In this article, we’ve gone through the eight types of survey bias that can seriously mess up the quality of your data. We’ve also shared some tips on what you can do to avoid them.

Of course, we are only human, and it’s hard to eliminate bias 100%. But with a bit of awareness, soul searching, and test surveys, you can discover the most egregious biases and eliminate them before they can do some serious damage. Design the best bias-free conversational surveys that work with SurveySparrow!

FAQs

In survey design, which aspect best explains ways to minimize response bias?

To reduce bias in surveys, ask questions in a straightforward, unbiased way. Think of it like having a chat where you’re just listening, not guiding the conversation. Also, the order of questions matters – don’t let earlier questions influence the answers to later ones.

Which type of survey is most likely to be biased?

Surveys, where people talk about their habits or opinions (like how often they exercise), are most likely to have bias. This is because we often see ourselves and our actions in a certain light, which might not be completely accurate.

If a public opinion survey suffers from selection bias, what might be the reason?

If a public opinion survey has selection bias, it usually means not everyone has an equal chance to take part. Imagine only asking people in one city about a national issue – their views might not represent the whole country.

How bias can affect the outcome of a survey?

Bias in surveys can twist the results, making them less true to what people think or do. It’s like wearing colored glasses – they can change how you see things, making the survey results less clear and less accurate.