Biased Questions: How to Identify & Fix Them in Surveys
A single pejorative phrase before the question, or one extra option in the answer set, is enough to skew survey results in a direction you didn't intend. Biased questions are the most common — and most invisible — survey-design mistake. Here's what they look like, why they happen, and how to fix them with real examples.
On this page
Designing a customer feedback survey in way that provides and concise data you can use in you business is not always simple.
Using Biased Questions is a common mistake among survey designers and this post outlines what they are, how to identify them and how to fix them.
What are Biased questions
Biased questions are ones that, intentionally or unintentionally, may influence the way the respondent will answer. Take this example:
Example of an Intentionally Biased Survey Question
By prefacing the question (“Overall how satisfied…”) with the pejorative statement “… is continually striving to achieve 5-STAR customer satisfaction…” the company has subtly altered the respondents’ feelings towards their experience.
Question bias can also be caused through the options the respondent is offered.
Here’s a perfectly balanced question turned into a biased question through the use of limited response options:
Balanced Survey Question Turned into Biased Question Through Response Options
Of course no company wanting to really understand customer loyalty would write the question like this but it demonstrates what can happen.
Why Biased Questions are a Problem in Surveys
The main reasons such questions are a problem in surveys is they distort the data collected.
Most of the time surveys and questionnaires are trying to accurately understand a situation or perception of a customer or respondent. By using biased questions the organisation writing the survey is distorting the data collected making it less reliable and accurate.
Interestingly not only can the data be distorted it can also be distorted in unknown ways.
Take the example question above.
Typically this style of distortion is done by organisation trying to maximise some feedback score but it’s a mistake because the respondent could go either way:
Reduce the score they were going to give: Some people will read the question and their expectations will rise, depressing the score they give: “well they were going for 5-star and that was only 3 star.”
Increase the score they were going to give: other people will subtly increase their score because they don’t want to disappoint the company by marking them down.
The problem is that you will never really know the effect.
Why Do Biased Questions Occur?
There are two main reasons biased questions turn up in surveys:
Unintentionally
Often it can slip into survey questions unintentionally, especially where the question writer has a strong affiliation with the organisation or subject of the survey and want it to be seen in a positive light, or not in a negative way.
Intentionally
However, just as often questions are written in a biased manner to attempt to influence either the response or the respondent. so the results of the survey analysis support one or other of the writer’s agendas. This is termed Push Polling.
What are not Biased Questions
There are plenty of poor ways to word survey questions that are not necessarily biased including the following types:
Just because a question is not well written does not mean it is biased.
Biased Question Examples
Here are a few more biased question examples so you can get the feel for them.
Loaded Questions
Example Biased Question: Loaded Question
This question comes from the famous BBC comedy “Yes, Minister”. There are several pejorative words in this question: kill, danger, guns. Which biases people to answer in the affirmative – which was the intention.
Example Biased Question: Loaded Question
Unless this question occurs AFTER someone has already expressed dislike of the service, e.g. they have answered the Net Promoter question as a detractor, you’re inviting them to find fault where there might have been none.
Example Biased Question: Loaded Question
Adding “disrespectable “ to the question clearly biases the response.
Language That Excludes Particular Groups
For example the use of gendered language in survey questions. Surveys that use language such as “he or she” or “him or her” to refer to participants may exclude individuals who do not identify as male or female, such as those who are non-binary or gender non-conforming. This can lead to a lack of representation of these groups in the survey results, potentially leading to inaccurate conclusions about the experiences or perspectives of those groups.
Other examples of this issue include racial, political or other social groupings.
How to Spot a Biased Question
Unlike Double Barrelled Questions, which are easy to spot, it can sometimes be difficult to spot a biased question in a survey because the form bias can occur in so many different ways.
In general if you see emotive, pejorative or assumptive words of any type or limited response options in a survey question, there is a chance you have created a biased question.
Even if you only suspect the question to be a problem you should attempt to correct it.
How to fix Biased Questions
Fixing biased survey questions can sometimes be a little tricky.
The best approach is to simply re-write the question in a way that is as neutral as possible and eliminate emotional, contentious and pejorative wording.
For example to fix this biased question:
Biased Question Example
You could rephrase it as:
Corrected Biased Question
Biased Question FAQ
What are Leading Questions
Leading questions are a type of biased question intended to intentionally influences the respondent. For example this famous question from the BBC comedy “Yes, Minister”.
“Do you think there’s a danger in giving young people guns and teaching them how to kill? “
