Retour Back
An Experiment in Survey Skew and Bias

An Experiment in Survey Skew and Bias

5  mins 5 mins
|
By MIS Group - 09/01/2024

Crafting effective survey questions is essential to obtaining accurate and unbiased responses. Before writing a survey, consider how to phrase questions and what pitfalls to avoid to ensure you recruit the right respondents and avoid skewing the results.

To address common survey-writing challenges, we conducted an experiment by presenting two different surveys to our panellists. The first survey was designed to elicit skewed results, while the second aimed to be neutral. Both surveys were completed by 1,600 individuals in France, aged 18 to 65, representing the general population.

The findings were enlightening. Let's delve into the results...

Avoid Complex Vocabulary

Our experiment confirmed that using complex scientific terms can significantly alter responses. For instance, participants were less likely to recognise the health benefits of 'thiamine' compared to 'vitamin B1'. The unfamiliarity with scientific names like thiamine, riboflavin, niacin, pantothenic acid, and pyridoxine often led participants to believe these substances had no health benefits. Simplifying language to common terms can lead to more accurate responses.

Avoid Double Negatives

Double negatives can confuse respondents and skew results. We tested statements like "French people are interested in politics" versus "It would be incorrect to say that French people are not interested in politics." Despite both statements meaning the same, 51% agreed with the first, while 64% agreed with the second. This highlights the complexity double negatives introduce and their impact on survey accuracy.

Avoid Including the Answer in the Question

In screener questions, revealing the desired answer can bias respondents. For example, when recruiting participants who drink hot chocolate for breakfast, asking directly ("Do you drink hot chocolate for breakfast?") led to 43% affirmative responses. However, when hot chocolate was one option among many, only 28% selected it. Concealing the target answer among other options prevents skewed recruitment.

Avoid Changing the Order of Satisfaction Scales

We tested satisfaction scales by asking participants about a well-known coffee brand. Half saw a scale from "Excellent brand" to "Terrible brand," and the other half saw "Terrible brand" to "Excellent brand." The results were consistent across both groups. However, maintaining a consistent order in satisfaction scales within a survey helps avoid confusing respondents and ensures reliable data.

Avoid Option Lists That Are Too Short

When asking for opinions, providing a balanced list of adjectives is crucial. If options are predominantly positive, responses will skew positively. Additionally, respondents may choose "Other" if they feel the provided options don't reflect their views, leading to an increased volume of open-ended responses. Ensuring a comprehensive list of adjectives allows for more accurate data collection.

Avoid Text-Only Surveys

Incorporating images or videos can enhance survey accuracy, especially for brand recognition questions. While well-known brands like Nike don't require visual aids for recognition, lesser-known brands like Intel benefit from logo inclusion. Visual aids cater to more visual respondents and can lead to more accurate responses.

Avoid Influential Imagery

We tested the impact of imagery by showing participants different photos of French actor Gérard Depardieu. One group saw a neutral photo, while the other saw him with a Russian passport. The neutral photo group had a more positive view (62% vs. 50%). Using neutral imagery avoids influencing respondents' opinions.

Avoid Neglecting the Random Ordering Option

Randomising response options can prevent biases. For example, when listing 19 options for participants to describe their perfect shoe, randomising the order prevents the top options from being selected more frequently. In cases where order matters, such as alphabetical listings, logical order can be maintained.

Avoid Poorly-Constructed Price Questions

Price questions can be tricky. We tested price brackets for a pod coffee machine. One group saw six brackets starting with "Less than €40," while the other saw seven brackets starting with "Less than €30" and "Between €30 and €39." Contrary to logic, 20% of the first group chose "Less than €40," whereas 26% of the second group chose the combined lower brackets. Keeping price questions simple and straightforward is crucial.

See also our other articles

The Power of Trick Questions: Enhancing Data Integrity at Made in Surveys
The Power of Trick Questions: Enhancing Data Integrity at Made in Surveys
Quant

At Made in Surveys, we take pride in the quality of the data we collect. However, maintaining this standard isn't without its challenges. On average, we find ourselves deleting about 20% of survey responses. This happens because sometimes unserious participants manage to slip through and respond to our questionnaires. To combat this, our dedicat...

See more
4 mins
Maximising Ad Campaign Impact: The Power of 
Consumer Testing
Maximising Ad Campaign Impact: The Power of Consumer Testing
Quant

We often discuss the value of ad campaign tests, but how effective are they really? And how can they be utilised to achieve quick and impactful results?

The Power of Testing Campaigns

Testing a campaign provides concrete and measurable outcomes. It's the most reliable way to clear any unc...

See more
2 mins