An Experiment in Survey Skew and Bias

Crafting effective survey questions is essential to obtaining accurate and unbiased responses. Before writing a survey, consider how to phrase questions and what pitfalls to avoid to ensure you recruit the right respondents and avoid skewing the results.

To address common survey-writing challenges, we conducted an experiment by presenting two different surveys to our panellists. The first survey was designed to elicit skewed results, while the second aimed to be neutral. Both surveys were completed by 1,600 individuals in France, aged 18 to 65, representing the general population.

The findings were enlightening. Let's delve into the results...

Our experiment confirmed that using complex scientific terms can significantly alter responses. For instance, participants were less likely to recognise the health benefits of 'thiamine' compared to 'vitamin B1'. The unfamiliarity with scientific names like thiamine, riboflavin, niacin, pantothenic acid, and pyridoxine often led participants to believe these substances had no health benefits. Simplifying language to common terms can lead to more accurate responses.

Double negatives can confuse respondents and skew results. We tested statements like "French people are interested in politics" versus "It would be incorrect to say that French people are not interested in politics." Despite both statements meaning the same, 51% agreed with the first, while 64% agreed with the second. This highlights the complexity double negatives introduce and their impact on survey accuracy.

In screener questions, revealing the desired answer can bias respondents. For example, when recruiting participants who drink hot chocolate for breakfast, asking directly ("Do you drink hot chocolate for breakfast?") led to 43% affirmative responses. However, when hot chocolate was one option among many, only 28% selected it. Concealing the target answer among other options prevents skewed recruitment.

We tested satisfaction scales by asking participants about a well-known coffee brand. Half saw a scale from "Excellent brand" to "Terrible brand," and the other half saw "Terrible brand" to "Excellent brand." The results were consistent across both groups. However, maintaining a consistent order in satisfaction scales within a survey helps avoid confusing respondents and ensures reliable data.

When asking for opinions, providing a balanced list of adjectives is crucial. If options are predominantly positive, responses will skew positively. Additionally, respondents may choose "Other" if they feel the provided options don't reflect their views, leading to an increased volume of open-ended responses. Ensuring a comprehensive list of adjectives allows for more accurate data collection.

Incorporating images or videos can enhance survey accuracy, especially for brand recognition questions. While well-known brands like Nike don't require visual aids for recognition, lesser-known brands like Intel benefit from logo inclusion. Visual aids cater to more visual respondents and can lead to more accurate responses.

We tested the impact of imagery by showing participants different photos of French actor Gérard Depardieu. One group saw a neutral photo, while the other saw him with a Russian passport. The neutral photo group had a more positive view (62% vs. 50%). Using neutral imagery avoids influencing respondents' opinions.

Randomising response options can prevent biases. For example, when listing 19 options for participants to describe their perfect shoe, randomising the order prevents the top options from being selected more frequently. In cases where order matters, such as alphabetical listings, logical order can be maintained.

Price questions can be tricky. We tested price brackets for a pod coffee machine. One group saw six brackets starting with "Less than €40," while the other saw seven brackets starting with "Less than €30" and "Between €30 and €39." Contrary to logic, 20% of the first group chose "Less than €40," whereas 26% of the second group chose the combined lower brackets. Keeping price questions simple and straightforward is crucial.