Retour Back
The Power of Trick Questions: Enhancing Data Integrity at Made in Surveys

The Power of Trick Questions: Enhancing Data Integrity at Made in Surveys

4  mins 4 mins
|
By MIS Group - 16/07/2024

At Made in Surveys, we take pride in the quality of the data we collect. However, maintaining this standard isn't without its challenges. On average, we find ourselves deleting about 20% of survey responses. This happens because sometimes unserious participants manage to slip through and respond to our questionnaires. To combat this, our dedicated Research Managers are on a constant mission to uphold data integrity.

Meticulous Data Cleaning Process 

Once a survey is completed, our team immediately starts a meticulous cleaning process. They scrutinise everything from respondent locations and response times to the consistency of data and relevance of answers to open-ended questions. This manual checking ensures that only high-quality, reliable data makes it into our databases. It's a time-intensive task, particularly in areas with complex survey targets where we often need to anticipate a surplus of responses.

To streamline this process, we've implemented various proactive measures. These include using IP location blocking and setting minimum response times to swiftly identify and restrict responses from unreliable sources in real-time. This helps minimise the number of responses that require extensive cleaning after the fact.

Harnessing "Trick Questions" for Data Integrity

One of our key strategies is the use of "trick questions." These serve dual purposes: verifying participant authenticity and monitoring their engagement throughout the survey. We utilise two types of trick questions: verification questions and attention checks.

a. Verification Questions: 

Verification questions are strategically placed at the beginning of our questionnaires. They serve as a gatekeeper, blocking participants who cannot provide a valid answer. Alongside traditional Captcha tools (which prevent automated bots from skewing results) we've innovated with video questions. These ensure that respondents are not only present but actively engaged with multimedia content, reinforcing the authenticity of their responses.

b. Attention Checks: 

Equally important are attention checks sprinkled throughout the survey. These come in various forms: 

  • Repeated questions :  By asking the same question at both the beginning and end of a survey, we catch respondents whose answers don't align, signalling potential inconsistency or inattentiveness. 
  • Instructional questions : Introduced mid-survey, these straightforward tasks gauge respondent alertness such as asking them to identify a specific detail (e.g. the colour blue). 
  • Logical questions : Designed to ensure continuous engagement, these types of queries, (such as asking what day it is) filter out those who aren't fully engaged with the survey content. 

While effective, it's crucial to use these tools judiciously. Misapplication can inadvertently penalise genuine respondents. For example, altering the order of answers in repeated questions may lead to valid differences in interpretation, which shouldn't result in data deletion.

Commitment to Data Excellence 

At Made in Surveys, our commitment to data excellence extends beyond these methods. Whether through automated checks or meticulous manual verification, we strive to deliver datasets of the highest integrity. We're always exploring new ways to innovate and refine our processes, ensuring that the insights we provide are based on robust, reliable data.

What About You? 

How do you ensure the quality of the data you collect? Share your strategies with us—we're always eager to learn and improve!

See also our other articles

An Experiment in Survey Skew and Bias
An Experiment in Survey Skew and Bias
Quant

Crafting effective survey questions is essential to obtaining accurate and unbiased responses. Before writing a survey, consider how to phrase questions and what pitfalls to avoid to ensure you recruit the right respondents and avoid skewing the results. To address common survey-writing challenges, we conducted an experiment by presenting two...

See more
5 mins
The Multiverse of Market Research: 
An Overview of Survey Methodologies
The Multiverse of Market Research: An Overview of Survey Methodologies
Quant

Market research is a surprisingly diverse tool for understanding consumer behaviour, opinions and trends. It relies on a variety of different data sets that are analysed statistically to uncover insightful patterns and correlations. To collect the necessary data, researchers can employ many different survey methodologies, each with distinct feat...

See more
5 mins
Maximising Ad Campaign Impact: The Power of 
Consumer Testing
Maximising Ad Campaign Impact: The Power of Consumer Testing
Quant

We often discuss the value of ad campaign tests, but how effective are they really? And how can they be utilised to achieve quick and impactful results?

The Power of Testing Campaigns

Testing a campaign provides concrete and measurable outcomes. It's the most reliable way to clear any unc...

See more
2 mins