The Power of Trick Questions: Enhancing Data Integrity at Made in Surveys

At Made in Surveys, we take pride in the quality of the data we collect. However, maintaining this standard isn't without its challenges. On average, we find ourselves deleting about 20% of survey responses. This happens because sometimes unserious participants manage to slip through and respond to our questionnaires. To combat this, our dedicated Research Managers are on a constant mission to uphold data integrity.

Once a survey is completed, our team immediately starts a meticulous cleaning process. They scrutinise everything from respondent locations and response times to the consistency of data and relevance of answers to open-ended questions. This manual checking ensures that only high-quality, reliable data makes it into our databases. It's a time-intensive task, particularly in areas with complex survey targets where we often need to anticipate a surplus of responses.

To streamline this process, we've implemented various proactive measures. These include using IP location blocking and setting minimum response times to swiftly identify and restrict responses from unreliable sources in real-time. This helps minimise the number of responses that require extensive cleaning after the fact.

One of our key strategies is the use of "trick questions." These serve dual purposes: verifying participant authenticity and monitoring their engagement throughout the survey. We utilise two types of trick questions: verification questions and attention checks.

Verification questions are strategically placed at the beginning of our questionnaires. They serve as a gatekeeper, blocking participants who cannot provide a valid answer. Alongside traditional Captcha tools (which prevent automated bots from skewing results) we've innovated with video questions. These ensure that respondents are not only present but actively engaged with multimedia content, reinforcing the authenticity of their responses.

Equally important are attention checks sprinkled throughout the survey. These come in various forms: 

• Repeated questions :  By asking the same question at both the beginning and end of a survey, we catch respondents whose answers don't align, signalling potential inconsistency or inattentiveness. 
• Instructional questions : Introduced mid-survey, these straightforward tasks gauge respondent alertness such as asking them to identify a specific detail (e.g. the colour blue). 
• Logical questions : Designed to ensure continuous engagement, these types of queries, (such as asking what day it is) filter out those who aren't fully engaged with the survey content. 

While effective, it's crucial to use these tools judiciously. Misapplication can inadvertently penalise genuine respondents. For example, altering the order of answers in repeated questions may lead to valid differences in interpretation, which shouldn't result in data deletion.

At Made in Surveys, our commitment to data excellence extends beyond these methods. Whether through automated checks or meticulous manual verification, we strive to deliver datasets of the highest integrity. We're always exploring new ways to innovate and refine our processes, ensuring that the insights we provide are based on robust, reliable data.

How do you ensure the quality of the data you collect? Share your strategies with us—we're always eager to learn and improve!