Overcoming Research Data Quality Challenges

The demand for online surveys is at an all-time high, which is a good sign that the market research industry is thriving. However, this demand creates a serious supply issue that makes feasibility challenging, especially for hard-to-reach populations. Research data quality can be a real issue.

Why current solutions fall short

When demand for sample is high, sample providers start scraping the bottom of the barrel to get clients as much sample as they can. Fraudsters thrive in this environment, often getting in and out of several surveys and earning their rewards before anyone has a chance to blacklist them. Because of this, having quality check measures in place is even more important when demand for sample is high.

Leaving quality checks up to the sample providers and thinking this will give us flawless sample is simply not realistic. It never has been, and it never will be. Every insights provider should only work with sample companies that deliver quality respondents on a consistent basis, but there are always some fraudsters who will find a way to cheat the system. Despite multiple sample providers claiming they provide better respondents with less cleaning needed, we still see fraudulent responses in every study. Depending on the population, this typically ranges from 5% to 30%.

Similarly, relying on third-party fraud prevention solutions is not good enough. Despite some hefty claims being made by the handful of major fraud prevention solutions, the fact is that they are always at least one step behind the “bad guys.” Even the Insights Association acknowledges that these fraud prevention solutions are not fully reliable, saying that while evolving quickly, this technology seems pretty nascent, pretty young, pretty immature at finding fraud.

Ensuring quality: A multi-layered approach

Sample providers and third-party solutions are pieces of the puzzle, but don’t completely solve the fraudster issue. There are a couple of very important steps that not everyone takes, because they believe the previously mentioned solutions are “good enough:”

• The first step is designing the questionnaire with multiple quality fails in place. These can include red herring questions, validating an earlier survey response, open-ended response review, among many others. Both intelligent instrument design and intentional survey programing can be your first line of defense.
• The second step is embedded fraud prevention and duplicate detection programs that identify a potential fraudulent response before they even start the survey. There are a multitude of ways that these respondents can be identified and removed, and finding and building quality metrics and respondent scoring algorithms are critical to success.
• The third step is to have a process in place to carefully review the data and identify any fraudsters who slipped through the cracks. I would argue that these are the most important steps in the entire research process. Both expert syntactic analytics and text entry analysis are key tools to ensure the responses meet quality standards whether they are fraud or inattentive respondents.

In addition, the most effective data integrity programs measure and score sample from providers across multiple measures of quality by type of project and sample to ensure the best mix of sample providers are used on any given study.

Insights are only as good as the data they are based on. We must keep fighting the battle on fraudsters for the sake of high-quality insights. The best way to do this in today’s world is to use a multi-pronged approach which must include the latest AI+HI™ from data managers who are trained at identifying and eliminating bad data. Otherwise, we would find ourselves in a “garbage in, garbage out” situation which not only impacts data quality and insights, but can impact client decisions and undermine the entire purpose for conducting research.

Interested in CMB’s data quality measures? Contact us to learn more and work with us.