During a recent data collection orientation for new CMB employees, someone asked me how we select the online sample providers we work with on a regular basis. Each week, my Field Services team receives multiple requests from sample providers—some we know from conferences, others from what we’ve read in industry publications, and some that are entirely new to us.

When vetting new sample providers, a good place to start is the ESOMAR 28 Questions to Help Buyers of Online Samples. Per the site, these questions “help research buyers think about issues related to online samples.”

An online sample provider should be able to answer the ESOMAR 28 questions; consider red flagging any that won’t. If their answers are too brief and don’t provide much insight into their procedures, it’s okay to ask them for more information, or just move along to the next.

While all 28 questions are valuable, here are a few that I pay close attention to:

Please describe and explain the type(s) of online sample sources from which you get respondents. Are these databases?  Actively managed research panels?  Direct marketing lists?  Social networks?  Web intercept (also known as river) samples?  

Many online sample providers use multiple methods, so these options aren’t always exclusive. I’m a firm believer in knowing where the sample is coming from, but there isn’t necessarily one “right” answer to this question. Depending on the project and the population you are looking for, different methods may need to be used to get the desired results.

Are your sample source(s) used solely for market research? If not, what other purposes are they used for? 

Beware of providers that use sample sources for non-research purposes. If a provider states that they are using their sample for something other than research, at the very least you should probe them for more details so that you feel comfortable in what those other purposes are. Otherwise, pass on the provider.

Do you employ a survey router? 

A survey router is software that directs potential respondents to a questionnaire for which they may qualify. There are pros and cons to survey routers, and they have become such a touchy subject that several of the ESOMAR 28 questions are devoted to the topic of routers. I’m not a big fan of survey routers, since they can be easily abused by dishonest respondents. If a company uses a survey router as part of their standard practice, be sure you have a very clear understanding of how the router is used as well as any restrictions they place on router usage.

You should also be wary of any sample provider who tells you that your quality control (QC) measures are too strict. This happened to me a few years ago and, needless to say, it ended our relationship with the company. This is not to say that QC measures can’t be too restrictive, and in those cases you can actually be throwing out good data.

At CMB, we did a lot of research prior to implementing our QC standards.  We consulted peers and sample providers to get a good understanding of what was fair and reasonable in the market. We investigated speeding criteria, red herring options, and how to look at open-ended responses. We revisit these standards on a regular basis to make sure they are still relevant.

Since each of our tried and true providers support our QC standards, when a new (to us) sample provider tells us we’re rejecting too many of their panelists due to poor quality, you can understand how that raises a red flag. Legitimate sample providers will appreciate the feedback on “bad” respondents because it helps them to improve the quality of their sample.

There are tons of online sample providers in the marketplace, but not every partner is a good fit for everyone. While I won’t make specific recommendations, I urge you to consider the three questions I referenced above when selecting your partner.

At Chadwick Martin Bailey, we’ve worked hard to establish trusted relationships with a handful of online sample providers. They’re dedicated to delivering high quality sample and have a true “partnership” mentality.

In my world of data collection, recommending the best sample providers to my internal clients is extremely important. This is key to providing our clients with sound insights and recommendations that support confident, strategic decision-making.