Blog



The Gold Standard: Probability Sampling

While probability sampling is rarely used with online research, it helps to keep probability sampling in mind because many of the techniques to improve the representativeness of online surveys mimic or emulate the key characteristics of probability sampling. 

Probability sampling has two key requirements

Probability sampling has two key requirements:

  1. COVERAGE – A known and non-zero chance of selecting any member of the target population. (Roger Sapsford and others argue it has to be an equal chance of selection.)
  2. EXTERNAL SELECTION - Random selection of members of the population to participate in the survey.

The requirement for external selection is frequently forgotten, yet it is vital. We have selected someone to participate in this survey, and we are going to keep reminding them of that fact until they finally participate. To maximize participation among those selected, surveys of mail probability samples often have five rounds of notification (an invitation and four reminders); telephone surveys may call a landline number dozens of times using automatic dialers; online surveys may follow-up an invitation with one or more reminder emails. 

Failure to follow through on external selection may lead to sampling bias. For instance, back in 1993, I worked with Bill Ablondi of Strategy Analytics on a landmark mobile professional study. The protocol was to call the professionals over a dozen times on their office numbers (this was before the age of affordable cell phones): otherwise our sample would have been skewed to those professionals who were in the office more (i.e., less mobile). 

Some critics argue that today’s probability sampling, with its low response rates, no longer qualifies as a scientific sample.

Yet this view is controversial, and repeated studies show that probability sampling is remarkably robust – even as response rates have dramatically declined, and the use of weighting has increased to compensate, the results have maintained similar accuracy.

It is possible to conduct online probability sampling – you just need a complete list of the target market being studied. This is surprisingly easy for e-commerce companies: Amazon, Ebay, and JetBlue can all readily conduct probability surveys by randomly selecting email addresses from their databases and then sending an invitation and a series of reminders. This has the greatest chance of minimizing sampling bias. 

Most organizations are not so lucky. And even Amazon, Ebay, and JetBlue can’t conduct online probability sampling on their own if they want to survey book readers, bargain hunters, or travelers in general. 

For online surveys representative of wider markets, you will have to rely on other methods, which may or may not approximate some of the characteristics of probability samples.

This is an excerpt from the free Researchscape white paper, "Improving the Representativeness of Online Surveys”. Download your own copy now.

 

Recent Headlines

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. PDI Releases New C-Store Shopper Report Jeffrey Henning 26-Jun-2019
  4. Issues with Grid Questions Jeffrey Henning 24-Jun-2019


White Papers


Consumer Research

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  4. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019


From Jeffrey Henning's Blog

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Asking Too Much Jeffrey Henning 07-Jun-2019
  3. Helping You Through the Entire Survey Research Process Jeffrey Henning 02-Mar-2019
  4. Market Research Conference Calendar - 2019 Jeffrey Henning 12-Dec-2018