Blog



The Gold Standard: Probability Sampling

While probability sampling is rarely used with online research, it helps to keep probability sampling in mind because many of the techniques to improve the representativeness of online surveys mimic or emulate the key characteristics of probability sampling. 

Probability sampling has two key requirements

Probability sampling has two key requirements:

  1. COVERAGE – A known and non-zero chance of selecting any member of the target population. (Roger Sapsford and others argue it has to be an equal chance of selection.)
  2. EXTERNAL SELECTION - Random selection of members of the population to participate in the survey.

The requirement for external selection is frequently forgotten, yet it is vital. We have selected someone to participate in this survey, and we are going to keep reminding them of that fact until they finally participate. To maximize participation among those selected, surveys of mail probability samples often have five rounds of notification (an invitation and four reminders); telephone surveys may call a landline number dozens of times using automatic dialers; online surveys may follow-up an invitation with one or more reminder emails. 

Failure to follow through on external selection may lead to sampling bias. For instance, back in 1993, I worked with Bill Ablondi of Strategy Analytics on a landmark mobile professional study. The protocol was to call the professionals over a dozen times on their office numbers (this was before the age of affordable cell phones): otherwise our sample would have been skewed to those professionals who were in the office more (i.e., less mobile). 

Some critics argue that today’s probability sampling, with its low response rates, no longer qualifies as a scientific sample.

Yet this view is controversial, and repeated studies show that probability sampling is remarkably robust – even as response rates have dramatically declined, and the use of weighting has increased to compensate, the results have maintained similar accuracy.

It is possible to conduct online probability sampling – you just need a complete list of the target market being studied. This is surprisingly easy for e-commerce companies: Amazon, Ebay, and JetBlue can all readily conduct probability surveys by randomly selecting email addresses from their databases and then sending an invitation and a series of reminders. This has the greatest chance of minimizing sampling bias. 

Most organizations are not so lucky. And even Amazon, Ebay, and JetBlue can’t conduct online probability sampling on their own if they want to survey book readers, bargain hunters, or travelers in general. 

For online surveys representative of wider markets, you will have to rely on other methods, which may or may not approximate some of the characteristics of probability samples.

This is an excerpt from the free Researchscape white paper, "Improving the Representativeness of Online Surveys”. Download your own copy now.

 

Recent Headlines



White Papers