Blog



Early Dashboard Results Skewed Until Quota Sampling Kicks In

To improve the representativeness of our online surveys, we – like most survey researchers – use quota sampling to closely mirror the U.S. population. In our case, we may go a little overboard. We allocate 32 quota cells for each combination of age range, gender, and region, and stop accepting results after we’ve achieved the quota limit for each cell. For instance, in a survey of 2,000 U.S. adults aged from 18- to 71-years old, the quota cell for 18- to 22-year-old males in the Northeast will require 19 completes, and the quota cell for 36- to 55-year-old females in the South (a wider age range in a more populous region) will require 158 completes. Once the target for a quota cell is achieved, subsequent respondents of that demographic are screened out of the survey.

If we didn’t use quota cells, our samples would over-represent middle-aged adults (and conversely under-represent 18-22 year olds and 56-71 year olds) and would over-represent women (and conversely under-represent men).

Because the demographics of online panels don’t exactly mimic the U.S. population, early incoming results to live surveys exhibit exactly the bias we are seeking to correct for! As a result, early dashboards can be misleading about final results, in those cases where attitudes and behaviors prove to differ by age or by gender.

Here are how our quota cells typically fill up, from first to last, in order of hitting the target number of completes:

  1. 36-55, Female, South
  2. 23-35, Female, South
  3. 36-55, Female, Northeast
  4. 36-55, Female, Midwest
  5. 36-55, Female, West
  6. 23-35, Female, West
  7. 56-71, Female, South
  8. 23-35, Female, Northeast
  9. 23-35, Female, Midwest
  10. 36-55, Male, South
  11. 23-35, Male, South
  12. 23-35, Male, West
  13. 23-35, Male, Northeast
  14. 36-55, Male, West
  15. 56-71, Female, Midwest
  16. 36-55, Male, Northeast
  17. 56-71, Female, West
  18. 36-55, Male, Midwest
  19. 56-71, Female, Northeast
  20. 23-35, Male, Midwest
  21. 56-71, Male, South
  22. 18-22, Female, South
  23. 56-71, Male, Midwest
  24. 56-71, Male, West
  25. 56-71, Male, Northeast
  26. 18-22, Female, West
  27. 18-22, Male, South
  28. 18-22, Female, Midwest
  29. 18-22, Female, Northeast
  30. 18-22, Male, West
  31. 18-22, Male, Northeast
  32. 18-22, Male, Midwest

This means surveys can get 80% of their results right away and then slowly march on trying to fill up the final quota cells.

But, by eliminating a known source of bias, the wait ends up being worth it.

For more on quota sampling and other techniques for improving the quality of online research, see the free Researchscape white paper, "Improving the Representativeness of Online Surveys”. Download your own copy now.

Email Address*  
Company*
First Name*  
Last Name*  
 


 

Recent Headlines

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. PDI Releases New C-Store Shopper Report Jeffrey Henning 26-Jun-2019
  4. Issues with Grid Questions Jeffrey Henning 24-Jun-2019


White Papers


Consumer Research

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  4. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019


From Jeffrey Henning's Blog

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Asking Too Much Jeffrey Henning 07-Jun-2019
  3. Helping You Through the Entire Survey Research Process Jeffrey Henning 02-Mar-2019
  4. Market Research Conference Calendar - 2019 Jeffrey Henning 12-Dec-2018