Blog



The Paradox of Surveys of Small Population Sizes

Unfortunately for market researchers, the science of sampling error is often counterintuitive. For instance, the public considers convenience samples with tens of thousands of responses to be more accurate than random samples with 400 responses, not realizing that the random sample is far more accurate. Surveys of small populations are similarly counterintuitive, as some of my clients have bumped up against this year.

Three of my business-to-business clients had an average of approximately 100 customers that they surveyed. All overestimated the precision of the results based on the number of responses they received. One client had only 20 responses, which has a margin of error of 20%. In other words, a response of 50% to a question (e.g., 50% are in favor of feature X) has an actual range of 30% to 70% for the entire population of 100 clients, 19 times out of 20 (95% confidence). Another client had a 50% response rate to its census of customers – but this still provides a margin of error of 10% for a population of 100.

Margin of Error Declines Rapidly As Sample Size Increases

What makes this particularly counterintuitive is that 400 responses to a probability survey of the U.S. adult population of 235 million (a response from less than 0.0002% of the population) has a margin of error of 5%. To achieve that margin of error with a population of 100, you need to survey 80 people!

The key takeaway for B2B surveys is that much of the emphasis during fielding needs to be on increasing the response rate. You should implement multiple rounds of reminders, using multiple modes of contact (online and telephone at least) to drive that response rate higher.

Otherwise your survey results will only be qualitative, or directional, in nature.

 

Recent Headlines

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  3. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019
  4. Asking Too Much Jeffrey Henning 07-Jun-2019


White Papers


Consumer Research

  1. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  2. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019
  3. Vendor Communications Email Survey Jeffrey Henning 27-Apr-2019
  4. TTRPG, Gamebook, and Interactive Fiction Survey Jeffrey Henning 27-Apr-2019


From Jeffrey Henning's Blog

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Asking Too Much Jeffrey Henning 07-Jun-2019
  3. Helping You Through the Entire Survey Research Process Jeffrey Henning 02-Mar-2019
  4. Market Research Conference Calendar - 2019 Jeffrey Henning 12-Dec-2018