Blog



10 Tips to Sample Better

In a webinar today, Annie Pettit, the chief research officer of Peanut Labs, shared 10 tips to make better use of survey sampling.

1. Have a SMART objective. Annie advocates setting a goal that is S.M.A.R.T.: Specific, Measurable, Attainable, Relevant, and Time-bound. For instance, instead of something vague like "learn about our brand" or "understand what brand users think of our products", be specific: "understand who uses SKU [Shelf-Keeping-Unit, a reference for retailers] XYZ at least once per week and why they choose our SKU over competitive SKUs."  

2. Define the target group characteristics. Be precise when screening respondents. For instance, coffee usage, coffee purchase, and coffee consumption are not the same. It's the difference between someone who might use coffee grinds in their garden, someone who might buy coffee for their spouse, and someone who actually drinks coffee.  

3. Define who is not in your target group. Don't exclude people from your sample out of habit. Including them can help you answer questions like: Why are non-users rejecting your category? Why are competitive users rejecting your brand? Why aren't low volume users translating into high volume users? Why don't switchers remain loyal to your brand? 

4. Slower is better. Yes, you can field an online survey in an hour. Or a few hours. Or a day. But by doing so you skew your results towards those people who were at their computer at that moment. "The best thing you can do," said Annie, "is to leave your study open for a full week. That way you hit everyone, not just people who are on their computers right this second."  

5. Stay in field. Similarly, speed can compromise quality. If you are using quota sampling, budget the extra time to let the harder-to-reach cells fill up. For instance, it can be difficult to recruit younger respondents. "You should simply wait for those people. It is the only alternative – only close fielding when you hit at least 90% of every target. Then, if you have to weight a group, the weights are small and less likely to bias the results. Tell your kids to stay in school, and tell yourself to stay in field!" 

6. Use a large sample size. Imagine that 43% of the sample likes product concept 1, and 30% likes product concept 2. If you have 100 responses, this finding falls within the margin of error. If you have 400 responses, then this difference is more likely to be meaningful. Another benefit of larger sample sizes is that the subgroups within the sample are then larger, making it easier to compare the results from different subgroups.  

7. Sample based on return rates. When inviting participants to take your survey, oversample those with lower response rates: men, younger people, non-whites, and those with higher income.  

8. Pre-test any new targeting criteria. What is a heavy user of Starbucks coffee? Is it someone who purchases 1+ times per week, 5+ times per week, or 10+ times per week? You want to make sure you have enough responses in the loyal customer segments that you are studying. Pre-test the most important questions to ensure that you have identified the right group and will have a sufficiently large sample size at the end. "Pre-test 2 or 3 alternatives to make sure you have enough people in your important buckets," said Annie.  

9. Forget preconceived notions – Given the research objectives, embrace the sampling method and research methodology that makes the most sense.  

10. Accept the errors – "Where do errors enter a research project?" Annie asked. "Everywhere!" A project can have improperly operationalized objectives, biased questions, incomplete sample frames, non-response bias, self-selection, data editing and recording errors, poor data analysis, and weak report writing. You might miscode a response in the SPSS or SAS file or use unclear language. Be vigilant – when you find out something with your research didn't work out the way you hoped, be ready to think about the problems that might have arisen, not just the obvious problem of sampling bias.

For better sampling, plan ahead, pre-test as much as possible, leave sufficient fielding time in your schedule, use a sufficiently large sample size, and understand how to recognize and work around errors.

 

Recent Headlines

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. PDI Releases New C-Store Shopper Report Jeffrey Henning 26-Jun-2019
  4. Issues with Grid Questions Jeffrey Henning 24-Jun-2019


White Papers


Consumer Research

  1. Postgraduate Degree Survey Jeffrey Henning 10-Jul-2019
  2. Voice Assistant Survey: Let the Young Speak Jeffrey Henning 09-Jul-2019
  3. Grocery Shopping Survey Jeffrey Henning 22-Jun-2019
  4. 2019 Summer Vacation Survey Jeffrey Henning 11-Jun-2019


From Jeffrey Henning's Blog

  1. Issues with Grid Questions Jeffrey Henning 24-Jun-2019
  2. Asking Too Much Jeffrey Henning 07-Jun-2019
  3. Helping You Through the Entire Survey Research Process Jeffrey Henning 02-Mar-2019
  4. Market Research Conference Calendar - 2019 Jeffrey Henning 12-Dec-2018