In a webinar today, Annie Pettit, the chief research officer of Peanut Labs, shared 10 tips to make better use of survey sampling.
1. Have a SMART objective. Annie advocates setting a goal that is S.M.A.R.T.: Specific, Measurable, Attainable, Relevant, and Time-bound. For instance, instead of something vague like “learn about our brand” or “understand what brand users think of our products”, be specific: “understand who uses SKU [Shelf-Keeping-Unit, a reference for retailers] XYZ at least once per week and why they choose our SKU over competitive SKUs.”
2. Define the target group characteristics. Be precise when screening respondents. For instance, coffee usage, coffee purchase, and coffee consumption are not the same. It’s the difference between someone who might use coffee grinds in their garden, someone who might buy coffee for their spouse, and someone who actually drinks coffee.
3. Define who is not in your target group. Don’t exclude people from your sample out of habit. Including them can help you answer questions like: Why are non-users rejecting your category? Why are competitive users rejecting your brand? Why aren’t low volume users translating into high volume users? Why don’t switchers remain loyal to your brand?
4. Slower is better. Yes, you can field an online survey in an hour. Or a few hours. Or a day. But by doing so you skew your results towards those people who were at their computer at that moment. “The best thing you can do,” said Annie, “is to leave your study open for a full week. That way you hit everyone, not just people who are on their computers right this second.”
5. Stay in field. Similarly, speed can compromise quality. If you are using quota sampling, budget the extra time to let the harder-to-reach cells fill up. For instance, it can be difficult to recruit younger respondents. “You should simply wait for those people. It is the only alternative – only close fielding when you hit at least 90% of every target. Then, if you have to weight a group, the weights are small and less likely to bias the results. Tell your kids to stay in school, and tell yourself to stay in field!”
6. Use a large sample size. Imagine that 43% of the sample likes product concept 1, and 30% likes product concept 2. If you have 100 responses, this finding falls within the margin of error. If you have 400 responses, then this difference is more likely to be meaningful. Another benefit of larger sample sizes is that the subgroups within the sample are then larger, making it easier to compare the results from different subgroups.
7. Sample based on return rates. When inviting participants to take your survey, oversample those with lower response rates: men, younger people, non-whites, and those with higher income.
8. Pre-test any new targeting criteria. What is a heavy user of Starbucks coffee? Is it someone who purchases 1+ times per week, 5+ times per week, or 10+ times per week? You want to make sure you have enough responses in the loyal customer segments that you are studying. Pre-test the most important questions to ensure that you have identified the right group and will have a sufficiently large sample size at the end. “Pre-test 2 or 3 alternatives to make sure you have enough people in your important buckets,” said Annie.
9. Forget preconceived notions – Given the research objectives, embrace the sampling method and research methodology that makes the most sense.
10. Accept the errors – “Where do errors enter a research project?” Annie asked. “Everywhere!” A project can have improperly operationalized objectives, biased questions, incomplete sample frames, non-response bias, self-selection, data editing and recording errors, poor data analysis, and weak report writing. You might miscode a response in the SPSS or SAS file or use unclear language. Be vigilant – when you find out something with your research didn’t work out the way you hoped, be ready to think about the problems that might have arisen, not just the obvious problem of sampling bias.
For better sampling, plan ahead, pre-test as much as possible, leave sufficient fielding time in your schedule, use a sufficiently large sample size, and understand how to recognize and work around errors.
Author Notes:
Jeffrey Henning
