Jon Krosnick, Professor of Communication, Political Science, and Psychology, Stanford University spoke at the NYAAPOR meeting on May 15th. On the topic of polling, Jon started by pointing out the crisis in the public image of polling, citing Nate Silver’s comment that polls are failing.
Jon explained that we as an industry can proceed in one of two directions. One path, he described at ‘the good path’ contrasting with the other path leading to ‘the death of surveys’.
I was able to catch up with Scott McDonald, President of Nomos Research, after the presentation for his commentary on three key themes.
Question Design
Jon posited that question wording matters for the purpose of reliability in polls. He gave a number of examples of different polling organizations and how some of their question wording was ‘leading’, even ‘disrespectful’. Verbatim, open-ended questions were offered as one solution to avoid these issues. Jon opined that an organization like AAPOR could develop and promote a set of standards on question design to enhance reliability and polling rigor.
Scott commented that Jon assumes that all of these polling organizations are motivated by competitive differentiation in their selection of wording, but since the public doesn’t usually perceive quality differences based on this dimension, I doubt that it is effective way to brand those organizations. However the pollsters do like to be cited in other media – and sometimes the best way to do that is to generate a surprising statistic.
Survey Error
Jon discussed sources of survey error, showing slides from research comparing probability studies (both phone and online) to opt-in studies. He referenced the ‘heroin dealers’ in the polling industry, pointing out that opt-in studies showed tremendous variance (e.g. 80% of Americans reported that they visited a movie theater in the last thirty days). This survey error can be a compromising factor in poll reliability.
The contention that the randomness of the sample design that matters is widely accepted (if not as accepted as it used to be), and I think Jon is also right that no amount of post-sample weighting can correct for shortcomings in sample design – despite the various claims made in the industry to the contrary. The problem is that real random surveys are usually very expensive and most of the business questions being addressed have narrow audiences that would be much too cost prohibitive to target through random sampling. One of our clients has a total addressable market of just 16,000 organizations: it has to be targeted nonrandomly.
Real-World Implementation
I posed the question to Jon on how to communicate probability sampling and representativeness in layman’s terms. For context, margin of error and sample size depicted in a table is relatively digestible and clients adjust budgets accordingly. Jon agreed that probability/random sampling is a topic that is less accessible to a broad audience.
Many of the things we try to measure are so small and fragmented that they would require gigantic (and very expensive) random samples. What’s more, many of these events (say, for example, a visit to a website) are transient enough and low enough in salience as to be quickly forgotten. Hence the vast difference between passive measurement of things like radio stations (which picks up the fragmentation) and self-reports (which tend to favor the bigger brands). When a survey organization needs to measure highly fragmented phenomena, they are pushed to super-large samples — and through that door to samples that don’t meet the standards of random sampling.
After the presentation, Jon took the stage with his band, Charged Particles, along with special guest, Paul McCandless. Certainly one of the most improbable juxtapositions of any talk I’ve attended! I left the event with a signed ‘Sparks’ CD and new insight into percussion—and probability.
Author Notes:
Jeffrey Henning
