Presidential elections present a unique opportunity for polling firms to embarrass themselves, and in 2012 it was Gallup’s turn. Gallup’s last pre-election estimate of the 2012 popular vote for the Presidential election predicted a race that was too close to call, with Romney at 49% of the vote and Obama at 48%, both with a margin of sampling error of plus or minus 2 percentage points. The final result was Romney 47%, Obama 51%, at the edge of the margin of error.
In a review of its methodology announced today, Gallup concluded that 4 factors accounted for most of the measurement error.
- Gallup’s likely voter estimating exaggerated support for Romney. Pre-election polling measures “a population that doesn’t exist yet,” that of voters in the upcoming election. Likely voter estimation is an attempt to identify this population, and is based on scoring respondents for voting behavior, attitudes and intended action. In this case, much of the skew towards Romney came from Gallup’s question, “How much thought have you given to the upcoming election for president — quite a lot, some, or only a little?”
- Gallup’s regional representation missed Obama voters. The organization weighted to U.S. Census regions, but underrepresented the Eastern time zone within the Midwest and South and underrepresented the Pacific time zone within the West.
- Gallup’s race and ethnicity questions were nonstandard. Gallup weighted its data according to race estimates from the Current Population Survey of the U.S. Census Bureau but used its own question battery (designed for telephone) rather than the CPS questions (which were designed for mail). The result was disproportionate respondent identification as multiracial, American Indian and Alaska Native.
- Gallup’s shift to listed landlines instead of RDD sampling. By relying on listed numbers instead of a probability sample using Random Digital Dialing, Gallup skewed its results to older, more Republican voters.
Too much of the discussion of survey error cursorily ascribes it to sampling error, when there are many types of error at play. Kudos to Gallup for its candid review of 13 possible sources of error in its work.
Author Notes:
Jeffrey Henning
