If political polls are driving you crazy, read this
As the election approaches, I’ve been musing over the nature of political polling. There are many folks who make a living reporting on poll results, and a few who actually do solid analysis using such polls (with Nate Silver at 538 being the most prominent and sophisticated example). Unfortunately, there are problems inherent in the enterprise of measuring public opinion that make it impossible to say with certainty what the outcome will be (at least for a close presidential election like this one promises to be).
There are two goals of a poll: 1) to create a “snapshot” of public opinion during the period over which the poll was conducted, and 2) to predict who will win the election. It’s important to distinguish these two goals.
Taking a snapshot of public opinion seems straightforward, but it’s getting harder to do so, as different parts of the public change their preferences about answering calls from strangers (in part aided by technologies like call-waiting, which are becoming more widespread). People are increasingly shifting to not having landline phones, and that may also introduce biases into the results.
The spread of polling results is largely the result of differences in how factors such as these are treated by the pollsters (and there are always buried assumptions and judgment calls in such analyses). So it’s not at all clear that the snapshot of public preferences is accurate, even for the pollsters who are most sophisticated and use human interviewers and careful statistical methods. Analysts try to adjust for these variations by taking averages of polling results, but such methods only work when there is no systematic bias affecting the results (as an aside, the “margin of error” that is commonly reported for polls is a simple statistical measure based on the number of respondents and does not reflect the kinds of structural biases I describe above).
I want to turn now to the second goal, which has received comparatively little attention (it’s the one that prompted me to write this post in the first place). Doing a prediction of what will happen is fraught with problems, unless you are dealing with a physical system like planets orbiting the sun, and it is in attempting to do predictions that I think most pollsters get into real trouble.
The methods used to convert samples of “registered voters” to samples of “likely voters” are where the snapshot formally becomes a prediction. In that conversion the pollster needs to decide who will submit a valid ballot by Tuesday November 6th. While it is possible to make educated guesses based on historical data, each election is different. Will an energized Republican base offset increased enthusiasm in the hispanic community? Will Hurricane Sandy make voting difficult in some states? Will undecided voters break for the incumbent or the challenger? Will efforts to require photo ID reduce turnout of certain voter groups, and if so, how much? Will someone be able to manipulate the voting results? None of these things will be known with certainty until after November 6th, and history may give no guidance at all.
So I’m convinced that polling simply can’t tell us with certainty who will win the election, at least when it’s close. According to Nate Silver, the state level polls suggest an electoral advantage for President Obama (with about 75% probability today), but what the polls can’t say is what will actually happen on November 6th, and the margins in many states are small enough that the election could go either way.
And that’s where we, the people come in. Our choices are what determine the future. In an election where polls are close, your vote really does count. What we decide to do will be what makes the difference on Tuesday. So don’t get hung up on contradictory polling results, just go out and vote!
“The best way to predict the future is to invent it.” –Alan Kay
Addendum, October 30, 2012: The statistician Andrew Gelman wrote a nice piece in The New York Times that analyzes what “too close to call” means in the context of this election. Highly recommended reading. It addresses the ostensible contradiction that President Obama has a 75% chance of winning but that the election could go either way based on unpredictable factors.
