Trump-Biden 2020: The polls were off, but they're not crystal balls and aren't meant to be.
The news cycle is jampacked with polls. But have you ever wondered how polls actually work and what they mean? USA TODAY
All the information in the world is at our fingertips 24 hours a day, so we think election outcomes should be, too. But polling is not predictive.
There are plenty of headlines proclaiming disaster for the polling field. In the last two election cycles, preelection polls have underestimated electoral support for President Donald Trump. In 2016, the problem seemed to be isolated to polls in specific states. In 2020, although vote totals are not certified, it’s clear that many state and national polls underestimated the Trump vote in the horse race between him and Democrat Joe Biden.
Certainly, pollsters have work to do. But pollsters did not create the expectation that horse race polling will elucidate exactly what is to come on Election Day all by themselves. The media, polling aggregators, forecasters and consumers are all part of the problem, and all need to participate in the solution.
As I have watched and participated in the field in all four capacities over the last 15 years, the electoral polling industry has developed into an ecosystem: Pollsters want the media attention that comes with putting out reliable election estimates and getting picked up in aggregates and forecast models. Forecasters and polling aggregators (like RealClearPolitics.com and FiveThirtyEight.com) want polls to feed their models. The media wants polls, aggregates and forecasts to drive attention to their content. And as consumers, we think that since all the information in the world is at our fingertips 24 hours a day, future election outcomes should be, too.
Highlight uncertainties in polling
This ecosystem regularly pushes beyond what polls are capable of doing. Pollsters have already begun the introspection required by the pattern of misses, some more dire in their conclusions than others, but it is not clear that any part of the ecosystem fully recognizes how intertwined all of their actions are in perpetuating the idea that polls are predictive.
Pollsters like to say that polls are snapshots, not predictions. At the same time, horse race numbers are interpreted as predictive in part due to choices pollsters make. “Likely voter” models are pollsters’ attempt to show a sample of a future electorate — which in practice means trying to predict who among all the respondents will actually vote.
To non-technical audiences, the implication is that since “likely voters” are the predicted electorate, the likely voter result is a prediction. Providing multiple estimates, or ranges of estimates, of what the likely electorate might look like would illustrate the uncertainty involved. Clearer and more prominent discussions of uncertainty in press releases would also serve the goal.
Aggregators and forecasters can help the overall polling ecosystem by using only high-quality data and thereby playing a gatekeeping role. Not all polls deserve to see the light of day, yet low quality polls flood aggregates and forecasts. How much attention and understanding of the nuances of the potential electorate do polls offer if they are just churning out repeated large samples day after day?
Eliminating bad or redundant data could also improve aggregate and forecast models by allowing fewer, higher quality data points to define the range of possible outcomes. For example, the final Iowa Poll conducted by Selzer & Co. in 2020, which waslittle impact on win probabilities. Using only high-quality polls such as Selzer’s would have allowed the outlier poll to put more uncertainty into the models.
media relies heavily on polls, aggregates and forecasts to feed 24-hour news cycles, often without enough discussion of margins of error and uncertainty. In the wake of 2016, pollsters tried to balance communicating adjustments and not overselling fixes to show the lessons of 2016 were taken seriously. While most pollsters did emphasize that new issues could crop up, some prominent polling headlines in 2020 implied the risk of misjudging outcomes this year was lower (repeatedly), and the narrative of “it’s fixed” took hold.
Shift focus to why people vote
To fill the news cycle without focusing on the horse race, media should spend more time on the non-electoral results in polls. (Pollsters: Don’t lead with the election in your press release if you want the media to look at the other results.) There are often far more interesting questions in the polls that get back to the “why” of voting. This also drives polling more back to what it’s great at: Answering the “why” of people’s votes and opinions, rather than simply bean counting vote choices.
Finally, consumers need to change how they think about what polls or poll-based forecasts say. Stop. Most consumers paying attention to polls know the uncertainties and reasons to not take it all as predictive, yet they still refresh FiveThirtyEight.com every 5.38 seconds to see what changes and what new polls are up. Use polls for what they are good at — general pictures of what the public thinks. Predicting the future just isn’t possible, and polls and forecasts do not make it possible.