Editor’s note: This commentary is by Rich Clark, a professor of political science at Castleton University and an editor of “Polling America: An Encyclopedia of Public Opinion.” He was the director of the Castleton Polling Institute.
After the 1996 presidential election where Bill Clinton easily defeated Sen. Bob Dole to win a second term in the White House, my professor at the University of Connecticut and director of the Roper Center for Public Opinion Research wrote an opinion piece for The Wall Street Journal titled, “The Pollsters’ Waterloo” (Nov. 19, 1996). Ladd’s obituary for the polling industry was by no means the first nor would it be the last, but because it came from someone so closely tied to the polling industry, it was profound and stinging for the industry.
Since 1996, we’ve heard many times of the death of polling, most resoundingly after the 2016 election, even while the national polls performed extremely well by the standard of predicting the final election outcome. Nevertheless, the cries for ending polling were vociferous. And again, on Nov. 3, 2020, before the votes were tallied and America had any idea how closely polling averages approximated the election outcomes, the cries went out again that polling is dead.
To paraphrase Mark Twain, it seems the new reports of the death of polls are greatly exaggerated.
I will be the first to note that polling as a means for measuring behavior, such as voting, is merely an inexact approximation. Polls are a snapshot of public sentiment, and not necessarily predictive of future behavior. Furthermore, polling as a measure of public attitudes is almost equally as problematic, except that there is no other option. Polling is like the dog that can sing; it doesn’t sing well, but it’s the only dog that can. Unlike a singing dog, however, our democracy benefits from measures of public attitudes, even if those measures are approximations.
Winston Churchill said of democracy that it is the worst form of government, except for all of the others. That idea can be attributed to polling as well; it is the worst way to measure public opinion, except for every other way that has been tried.
While polling is the best means for measuring public attitudes, it is a methodology riddled with error. The job of the pollster is to minimize measurement error to the greatest extent possible. The error that threatens polling results come from (a) the fact that our sample frames from which we select respondent are not always complete, (b) the fact that random sampling is prone to error by the very nature of probability statistics, (c) the fact that many of those sampled in our surveys do not participate in the polls, and (d) the fact that sometimes respondents do not fully understand what they are being asked or they understand but do not feel comfortable giving an accurate response.
We commonly discuss margins of error when assessing polling accuracy, but we often forget that the error terms around polling estimates are based on a 95% confidence level; in other words, we should expect that one out of every 20 samples is completely off the mark. If we are 95% confident that the true public support for a candidate is within 4 percentage points of X, then we should expect that true public support is beyond that margin of error 5% of the time.
As for non-response, this has been an increasingly vexing problem for pollsters. Response rates are suffering from the fact that it is becoming more difficult to reach the people we need to survey, and many of those that we do reach refuse to participate in the polls. When George Gallup, Elmo Roper, and the other pioneers of public opinion polling started in the 1930s and 1940s, Americans were very cooperative, and refusal rates were low. But as the number of polls grew and the number of solicitations from marketers and other entities competed for the public’s time and attention, cooperation rates began to decline. The public is deluged by requests for their survey participation, and they do not feel any civic obligation to participate in public opinion polls any longer.
Additionally, technology has given consumers more tools to screen unsolicited intrusions on their phones, beginning with answering machines to call blocking. caller ID, and the Telephone Consumer Protection Act (TCPA of 1991). Pollsters have tried to adapt to non-response problems by making greater efforts to reach those who are hardest to reach (particularly young adults) and by using post-data-collection weights to adjust for differences in response by certain demographic groups. If, however, we fail to include a representative demographic in our polls at all, no amount to weighting can adjust for that omission.
Even with these barriers, the polling industry has continued to grow. With every advancement in technology, pollsters have adapted, using the latest means of communication available to reach the public. When, in the 1970s, most households had acquired landline telephones, pollsters shifted their methods from in-person polling to polling by telephone. And when cellphones began to replace landline phones, pollsters adapted by including dual-frame sampling techniques to include both landline and cell phones. As more Americans connected to the internet, pollsters explored web-based polling, and they are currently exploring polling with smart phone apps. The industry that measures public opinion continues to adjust to the way that the public communicates with one another.
After the 2016 election, the American Association for Public Opinion Research (AAPOR) convened a panel of prominent scholars and practitioners of public opinion research to explore the problems that polling faced and to recommend a path forward. While the national polls in 2016 performed well, predicting correctly that Hillary Clinton would win the popular vote by a small margin, many state polls did not accurately reflect state results. Of course, we do not elect presidents based on the national popular vote, so the accuracy of state polls is critical to our understanding of who will win a presidential race.
It is proper that a professional organization like AAPOR should have such a post-election analysis. This is good for the industry, and it is important for helping to restore trust in the polls. While the 2016 report has been great for helping pollsters move forward, it has done little to rebuild public trust, sadly. In 2020, facts matter less than they once did.
Trust in polling is understandably low. I say “understandably” not in agreement with popular sentiment that polling is dead, but rather because of the way polls have been discussed in the media. Pollsters have been treated as oracles, and polls have been portrayed as precise instruments, which they are not. Given the high expectation of precision, it is inevitable that polls will disappoint.
So now, we need to consider why horse race polls matter. For pollsters, the degree to which our population estimates reflect the actual vote is an assessment of the validity of our methods for measuring attitudes, where there is no empirical outcome to test against. However, for the media and for the public, the estimates of which candidate is ahead and by what margin is fascinating, but it also detracts from the important substantive issues in the election. Ultimately, the election will tell whom the voters chose and by what margin. What polling gives the media and, by extension, the public is the explanatory information about who within the public favored which candidate and why. Polling tells us what the voters want by casting their ballots. The election results tell us who won; polling tells us why.
So rather than burying polling, we should consider expecting less of polls as a means of predicting election outcomes and focus on what polls offer that the ballot boxes cannot provide. We should not expect polls to be prophetic but rather illuminating. We should use them to help us better understand the American electorate and, maybe more importantly, the general public that includes those who vote and the more than one-third of eligible voters who do not vote.
