Last month, Survey Practiceposted an article by David Moore that noted the “convergence mystery” – the fact that while pre-election polls show wide disparities during the month of October, they tended to converge in their final pre-election predictions. Moore’s data, presented in the article, was based on weekly averages. The data are in last month’s article.

We asked numerous experts their views of why the polls showed this convergence, which included invitations to virtually all the major media pollsters. None of the latter chose to offer an explanation.

Mark Blumenthal posted an article on pollster.com in response to the article. David Moore contributed more information about the convergence mystery on pollster.com.

Our thanks to the experts who provided their responses to Survey Practice.

Paul J. Lavrakas
Independent Consultant

There are some reasons that the convergence that Moore notes occurred at the end of the 2008 presidential election campaign would be expected to have happened, and none of them suggest any collusion or conspiracy among the pollsters.

  1. Screening and weighting pre-election polls is both an art and a science. Responsible pollsters should always be making explicit decisions about whether the approaches they used in the past (including the last poll they conducted) to screen their samples and weight their data are the best approaches for their most recent poll. There is nothing suspicious about pollsters changing their screening techniques or weighting algorithms based on what they think about the appropriateness of their prior approaches. Taking into account what other polls on the same topic are reporting is one useful and appropriate piece of information in deciding how to “tweak” the screening and weighting used in one’s most recent poll.
  2. As shown in the table in Moore article, the variance among the polls represented in the table is decreasing each seven-day period through the month of October, with the exception of the 10/21–10/27 time period. Without this one anomaly, the final variance is wholly in line with the convergence that was occurring consistently during the month prior to the election. As voters become more certain of how they will vote, the polls should reflect this with less volatility across different polls and different polling methods.
  3. The economic crisis seems likely to have increased the accuracy of pollsters if it made likely voters more certain of how they would vote. (Data exist that can test whether voters were more certain about how they would vote in the 2008 election versus voter certainty in prior elections, especially right before the election, but I do not have access to those data.) The economic crisis also may have driven an increase in early voting, especially for those who chose to vote for Obama, as early voting can have a cathartic effect for anyone concerned about the urgent need for changing the country’s top leadership — that is, one may immediately “feel better” about the disastrous economy once one votes for the future change (i.e., the new president) that one hopes will be able to resolve the crisis.

Michael Traugott
University of Michigan

David Moore has raised some interesting questions in his posting about the convergence of pre-election estimates of the outcome the 2008 election as Election Day approached. And he and Mark Blumenthal have expanded upon them in a subsequent exchange on Pollster.com. The original observation is not new, although the previous explanations offered for the phenomenon may not be any more compelling than the “methodological fixes” that are now proposed. But at least the original explanations have a sounder theoretical basis and merit some attention as plausible alternative hypotheses about why the estimates converge.

Moore begins with an observation about the convergence of pre-election estimates in the last week of the campaign, suggesting that Barack Obama would win by a smaller margin that the estimates from the previous weeks indicated. But he also comments on the fact that the variance in the estimates is reduced in the final week compared to the preceding weeks and asks why.

The prior explanations for convergence center on the forces that have historically (post-World War II) produced an apparent equilibrium in electoral outcomes in the American two-party system. They begin with a concept of the normal vote developed by Phil Converse (1966), one of the co-authors of The American Voter. Based upon the standing division between Democrats and Republicans in the 1960’s in the United States, he estimated that – taking into account the higher turnout levels of the Republicans and the greater propensity of the Democrats to defect – the outcome of the prototypical campaign (involving a prototypical pairing of candidates and issue agendas) would be 54% Democratic and 46% Republican. In addition to modeling the dynamics of the system at the aggregate level, the normal vote provides the context for explaining how the outcome of any specific election differs from this baseline expectation by looking at the short term deviations from this expected outcome due to the specific pairing of candidates and the issues that defined the campaign, measured as the voters’ reaction to both. While the relative weights for defection and turnout have been adjusted slightly in the intervening period, the basic theoretical underpinnings of the model remain valuable to this day.

As an extension of this model, Gelman and King (1993) wrote a well-reasoned and provocative essay that asked why we pay any attention to polling at all during the campaign when an historical review of post-World War II election returns, in conjunction with the normal vote model, suggested what the outcome would be minus the distraction of variations in the polls along the way. They asked the question “How can political scientists reliably predict the outcome of presidential elections months in advance of election day in the face of active campaigns?” They expanded on the explanation for the “typical” outcome, indicating that the variability of the polls during the campaign was a result of citizens starting the campaign generally poorly informed, and then the stability and convergence of preferences comes about because of the campaigns’ effect in informing voters through a variety of mechanisms. Their implication is that as citizens learn more about the candidates and what they stand for, their opinions crystallize and become more stable, hence the reduction in variability in the preference distribution. While there are other factors that contribute to the variations in estimates, including house effects (systematic methodological differences between polling firms), sampling error, and question wording differences, this crystallization of support for a candidate is a central explanatory factor.

There have been some “landslide” elections in the post-war period, including Johnson-Goldwater and Reagan-Mondale. But most of the outcomes have fallen very close to the expected outcome estimated through the normal vote. When there have been significant third party candidacies, the outcomes between the leading candidates have often been closer. Of course, the 2000 election demonstrated how close outcomes can be under specific sets of circumstances, and the 2008 senatorial election in Minnesota illustrates this from the most recent campaign. But these outcomes reflect the impact of the short term forces determined by who the candidates are and how the issue agenda is defined.

In this light, the outcome of the 2008 presidential election seems pretty “normal,” despite the discussion early in the general election campaign of whether Barack Obama was “underperforming” and then later in the campaign, after the debates, whether he was pulling out to a runaway lead. Even though his final lead was of the expected magnitude according to the normal vote model, it masked a significant shift in a growth in support for the Democrats as a party and for their standard bearer compared to John Kerry’s effort in 2004.

Mark Blumenthal adds his own twist to the convergence phenomenon by suggesting that the explanation lies in the adjustments or “fixing” that pollsters who produced earlier outlier estimates make by examining their procedures and modifying them. This may at one level accurately describe how the pollsters respond to having produced an estimate that is quite different than the majority of the others, but at another level it could be taken to imply devious or nefarious behavior that would suggest to some a fudging of the data the next time an estimate is produced. This is also a potential alternative explanation, but one that would be difficult to confirm empirically for obvious reasons; and I don’t share this view of commercial pollsters. A better prospect for explaining convergence would be to update the Gelman and King results from 15 years ago, with more emphasis on media effects as suggested by Moore and Blumenthal when they note that the variance jumps in the short period after debates. This is a function of direct exposure to the event plus the follow up media coverage of them, especially given their important role in the 2008 campaign.

Converse, P.E. 1966. The concept of a normal vote. In: (C. Campbell, and Miller, eds.) Elections and the political order. New York, John Wiley and Sons. pp. 9–39.

Gelman, A. and G. King. 1993. Why are american presidential election campaign polls so variable when votes are so predictable? British Journal of Political Science 4: 409–451.

Micheline Blum
City University of New York

The final three days of national polling before the 2008 Presidential election produced results close to the final election outcome, and which appear to have converged from earlier disparate results. While some earlier volatility in the vote and variation in the polls might be expected over the final weeks, David Moore points out the appearance of convergence in only the last three days and questions the reasons for the accuracy of the later polls and the divergence of the earlier polls.

Some of the reasons for the convergence would be welcome news, while other explanations would not reflect as well on the results. The most reasonable explanations for the “convergence mystery,” beyond the solidifying of the vote choice itself in only the final three days, would seem to be:

  1. Allocation of the undecided on the final pre-election polls only, with more of the undecided allotted to the candidate previously “underestimated” by the particular organization and in the direction of the average of the polls.
  2. Participation of a few influential “outlier” polls and organizations in the weeks leading up to the election but not in the final 3 days.
  3. Refinement of, or changes to, the likely voter models, including index questions, weighting formulas and turnout estimates by “outlier” polls.

As I was not privy to the information necessary to examine the last of these, I have chosen to concentrate on reasons 1 and 2 only.

1. Allocation of the undecided vote.

In the last 3 days before the 2008 election 20 national polls conducted by 19 organizations appeared on Pollster.com and were cited by David Moore in Survey Practice. Of these, 7 polls allocated the undecided, (1 poll did not report, but did not allocate the undecided). Allocation allowed the pollsters to apportion more of the undecided to the candidate they had underestimated on previous polls, in comparison with other polling organizations. Six of the seven reported the vote before allocation on their websites. The undecided vote on these six polls ranged from 2–7 points, with an average of 5 points. Three allotted more to McCain, one gave more to Obama, and two allotted the vote evenly. The 3 polls giving more to McCain showed a larger pre-allocation margin than did the 1 poll which allocated more to Obama. This allocation toward the mean clearly accounted for some of the “convergence” and the reduction in the variance. In the previous 4 days, there were 13 polls done by 11 organizations, and only 1 poll allotted or did not report the undecided. Had some of these polls allocated the undecided in the direction of the average, the variance would have been lower and they would have appeared to converge earlier.

2. The influence of the “outlier” polls.

Much of the variance seen in the polls done before the last three days is due to very few “outlier” polls, and about half of this group either did not release polls in the last three days or were among those which allocated the undecided in their final polls.

The range in the Obama-McCain margin in the final three days was from 5 to 11 points. In the previous four days, 3 of the 13 polls released (by 11 organizations) had margins beyond the 5–11 range and could be considered “outliers.” All three had margins of less than 5. No poll had a margin larger than 11 in that four day period. If we remove those 3 “outliers” from the 13 reporting during those four days, the variance is reduced to only 3.3 pts.

Similarly, if we look at the previous week (10/21–27), when we could expect more true variance due to voter uncertainty and volatility, only 7 of 29 polls, conducted by 6 of 24 polling organizations, were “outliers.” Three of the six outliers were the same three seen above.

Of the six organizations with outlier polls, three reported margins larger than 11 points, and all three consistently showed larger margins in their polls. In the final three days, however, one of these organizations allocated the undecided, giving more to McCain, one did not release a poll, and one had a margin of exactly11 points on its final poll. Of the three organizations releasing four polls with margins smaller than 5 points, one organization (with 2 “outlier” polls) allocated the undecided in the final three days, one did not release a poll in the final three days, and one “converged.” So, of the 6 organizations, 2 organizations (accounting for 3 “outlier” polls) allocated undecided in the final three days in the direction of the previously underestimated candidate, 2 did not release polls in last 3 days, and only 2 “converged.” If we remove the 7 “outlier” polls from the 29 released in the week of 10/21–27, the variance is reduced to only 2.9 points.

Moving beyond the last two weeks, we get to the two most extreme outliers seen in the final month of polling, both cited by David in his analysis. These were the only 2 organizations (out of 26 organizations) to report margins of less than 3 points or more than 13 points, and those results were both more than two weeks before the election. Again the pattern seen above is repeated, with one of the two most extreme outliers not releasing a poll in the last three days and the other allocating more of the undecided in the direction of the average of the polls.

Basically then, both of the explanations examined, the allocation of the undecided by seven organizations in the final three days and the absence or favorable allocation of a few “outlier” organizations, appear to be major contributors to the “convergence” seen. Apportioning the undecided in the favorable direction and the absence of previous outliers virtually guarantees less variance and the appearance of “convergence.” So, perhaps, rather than convergence, what we saw was that much of the earlier variance was due to a few outliers and that the final three days benefited from their absence or favorable apportionment of their undecided vote. This may help solve the mystery, but it is both good and bad news in evaluating the 2008 pre-election polls. The bad news is that some of the final poll results and the “convergence” seen and lauded must admit to being assisted by favorable allocation of the undecided vote. The good news is that there were few outlier polls, and, in fact, most of the national polls were very accurate predictors for more than just the final three days.

Cliff Zukin
Rutgers University

The question of why pre-election poll estimates of vote intention converge at the end of a campaign is an interesting one. I suspect the answer may have less to do with our methods than the phenomenon we are measuring. Absent some major occurrence in the campaign, one would expect citizens to become more certain about their behaviors the closer it gets to election day-both in whether and for whom to vote. If so, we might expect less variation in poll estimates (convergence). Screens to identify likely voters probably produce more consistent results from poll to poll, among other judgments that pollsters make at the end of the campaign.

Don Dresser
Sierra Club

Part of the answer is probably just sample size. The obvious part of this comes from some pollsters who deliberately choose to take a bigger sample for the final poll. Harris, ABC, and Pew in particular all ran polls of around 2400 respondents in the last couple of days, but had been doing polls closer to 1200 earlier in the cycle. I suspect there may be an additional piece (which is still basically sample size) that says everyone makes sure they have a poll in the last few days – but pollsters who consistently run large polls may be somewhat less frequent to put out additional results, while those who consistently run smaller polls can afford to do them more often, so the smaller polls may make up a larger fraction of the earlier polling.

I don’t think there is enough in sample size to account for all of the convergence you note – but I suspect it tells part of the story.

We welcome your comments on this issue.