Prospects for Pre-election Polls in the Early 2012 Presidential Primaries

Voting in Iowa and New Hampshire is only a few weeks away, but variance in their early estimates suggest that there are storm clouds on the horizon for the pre-election pollsters – and therefore for everyone involved in the polling and survey research business. Based upon a variety of analyses conducted after the 2008 campaign and the problems that pre-election pollsters had in those primaries, we know that the image of the entire industry rests substantially on the performance of the pre-election polls.^[1] This is due partly to the fact that pre-election polls have a peculiar external validation in the actual outcome of the election that most other polls do not have. In addition, the way that the news media report on “the polls” in the aftermath of an estimation error, especially when they systematically get the winner wrong as they did in New Hampshire, lays generic blame on the method and all those who apply it.

In 2008, because of an unusual perspective on estimation issues gleaned through the long series of Democratic primary polls in 2008 due to the extended contest between Barack Obama and Hillary Clinton, we learned that the pre-election polls systematically underestimated the winner’s share of the vote.^[2] While they rarely got the wrong winner, they did underestimate actual vote share by an amount that was typically greater than sampling error would suggest.

Why could the same thing happen again in this cycle? The contextual conditions are almost the same as in 2008. Some states again tried to jump the queue established by the Republican National Committee. Iowa and New Hampshire responded by moving up their dates about a month, from February to early January, on the 3^rd and 10^th respectively. That means candidates will be campaigning across the holidays, and pollsters will have to interview prospective primary voters in that period as well.

There is also a peculiar dynamic to the Republican contest this year that many analysts and journalists have characterized as “anybody but Romney.”^[3] National poll data showed this when they reported a changing set of second-place contenders, each of whom has fallen back under increased press scrutiny as they rose in prominence. Newt Gingrich is the latest to climb to second place, and he is receiving more intense coverage of his personal finances and contacts inside the Beltway. In the next month, voters in Iowa and New Hampshire will focus more sharply on the alternatives they have, receive stepped up contact from every candidate’s workers on the ground, and the television advertising will build to a crescendo.

The national poll data do not provide a useful perspective on what is going on in Iowa and New Hampshire, and in fact they may be confusing some journalists who are covering the first two events. We know that these two electorates have a larger proportion of whites and conservatives than the country as a whole. The Iowa caucus is a closed event limited to declared Republicans, while the New Hampshire primary is limited to Republicans and Undeclared voters who ask for a Republican ballot. National polls that include Democrats and Republicans in their samples do not provide any guidance about what might happen in the caucus and primary in these two states.

As of this writing, there are five published poll results for New Hampshire. Each one shows Romney in the lead, by a margin ranging from 2 to 27 percentage points. The smallest margin comes from a poll that seems to be an outlier with just a two-point lead,^[4] but the most recent result, from Rasmussen Reports, shows the next smallest margin of 10 percentage points. These two polls conducted by IVR telephone procedures are producing the highest levels of support for Gingrich. Combined, the data may suggest a slight decline in support for Romney and some increase in support for Gingrich across the two-week period under consideration.

How could we make further sense of these differences? The answer lies in getting more information about the methodology of each poll. At one extreme, the University of New Hampshire/WMUR present their poll data in a 53-page press release, including graphics. Magellan Strategies presents their results in three documents consisting of more than 100 pages, most of which consists of detailed cross-tabulations. At the other end, Rasmussen Reports has a four-page release with some embedded links to a general description of their survey methodology but not this poll specifically.

Figure 1

Three of the polling organizations (Magellan Strategies, the University of New Hampshire, and Suffolk University) provide information for calculating the ratio of Republicans to Undeclared/Independents in their samples, ranging from 53% to 60%. However, this ratio is uncorrelated with support for either Romney or Gingrich. None of the organizations provides details about the key element of their methods that could help a reader understand why they might have produced different distributions of candidate preference – how they defined “likely voters.”

Polling firms that do pre-election polling are doing a better job of disclosing more information than in the past, but they are still not where they should be. The next month is going to test their methods and accuracy in a difficult political environment, and it is very likely that there will be some estimates that differ substantially from the outcome in New Hampshire. In preparation for the critical commentary that will follow, we should expect polling organizations to be more forthcoming about their methods now rather than trying to recover such information after the fact.