Introduction
We examine pre-election polls for what leading experts considered the most competitive US Senate races in 2014 to test the association between samples’ proportions of cell-phone respondents and polls’ accuracy in predicting election outcomes. Figuring out the optimum percentage of cell-phone-only users in one’s sample and actually reaching such respondents provide great challenges to pollsters.
Challenges
The percentage of cell-phone-only U.S. households has risen dramatically, from single-digit figures a decade ago to 43 percent for adults according to recent estimates (Blumberg and Luke 2014). Because of rapid changes in technology and consumer behavior, the ratio of cell and landline users in the population is a constantly moving target. Adding to pollsters’ difficulty, cell-phone-only voters “have significantly different, and more Democratic, political views than those with land lines” (Harwood 2012). Further, younger adults, who are disproportionately heavy cell-phone users (DeSilver 2014), vary widely in their turnout between presidential and midterm election years. Under-30 voters composed 12 percent of the electorate in each of the midterm election years of 2006, 2010, and 2014, whereas their shares of the 2008 and 2012 presidential electorates were 18 percent and 19 percent, respectively (Scott 2014).
Yet another layer of complexity arises from restrictions (or the lack thereof) on who pollsters can call, depending on telephone methodology. Pollsters who engage in mass calling with interactive voice response (IVR) technology, in which interviewees receive pre-recorded instructions over the phone and respond by pressing their keypad, cannot call cell phones, as per Federal Communications Commission (FCC) regulations (Harwood 2012). IVR pollsters, therefore, can only call landline numbers and attempt to reach cell-phone-only voters via alternative means (e.g., Internet panels), thus increasing the complexity of their task. Polling organizations that use live interviewers (i.e., “real people”) to conduct phone surveys can call anyone, regardless of recipients’ type of phone. This aspect of polling is important to note, as it explains why some pollsters augment their phone surveys with Internet components.
Current Practices and Issues Associated with Them
In light of these challenging factors, a “magic percentage” that cell-phone-only respondents should compose of a sample for a given state in a given year would be desirable, but is unlikely to be attained. At this point, some pollsters have recommended approximate ratios of cell-phone respondents to seek. Writing roughly a year ago, Brian Stryker of Anzalone Liszt Grove Research issued the following recommendation: “In 2014, dial a minimum of 15 percent cell-only” (Stryker 2014). In the immediate aftermath of the last presidential election, the firm Public Opinion Strategies (2012) posted the following on its website: “…our prescription includes doing at least one-third of the interviews with cell phone respondents going forward, adjusting as required, ensuring that we include enough younger voters in our sampling…”. Given the relatively high percentage recommended by POS for cell phones and the fact the memo was written in the context of a presidential election, one could surmise the 33 percent figure was probably intended for presidential-election years.
Within the technical literature, Fahimi (2014) noted that prevalence data on cell-phone-only households have become more available and that certain formulas can be applied to determine ratios of cell and landline phones to call, all the while minimizing certain types of errors and taking into account cost. If implemented, such techniques may well improve poll accuracy. Even if cell-phone-only prevalence were perfectly known for geographic units, however, additional issues unique to pre-election polling would still remain. Determining which respondents are likely voters is always challenging and, if young-voter turnout were expected to be low, it conceivably could make sense to aim for a cell-phone-only proportion below the known proportion in a given area.
The Present Study
To get an idea of how polls’ cell/landline telephone composition was related to their accuracy in 2014, we examined pre-election polls for what the website Real Clear Politics considered the year’s most competitive US Senate races. Given young voters’ disproportionate use of cell phones and their recent pattern of low turnout in midterm elections, we predicted that an increased share of cell-phone-only respondents in these polls would be associated with a greater prediction error in a pro-Democratic direction. In addition, we tested whether a younger age might explain some of this association, predicting that partialling out the percentage of 18–29 years-old would reduce the correlation between cell-phone-only use and prediction error somewhat.
Methods
Competitive US Senate races were defined as those Real Clear Politics listed as “toss-ups” or to “lean toward” one party or the other. Ten races met this criterion: Alaska, Arkansas, Colorado, Georgia, Iowa, Kansas (in which the Independent candidate was analyzed as the Democrat), Kentucky, New Hampshire, North Carolina, and Virginia. We also examined toss-up/lean governor’s races from the same source, but there appeared to be less polling in these races than in senatorial ones. We therefore did not include gubernatorial polls in our main analyses, but presented a supplemental case study of the governor’s race in one state, Massachusetts, which featured a relatively large number of pre-election polls.
We attempted to include all U.S. Senate polls in the target states from October 15 until Election Day (November 4), as listed by Real Clear Politics. We chose this time window because polls closest to Election Day are most likely to yield accurate predictions. However, if a given pollster conducted more than one survey post-October 15, we used only the most-recent one. Polls conducted fully by Internet (e.g., YouGov) were not included. Fifty-six potentially usable polls were identified, but for only 41 of these were estimates of cell-phone-only proportions of the respective samples available.
The percentage of a given sample composed of cell-phone-only households was gleaned from pollsters’ online technical reports or, in a few cases, by contacting pollsters directly. Some polls’ reports clearly stated the percentages of respondents whose households were cell-phone-only, landline only, or both. Other polls’ documentation, however, only stated how many completed calls were made on landlines and on cell phones, where the latter would only be an approximation of cell-phone-only prevalence because some people interviewed by cell phone could be from dual-use households. If an IVR pollster reported using an Internet panel to represent cell-phone-only respondents (in this case, a mixed-mode approach combining the Internet with landline calls), the share of the total sample composed by the panel was considered the cell-phone-only percentage.
The percentage of a sample composed of cell-phone-only households carries added meaning when compared to the cell-phone-only prevalence of the state in which the survey was conducted. For example, if 30 percent of households in a poll had only cell phones, the implications would likely differ if the state was known to have 25 percent vs. 35 percent cell-phone-only households. Therefore, we subtracted the relevant cell-phone-only state-wide base rate from each sample-based cell-phone-only percentage. Two sources of states’ telephone breakdowns were Marketing Systems Group/Genesys Sampling (2014), whose state-by-state “Percent Wireless Household” estimates were from October 2013; and the National Health Interview Survey (National Center for Health Statistics 2014), whose “Wireless-Only” percentages were also as of 2013. Using all 50 states plus the District of Columbia as units of analysis, the two sources’ estimates correlated strongly (r=0.66, p<0.001). However, looking only at the ten states in our study, the correlation was much weaker (r=0.19, p>0.50), due primarily to large discrepancies in estimated cell-phone-only usage in four states (Marketing Systems Group first, National Health Interview Survey second): Iowa (26 percent vs. 48 percent), Kansas (31 percent vs. 44 percent), Arkansas (43 percent vs. 54 percent), and Alaska (41 percent vs. 32 percent). Nevertheless, the two estimates for a given state were averaged to provide a single estimate.
Finally, to guard against possible confounding of cell-phone-only and age representation (i.e., samples with large shares of cell-phone-only respondents also having large shares of younger voters), we recorded age information from samples where available. Pollsters were not consistent in how they categorized age. In 20 reports, the youngest respondents were categorized as 18–29 years-old, with far fewer pollsters using intervals such as 18–34 or 35 (9 polls), or others. Using only studies that reported percentages of respondents 18–29 (for consistency and because of their relative availability), correlational analysis indeed showed that the greater samples’ share of cell-phone-only households, the greater their share of 18–29 years-old (r=0.58, p<0.01). We therefore controlled for age when examining associations between samples’ cell-phone percentages and predictive accuracy in the elections.
Results
Each poll’s accuracy was defined by the following formula, so that higher positive scores indicated greater error in stating Democratic candidates’ chances and more pronounced negative scores represented greater error in the Republicans’ direction:
(% Favor D in Poll – % Favor R in Poll) – (% Vote D in Election – % Vote R in Election)
To take a concrete example, Public Policy Polling’s final pre-election survey in the North Carolina US Senate race (Nov. 1–3) showed Democratic candidate Kay Hagan leading Republican Thom Tillis, 46–44 percent (D +2). However, the actual election result was that Hagan lost to Tillis, 47.3–49.0 (D −1.7). By this standard, the poll was +3.7 percentage points “too Democratic” (2 – [−1.7]). An example in the opposite direction was the Oct. 27–28 Vox Populi poll in New Hampshire, showing Democrat Jeanne Shaheen trailing Republican Scott Brown, 45–49 (D –4). Shaheen won 51.6–48.4 (D +3.2), making the poll “too Republican” by 7.2 percentage points in absolute value ([−4]−3.2=−7.2).
Across the full set of 56 eligible polls (including where cell-phone data were not available), the average error was 4.52 percentage points (SD=5.28). States with the largest mean errors (all positive and thus pro-Democratic) were those in which Republican candidates pulled away at the end and won larger victories than anticipated by most polls (Arkansas, 9.40; Kansas, 12.80; Kentucky, 8.00) and in which a Republican candidate consistently trailed by sizable margins in the polls, but only narrowly lost on Election Night (Virginia, 8.70).
We conducted several correlational analyses between polls’ shares of cell-phone-only respondents (subtracting the established cell-phone-only state base rate) and their accuracy/error. In the full set of 41 polls, cell-phone share and poll accuracy/error were uncorrelated, r=−0.04. However, dividing the polls based on states’ partisan tendencies yielded some suggestive findings. The Cook Political Report’s (2014) Partisan Voter Index (PVI) was used for this purpose. The PVI compares each state’s voting in a given election year (e.g., 53 percent for the GOP candidate) to that of the nation as a whole, to estimate any Democratic or Republican lean. Even though 2014 PVI values were available, we used 2010 scores to avoid confounding our analyses with any 2014 statistics.
Figure 1 shows associations between cell-phone shares and polling accuracy/error in moderately (PVI from R +2 to R +7; 13 polls) and strongly (PVI from R +9 to R +13; 12 polls) Republican-leaning states. In each set of states, the greater a poll’s cell-phone-only share, the greater the poll’s error in a Democratic-leaning direction. Conversely, smaller shares of cell-phone-only voters in these samples made them more accurate. Correlations were r=0.41 and r=0.30 in the moderately and strongly Republican-leaning states, respectively, but because of the small number of cases, neither was statistically significant.[1]
There was one apparent outlier issue. A North Carolina survey by Harper Polling (2014) used an IVR method with no provision for contacting cell-phone-only respondents mentioned in the associated documentation. Even with an apparent zero-percent rate of cell-phone surveys, the Harper poll’s result (Tillis [R] +2) was quite accurate. Removing this data-point greatly reduces the correlation in the moderate GOP-leaning category to r=0.125.
In states ranging from neutral (PVI=0) to slightly Democratic leaning (PVI up to D +2), there was a slight trend (r=−0.10) for larger shares of cell-phone-only respondents to correlate with more accurate polls when states’ baseline cell-phone-only percentages were based on an average of the two available sources (Marketing Systems Group and National Health Interview Survey). When analyses were conducted separately for each source, the results diverged considerably.[2] We thus conclude that the effect of polls having different cell-phone percentages in the neutral to slightly Democratic states cannot be determined at this time.
The most robust correlation was the positive one (r=0.30, Figure 1, right-hand side) between polls’ cell-phone share and pro-Democratic error, within the strongly GOP-leaning states. To test whether age might explain some of this association, a partial correlation controlled for share of 18–29 years-old in each sample (among samples that categorized age that way). In samples with the relevant age data, the bivariate correlation between cell-phone share and pro-Democratic error in the strongly Republican states was r=0.18; partialling out the percentage of 18–29 years-old had little effect, leaving this correlation at r=0.20.
In-Depth Gubernatorial Case Study
As noted earlier, there were fewer polls for governor’s races than for US Senate contests. Further, some states featured competitive elections for both governor and senator, so that including data from both kinds of elections (e.g., the Colorado governor’s race and Colorado senator’s race) in the same analysis would have created a problem with non-independent observations. One state whose 2014 governor’s race was competitive, but whose senate race was not, was Massachusetts. Seven telephone-based gubernatorial polls were conducted between October 15 and Election Day (see list), for five of which the cell-phone percentages were available or could reasonably be inferred. For example, Western New England University Polling Institute’s (2014) Massachusetts poll recruited landline and cell-phone participants and then, “weighted [the data] to adjust for cell phone and landline usage based on state-level estimates for Massachusetts from the National Center for Health Statistics.” The NCHS estimate for Massachusetts was 26.1 percent, thus strongly suggesting that cell-phone-only voters composed 26.1 percent of the poll’s sample. (The other source of state-wide estimates, Marketing Systems Group, had a very similar percentage for Massachusetts at 27.7 percent.)
We can thus examine how far a given poll’s cell-phone percentage in the Massachusetts governor’s race differed (in either direction) from the state-wide baseline of roughly 27 percent and relate degree of cell-phone deviation to the poll’s accuracy. The actual electoral result was that Republican Charlie Baker received 48.5 percent of the vote, whereas Democrat Martha Coakley received 46.6 percent. At one extreme end of polling strategies, the Emerson College Polling Society (2014) collected data via IVR (automated) methods, thus missing cell-phone-only respondents. This poll exaggerated Baker’s likely margin with a 48-42 reading in his favor. It appears, however, that Emerson College’s poll initially was even more favorable to Baker. To compensate for the lack of cell-phone respondents, Emerson followed a policy that “survey data is [sic] weighted with a 3 point decrease in Conservative opinion and a 3 point increase in Liberal opinion to offset the bias in land line only telephone polls.” Without this adjustment, therefore, Baker apparently would have led 51-39 in the poll.
Public Policy Polling (2014) composed all of its 2014 samples with 80 percent of respondents drawn from landlines and 20 percent from an Internet panel of cell-phone-only users. This 20 percent cell-phone share was somewhat short of the Massachusetts baseline of 27 percent, and PPP’s 4-point margin for Baker (46-42) slightly overstated Baker’s margin by roughly 2 percentage points. The aforementioned Western New England University poll, weighted to match the NCHS estimate for Massachusetts of 26.1 percent cell users, had Baker up 46-41, thus overstating his eventual 2-point victory by 3 percentage points. The remaining two polls – Suffolk University/Boston Herald and University of Massachusetts-Lowell/7 News – used a much higher cell-phone percentage, namely 35 percent. These polls tended to be quite accurate, although the differences from some of the other polls were not large. Suffolk estimated Baker’s victory to be 46-43, only a 1-point overstatement of his eventual actual margin, whereas UMass-Lowell predicted Baker would win 45-41, a 2-point overstatement. In Democratic-leaning states such as Massachusetts, therefore, exceeding state-wide cell-phone base rates may be necessary to measure candidates’ standing accurately.
Discussion
Findings from our investigation suggest that in 2014’s closest US Senate races held in Republican-leaning states, increasing the share of cell-phone-only respondents in pre-election polls increased the poll’s prediction error in a pro-Democratic direction. These findings, of course, must be interpreted with caution due to the small number of cases for analysis.
Setting a poll’s cell-phone percentage to match an established figure (e.g., reports from Marketing Systems Group or the National Health Interview Survey) appears to be a completely reasonable approach. However, should pollsters expect disproportionately high or low turnout by demographic subgroups on a dimension correlated with cell-phone usage (e.g., low participation by younger voters), then deviation from state cell-phone rates may be warranted (in this case, surveying a smaller share of cell-phone-only users than the state-wide rate). To aid in such projections, the Pew Research Center regularly surveys Americans on how closely they are following upcoming elections, with age breakdowns (Cillizza 2014).
There are even more complex issues pertaining to cell phones that we have not addressed, such as how to deal with individuals who use both cell phones and landlines. Some researchers will hang up on people reached by cell phone but who also use landlines, to isolate cell-phone only respondents (Harwood 2012). Others (Fahimi 2014) argue that modern analytic methods make doing so unnecessary and that greater cost-saving can be achieved. In July 2015, Langer Research Associates, the pollster for Washington Post/ABC News surveys, informed the Huffington Post of its switch to an “overlapping dual-frame design, which can be thought of as three frames – individuals who can be contacted on cell phones only, those who can be contacted on landlines only, and those who can be contacted on either. The total number of cell interviews continues to be set to reach the cell-phone-only target derived from NHIS estimates” (quoted in Blumenthal, Edwards-Levy, and Velencia 2015). Approaches for dealing with phone-usage patterns thus appear to continue to evolve rapidly.
Our recommendation is to consult state-wide estimates of cell-phone-only usage in determining targeted percentages of such respondents for a survey. However, other factors such as a given state’s political leaning and advance indicators of differential electoral interest by age, should also be considered. Such considerations may lead pollsters to raise or lower their targeted percentage of cell-phone-only voters, beyond state-wide base rates for different types of phone usage. We have emphasized age as a likely demographic correlate of different types of phone usage, but other demographic markers may be identified and contribute to determining the proper cell-phone ratio in a pre-election survey.
For purposes of sensitivity analysis, these correlations were also computed separately for the two sources of states’ cell-phone use baselines (Marketing Systems Group and National Health Interview Survey), supplementing the main analyses, which averaged the two sources. With MSG as the baseline, the correlation between surveys’ pro-Democratic error in forecasting the elections and cell-phone percentage relative to MSG were r=0.42 in the moderately Republican states and r=0.49 in the strongly Republican states. With NHIS as the baseline, these correlations were r=0.40 in the moderately Republican states and r=0.10 in the strongly Republican states.
In states with political leanings ranging from relatively even to slightly pro-Democratic, the correlations were r=0.49 (p=0.053) with the MSG baseline, and r=−0.53 (p<0.05) with the NHIS baseline. Three states (Colorado, Iowa, and New Hampshire) comprised this category. The main source of difficulty appears to be in Iowa, where MSG put the wireless percentage at 26 and NHIS put it at 48.