Cell phones have produced a dramatic decline in household coverage in telephone surveys using RDD landline sample. Dual frame (landline and cell phone) samples are increasingly used in telephone surveys to address this issue. However, there is no consensus on the best design for dual frame surveys: segmented designs which screen out respondents with landlines from the cell phone (wireless) sampling frame, or overlapping designs which include dual users from both wireless and landline samples.
A national telephone survey of 2,501 adults was conducted in May 2009 using an overlapping dual frame design. The two designs can be compared by including (overlapping design) or excluding (segmented design) the dual users from the wireless sample. A comparison of the sample characteristics of this survey to Census population estimates finds the overlapping design generally yields results closer to population parameters than the segmented design. The study also finds significant differences in the characteristics of dual users from the two samples, explaining why the segmented design yields different and potentially less accurate population estimates.
Background: Cell Phones and Coverage in Telephone Surveys
As a result of the widespread adoption of cellular telephones, 22.9% of adults in the United States were “wireless only” in the second half of 2009 (Blumberg and Luke 2010). The coverage error associated with exclusively landline samples, along with significant differences in the characteristics of wireless only and landline populations, have encouraged the use of dual frame surveys. Unfortunately, while there is increasing demand for dual frame designs to address coverage limits in traditional RDD surveys, there is no consensus on the best design for a dual frame survey (AAPOR 2008).
The primary choices are: 1) segmented sample designs which draw mutually exclusive samples of landline and wireless only households; and 2) overlapping sample designs which include dual user households from both landline and wireless samples. Dual users, persons with both landline and wireless telephones in the household, are drawn exclusively from the landline frame in the segmented design but from both frames in the overlapping design. The segmented design has the advantage of simplicity in sample weighting since each survey respondent is drawn from a mutually exclusive sample segment.
If the characteristics of the dual users were essentially the same in landline and wireless frames, then the sample weighting simplicity of the segmented design would clearly be preferable. However, if there are significant differences in the characteristics of dual users between sampling frames, this would suggest differential propensity to respond by type of telephone among subpopulations of the dual user population. Under these circumstances, the exclusion of the dual users from the wireless frame would produce biases in the population estimates for dual users. Therefore the critical question in choosing a dual frame design is whether there are significant differences between dual users from the two frames.
The 2009 Traffic Safety Culture Index survey for the AAA Foundation for Traffic Safety, an overlapping dual frame survey, provided an opportunity to compare the characteristics of dual users from landline and wireless samples, and to test the impact of the type of dual-frame design on population estimates. The first sample consisted of a total of 1500 completed telephone interviews with adults aged 16 and older from a national, list-assisted RDD sample of households with landlines. The second sample consisted of 1001 completed interviews with adults aged 16 and older from a national sample of cell phone banks. For study specific reasons, the landline sample was stratified to oversample non-Metropolitan Statistical Area (rural) counties as classified by the U.S. Census. The data presented in this article were weighted to correct for the disproportionate sampling of rural areas in the landline sample.
The overall response rate (AAPOR 3) and cooperation rate (AAPOR 3) in the landline sample were 25.0% and 62.9%, respectively, and in the cell phone sample, they were 14.8% and 45.3%, respectively.
Are Dual Users from Landline and Wireless Samples the Same?
Although those with both landline and cell phones should be covered by both samples, there may be a difference in the likelihood of reaching segments of this population or a difference in their propensity to respond by type of phone. In order to test this hypothesis, we compared the characteristics of the dual user respondents from the two samples.
There is a significant difference in the age of respondents from the dual user segments of the landline and wireless samples (Table 1). Only 17.6% of landline dual users are under the age of 35, compared to 32.8% of wireless dual users. By contrast, 41.7% of dual users in the landline sample are 55 and over, compared to 30.7% of dual users in the wireless sample. Dual users from the wireless sample are more likely to never have been married, 26.6% to 12.4%, which probably reflects the age differences.
The majority of wireless dual users are males (51.6%), while males represent only 36.9% of dual users in the landline sample. White, non-Hispanic respondents represent 80.1% of the landline dual users, but only 67.4% of wireless dual users. Similarly, 78.1% of landline dual users report only English is spoken in their household compared to 71.3% of dual users in the wireless sample.
Dual users from the landline sample are more likely to own their residence (79.5%) than dual users from the cell phone sample (68.7%). Similarly, dual users from the landline sample are more likely to live in a small town or in the country (44.9%) compared to dual users from the wireless sample (35.3%). Although landline dual users are less likely to have less than a high school education than dual users from the wireless sample (5.6% vs. 15.0%), the differences in educational attainment are not as striking as other demographic differences.
In general, the demographic characteristics of dual users from the landline and wireless samples are quite distinct and the differences are statistically significant. Based on these differences, it appears that the dual user segment from a landline sample may not adequately represent the dual user segment of the general household population.
Overlapping or Segmented Design: Implications for Population Estimates
The difference in the demographics between dual users from the two samples suggests that there is a difference in the propensity to respond by type of phone among some key demographic segments. If the overlapping design yields a more representative sample of the general population, then it should produce survey estimates closer to population parameters without post-stratification weights. In order to test this hypothesis, we have compared both the segmented and overlapping sample approaches, weighted only to correct for disproportionate geographic sampling and estimated size of the telephone strata, to population estimates from the 2005–2007 Census PUMS for adults 16 and older (Table 2).
The overlapping dual frame design includes all of the respondents from the landline and wireless sample, which we have stratified by four telephone usage categories (landline only, dual users who are landline mostly, wireless mostly, and wireless only). The size of the completed wireless sample (N=1001) was selected so that it would yield a sample of wireless only respondents which would be roughly proportionate in the total sample of 2,501 interviews to the expected population. The four telephone strata in the combined sample are then weighted to NCHS estimates of the size of the four population segments of the population. In the case of the segmented frame design, the dual users from the wireless sample are excluded from the final sample as if they had been screened out at the beginning of the cell phone interview. Two estimates have been made using the segmented sample, one with the four category stratification including wireless mostly and landline mostly among dual users, and the other only using three categories (landline only, dual users, and wireless only).
The overlapping sample stratified by the four telephone use types comes closer to almost all of the Census estimates than either of the segmented dual frame models. For example, the Census estimates that 15.4% of the US population aged 16 and older is 16–24 years old. The 16–24 year olds represent 13.9% of the sample in the overlapping design before weighting to demographics. By contrast, 16–24 year olds represent only 11.3% and 11.4% in either of the segmented designs where the dual users from the wireless sample are excluded.
In Census estimates, males represent 48.3% of the population. In the overlapping dual frame design, males only represent 43.8% of the sample before demographic weighting. However, in both segmented dual frame designs, only 41% are male.
White non-Hispanics account for 69.1% of the population in Census estimates. In the overlapping dual frame design, white non-Hispanics represent a slightly higher 70.9% of the sample. However, in both segmented dual frame designs, white non-Hispanics represent a still higher 73.4%.
The never married represent 28.0% in the Census estimates. In the overlapping dual frame design, they represent a slightly smaller 26.1% of the sample. However, in the segmented dual frame designs, only 23.2% and 23.3% are never married.
The proportion of the population 16 and older that rents rather than owns is 29.2% according to the Census. In the overlapping dual frame design, the renters represent 29.5% of the sample (the answer categories for home ownership status were slightly different than the census questions). In the segmented dual frame designs, the renters represent a slightly smaller 28.3% and 28.4% of the sample.
Persons who have not completed high school represent 18.7% of the population aged 16 and older according to the Census. In the overlapping dual frame design, those without a high school degree represent only 11.7%. However, in both segmented dual frame designs, those without a high school degree represent only 9.7% of the sample.
Only in the case of language spoken in the household does the segmented frame appear closer to the Census estimate.
The findings from this national dual frame survey with an overlapping sample design suggest two things about dual users in telephone surveys. First, the differences by sample source in the characteristics of dual users interviewed suggest that there is a significant difference in propensity to respond to landline and cell phone contacts among certain demographic subsets of the dual user population. This differential propensity to respond by phone type means that certain population subsets are likely to be under-represented in landline samples, even though they are covered by landline sampling frames, and may be reachable by landline calls.
Second, the overlapping sample design for dual frame surveys corrects for this apparent differential propensity to respond. The availability, at least at the national level, of estimates of the size of the four telephone household usage segments allows us to weight the sample from the overlapping frames to its population size. And, at least in this survey, whose wireless sample size was calculated to yield a self-weighting sample of wireless only respondents, a simple combination of the overlapping samples from the two frames, weighted only to the projected size of the telephone usage strata, seems to yield population estimates that are more consistent with Census estimates than a segmented dual frame would have done.
We believe that these findings, if confirmed by further research, would suggest that overlapping dual frame surveys may be the most effective solution to issues of both coverage and propensity to respond to telephone surveys, as a result of the widespread use of wireless telephones in America.