Sampling Cell Phone Only Households: A Comparison of Demographic and Behavioral Characteristics from ABS and Cell Phone Samples

Address based sampling (ABS) represents an alternative to RDD screening of cell phone exchanges to identify cell phone only households for dual frame telephone surveys. Although both approaches provide coverage for cell phone only households, do the differences in frame, mode and screening introduce biases in the achieved sample? The authors compare the demographic and behavioral characteristics between two national samples of cell phone only adults, one sampled by ABS and one by cell phone RDD.

Background

Random digit dialing (RDD) telephone surveys remain the dominant form of probability samples for general population surveys in the United States. However, the continued growth of cell phone only households (27.8% of adults in the second half of 2010) has eroded the coverage of traditional RDD landline sampling (Blumberg and Luke 2011). This coverage error can be corrected by dual frame – landline and cell phone – samples. The higher cost of screening RDD cell phone samples has slowed the adoption of dual frame designs and encouraged alternative approaches to sampling cell phone only households.

Address based sampling (ABS) has emerged as the major alternative (Link et al. 2009). ABS is based on the U.S. Postal Service’s Delivery Sequence File (DSF), which is estimated to cover 97% of U.S. households (Iannacchione, Staab, and Redden 2003; O’Muircheartaigh, Eckman, and Weiss 2003; Staab and Iannacchione 2004). A random sample of addresses from the DSF can be drawn within any geographically defined sampling area and used for face-to-face or mail surveys. These samples can also be used for telephone surveys if the addresses are matched to telephone listings. Those addresses with phone number matches can be dialed in a similar fashion as traditional RDD. Addresses unable to be matched to a phone number are assumed to be cell phone only, unlisted, no phone, or vacant households. The phone numbers for these unmatched households can be obtained through a short mail questionnaire. Thus the ABS sampling frame permits a mixed-mode solution to the cell phone only problem.

The decision to use only the unmatched sample to identify cell phone only households was a result of the increased efficiency of eliminating more than fifty percent of the total sample matched to known landline households. We recognize that coverage error would be introduced if a significant proportion of the matched phone numbers were incorrect as a result of the matching algorithms used by the sample vendor. If these mismatches are greater among apartment addresses because the matching algorithms for multi-unit dwellings may be less precise, the likelihood of coverage error increases because the majority of cell phone only adults are renters (Blumberg and Luke 2011).

The procedures used in ABS minimize the chances for incorrect matching. First, ABS is based on the USPS’s Delivery Sequence File that uses uniform address formatting, unlike many other list samples. Second, for group housing addresses in ABS samples, our sample vendor requires an exact match between addresses before a phone number can be appended. Hence, if the addresses, including apartment numbers, are not identical, the sample is classified as unmatched sample.

Of the 10,000 addresses in the ABS sample, 25% had apartment numbers. As expected, the rate of matched phone numbers was substantially lower among these addresses (31%) than among non-apartment addresses (58%). Therefore, while there is some risk of coverage error in using only the unmatched sample to identify cell phone only adults, there is no evidence based on this sample that it introduces significant error.

The mixed mode ABS approach may introduce different selection biases than the single mode cell phone screening. Unfortunately, there is little evidence available to evaluate the sample bias associated with these two alternative approaches. In this paper, we have used ABS to generate a national cell phone only sample for comparison to a national cell phone only sample drawn from an RDD cell phone sample. We do not address differences in the cost, timeliness, or response rate between the two modes. Our research question is whether the two methods yield equivalent samples of cell phone only respondents with the same demographic and behavioral characteristics.

Methods

For this experiment, a national sample of 10,000 household addresses was drawn from the Delivery Sequence File by Marketing Systems Group (MSG). The address sample was matched to listed telephone numbers. There was a phone match for 5,136 addresses; 4,864 addresses were unable to be matched.

A random sample of 4,000 unmatched addresses was sent a cover letter, a one page (two-sided) questionnaire, and a business reply envelope. The cover letter and questionnaire were in English only. An incentive of five dollars was offered for completing and returning the questionnaire. A telephone number was requested for a possible follow-up study.

Eleven percent of the mail questionnaires were returned after two mailings. Among those who returned the questionnaire, nearly half (49%) were cell phone only households. Dual users were not far behind at 43% of returns, with a small proportion of landline only (5%) and no phone (1%) households making up the remainder.

Seventy-six percent of respondents from the unmatched sample who returned the mail questionnaire provided a telephone number. The proportion providing phone numbers was higher (80%) in cell phone only households. Follow-up telephone interviews were completed with nearly six out of ten cell phone numbers (59%). Refusals were rare (4%). However, some numbers were bad (6%), others were never reached (10%), and more (21%) were still in callback status at the end of the field period.

A national cell phone only sample from the 2010 Traffic Safety Culture Index survey, which used an overlapping dual frame design, was used for comparison. This cell phone only sample included 419 completed interviews with respondents aged 16 and older. Additionally, the June 2010 estimates of demographics for cell phone only adults (18 and older) from the National Health Interview Survey (NHIS) are shown for comparison.

Demographic Comparisons

The proportion of males in the unweighted ABS cell phone only sample was 44% compared to 54% in the unweighted RDD cell phone sample (Figure 1). The proportion of persons aged 16-34 in the ABS sample was 46% compared to 52% in the RDD cell phone sample. In the ABS sample, 39% had an educational attainment of high school or less compared to 44% in the RDD sample. The proportions of males (54% to 51%), 16-34 year olds (52% to 54%), and high school graduates or less (44% to 43%) in the RDD cell phone sample were nearly identical to the NHIS estimates. The proportion of black, non-Hispanic respondents in the ABS sample was 14%, similar to 16% in the RDD cell only sample, and 13% for the NHIS cell only estimates. The age and gender differences between the ABS and cell phone samples are statistically significant. These demographic differences are consistent with the self-selection bias usually associated with mail surveys with limited follow-up.

Behavioral Comparisons

Nine out of ten of the cell phone only respondents from the unweighted ABS sample (91%) reported driving in the past 30 days, compared to 82% in the unweighted RDD cell phone sample (Figure 2). Since the ABS mail survey was labeled as a “Driving Survey” with most of the questions about driving, this may reflect a topic salience bias which should be more pronounced in a self-administered mail survey than a telephone survey. Among those who had driven in the past 30 days, a higher proportion of the respondents from the ABS sample reported talking on their cell phone when driving in the past 30 days (87%) compared to those from the cell phone sample (81%). Similarly, a higher proportion of the ABS cell phone only respondents reported reading or sending a text message or e-mail while they were driving (45%) than in the cell phone sample (39%).

The proportion of respondents who reported driving without wearing seatbelts in the past 30 days was somewhat lower in the ABS sample (20%) than in the RDD cell sample (28%). The proportion of cell phone only respondents who reported that they drove 15 miles per hour over the speed limit on residential streets in the past 30 days was lower in the ABS sample (14%) than in the RDD cell sample (29%). Similarly, a smaller proportion of the ABS sample (42%) than the cell sample (48%) reported driving 15 miles per hour over the speed limit on freeways in the past 30 days. Finally, the proportion who had driven through a light that had just turned red, when they could have stopped, in the past thirty days was lower in the ABS sample (34%) than the cell phone sample (39%).

The two samples were similar in the proportion of respondents who had driven when they were so tired they were having a hard time keeping their eyes open (28% for ABS and 30% for cell RDD). The proportion of alcohol drinkers was the same for the ABS sample (63%) and the RDD cell sample (63%) as were the proportion that reported that they had driven in the past year when they thought their alcohol level might have been close or possibly over the legal limit (26% ABS, 27% RDD).

Finally, the cell phone only drivers from the ABS sample (20%) are somewhat less likely than those from the RDD cell phone sample (31%) to report having been stopped by the police for any sort of moving violation in the past 2 years. By contrast, the drivers in the ABS sample are more likely (18%) than those in the RDD cell sample (13%) to report having been in an accident when they were driving in the past 2 years.

Although there was a significant demographic bias between the two cell phone only samples, this did not seem to significantly affect the behavioral measures.^[1] The proportion of drivers in the two unweighted samples (91% ABS, 82% RDD) remained about the same after sample balancing the two samples (90%, 82% respectively). Other measures remained unchanged or only differed by a few percentage points (Figure 3). Weighted or unweighted, the differences in the behavioral measures were small, and there is no consistent direction of the differences, suggesting sampling variability rather than true differences.

Discussion

The results of this study indicate that the demographic characteristics of cell phone only adults identified from ABS are less consistent with the national NHIS estimates than those from cell RDD. The differences appear to reflect the biases found in mail surveys, which the ABS cell phone sample depends on. However, the differences are not large, and thus may be corrected with post-stratification weighting.

The disparity in behavioral characteristics between the two samples was small yet not corrected through the use of post-stratification weights. Hence, the differences in the behavioral measures in the two samples do not appear to a result of the demographic differences in the unweighted sample.

The results from this study suggest that we cannot assume that sampling cell phone only adults from ABS and RDD samples will yield comparable estimates, even though both approaches provide coverage for cell phone only households. At minimum, the mail component of the ABS design appears to introduce a demographic bias, which has long been associated with mail surveys. Additionally, a telephone survey can mask the content of the survey better than can a self-administered survey, which may have introduced a salience bias in this study. In summary, although ABS may offer a potentially lower cost alternative for obtaining a cell phone only sample, the multi-mode ABS approach may introduce more demographic and topic salience bias in the survey measures than the RDD cell phone sample. Although this did not appear to affect subsequent behavioral measures in this survey, researchers should be cautious about the potential affect of sample choice on survey estimates.