As landline telephone coverage rates continue to decline, researchers are increasingly turning to list-assisted mail surveys as an alternative to random-digit-dial (RDD) telephone surveys. However, limited empirical work has been conducted to determine whether these list-assisted mail surveys represent an improvement over RDD for surveys that attempt to target specific subpopulations. This paper compares the results of an existing RDD survey of young adults with those of a concurrent list-assisted mail administration. We found that the mail administration resulted in improved coverage of the target population, increased efficiency of the administration, and enhanced representativeness of the target population, without reducing the validity of the key survey metric.
Over the past decade, the quality of the estimates produced by RDD telephone surveys has increasingly been questioned. In particular, the potential for noncoverage error has grown as cell-only households have become progressively more common. Among young adults, cell-only households are especially widespread; as of mid-2010, 40 percent of young adults ages 18 through 24 lived in a cell-only household, compared to 25 percent of all adults (Blumberg and Luke 2010). In addition, various characteristics and behaviors have been found to differ significantly between young adults living in cell-only households and those living in landline households (Blumberg and Luke 2007).
RDD surveys have also become increasingly expensive and inefficient as response rates have declined over the past few decades (Curtin, Presser, and Singer 2005). In RDD surveys of young adults, further inefficiency arises from the difficulty of successfully determining, prior to contact efforts, which phone numbers represent young-adult households; as a result, RDD surveys of young adults may waste resources by attempting to contact households that do not include members of the target population.
In the face of these challenges, survey practitioners are considering alternatives to traditional, landline-only RDD surveys (Link et al. 2008; Sherr et al. 2009). The two most common approaches are to add a cell phone sample to a landline sample or to switch to a list-based, mail sample. A list-assisted mail approach was chosen for this study because of its ability to efficiently identify young adults prior to contact efforts through the use of appended demographic data, while also reaching cell-only households. A cell-phone sample was not employed because of the low response rate and high costs associated with this approach (Brick et al. 2007; Link et al. 2007).
However, limited empirical work has been conducted to verify that list-assisted mail surveys represent an improvement over RDD for surveys of specific target populations. The present study addresses this gap in the literature by assessing whether a list-assisted mail administration yields superior data quality compared to an existing RDD survey of young adults. The study compares the two administrations in terms of
- coverage of the target population,
- efficiency of the administration,
- representativeness of the respondents, and
- validity of the resulting survey estimates.
Methodology
The RDD Telephone Survey The Youth Poll has been conducted by the Department of Defense (DoD) for over thirty years. The results presented in this article are from the December 2008 administration. The survey is conducted biannually using a list-assisted, landline-only RDD sample of young adults. Interviews are completed using computer-assisted telephone interviewing (CATI). The target population is non-institutionalized, English-speaking youth ages 16 to 24 who live in the United States and have never served in the U.S. Military.
The Youth Poll asks young people about their career and education plans, particularly as these relate to potential military service. The key metric produced by this survey is “military propensity” – defined as the proportion of youth responding that they will definitely or probably be serving in the U.S. Military in the next few years.
The List-Assisted Mail Survey The mail survey was administered in December 2008. It was conducted concurrently with the RDD Youth Poll in order to maximize comparability. The mail survey had the same target population as the RDD survey. The frame included approximately 28 million cases; address, name, and age information were available for each case. The frame was created by the DoD using government administrative lists, as well as a list purchased from a private vendor (see Table 1). The source lists were merged and purged of duplicate listings to create the frame.
Ten thousand cases were randomly sampled from the frame. Over a two-month period, sample members were sent a notification letter, an initial survey package, a reminder/thank you postcard, and two replacement survey packages. The initial survey package included a two-dollar incentive. The survey instrument contained a subset of the RDD questionnaire items, including the military propensity item.
Results
Coverage While the RDD survey sample was randomly drawn from the U.S. landline household population, the mail survey sample was drawn from lists created by multiple sources. As a result, the coverage properties and “randomness” of the mail sample are less clear. Despite this limitation, coverage estimates for the two surveys have been calculated and compared to one another. The estimated coverage rate of the mail survey (73 percent) was higher than that of the RDD survey (67 percent).
The estimated 73 percent coverage rate for the mail survey was computed by comparing the mail frame with the July 2008 Population Estimates Program (PEP) data produced by the Census Bureau. PEP data was selected as the benchmark because it includes estimates of the institutionalized and enlisted populations, which were included in the mail frame but are not part of Current Population Survey (CPS) estimates.
The estimated 67 percent coverage rate for the RDD survey is based on the Centers for Disease Control and Prevention (CDC) July – December 2008 data, which estimated that about one-third of 18 to 24 year olds live in cell-only households (Blumberg and Luke 2009). Due to a lack of available data, this estimate does not take into account the telephone status of 16- to 17-year-olds or the proportion of 18- to 24-year-olds that live in households with no telephone at all.
Efficiency The mail administration was more efficient than the RDD administration (see Table 2). The cost per mail respondent was less than half the cost per RDD respondent. The mail administration was more successful than the RDD administration at targeting the population of interest; about one-fifth of the mail sample resulted in completed questionnaires, while only 1 percent of the RDD sample resulted in completed interviews. The inefficiency of the RDD survey stemmed from the difficulty of reaching the target population. About one-third of the dialed numbers were nonworking; another fifth were tied to households that did not have a young adult living in them. In contrast, a much smaller proportion of mail cases were undeliverable or resulted in self-reports of ineligibility.
Respondent Profile Generally, the RDD and mail survey respondents had similar unweighted demographic characteristics to one another and to December 2008 CPS estimates. However, educational enrollment and attainment were considerably higher among mail survey respondents compared to RDD respondents and CPS estimates. This difference was accounted for in the weighting of the mail survey.
Respondents to the two surveys also differed in their phone usage. Mail respondents were considerably more likely than RDD respondents to have only a cell phone. RDD respondents were considerably more likely than mail respondents to have only a landline. Among respondents that had both a landline and a cell phone, mail respondents were more likely than RDD respondents to report using their cell phone to make all or almost all of their calls (61% and 44%, respectively); although these heavy cell phone users are covered by the RDD frame, they are likely difficult to reach on a landline.
Key Metric: Military Propensity The key metric resulting from this survey is military propensity – defined as the proportion of youth responding that they will definitely or probably be serving in the U.S. Military in the next few years. The weighted estimate of military propensity resulting from the mail survey was significantly lower than that resulting from the RDD survey (7 percent versus 11 percent). We believe the mail survey estimate was lower because of improved representation of cell-only, cell-mostly, and no-phone youth in the mail survey.
Cell-mostly youth were significantly less likely to report interest in military service in the mail administration than they were in the RDD administration. Military propensity was statistically equivalent in the two administrations for all other phone usage groups.
Among mail survey respondents, youth who used their cell phone at least as much as their landline were considerably less likely to be interested in military service than were landline-only or landline-mostly youth (Table 4). In particular, cell-mostly youth were significantly less likely to be propensed than were youth in other landline/cell usage subgroups.
Conclusions and Plans for Future Research
The mail administration appears to have improved upon the weaknesses of the RDD administration, while maintaining valid estimates of the key survey metric. The mail survey was more cost-effective and more efficient than the RDD survey. It also had a higher estimated coverage rate than the RDD survey. Although the military propensity estimate resulting from the mail survey was significantly lower than the one resulting from the RDD survey, this was likely due to the improved coverage of cell-only and cell-mostly youth.
Additional concurrent mail and RDD administrations are currently being conducted to determine whether the mail survey can replace the RDD survey, with a focus on further increasing coverage and efficiency while maintaining valid estimates.