Building a New Foundation: Transitioning to Address Based Sampling after Nearly 30 Years of RDD

Michael W. Link, Gail Daily, Charles D. Shuttles, L. Tracie Yancey, Anh Thu Burks, and H. Christine Bourquin
The Nielsen Company

Address based sampling (ABS), the use of a comprehensive database of addresses for sampling of residential households, is garnering considerable attention by survey researchers as a potential alternative to random digit dialing (RDD) surveys. For nearly 30 years, the Nielsen TV Ratings Diary Survey was one of the world’s largest RDD surveys (in recent years screening more than 5 million telephone numbers annually). In November 2008, the TV Ratings Diary moved from a landline telephone frame to an ABS frame, becoming the first major survey research effort to make this important transition. The new TV Ratings design uses ABS with a multi mode data collection approach, which includes Web, mail, and telephone recruitment tools. We assess the success of this transition by comparing the March 2009 ABS measurement to the February 2008 RDD design. The lessons learned from this pioneering effort will further the understanding the industry has for the potential uses of this new approach. Some of the key findings included:

Allows researchers to reach cell phone only households

Significant step in improving representation of younger homes

Improves coverage but not necessarily response rate in all cases

Key sample indicators on ABS file are more accurate than corresponding indicators on landline frame — with addition of geocoded information can be a powerful tool for addressing racial/ethnic imbalances

Using the U.S. Postal Delivery Sequence File for Mixed-Mode Studies: Report on Measurement Differences Between Mail and Telephone Responses in the SHAPE Study

Todd Rockwood, Melissa Constantine, and Michael Davern
University of Minnesota

Timothy Beebe
Mayo Clinic

Sheldon Swaney
Hennepin County Minnesota

The SHAPE study is a large general population (n=7500) public health screening survey conducted in Hennepin County Minnesota by the Hennepin County Department of Health every three years ( http://tinyurl.com/6w9m3o). The 2006 administration of the SHAPE study utilized the DSF as the primary sampling frame. Additionally, a mode of administration experiment was conducted in which a sub-sample of respondents were randomized to either the mail (n=1848, RR1 70%) or telephone (n=560, RR 27%) mode of administration.

Socio-demographic characteristics

Differences:

  • Higher percentage of females (65%) by mail than phone (58%, X2p=0.01)
  • Higher percentage born in the US (93%) by mail than phone (89%, X2p=0.01)
  • Higher percentage live in urban core (as opposed to suburbs, 45%) by mail than phone (40%, X2p=0.01)

No Differences:

  • Race/Ethnicity
  • Educational status
  • Age

Questions by topical area:

General Health Status – 11 items/3 demonstrate significant differences

Depression – 9 items/7 demonstrate significant differences

Insurance/Health Services – 6 items/4 demonstrate significant differences

Health Behaviors/Screening Tests – 3 items/3 demonstrate significant differences

Evaluation of community – 10 items/7 demonstrate significant differences

Public/Medical Assistance (Welfare) — 11 items/2 demonstrate significant differences

Discrimination – 10 items/4 demonstrate significant differences

Overall, 50% of the topical items evaluated demonstrate significant differences between the mail and telephone modes of administration. In general the differences are consistent with what would be expected based on models of mode of administration effects.

Using Address-based Sampling to Survey the General Public by Mail vs. Web plus Mail

Benjamin L. Messer and Don A. Dillman
Washington State University

Our purpose in this study was to determine the extent to which households in an address-based sample would respond to a general public survey via the Internet when the survey request was sent by postal mail. We obtained a statewide random sample of addresses from the USPS Delivery Sequence File (DSF) to conduct the Washington Community Survey (WCS) in the summer and fall of 2008 using both mail and Internet survey modes. Nine experimental treatment groups were designed to test the overall differences between and the effects of different contact procedures, including a $5 incentive and Internet instruction card, on mail and Internet response. The groups were comprised of a mail-only treatment, three mail preference treatments, and five Internet preference treatments. Respondents in the “preference” groups were mailed a request to complete the survey via a designated mode, mail or Internet, and three weeks later non-respondents were asked to respond by the alternate mode. Among the results:

The $5 mail only group achieved the highest overall response rate of 56.7%, while the mail-preference groups with a $5 incentive were close behind, ranging from 53.6% to 55%. Response rates for the Internet groups with a $5 incentive were significantly lower, ranging from 42.8% to 46.3%. However, offering the Internet as the first mode, as in the Internet preference groups, resulted in two-thirds responding via the Internet and the remaining one-third via postal mail. Offering mail as the first survey mode resulted in very few Internet responses, ranging from only 2% to 5.7% of the total.

The illustrated Internet card, which provided instructions and encouragement to respond via the Internet, did not increase response rates for the two Internet groups that received it. In addition, we found very few significant differences between respondents who received an Internet card vs. those who did not; the only differences were on gender, education, and marital status.

The $5 cash incentive significantly increased response rates for both mail (52.5% vs. 39.2%) and Internet respondents (31.3% vs. 13.4%) but had a larger impact on Internet response with a difference of 17.9% for the Internet and only 13.3% for the mail. Respondents to the $5 incentive groups were very similar to respondents who did not receive the $5, regardless of survey mode; there was a significant difference only for income among mail respondents and on education among Internet respondents.

Respondents to the Internet in the Internet preference groups were, on average, younger, more educated, married, and employed with fewer people in the household and higher incomes compared to the mail follow-up respondents in these groups. The differences persisted but were much smaller when comparing the Internet respondents to the mail respondents in the mail preference groups. However, these differences become negligible when Internet and mail follow-up respondents in the Internet preference groups were combined and compared to mail respondents in the mail preference groups.

Finally, neither WCS Internet nor mail respondents were completely representative of the Washington population when compared with 2007 American Community Survey (ACS) data for Washington, even with weighting on gender and age. The mail respondents were closer overall but both types of respondents were more educated and more likely to be employed, have children in the household, larger household sizes, and higher incomes compared to ACS respondents. All WCS respondents were representative of the cell-only population in Washington, estimated at about 18%, but were over-representative of households with Internet access by 12% to 24%.

Comparing Random Digit Dial (RDD) and United States Postal Service (USPS) Address-Based Sample Designs for a General Population Survey: The 2008 Massachusetts Health Insurance Survey

Susan Sherr and David Dutwin
SSRS

Timothy Triplett, Doug Wissoker and Sharon Long
Urban Institute

In the summer of 2008, the Urban Institute and Social Science Research Solutions (SSRS) conducted the Massachusetts Health Insurance Survey (HIS) on behalf of The Massachusetts Division of Health Care Finance and Policy. The goal of the Massachusetts HIS is to document health insurance coverage and access among Massachusetts residents. In an effort to include cell-phone only households in the study, the 2008 HIS employed a dual-sample-frame design that combined a random-digit-dial (RDD) telephone sample and an address-based (AB) household sample. In addition, the AB sample and the RDD sample were each divided into two strata: (1) sample records with both an address and matching telephone number and (2) sample records with either a phone number (RDD) or an address (ABS), but not both. Survey respondents could choose to complete the survey by telephone, web or mail. A total of 4,910 interviews were completed across both sample frames. In comparing results from the RDD and ABS samples, we found that:

The sample yield was better and more efficient in the ABS sample. As a result, the cost per interview was lower with the ABS sample.

Almost half of all respondents–52% of ABS respondents and 34% of RDD respondents–completed the survey online. The popularity of a web questionnaire helped keep overall survey costs low.

The ABS response rate (34.7%) was lower than RDD (42.0%). However, there was no significant difference by sample frame or by mode in either breakoffs or incompletes.

8.5% of ABS respondents were from cell-phone only households, a figure that is close to the recent NIH estimate for Massachusetts.

Unweighted demographic characteristics of the ABS respondents were closer to American Community Survey population counts than was true for RDD respondents, indicating a reduction in coverage bias with the ABS. Nevertheless, nonresponse bias among underrepresented groups was present in both sample frames.

We conclude that the ABS offered a number of advantages over the RDD sample, including interviews with cell-only households, better sample yields, and lower costs per interview. The better coverage in the ABS sample results in smaller weights, which resulted in smaller design effects and less sensitivity in estimates of key variables due to weighting as compared to the RDD sample.

Performance Rates of CPO Subsequent Survey Households Identified Via Address Frames

Anna Fleeman and Nicole Wasikowski
Arbitron Inc.

Findings from Arbitron’s two large address-based-sample (ABS) studies, fielded in 2007 and 2008, suggest that Cell-Phone-Only (CPO) households can be efficiently identified and less expensively than cellular RDD. A short questionnaire asking about media-related behaviors and cell/landline ownership was sent to the address sample unable to be matched to a phone number (n=20,094). Sampled households responded at encouraging levels with approximately 35% reporting CPO status and providing a cell phone number on which to reach them (n=2,058). After constructing the CPO sample pool via the questionnaire mailing, Arbitron then began placing radio listening diaries and encouraging diary return in these CPO households following standard Radio Ratings methodology. Highlights are as follows:

70% of CPO households consented to participate in the diary-based Radio Ratings, nearly double the landline sample (control).

Of the ~2,700 radio-listening diaries sent to consenting households, 72% were returned, which was several points greater than landline sample (p≤0.00).

More than double the number of 18–34 year old diarykeepers in CPO sample than landline sample (p≤0.00).

Yield of CPO households in ABS greater than cellular RDD: 10% versus 4%.

Using ABS to identify CPO households allows for pre-alert mailings and targeted incentives prior to phone contact.

Because respondents provided their cell phone number on the questionnaire, there is no need to hand-dial as with cellular RDD.

Best of both worlds: maximize mail and phone contact while identifying CPO households.

Using an address frame to include CPO households in Arbitron’s Radio Ratings is financially and methodologically better than relying on cellular RDD.

Modeling the Need for Traditional vs. Commercially-Available Address Listings for In-Person Surveys: Results from a National Validation of Addresses

Ned English, Colm O’Muircheartaigh, Michael Latterner, Stephanie Eckman, and Katie Dekker
NORC

NORC conducted a national validation of USPS delivery sequence file (or DSF) addresses with three goals: first, to create updated coverage estimates for urban, rural, and suburban areas, as an extension of previous research; second, to learn about DSF coverage in areas that have experienced growth during the past decade, and so have transitioned from rural to suburban or urban; finally, to develop a model to predict DSF coverage based on information available in advance of data collection.

NORC field staff checked the quality of the DSF, confirming addresses that existed, rejecting addresses not present, and adding any new addresses not on the DSF. Based on these results we measured the quality of the DSF in a variety of environments, and determined the relative under- and over- coverage. Our results were as follows:

Urban areas generally have better DSF coverage than rural areas.

Rural areas do not have universally poor coverage.

Traditional listing is not always necessary in rural areas, as rural areas have started to be better represented on the DSF then previously.

Population density and the percent of addresses city style are effective predictors of coverage.

Over-coverage appears to be more haphazard than under-coverage, as the former is dependent on geocoding database quality at the local scale.

Address Based Sampling and Address Matching: Experience from REACH U.S.

Katie Dekker and Whitney Murphy
NORC at the University of Chicago

The address-based sampling approach relies heavily on accurate matching of addresses to working telephone numbers. After selecting a sample of addresses, NORC matched selected addresses to telephone numbers using commercial vendors. During the screener portion of the telephone interview, respondents were asked to verify that they live at the selected address. The results of the screening process allow us to assess how well our vendor is able to match addresses and help us to determine whether we can safely eliminate the screener question in future rounds.

At the time of this analysis, our vendor was able to match approximately 70% of the sampled addresses to telephone numbers.

Of the phone numbers that were matched, about 91% of the respondents confirmed the address in the phone interview.

Based on our findings, we do not have the confidence in our match rates to eliminate this screener verification question.

Evaluation of Address Based Sampling (ABS) Frame Supplementation Methods for In-Person Household Surveys

Joseph P. McMichael, Jamie L. Ridenhour, Bonnie E. Shook-Sa and Vincent G. Iannacchione
RTI International

Survey researchers are increasingly looking to Address Based Sampling (ABS) as a less costly alternative to field enumerated sampling frames. Although research suggests that the national household coverage of an ABS frame is high, coverage is not evenly distributed leading to a disparity in coverage between rural and urban areas. The undercoverage of the ABS frame, particularly in rural areas, can create bias in surveys utilizing only an ABS frame. In-person surveys can use field-implemented supplementation methods to increase coverage and reduce this potential for bias. The 2008 American National Election Survey (ANES) is a national, in-person survey that used a frame supplementation procedure called the Check for Housing Units Missed or CHUM (McMichael et al. 2008). This procedure is a series of protocols to systematically identify dwelling units missing from the frame. Evaluation of the CHUM has shown it is a successful method for improving coverage, though more work can be done to improve the training and monitoring of field staff. Future research on the CHUM will focus on improving training for field staff, evaluating cost, and examining bias reduction.

References

McMichael, J., J. Ridenhour and B. Shook-Sa. 2008. A robust procedure to supplement the coverage of address-based sampling frames for household surveys. Proceedings of the American Statistical Association, Section on Survey Research Methods.

Multi-Mode Surveys Using Address Based Sampling: The Design and Preliminary Experience of REACH U.S. Risk Factor Survey

Martin Barron
NORC at the University of Chicago

This presentation describes the design of the Racial and Ethnic Approaches to Community Health Across the U.S. Risk Factor Survey (REACH RFS). REACH RFS is one of the first large-scale surveys to employ a multimode ABS approach. REACH RFS is a project sponsored by the Centers for Disease Control and Prevention to measure the performance of 28 community-based programs designed to reduce health disparities among racial and ethnic minority populations. REACH RFS will employ ABS combined with data collection via telephone, mail, and face-to-face interviews. In this presentation, we discuss the ABS design and practical implications of the REACH RFS design. In summary:

REACH RFS considered a number of designs but determined an ABS design to be most appropriate.

In designing our ABS approach, the main priorities were to maximize coverage, interviews via the telephone, and response rates.

The final design calls for attempting to contact households via telephone first. If that fails, attempts will be made by mail and face-to-face.

Though it is still too early to gauge the success of this design, we note that this ABS design requires considerably more field time than a traditional RDD survey. We further note that the multi-mode design can quickly lead to an extremely complex design, particularly if respondents are allowed to switch back and forth between modes. Nevertheless, NORC believe that ABS has significant potential for the REACH Risk Factor Survey.