Computing “e” in Self-Administered Address-Based Sampling Studies

In Standard Definitions (American Association for Public Opinion Research 2016), two of the formulae for computing response rates (RR3 and RR4) include a factor, e, that accounts for the proportion of cases with unknown eligibility that are assumed to be eligible. This note addresses challenges with computing e in self-administered household surveys in which address-based sampling (ABS) was used to select the sample of addresses.

In recent years, ABS has become a widely used approach to obtaining samples for household surveys in the United States. ABS begins with a sampling frame that generally uses as its basis mailing address files maintained by the U.S. Postal Service (USPS). Because the USPS uses the database on which these files are based to manage its mail delivery services, efforts are made to keep these files up-to-date. However, the USPS database depends on input from the mail carriers. In addition, survey research organizations are only able to access these files through third-party vendors who license them. The ABS frames generally have nearly complete coverage of household addresses (Link et al. 2008); however, inaccuracies still exist due to lags in timing or failures to update statuses of addresses.

In general terms, each sampled address can be classified into exactly one of three possible address eligibility categories: (1) addresses known to be eligible addresses (where “eligible” here is defined in terms of the address being associated with a household, regardless of whether the household is eligible for the particular study); (2) addresses known to be ineligible; and (3) addresses with unknown eligibility. Eligible addresses include those who respond to the survey (complete or partial respondents) as well as those who fail to respond but provide information to indicate that a household resides at the address (e.g. those who call in to refuse or return the survey with a comment indicating they are refusing). Ineligible addresses include those whose survey mailings (at the initial contact stage) are returned by the USPS as nondeliverable as well as those who call in (or contact the survey organization by mail, email, etc.) to indicate that the address is associated with only a business. All remaining addresses have unknown eligibility and are referred to as unresolved. Addresses known to be eligible or ineligible are collectively referred to as resolved.

Various approaches have been used to compute e (Brick, Montaquila, and Scheuren 2002; Ezzati-Rice et al. 2000; Frankel and Wiseman 1983; Kennedy, Keeter, and Dimock 2008; Shapiro et al. 1995). These approaches may be reasonable for surveys in which a large proportion of sampled cases are resolved. However, in ABS studies, it is often the case that only a minority of addresses are resolved, a situation in which the assumptions underlying the existing methods may not hold.

The calculation of e we propose involves “backing out” the estimated eligibility rate using estimated totals of eligible and unresolved addresses from the survey (${\widehat{T}}_{\text{elig}}$ and ${\widehat{T}}_{\text{unr}}$, respectively) and a reliable external estimate (${\widehat{T}}_{\text{ext}}$) such as an estimate of the number of households from the American Community Survey (ACS). It is necessary for this external estimate to be an estimated number of households (as opposed to housing units), for it to correspond to the geographic scope and the timing of the survey, and for it to come from a reliable source. In addition to the ACS, other sources that could be considered (depending on the particular survey) include the Current Population Survey and the decennial census. Here, ${\widehat{T}}_{\text{elig}}$ and ${\widehat{T}}_{\text{unr}}$ are sums of weights (generally base weights) of eligible and unresolved addresses, respectively. We note that the quantity ${\widehat{T}}_{\text{elig}} + e{\widehat{T}}_{\text{unr}}$ provides an estimate of the number of eligible addresses (i.e. households), as does ${\widehat{T}}_{\text{ext}}$. Setting these two estimators equal to each other and solving for e yields the “backing out” estimate

$$e_{B} = \frac{1}{{\widehat{T}}_{\text{unr}}}\left( {\widehat{T}}_{\text{ext}} - {\widehat{T}}_{\text{elig}} \right)$$

It is mathematically possible for $e_{B}$ to be negative (if ${\widehat{T}}_{\text{elig}}$ exceeds ${\widehat{T}}_{\text{ext}}$) or to exceed 1 (if ${\widehat{T}}_{\text{ext}} > {\widehat{T}}_{\text{elig}} + {\widehat{T}}_{\text{unr}}$). In such cases, we recommend setting $e_{B}$ equal to 0 or 1, respectively.

Author’s Contact Information

Jill M. DeMatteis, Westat, 1600 Research Blvd., Rockville, MD 20850. jilldematteis@westat.com; (301) 517-4046.