The 2013 Census Test: Piloting Methods to Reduce 2020 Census Costs

Introduction

Decennial censuses in the United States are increasingly expensive. The most recent – the 2010 Census – cost nearly $12.7 billion (U.S. Census Bureau 2013). After the 2010 Census, then US Census Bureau Director, Dr. Robert Groves, said in a prepared statement to a US Senate subcommittee, “We are attempting to design a 2020 Census that costs less per housing unit than the 2010 Census while maintaining the quality of the results” (Groves 2011). Major goals of methodological research for the 2020 Census are to reduce cost and maintain high quality data.

This paper discusses the 2013 Census Test, a pilot study conducted by the Census Bureau that lead off a series of decennial census experiments examining ways to reduce the costs of in-person interviews with households that do not respond to the decennial census by Web or mail. These in-person interviews – known as “nonresponse follow up” (NRFU) – are a major cost driver for the census, amounting to $1.6 billion in 2010 (Walker et al. 2012). The 2013 Census Test examined the feasibility of three potential cost-saving methods:

1. Use administrative records to both determine housing unit occupancy status and directly enumerate occupied households or to determine the level of effort (e.g., number of contact attempts) expended on enumerating them in person.
2. Use an adaptive approach to manage in-person contact attempts.
3. Attempt telephone contacts prior to in-person visits.

The study provided a foundation for subsequent research into 2020 Census methods by testing the feasibility of new NRFU methods and identifying issues to be addressed before deciding whether to adopt them.

Background

Managing NRFU costs requires reducing the in-person interview workload and gaining better control over field operations. The National Research Council’s (NRC) recommendations for the 2020 Census include “field reengineering,” which covers a broad class of infrastructure development and management improvements (Cook, Norwood, and Cork 2011). In this study, it involves tailoring the number of contact attempts expended on a housing unit based on information known about the address and prioritizing enumerator contact attempts on a daily basis through modeling and automation of case assignments. Tailoring contacts and prioritizing daily case assignments are components of “adaptive” or “responsive” survey design (e.g., Groves and Heeringa 2006; Mohl and Laflamme 2007; Schouten, Calinescu, and Luiten 2013). An adaptive design approach makes use of auxiliary frame data and paradata to monitor and control data collection efforts in service of cost and quality goals.

The NRC panel also urged the Census Bureau to employ administrative records to improve the enumeration process. Since a great many US residents provide information to the federal government – for example, to participate in health care programs or to file taxes – this information could aid in decennial census enumeration, thereby reducing burden on the public. Using records to enumerate households directly would also achieve great cost savings, since NRFU contacts would be limited to addresses for which no records
exist. An alternative use of records involves attempting in-person contacts with all nonrespondents but adjusting the number of contact attempts according to whether record information is available. If suitable record information exists, the Census Bureau could expend less effort trying to obtain an interview. This option would preserve the choice for US residents to respond to an in-person census appeal. The 2013 Census Test examined these two types of record uses in order to see if they could be implemented in an actual field context.

The choice of data collection mode also has implications for NRFU costs. Surveys employing multiple modes often try contacts using cheaper methods before moving to more labor-intensive approaches. For example, the American Community Survey first offers respondents a mail or internet self-response option and then contacts by telephone before sending field representatives for an in-person interview (U.S. Census Bureau 2014). The NRC has urged the Census Bureau to continue examining the use of telephone in decennial census operations (Brown et al. 2010). The 2013 Census Test appended telephone numbers to sample addresses where possible and explored different calling approaches to see if it would decrease the number of costly personal visits.

Test Design

The pilot study involved four experimental treatments for NRFU case management, summarized in Table 1.

Administrative Records

Two treatments used administrative records to reduce the field workload size before interviewing. Referred to as the “Reduced Workload Group,” they employed administrative records to identify and remove some vacant and occupied housing units from the field workload. Sample unit occupancy status – either occupied or vacant – was determined using United States Postal Service (USPS) information on whether advance letters sent to sample households were deliverable, combined with administrative records information from the Internal Revenue Service U.S. Individual Income Tax Return Form 1040, Medicare Enrollment Database and Targus Federal Consumer file. For sample units judged to be occupied, the research team constructed a count of occupants from record data.

By contrast, the two treatments in the “Full Workload Group,” did not use administrative records directly to identify occupancy status or enumerate occupied households. Comparing the two groups allowed us to observe operational issues posed by using USPS data and administrative records to determine occupancy status and enumerate occupied housing units in real time. In addition, we could observe differences in difficulty of fieldwork between the two groups.

Adaptive Case Management Approach

As shown in Table 1, the 2013 Census Test also contrasted two ways of managing interview contact effort. Cases in the “Fixed Group” all received up to three personal visits, while cases in the “Adaptive Group” had a varied overall level of effort. In the Adaptive Group, cases with adequate administrative records information to enumerate the household received only one personal visit, while those without this administrative record designation received up to three personal visits.^[1]

In addition to comparing one versus three contact attempts, we compared two methods of interviewer case assignments. In the Fixed Group – similar to previous decennial censuses – interviewers each received a batch of cases to work and made contacts largely at their own discretion, within broad supervisory guidelines. By contrast, the team trained and equipped Adaptive Group interviewers to start work with seven “high priority” cases each day. Response propensity models, based on contact data from the 2010 Census and on paradata gathered during the test, determined high priority cases – those most likely to yield an interview on the next contact attempt. The case management system updated interviewers’ high priority cases each day when they transmitted completed work from their laptops. In comparing the Fixed and Adaptive groups, we wanted to see if the case management system could use response propensity scores to assign new cases each night and how interviewers handled these assignments, compared to those in the Fixed Group who had more personal control over which cases to contact.

Telephone-First Mode Sequence

In the 2013 Census Test, we also set up a mode sequence, attempting data collection by telephone before resorting to in-person visits.^[2] Up to three telephone numbers from a variety of commercial sources were linked to sample addresses. We then tested two different methods of telephone calling. In the Fixed Group, in-person interviewers used their own phones to make telephone calls prior to face-to-face contact attempts. In the Adaptive Group, a computer-assisted telephone interviewing (CATI) operation was mounted for two weeks. A decennial census has not attempted such a CATI approach its mammoth scale makes setting up a CATI infrastructure complex and expensive. The test was intended to see how it might work on a small scale. After two weeks of calling, nonresponding CATI cases were transferred to in-person interviewers.

Sample and Fieldwork

A sample of 2,077 housing units in eight block groups in Philadelphia, Pennsylvania, was selected for this pilot study. The block groups were matched on demographic characteristics and randomly assigned to the Fixed or Adaptive Groups. Within selected block groups, sample units that had not responded to the 2010 Census were randomly assigned to the Full or Reduced Workload Groups. Data collection took place in December 2013. Eighteen Census interviewers worked on the study, administering a questionnaire like that employed in the 2010 Census.

Key Lessons

Case dispositions for the four experimental groups are displayed in Table 2. Using somewhat dated record information, 508 cases (24.5 percent) could have been enumerated using administrative records, 408 before fieldwork in the Reduced Workload Group and 100 after fieldwork in the Full Workload Group. Interviewers completed 41 cases (2.0 percent) using telephone before in-person interviewing began, fourteen by in-person interviewers in the Fixed Group and 27 by CATI interviewers in the Adaptive Group. Cases with a final outcome code (including, for example, vacant housing units and addresses that are not housing units) were counted as “complete.” Some 249 cases (12.0 percent) had an “incomplete” disposition code, due to being a refusal or reaching the maximum allowable number of contact attempts without an interview.

Administrative Records

The team was able to designate cases as occupied or vacant using administrative record and postal information and, in the Reduced Workload Group, remove these cases from the workload before interviewers attempted contact. In the Adaptive, Full Workload treatment, the team was able to use administrative record information to designate different numbers of maximum contact attempts.

Using administrative records to reduce the NRFU workload assumes accurate record information about sample units’ occupancy status and population count. Comparing cases with both records information and interview results in this test – and treating the interview results as accurate – we find records predicted occupancy status correctly in 83 percent of cases. For occupied sample units, record information on household population was within one person in 66 percent of cases (Walejko et al. 2014). Since the records used were dated, these results are heartening, but predictions of occupancy status and household size from administrative data need more work. The Census Bureau is exploring different records and different match rules in subsequent research.

Adaptive Case Management Approach

The Adaptive Group targeted the number of allowable contacts assigned to cases depending on the availability of administrative records information. Systems worked correctly to assign one contact to housing units that could be enumerated using record information or three contacts to those that did not have adequate record information. However, our attempt to control the number of contact attempts expended on each case ran into implementation problems. At the end of the field period, a number of cases had no data because enumerators did not obtain an interview on the same day. Interviewer debriefings uncovered confusion about the need to get a proxy interview if an interviewer did not obtain a household interview and reached the maximum number of contacts.

Controlling the number of contacts relied on enumerators recording attempts as they made them and transmitting these data each day. If interviewers did not, they could exceed the desired number of contact attempts. An issue going forward is how to ensure that interviewers comply with accurately recording all contact attempts.

The Adaptive Group prescribed seven high priority cases daily for interviewers, based on contact predictions from response propensity models. The ability of response propensity models to identify promising cases for daily contact, however, remains unclear after this pilot test because interviewers did not dutifully work priority cases. Future studies will examine different methods of focusing interviewers on cases believed to offer good contact opportunities.

Telephone-First Mode Sequence

We expected contacts attempted via telephone to reduce the number of expensive in-person contacts, but interviewers completed so few cases via telephone that we cannot make a definitive statement regarding the effectiveness of two different “telephone-first” strategies (See Table 2). Furthermore, the majority of phone call attempts in both groups did not reach anyone (Walejko et al. 2014). For the Census Bureau to judge telephone as a viable option for NRFU, we would need to identify better telephone number-address matches and examine better dialing methods than those employed in this pilot test.

Conclusion

The 2013 Census Test demonstrated the feasibility of several new methods for nonresponse follow-up in the 2020 Census. It set an agenda for subsequent tests that will help move toward decisions on how to conduct a 2020 Census that costs less per housing unit but maintains the quality of results obtained in 2010. For the broader survey community, this test introduced innovative data collection methods – particularly uses of administrative data in conjunction with adaptive contact strategies – that may have utility in other survey contexts. As survey organizations attempt to deal with declining respondent cooperation and rising costs, more refined applications of these approaches may yield positive results.