In surveys of businesses (establishment surveys), part of the response process model is identifying the person within the business the survey request should be sent to (Bavdaz 2010; Dillman, Smyth, and Christian 2009; Snijkers et al. 2013; Willimack and Nichols 2010). Before survey invitations are sent, the U.S. Bureau of Labor Statistics (BLS) usually engages in address refinement, the process of confirming or updating the survey frame with the correct contact information for each sampled establishment. The process of address refinement is likely to increase the likelihood of the survey being delivered to the correct respondent. The Occupational Employment and Wage Statistics (OEWS) survey conducted by the BLS sends an advance letter (see example in Appendix) to sampled establishments one month prior to the survey invitation to collect the most up-to-date contact information. The advance letter informs the respondent of the upcoming survey request, underscores the legitimacy of the survey (De Leeuw et al. 2007), and asks for the appropriate contact information for someone within the organization. Because respondents in establishment surveys must have the authority and capacity to provide the desired data, collecting accurate contact information—such as email addresses—is integral to the survey process. Additionally, previous research has shown that identifying the most appropriate respondent allows for more tailored communications, increasing the motivation to respond, and thereby improving response rates (Dillman, Smyth, and Christian 2009). However, the most effective way to collect this information is still up for discussion.
To explore one way of improving the address refinement process, OEWS conducted an experiment to test whether placing a telephone call to sampled units after they received the advance letter resulted in more responses to the advance letter with up-to-date email addresses. Researchers conducting the experiment posited that a follow-up phone call would increase the establishment’s likelihood of providing contact information by heightening the saliency of the request (Groves, Singer, and Corning 2000) and providing another mode of response. In mid-April 2022, an advance letter was sent through the United States Postal Service to all sampled units. The letters were addressed to the frame contact information, which is generated when a business registers for a state unemployment account or has recently interacted with OEWS. After two weeks, units satisfying the following two conditions were randomized into two groups, controlling for size and industry: (1) units that did not respond to the advance letter and (2) units that had not provided OEWS with contact information during a previous interaction. These conditions mean units with contact information collected prior to the implementation of the experiment were excluded from the experiment.
The test group (n=242) received at least one telephone call to collect contact information, and the control group (n=235) was not recontacted. The telephone calls were placed to the frame contact numbers by an analyst working in a state data collection agency. The analyst attempted to contact all businesses in the test group at least once, leaving a voice message when possible. After each business in the test group had an attempted contact, time allowed for some businesses to receive an additional phone call. The experiment lasted two weeks, after which survey invitations were sent to the address on file. A timeline of the events is presented in Table 1.
The analyses examined both the proportion of advance letters that were returned and the proportion of email addresses collected. Assessing the proportion of email addresses collected along with the proportion of advance notices returned provides more insight into the effectiveness of the treatment and considers contact information collected over the phone. Although other contact information is acceptable, previous research suggests that sending a survey invitation to an email address is a cheap and effective method of pushing respondents to the web (Langeland 2019). The results are presented in Figures 1 and 2. The percent of test units returning an advance letter in the treatment group was 62.4%, which was significantly higher than the control group 10.2% (p<.0001). The percent of test units providing an email address in the test group was 73.6%, again, significantly higher than the control group (15.3%; p<.0001). The Cramér’s V statistic measuring the relationship between the intervention and prenote response was 0.5813, and the relationship between the intervention and providing an email address was 0.5854, signaling strong associations between the intervention and receiving updated contact information. These results show that a follow-up phone call to collect updated contact information after receipt of an advance letter results in a large increase in response for units that have not previously provided contact information.
There are several limitations of this research. First, the study was conducted in a single state which limits generalizability across wider populations. The study was also conducted in a production environment which limited the level of paradata detail the interviewers were able to collect. A finer level of call log data indicating whether an interviewer spoke to a respondent, left a message, or was unable to reach someone would shed light on the mechanisms at work behind the intervention.
To address these limitations, this experiment is currently being replicated in another state with different demographics and characteristics. More extensive paradata such as the number of contacts made to a unit and whether an interviewer spoke to a respondent will be collected.
Contact Information
Josh Langeland, Ph.D.
U.S. Bureau of Labor Statistics
langeland.joshua@bls.gov
