Methods for Improving Response Rates in Two-Phase Mail Surveys

Introduction

As random digit dial (RDD) telephone surveys in the United States face declining coverage and response rates, an alternative approach using address-based sampling (ABS) with a frame derived from U.S. Postal Service data files to improve household coverage (Iannacchione 2011) is emerging. Vendors of the ABS lists can link the addresses to telephone numbers for some addresses, opening up the possibility of using telephone as a mode of data collection. Several empirical studies have examined both the feasibility and the viability of converting RDD telephone surveys to ABS-based self-administered surveys, e.g., Link et al. 2008.

In our study we screen to identify a subgroup using a method we refer to as two-phase mail survey, using an ABS-based sample of the resident population. Sampled households are mailed a short screening questionnaire to enumerate the members of the household who are in the target subgroup. Eligible members from the responding households are sampled and mailed a second-phase questionnaire about their activities. Studies that have used this approach include surveys of children (Brick, Williams, and Montaquila 2011), veterans (Han et al. 2010), and anglers (Mathiowetz et al. 2010).

Because the two-phase mail approach is relatively new, standard mail procedures such as those described in Dillman, Smyth, and Christian (2009) that involve up to five contacts with the sampled household may not be appropriate due to its effect in the second phase. Brick, Williams, and Montaquila (2011) found that follow up by mail had considerably higher response rates than follow up by telephone, but this does not mean telephones should play no role in two-phase surveys.

In this article we examine the results of experiments designed to explore the effectiveness of different procedures for initial contact and follow up in two-phase mail surveys.

Study Design

The National Marine Fisheries Service of the National Oceanic and Atmospheric Administration (NOAA Fisheries) conducts several data collection efforts to estimate saltwater fishing participation, effort, and catch in the United States. In 2009, NOAA began testing the use of an ABS frame to survey saltwater anglers and found it to be a feasible replacement for the current RDD survey (Mathiowetz et al. 2010). The study reported here used a stratified, two-phase sample design to identify and interview anglers. In the first phase, a stratified (by geography) random sample of households in two states was selected from the ABS frame. Sampled households were mailed a screener questionnaire designed to identify adults participating in saltwater fishing the previous year. Once selected, the sample addresses were processed to identify addresses for which a telephone number could be obtained: 6,518 of the 13,750 addresses (47 percent) were linked to a landline telephone number by the ABS sample vendor.

The sample unit for the second phase of the ABS was an adult angler. All adults (18 years or older) identified as saltwater anglers in the first phase were included in the second-phase sample. Westat was responsible for the survey’s sampling, material preparation, data collection, and processing.

Sampled households were randomly assigned to treatment conditions to assess differences in response propensity. Treatments were defined by the type of mail used to deliver survey instruments (first class or priority mail) and by the use of a postcard or telephone prompt (an outbound interactive voice response (IVR) message) as a thank you/reminder. The effect of the treatments was of special interest since they might affect both cost and response over the two phases.

The initial mailing for the first phase of the ABS survey occurred in early November 2010. It included an introductory letter stating the purpose, a screener questionnaire, a postage-paid return envelope, and a $1 cash incentive. One week following the initial mailing, a thank you/reminder postcard was sent to all mail treatment cases, and an IVR reminder to complete and return questionnaires was initiated for telephone treatment cases. A contact included leaving a message on an answering machine. Up to five call attempts were made. A second mailing, which included a second questionnaire and a refusal conversion letter but no incentive, was sent three weeks after the initial mailing for households that had not returned a completed screener.

The survey packet for those eligible for the second phase included an introductory letter stating the purpose, the extended survey questionnaire, a postage-paid return envelope, and a $1 cash incentive. Packets mailed in early January were sent to individual anglers. All anglers within a household received the same treatment. The questionnaires asked about fishing activity that occurred in November and December of 2010. One week following the initial mailing, a thank you/reminder postcard was sent to designated sampled addresses, or an IVR call was made following the same protocols as the first phase. Three weeks after the initial mailing, a second mailing was sent for all sampled individuals who had not yet responded. A third mailing was sent for sampled anglers in two treatment groups if they had not responded within two weeks of the first follow-up. The survey packet for this group included the same extended survey questionnaire, a post-paid return envelope, and a revised letter^[1].

Screener response rates were computed using the base weights (inverse of the probability of sampling the address) and method RR3 described by AAPOR (2011). The second phase response rate was computed assuming all the adults sampled from the first phase were eligible. Thus, RR1 method (AAPOR RR1) was used.

Findings from Experiments

Table 1 shows the response rates for the first-phase treatments for those addresses matched to a landline telephone number and for those without a matching telephone number. The sample sizes in the last column of the table are for the residential addresses assigned to the treatment at the first phase irrespective of the second-phase mailing assignment.

Among households for which a phone number could be matched to the address, the 1^st-pc-1^st treatment group response rate is lower (for α=0.05) than the rate for each of the other treatments, except the 3.6 percentage point difference with the 1^st-ivr-1^st treatment^[2]. That rate is not significantly different from any of the other treatments that used priority mail.

Among households whose phone number was not matched to an address, the SP-pc-1^st treatment is significantly higher than either the 1^st-pc-1^st or the 1^st-pc-SP treatment. The significant difference between the SP-pc-1^st treatment and the 1^st-pc-SP treatment among households without matching phone numbers is surprising considering the non-significant difference for those households with matching phone numbers. The outcome could be due to differences in the characteristics of the two types of households. However, we suspect the effect might not be replicated in other studies since we have no theoretical rationale to support it.

We also combined the treatment groups that used priority mail or an IVR prompt (within the match-status subsets) and compared the response rates of this and the 1^st-pc-1^st treatment groups. For both match-status subsets, the difference was about 5 percentage points and statistically significant. The 1^st-pc-1^st treatment involves repeated use of first class mail methods while the others used either priority mail or telephone prompts. The poorer performance of 1^st-pc-1^st is consistent with the discussion in Dillman, Smyth, and Christian (2009), who suggest varying the contact methods to increase participation in mail surveys.

We explored potential differences in substantive outcomes resulting from the different treatments. The key outcome of the screener was the identification of households with one or more adults who participated in saltwater fishing in the previous year. We found no evidence of differential nonresponse bias for this key statistic.

The cost of the mailing in the first phase of the study accounted for a large proportion of the fixed costs of the survey. In November 2010, first class mail cost for a large envelope was about $1.25 while priority mail cost about four times as much (roughly $5). The cost of the postcard was $0.31 while the IVR prompt cost about $0.20. The IVR cost goes down with volume since it includes set up cost. Thus, treatments that use priority mail are considerably more expensive than those using first class, especially those that use priority for the first mailing.

These results suggest that the use of the 1^st-ivr-1^st is a good practical option for increasing response rates at the first phase if a phone number is available. For households without phone numbers, the preferred contact method depends on the importance of higher response rates. Because there is no evidence of nonresponse bias associated with any of the treatment groups, the higher cost of priority mail, especially for the screener, may not be justified.

Second-Phase Rates

Table 2 presents response rates for second-phase data collection for adults who were identified as having fished in the last year. The right column shows that the sample sizes are substantially smaller for this phase because of screener nonresponse and non-eligibility; only about 30 percent of sampled households had an eligible adult angler. For this phase, the sample size is the number of anglers rather than households.

Among anglers in phone-matched addresses, the 1^st-pc-1^st treatment results in lower response rates than any other treatments. None of the contrasts are statistically significant for anglers in non-phone-matched addresses.

We checked for differences in the key outcome measure (recreational saltwater fishing by respondent) for the extended survey to see if the treatments might induce differential nonresponse bias. No statistically significant differences were observed.

Discussion

The findings from these experiments suggest that the general principles in Dillman, Smyth, and Christian (2009) for conducting mail surveys apply to two-phase survey designs, with some modifications. The principle regarding the use of compatible yet variable contact methods to increase response rates in mail surveys seems at least as relevant in two-phase as in one-phase designs. Repeated use of first class mail for all attempts performed poorly. Varying the contact method to include priority mail and telephone prompts performed well.

Although not discussed here, we also found that mailing a third copy of the questionnaire increased response rates. We found increases in the second phase response rate of 10 percentage points using a third mailing and a significantly higher effect of the third mailing when using priority mail. These findings are similar to those reported by Brick, Williams, and Montaquila (2011) who used Federal Express.

Our findings are of course limited to a particular implementation in two states in the U.S. and for surveys concerning recreational saltwater fishing. Much remains to be studied for two-phase mail designs. Based on the general positive outcomes here, however, we believe continued research in this area is warranted.