Introduction
The percentage of cellphone-only households has grown steadily in recent years. At the end of 2006, 15.8% of American homes had only wireless phones but, by the end of 2016, that percentage had risen to 50.8% (Blumberg and Luke 2007, 2017). The increase in cellphone-only households and the notable decline in response rates (Dutwin and Lavrakas 2016) have led researchers to adapt their methods to the realities of telephone surveys in the 21st century. The use of dual-frame telephone surveys that include landline and cellphone numbers, rather than just landline numbers, and prenotification or advance letters are methods that have been used to combat declining response rates.
With landline random digit dialing (RDD) and list-based studies, advance letters have been shown to be effective at increasing response and cooperation rates (de Leeuw et al. 2007; Richardson 2011) However, most studies assessing the impact of advance letters have been carried out using landlines; thus, the evidence in dual-frame samples is scarce and inconclusive to date. An experiment carried out in Norway showed that advance letters resulted in higher response rates and lower refusals in a dual-frame telephone sample combining landline and cellphone numbers. Specifically, response rates increased by 8% and refusals decreased by 27% (Kluch, Tortora, and Kluch 2015).
The aim of this study is to assess the impact of advance letters on cellphone responses in a statewide dual-frame survey examining perceptions and experiences with healthcare. Based on findings from landline RDD and list-based studies, we hypothesize that advance letters would increase cellphone response and cooperation rates (de Leeuw et al. 2007). We also hypothesize that advance letters would reduce the calling effort required to complete a survey (Hembroff et al. 2005). Finally, we examine whether advance letters impact the volume of missing data and the sample composition in order to determine whether any increase in response rates results in a reduction in data quality (as measured by item nonresponse) and/or an increase in nonresponse bias.
Methods
Study Design
Data for the experiment were collected from January 13 to February 17, 2017, as part of a statewide dual-frame survey of adults in a Midwestern state regarding their perceptions and experiences with healthcare. The random sample (provided by Marketing Systems Group) included cellphone numbers for which corresponding addresses were available.
To determine the impact of advance letters on cellphone response rates and data quality, the sample was randomly split into two groups. As shown in Figure 1, respondents in the treatment group (n = 600) were sent advance letters, while respondents in the control group (n = 600) did not receive prenotices.
Between three and eleven days prior to the first call attempts, staff at the Center for Social & Behavioral Research sent advance letters with first-class postage. Cellphone respondents’ letters were addressed to individuals registered as the phones’ users. The advance letter text is included in the supplemental materials.
The letters invited potential respondents to take part in the study, outlined its purpose and usefulness, and mentioned the importance of their participation. The letters also specified the institutions carrying out the study and provided contact information. Finally, they indicated the number from which they would receive the calls and the estimated dates of contact. All of the letters were in English and were not personalized. No incentives were offered.
Variables and Analysis
The effect of prenotification letters was analyzed on six different measures: response rates (RR3, (American Association for Public Opinion Research 2016); cooperation rates (COOP3, AAPOR 2016); contact rates (CON3, AAPOR 2016); number of call attempts needed to reach completions and refusals (range 1 to 11); nonsubstantive responses (“Refused” and “Don’t know”) and sample composition (i.e., demographic characteristics).
Chi-square tests were used to compare response, contact, and cooperation rates between the groups. Differences in call attempts and item nonresponse between the groups were tested using parametric (t-tests) and nonparametric tests (Mann-Whitney tests) and calculating effect sizes (Cohen’s d). In accordance with previous research (Koopman et al. 2013), only items for which nonresponse exceeded 5% were compared. Comparisons between the groups on selected demographics were performed using chi-square tests and t-tests. Data presented here are unweighted.
Findings
Response, Cooperation, and Contact Rates
The RR3 was 9.1 percentage points higher for the treatment group (31.3%) compared to the control group (22.2%), and this difference was statistically significant (X2[1] = 12.865, p < .01).
Next, we calculated contact and cooperation rates. We did so to determine whether this difference was due to the fact that the participants in the letter group were more reachable or, being equally reachable, more willing to complete the survey. Only cooperation rates were significantly different between the groups (X2[1] = 1.316, p = .339; and X2[1] = 22.356, p < .01, respectively).
The CON3 was comparable across groups (97.2% vs. 98.2%); however, the COOP3 was 10.3 points higher in the treatment group (87.8% vs. 77.5%) when compared to the control group. Hence, participants in the treatment group were equally reachable but were more willing to cooperate after being contacted than were participants in the control group. In summary, our results show the positive effect of advance letters on response and cooperation rates, even when only 58.9% of respondents who were sent a letter remembered receiving it and 82.2% of those who remembered receiving the letter reported reading it. This percentage is slightly lower than the rates obtained in landline samples (de Leeuw et al. 2007), suggesting that cellphone addresses may be less accurate or that cellphone-only households are less likely to open and/or read mail from unknown senders.
Number of Call Attempts Needed to Reach a Final Disposition
We also compared the number of call attempts needed to obtain completions and refusals. On average, the number of calls required to reach a completion was lower in the treatment group (M = 2.72, SD = 1.71), than in the control group (M = 2.88, SD = 1.87). Although this difference did not reach statistical significance (t[208] = 0.665, p = .507), a small effect size was found (Cohen's d = 0.09). Because the distribution of this variable was skewed, we supplemented this analysis with nonparametric tests that yielded similar results. The median number of call attempts was 2 on the treatment condition and 2.50 on the control condition (Mann-Whitney = 5136.000, z = -0.464, p = .643).
Among cases that resulted in refusals, the average number of calls required was higher in the treatment group (M = 4.88, SD = 2.94) than in the control group (M = 4.39, SD = 2.59). However, the difference in the means between the two groups was not statistically significant (t[37] = 0.543, p = .591), and the effect size was small (Cohen's d = 0.18). A Mann-Whitney test was performed taking into consideration the non-normal distribution of the variable. Similar to the findings obtained when comparing the means, the Mann-Whitney test indicated that those in the treatment group had a higher, although nonsignificant, number of call attempts (Mdn = 5.50) than those in the control group (Mdn = 4, Mann-Whitney = 199.500, z = -0.447, p = .662).
In summary, prenotification letters did not significantly reduce the number of calls to completion or final refusal, although a small effect was observed in both cases.
Nonsubstantive Responses
We explored the possibility that advance letters could have an effect on nonsubstantive responses. In general, these responses were very low (ranging from 0% and 3% for all of the questions except one). The only item for which "don't know" and "refusal" exceeded 5% was income (the question was worded as "What is your annual gross household income from all sources before taxes?"). We found no significant difference between the groups (X2[1] = 0.748, p = .475) as the percentage of these answers was similar in the treatment group (8.1%) and the control group (11.6%).
Respondents' Characteristics: Are There Differences Between the Groups?
The characteristics of the respondents who completed the survey in both groups are shown in Table 1. No statistically significant differences were found between the groups in four of the five sociodemographic variables (age, sex, education, and income level). However, there was a significant difference in race (X2[1] = 4.847, p = .05), with the treatment group having a higher percentage of whites (97.6% vs. 90.7%) than the control group. While not statistically different, the treatment group profile indicated a lower percentage of individuals with lower income levels (34.2% vs. 44.7%) and educational attainment (19.4% vs. 24.4%).
Discussion and Conclusions
Our results are consistent with previous studies using landline RDD and list-based samples showing the positive effects of advance letters to reduce nonresponse (de Leeuw et al. 2007). The response and cooperation rates were 9.1 and 10.3 percentage points higher for the treatment group compared to the control group. In accordance with the results obtained in most landline samples (Richardson 2011) (see (Von der Lippe, Schmich, and Lange 2011) for an exception), we found no effect of advance letters on contact rates indicating that respondents in the treatment group were as likely to pick up the phone as those in the control group.
Despite this increase in participation, the results do not allow us to rule out the possibility that advance letters may have introduced sample bias in dual-frame telephone surveys. Although only racial composition yielded significant differences between the groups with a greater percentage of white participants in the treatment group compared to the control group, our results suggest a greater presence of highly educated and high-income respondents in the treatment group. Although these differences were not statistically significant, they are consistent with the trend observed in previous research (Link 2005) in which advance letters helped to recruit respondents that are already easier to interview such as older, better educated, and less racially diverse individuals. However, further research is needed to explore the reliability of this possible bias.
In addition to increasing response and cooperation rates, our results provide evidence that sending advance letters in dual-frame surveys can help to reduce the calling effort needed to complete the survey. However, this effect was small and the consideration of time and money necessary to send the letters is a factor to consider when adopting this approach, especially taking into account that advance letters increased the number of call attempts required to reach final refusals by roughly 0.5 attempts which also increases overall survey costs.
Finally, we did not find an effect of the advance letters on nonsubstantive responses. This finding corroborates previous evidence from mail surveys (Koopman et al. 2013) suggesting that the increase in participation does not come at the cost of a reduction in data quality. Future studies may be able to expand on these findings analyzing other indicators of data quality in addition to item nonresponse (e.g., acquiescence, non-differentiation).
In conclusion, our findings build on previous research showing the positive effects that advance letters can have on response and cooperation rates in dual-frame telephone surveys. This result is useful for survey practitioners given the widespread use of phone surveys that combine cellphones and landlines and the scarcity of studies conducted with this sample frame. Although advance letters do not seem to affect nonsubstantive responses, their effect on sample composition should continue to be studied in future work.
Acknowledgement
The authors would like to thank Suzanne Bentler and Peter Damiano at the University of Iowa Public Policy Center who facilitated the project by allowing us to conduct this experiment while implementing the study.