In 2006, we began a multi-year effort to evaluate the possibility of transitioning from a regularly conducted telephone survey of general public satisfaction with Oregon transportation services to either a mail-only or Web-plus-mail mixed-mode design. Two primary objectives were: 1) to identify ways to counter the trend toward lower telephone survey response rates and 2) to evaluate the impact of alternative designs on data quality.
Three experiments were conducted for which unit response rates analyses have been reported elsewhere (Lesser and Newton 2007; Lesser et al. 2011; Lesser, Newton, and Yang 2011). In this study, we consider whether switching from telephone to mail or a combination of mail and Web will have a deleterious effect on item nonresponse, as previous research has suggested (e.g., de Leeuw, Hox, and Huisman 2003).
Thus, our purpose is to examine overall item nonresponse rates for each of the individual modes and mode combinations. Implications for replacing telephone by these alternatives are also discussed.
The experiments were conducted by the Oregon State University Survey Research Center (OSU-SRC) in three separate years: 2006, 2008, and 2010. The questionnaire was similar each year, asking random samples of Oregon households their opinions on the quality of various transportation modes within the state. The paper version of the questionnaire in all studies across all years consisted of 12 pages (including a cover page), and requested answers to 95 to 109 questions over the three years. The Web and telephone versions of the questionnaire for a specific year contained the same questions as the paper questionnaire.
We used a stratified sample design to facilitate comparisons among geographic regions of the state. Nonresponse adjustments were included in the analysis to account for the different response rates across regions as well as to bring the sample demographics more in line with the population demographics.
The 2006 and 2008 studies each included three treatments: 1) Telephone only, using RDD sampling and a total of 15 contacts 2) mail-only using address-based sampling (ABS) from the U.S. Postal Service Delivery Sequence File and 3) Web+mail, which used the ABS sample frame and began with a mail request to respond over the Web followed later by a paper request follow-up. Both treatments 2 and 3 used a pre-letter, a second contact requesting response by the designated mode, a follow-up postcard thank you/reminder, and a final contact that included a paper questionnaire irrespective of the original delivery mode.
The 2010 study also included three treatments, repeating the mail-only and Web+mail treatments from the previous two studies. However, we added a fifth contact to the Web+mail treatment in 2010. This contact included a paper questionnaire and the Web access information. The telephone treatment was eliminated and, in its place, we included a Web+mail choice treatment with four contacts. All contacts contained a paper questionnaire and Web access information.
Table 1 summarizes the final sample sizes and response rates for each of the treatments in each of the three years. The four contacts used in the mail-only mode provided the highest response rate in each of the three years. It can be seen that the response rate from the telephone survey achieved the lowest response rate. The Web+mail treatment also had a lower response rate than mail. Recall, in 2010 another contact was added to the Web+mail mode. The comparison in the table for 2010 reflects four mailings for the mail-only and the Web+mail choice mode, but five mailings for the Web+mail mode.
An important aspect of evaluating these results is the quality of answers obtained by the various modes. We compared item nonresponse rates across telephone, paper (using the mail) and Web modes (Table 2). For each year, the results from a Kruskal-Wallis test showed that the item nonresponse rates across the three modes were significantly different (p-value<0.001). As expected, the telephone treatments had the lowest average item nonresponse for both 2006 and 2008, which we attribute to the person-to-person connection and the ability of the interviewer to encourage response and to potentially answer a respondent’s questions. Although the Web mode did not force respondents to select a response category before they proceeded in answering the next question, we expected that the Web would also result in a low item nonresponse rate, which was supported by our findings. Although mail questionnaires produced the highest item nonresponse, the percent was only marginally higher than the other modes.
Table 3 presents the mean item nonresponse rates for each of the treatment groups by question type. Inasmuch as the Web is used only in combination with a mail alternative or follow up, we report the combined item non-response for the two modes as used together in those treatment groups. The item nonresponse rates for the mail-only and mixed-modes across years were quite similar. However, this likely results from the majority of respondents in the mixed-mode group having completed the questionnaire by paper. Item nonresponse rates were highest for the tabular and open-ended questions across modes.
Unit response rates, item nonresponse rates, and the representativeness of samples obtained from different modes are important considerations when evaluating the choice of survey modes. For telephone surveys, there are increasing challenges to obtaining a sample that mirrors the population. In addition, the use of a dual frame to include both cellular and landlines increase costs to develop a complete economical frame of telephone numbers in the U.S. Although the item nonresponse rate was lowest for the telephone mode, this paper also illustrated a lower unit response rate using the telephone mode as compared to mail.
The DSF file provides a nearly complete frame of delivery point addresses serviced by the U.S. Postal Service. The frame was useful to obtain a sample that closely mirrors the Oregon population. The mail-only mode provides the highest unit response rate but also the highest item nonresponse rate. However, the item nonresponse rate was only slightly higher than those respondents completing the questionnaire by Web. Imputation is a well established method to adjust for item nonresponse. Given the levels found in these studies, imputation could be easily adopted to adjust for the item nonresponse.
Delivering a survey by a mixed-mode design has an advantage since the mixed modes combine those individuals more likely to respond by Web with those responding by mail. The unit response rate improved over the three years examined in this paper but not as high as the mail only survey. The item nonresponse rate is lower than the paper delivery. Further studies should continue to examine unit response rates from the Web to determine if the benefits of offering both modes outweigh the costs involved.