Comparing Item Nonresponse across Different Delivery Modes in General Population Surveys

Virginia M. Lesser Oregon State University

Lydia A. Newton Oregon State University

Daniel Yang Oregon State University

In 2006, we began a multi-year effort to evaluate the possibility of transitioning from a regularly conducted telephone survey of general public satisfaction with Oregon transportation services to either a mail-only or Web-plus-mail mixed-mode design. Two primary objectives were: 1) to identify ways to counter the trend toward lower telephone survey response rates and 2) to evaluate the impact of alternative designs on data quality.

Three experiments were conducted for which unit response rates analyses have been reported elsewhere (Lesser and Newton, 2007; Lesser et al., 2011a; Lesser et al., 2011b). In this study, we consider whether switching from telephone to mail or a combination of mail and Web will have a deleterious effect on item nonresponse, as previous research has suggested (e.g., de Leeuw et al., 2003).

Thus, our purpose is to examine overall item nonresponse rates for each of the individual modes and mode combinations. Implications for replacing telephone by these alternatives are also discussed.

Methods

The experiments were conducted by the Oregon State University Survey Research Center (OSU-SRC) in three separate years: 2006, 2008, and 2010. The questionnaire was similar each year, asking random samples of Oregon households their opinions on the quality of various transportation modes within the state. The paper version of the questionnaire in all studies across all years consisted of 12 pages (including a cover page), and requested answers to 95 to 109 questions over the three years. The Web and telephone versions of the questionnaire for a specific year contained the same questions as the paper questionnaire.

We used a stratified sample design to facilitate comparisons among geographic regions of the state. Nonresponse adjustments were included in the analysis to account for the different response rates across regions as well as to bring the sample demographics more in line with the population demographics.

The 2006 and 2008 studies each included three treatments: 1) Telephone only, using RDD sampling and a total of 15 contacts 2) mail-only using address-based sampling (ABS) from the U.S. Postal Service Delivery Sequence File and 3) Web+mail, which used the ABS sample frame and began with a mail request to respond over the Web followed later by a paper request follow-up. Both treatments 2 and 3 used a pre-letter, a second contact requesting response by the designated mode, a follow-up postcard thank you/reminder, and a final contact that included a paper questionnaire irrespective of the original delivery mode.

The 2010 study also included three treatments, repeating the mail-only and Web+mail treatments from the previous two studies. However, we added a fifth contact to the Web+mail treatment in 2010. This contact included a paper questionnaire and the Web access information. The telephone treatment was eliminated and, in its place, we included a Web+mail choice treatment with four contacts. All contacts contained a paper questionnaire and Web access information.

Results

Table 1 summarizes the final sample sizes and response rates for each of the treatments in each of the three years. The four contacts used in the mail-only mode provided the highest response rate in each of the three years. It can be seen that the response rate from the telephone survey achieved the lowest response rate. The Web+mail treatment also had a lower response rate than mail. Recall, in 2010 another contact was added to the Web+mail mode. The comparison in the table for 2010 reflects four mailings for the mail-only and the Web+mail choice mode, but five mailings for the Web+mail mode.

Table 1 Sample Sizes and Adjusted Unit Response Rates by Mode, as reported in (Lesser and Newton 2007; Lesser et al., 2011a; Lesser et al., 2011b).

Mode Final Sample Size
(% complete by Web)
Adjusted Response
Rate (AAPOR-RR3)
2006
 Telephone 1,013 30.0
 Mail 389 31.9
 Web+Mail 339 (28%) 22.4
2008
 Telephone 1,026 31.4
 Mail 327 35.5
 Web+Mail 469 (32%) 25.7
2010
 Mail 411 40.9
 Web+Mail 761 (28%) 37.8
 Web+Mail Choice 736 (17%) 36.4

An important aspect of evaluating these results is the quality of answers obtained by the various modes. We compared item nonresponse rates across telephone, paper (using the mail) and Web modes (Table 2). For each year, the results from a Kruskal-Wallis test showed that the item nonresponse rates across the three modes were significantly different (p-value<0.001). As expected, the telephone treatments had the lowest average item nonresponse for both 2006 and 2008, which we attribute to the person-to-person connection and the ability of the interviewer to encourage response and to potentially answer a respondent’s questions. Although the Web mode did not force respondents to select a response category before they proceeded in answering the next question, we expected that the Web would also result in a low item nonresponse rate, which was supported by our findings. Although mail questionnaires produced the highest item nonresponse, the percent was only marginally higher than the other modes.

Table 2 Item Nonresponse Rate by Mode and Year.

Year Telephone Paper Web
2006 0.79 2.3 0.6
2008 0.87 3.0 1.5
2010 4.3 1.8

Table 3 presents the mean item nonresponse rates for each of the treatment groups by question type. Inasmuch as the Web is used only in combination with a mail alternative or follow up, we report the combined item non-response for the two modes as used together in those treatment groups. The item nonresponse rates for the mail-only and mixed-modes across years were quite similar. However, this likely results from the majority of respondents in the mixed-mode group having completed the questionnaire by paper. Item nonresponse rates were highest for the tabular and open-ended questions across modes.

Table 3 Average Item Non-response Rates by Mode and Question Type Across Years.

Question
Type
Mode/Year
Telephone


Mail


Web+Mail


Web+Mail
Choice
2006 2008 2006 2008 2010 2006 2008 2010 2010
Likert 0.4 0.6 1.9 1.8 2.4 2.4 1.7 2.6 2.4
Filter 0.2 0.4 2.1 1.3 0.9 2.4 1.4 2.7 2.7
Tabular 0.4 0.2 5.7 4.1 4.9 5.2 3.2 5.6 5.2
Open 3.0 2.6 7.2 5.5 3.2 6.3 4.3 4.1 4.3
Demo 2.3 4.1 3.7 3.7 4.0 4.1 3.6 4.3 3.3
Overall 0.8 0.9 3.8 3.0 3.5 3.8 2.6 4.1 3.9

Conclusion

Unit response rates, item nonresponse rates, and the representativeness of samples obtained from different modes are important considerations when evaluating the choice of survey modes. For telephone surveys, there are increasing challenges to obtaining a sample that mirrors the population. In addition, the use of a dual frame to include both cellular and landlines increase costs to develop a complete economical frame of telephone numbers in the U.S. Although the item nonresponse rate was lowest for the telephone mode, this paper also illustrated a lower unit response rate using the telephone mode as compared to mail.

The DSF file provides a nearly complete frame of delivery point addresses serviced by the U.S. Postal Service. The frame was useful to obtain a sample that closely mirrors the Oregon population. The mail-only mode provides the highest unit response rate but also the highest item nonresponse rate. However, the item nonresponse rate was only slightly higher than those respondents completing the questionnaire by Web. Imputation is a well established method to adjust for item nonresponse. Given the levels found in these studies, imputation could be easily adopted to adjust for the item nonresponse.

Delivering a survey by a mixed-mode design has an advantage since the mixed modes combine those individuals more likely to respond by Web with those responding by mail. The unit response rate improved over the three years examined in this paper but not as high as the mail only survey. The item nonresponse rate is lower than the paper delivery. Further studies should continue to examine unit response rates from the Web to determine if the benefits of offering both modes outweigh the costs involved.

References

de Leeuw et al. 2003
de Leeuw, E.D., J. Hox and M. Huisman. 2003. Prevention and treatment of item nonresponse. Journal of Official Statistics 19(2): 153–176.
Lesser and Newton 2007
Lesser, V.M. and L. Newton. Comparison of delivery methods in a survey distributed by Internet, mail, and telephone. Proceedings of the International Statistics Institute Meetings (ISI 2007). Lisbon, Portugal.
Lesser et al. 2011a
Lesser, V.M., A. Olstad, D.Yang and L. Newton. Comparing item nonresponse and responses across modes in general population surveys. Paper presented at the 66th annual conference of the American Association for Public Opinion Research (AAPOR 2011a), Phoenix, AZ.
Lesser et al. 2011b
Lesser, V.M., L. Newton, and D.Yang. Evaluating methodologies to increase Internet responses in mixed-mode surveys. Proceedings of the International Statistics Institute Meetings (ISI 2011b). Dublin, Ireland.

Comments on this article

View all comments


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.