A significant limitation of most data quality comparisons between Web and mail survey responses is that the individuals who choose to respond to one survey mode have different characteristics than those who select the alternative mode. As suggested by other papers in this issue of Survey Practice, such differences in, for example, Internet access, education, and income, may contribute to mode differences in item nonresponse. This makes it difficult to isolate how mode itself affects this aspect of survey quality.
The surveys of undergraduate students analyzed herein differ from general public surveys in this regard, inasmuch as all students in this fairly homogenous (for education and age) population have Internet access and are accessible by both postal mail and e-mail. Since college students are relatively highly Internet-literate and course work requires nearly daily use of the Internet, it is possible to randomly assign students to Web-only and mail-only treatment groups. In this paper, we determine if mail and Web item nonresponse rates are similar when randomly assigning individuals to a response mode. Additionally, we examine patterns of item nonresponse for different types of question formats.
Data and Methods
We examined results from two surveys of randomly selected undergraduate students at Washington State University (WSU). The surveys were conducted in the spring and fall of 2009, respectively. The first study focused on students’ general satisfaction with WSU and their educational experiences. The second study primarily asked about the effects of the economic downturn and university budget cuts. The spring questionnaire contained 36 numbered questions requesting as many as 100 responses (depending upon skips), and the fall questionnaire contained 33 numbered questions requesting up to 74 responses.
The paper and Web questionnaires were created using a unified mode design to standardize the visual appearance and reduce the potential for mode differences (Dillman, Smyth, and Christian 2009). Both studies used multiple experimental treatment groups to compare the effects of mode and implementation strategies on response rates (the experimental designs were detailed in Millar and Dillman 2011). Table 1 displays the response rates for each treatment, as well as the number of responses used in the following analyses from each treatment.
We limited the following analyses to responses resulting from the random assignment to a response mode. Those respondents who were offered a choice of response mode were excluded. We also excluded Web break-off respondents (there were no comparable mail break-offs). We did not include responses such as “don’t know” and “not sure” in the counts of item nonresponse. We calculated the percent of missing responses for each respondent, and then calculated an average, composite rate of missing responses by mode of response for each study. We used z tests for differences in proportions to determine if there are any statistically significant differences in the rates of item nonresponse across modes.
Item Nonresponse Rates by Mode
Table 2 displays the average rate of item nonresponse for each study, by mode. The table shows that although the mail item nonresponse rates in these studies were slightly higher than those of the Web, there were no statistically significant differences in the average rate of item nonresponse between the two modes. This suggests that mode differences are negligible when individuals are randomly assigned to a response mode.
Although the composite rates in Table 2 indicate there was no overall mode difference, several individual items within each survey exhibited significant mode differences in item nonresponse rates. In the first study, 19 percent of the items had significant mode differences in item nonresponse rates. Forty-nine percent of the questionnaire items in the second study showed statistically significant mode differences in item nonresponse rates, which is noticeably higher than the percentage of items in the first study. This high percentage is especially surprising because there was no statistically significant mode difference in the composite rates of item nonresponse for either study. This encouraged us to explore potential sources of mode differences in item nonresponse.
Effect of Question Format
We first examined whether any particular question formats were common among those items exhibiting significant mode differences in item nonresponse. In both studies, multi-item, battery list questions were a frequent source of mode differences in item nonresponse rates. In the first study, 37 percent of the items showing significant mode differences in item nonresponse rates were part of multi-item, battery list questions. In the second study, this percentage was 57 percent. In both surveys, mail respondents were more likely than Web respondents to skip items within a multi-item battery.
Another source of mode differences in the second survey was four qualitative, open-ended questions. In contrast to the multi-item trend, these questions produced higher item nonresponse rates in Web than in mail. This suggests there is no simple trend in the effects of question format. Mode effects on item nonresponse are more complicated than indicated by common perception, which is that mail consistently produces higher item nonresponse rates than Web.
We next calculated the average respondent item nonresponse rates for three question types across modes: multi-item lists, open-ended questions, and other question types (such as yes/no, ordinal scales, and nominal scales), as well as for questions of any format which were branched to in a skip pattern. These rates for each study are summarized in Table 3. The table indicates that, for the first study, the average rates of item nonresponse for each question type are not statistically different across modes. However, the second study exhibited significant mode differences in item nonresponse for open-ended questions as well as other question types. Surprisingly, the mode difference for multi-item questions was not statistically significant. Thus, although certain question types seemed to be associated with significant mode differences in item nonresponse rates, the composite rates for each question type do not consistently vary across modes in either study.
We also examined whether the composite rates of item nonresponse for open-ended, multi-item, and other question formats (shown in the first column of data in Table 3) are significantly different from each other, regardless of mode. These results indicate that open-ended questions have significantly higher rates of item nonresponse than both multi-item and other question formats for both studies. Additionally, in the first study, multi-item questions have a significantly higher rate of item nonresponse than other question formats.
Effect of Survey Topic
We also considered whether the change in survey topic could have affected mode differences in item nonresponse. The topic of economic troubles and budget cuts (which was more prevalent in study two) seemed to matter. Forty-nine percent of the items exhibiting significant mode differences in responses in the second study were items related to these topics. Since this topic appeared less in the first study, only five percent of the significant mode differences in this study were related to it. This could partially explain why we observed a higher percentage of individual items showing mode differences in item nonresponse in the second study. Mail respondents were less likely than Web respondents to answer questions related to economics/budget cuts. Perhaps these topics appeared more sensitive and respondents were more comfortable responding to such items via Web.
This paper shows that in studies in which sampled individuals are randomly assigned to a response mode, the composite rate of item nonresponse is the same for Web and mail. However, our deeper analysis suggested the story is more complex. We found that despite the overall similarity across modes, a good number of individual items, particularly in the second survey, exhibited significant mode differences in item nonresponse rates. Our data demonstrate there may be other factors which play a role in creating mode differences in item nonresponse rates. In particular, this analysis showed that question format and survey topic may impact item nonresponse depending upon the mode of study.
Further cognitive interview work could help illuminate why these factors may differentially affect response in mail and Web. Without such insights, we can only speculate as to why survey topic and question format may matter. For example, perhaps multi-item battery list items seem more burdensome to respondents completing paper questionnaires as opposed to online questionnaires. Also, responding over the Internet may be more comfortable for students when dealing with topics that can be more personally sensitive, such as financial well-being.
This paper suggests that when we use random assignment to response modes, there is not a great deal of difference between mail and Web in item nonresponse rates. However, this does not mean response mode is inconsequential. Other factors appear to create such differences. Therefore, surveyors should continue to consider how mixing modes may affect data quality, and invest in future research to disentangle the sources of mode differences.
This research was conducted in the Washington State University Social and Economic Sciences Research Center (SESRC) with support from the USDA-National Agricultural Statistics Service and the National Science Foundation- National Center for Science and Engineering Statistics under cooperative agreement no. 43-3AEU-5-80039.