Multi-Mode and Method Experiment in a Study of Nurses

We conducted a 4-arm randomized experiment to compare paper versus Web survey completion of a questionnaire to study working conditions in outpatient cancer settings. We used a mailing list of nurses licensed to practice in North Carolina that contained names and last known mailing addresses, and varied both data collection mode (mail and Web surveys) and method of implementation (mail survey only, mail survey with a delayed Web option, Web survey with a delayed mail option, and a full Web/mail option). Because e-mail addresses or telephone numbers were not available, all respondent contacts used first-class mail. We observed modest differences in response rates, with the paper-only completion group with the highest response rate. Demographic characteristics did not vary across groups. Cost per completion differences were significant between the groups; the mail only option was the most efficient solution.

Background

Surveys of health care providers proliferate. While the research literature has examined mode of response from physicians, few studies can be found that focus on nurse populations. As nurses comprise the largest single proportion of health care providers, their attitudes and beliefs are important when studying quality of care. Reported response rates to mailed nursing surveys vary between 17 and 61 percent, and few studies have examined multi-mode approaches to nurse surveys (Aiken et al. 2010; Flynn, Carryer, and Budge 2005).

There is general agreement on effective methods for mail survey implementation (Dillman 1978; Dillman, Smyth, and Christian 2009). While there are several reasons why one would want to implement a multi-mode survey (i.e., lower costs, improving response rates, speed of data collection, etc.) debates continue about effective implementation. Dillman and colleagues stated in 2009, “Although mixed-mode surveys are popular, they are not to be used without careful consideration of the costs, error, and management consequences, both positive and negative.” (p. 330). Authors have demonstrated how multi-mode designs can backfire when respondents are given a choice of mode (Griffin, Fischer, and Morgan 2010). Web-based surveys are popular among populations with computer and Internet access, and given their work settings, nurses appear to have Web access.

We conducted a pilot study to examine whether mode and method of survey administration would result in differences in several measures of quality. We evaluated how the differences affected response rates, differential patterns of response among demographic groups, and cost efficiencies. Findings from this study will inform future efforts to create a data collection instrument and design to maximize response quality and control costs.

Methods

We purchased a sample frame from the State of North Carolina Board of Nursing that contained the entire registry of licensed nurses currently practicing in North Carolina. The frame provided complete name, preferred mailing address, and selected employment information. We identified nurses who reported they practiced in the specialty of oncology nursing, and who also reported their practice setting was outside the acute care hospital. Our final sample frame of eligible nurses was 1,339, and we included all of these nurses into our pilot study.

We randomized nurses to one of four experimental groups: a) mailed paper survey only (hereinafter termed paper only); b) mailed paper survey with a delayed invitation to complete a Web-based survey (paper to Web); c) mailed invitation to the Web survey, with a delayed invitation to complete a paper survey (Web to paper), and; d) a mailed paper survey with an immediate option to complete a Web-based survey (paper or Web option). Table 1 describes the data collection protocols implemented. Only nonresponders received follow-up communications. All study materials used university stationery and were mailed with first-class postage. The paper survey and Web-based surveys were both designed using best practices for each mode, but were also designed to minimize any mode or presentation differences. Both questionnaires had identical questions and question sequence. The study commenced on April 19, 2010, and data collection completed on June 3, 2010.

Results

The overall American Association for Public Opinion Research (AAPOR) Response Rate Calculation #2 (RR2)^[1] was 30.5 percent. Table 2 shows the RR2 across the four experimental arms. Differences in response rates across all four groups were not statistically significant (F=2.07, 3df, p=0.10). The only difference found was that the Web to paper group had a significantly lower response rate to the paper only group. For the three arms that allowed for both Web and paper completion, we generally observed a higher proportion of responses from the paper survey. We received 33 percent of our responses from the Web and 67 percent from the paper survey.

We then examined respondent demographics across experimental groups (Table 3). No significant differences were observed in general for age, gender, ethnicity, or educational attainment. There was a significant difference in non-whites across groups, with the highest proportion found in the Web/Paper option (18.0 percent). Comparing this study demographics to the most recent (2006) statistics published for the entire nurse population in North Carolina, we found our respondents were older, more were female, and more were white. This pattern may reflect changing demographics, or demographic differences between oncology nurses working in outpatient facilities and the general nurse population. This pattern may also be due to differential nonresponse patterns, which are well documented in general population surveys (Groves and Couper 1998; Johnson et al. 2002). We were encouraged that despite a concern for differential nonresponse, the experimental treatments did not have an impact on that pattern.

Cost is an important factor to consider in assessing different data collection protocols. In our study design, we explored whether adding a Web-based survey would increase cost efficiencies in the data collection. Web-based surveys have the potential to increase cost efficiencies because they involve fewer costs that are variable by the number of respondents. In our study, the only variable cost associated with the Web-based survey was the required paper mailed invitations (because we did not have e-mail addresses).

For this analysis, we considered the paper only mode as the baseline for the cost comparisons. Using cost data for the execution of this study, we computed a total cost for each treatment group. Because there were some differences in response rates by group, we also computed a cost per completed interview. Managing and coordinating the methodological experiment added to the overall cost of the study. Therefore we have used proportional increases from the baseline cost (there were no decreases) rather than the total cost to measure the impact of a Web option on data collection cost. Table 4 presents these results.

The groups that included the Web-based survey had higher costs than the paper-only (baseline) group, with a range of increased costs between 28.2 and 36.1 percent. However, given the difference in the response for each group, when examined at a cost per complete, we find there was a much larger difference, between 51.5 and 108.2 percent. In all cases, the paper only mode was the most cost-effective group. The Web/paper option group was the most costly, given the added cost and the lower overall response.

Conclusion

In concluding this study, we note the following caveats:

1. Our tight timeline and cut-off period for response compelled us to mail reminders sooner than is typically recommended. It is possible that the increased response observed after the second mailing is merely an artifact of the first mailing. These increased mailings were thus less efficient and more costly compared with a more traditional 6-week time period for data collection.
2. One may speculate that the web to paper group’s response rate may have increased if we allowed a longer period for follow up.
3. No e-mails were available for this sample – which reduced our ability to deploy the Web-based survey. These results should not guide the design of a study where e-mail addresses are available.
4. Our sample size was 1,339. Higher sample sizes may have an impact on the cost efficiency – the more mail surveys used, the lower the cost benefit of using them.

With those points in mind, in a study of outpatient-based oncology nurses where e-mail addresses were unavailable, we observed higher responses and lower costs when we used a paper survey. We did not observe tangible improvements in response with the addition of a costly web option.

Acknowledgments

This research was supported by a Pathway to Independence Award from the National Institute of Nursing Research, National Institute of Health, R00 NR10750 (Friese), and through the University of Michigan’s Cancer Center Support Grant (5 P30 CA46592).