Obtaining Responses by Mail or Web: Response Rates and Data Consequences

This purpose of this study was to explore the willingness of the people who have obtained information from Cooperative Extension, a quasi-general public population, to respond to a customer satisfaction survey via the Internet when receiving the request by postal mail. A list of Extension clients was sampled and randomly assigned to three experimental treatments – the traditional mail-only treatment, a mail/Web choice treatment where the client received both a paper version and the url address to the survey, and a Web-preference treatment. For the latter, the client received only the url in the initial request and then a paper survey and the url were provided in the follow up request.

Results show that Web-preference respondents (who could respond by Web initially and then offered a paper survey later) had a lower response rate (52.6%) than did mail/Web choice and mail-only respondents (59.2% and 64.5%, respectively).

The results also show that respondents in the Web-preference treatment differed by mode. Those who responded by Web were more likely to have used Extension’s Solutions for Your Life Web^[1] site during the past year (36.2%) than did those who responded by mail (7.8%). Web respondents also had an average of 4 more contacts with Extension during the year than did mail respondents (8.6 and 4.1 contacts, respectively).

A larger proportion of Web respondents had at least some college education (85.3%) and were younger (mean age of 53.9 years) than mail respondents (67.7% and 59.8 years, respectively). Perceptions of service quality by Extension’s clients varied somewhat by mode of response with more Web respondents being “very satisfied” (76.2%) than were mail respondents (62.1%). The results show that survey mode resulted in differences in the demographic profile of respondents and in the substantive results.

When Web and mail respondents in the Web preference treatment were combined, these respondents were not significantly different with regard to demographic attributes or measures of satisfaction from those in the standard mail-only and mail/Web choice groups.

The finding of differences between those who responded via the Web and those responding by mail in the Web preference treatment should warn survey professionals to avoid relying on the Web alone to conduct surveys of the general public. The similarity of the results, however, between the mail only, mail/Web choice and Web preference treatments suggests that mixed-mode surveys can be considered as an option for collecting data from address-based and client lists. On the other hand, response rates were reduced for the two treatments involving the Internet, more so when clients were pushed to use the Web (as in the case of the Web preference treatment). Given this, researchers might need to increase the initial sample size for mixed-mode surveys to avoid problems of insufficient completes for the data analysis.