Mail surveys remain an effective and popular mode of data collection, given the relatively low and decreasing response rates of Web surveys (Sheehan 2001) and the unreliability of e-mail addresses as compared to physical addresses (Crawford et al. 2002; Shannon and Bradshaw 2002). This article addresses the implications of using return receipts/address updates to reduce the costs of self-administered mail surveys at Penn State Harrisburg (PSH). We build upon Dillman’s (1991, 2000) widely-used approach of multiple contact surveys to demonstrate the cost-effectiveness and improvement in completion rates using return receipts.
We found that having valid addresses has the potential of reducing costs associated with a mail survey (e.g. copy costs, postage, etc.). However, in determining the efficacy of the return receipt method a priori, two features of the survey must be considered: (1) the relative cost between the return receipt and the additional mailings, and (2) the ratio between usable (deliverable) and non-usable (un-deliverable) addresses (see appendix for a model that demonstrates the utility of return receipts). If the ratio of bad-to-good addresses is higher than the cost of the return receipt relative to the cost of additional mailings, then return receipts are cost-effective.
Illustrative Study: Mail Survey of PSH Alumni
We designed a self-administered, anonymous mail survey to determine factors affecting graduate students’ abilities to complete a degree program. A mail survey was utilized for this project instead of a web-based survey, in part because the University did not have current e-mail addresses of students who were no longer enrolled. The study population was the 6,430 students enrolled between 1995 and 2005, including both (a) 3,510 alumni and (b) 2,920 students who did not receive degrees and were not enrolled in Spring 2006 when the survey was distributed.
Many of the questions in the survey used four-point Likert scales and related to student issues such as finances, classes, faculty, advising, and personal experiences. Five mailings were implemented, including: (a) an introductory postcard, (b) a survey, (c) a reminder postcard, (d) a postcard to those who did not complete the degree program, a group with a low (4%) response rate compared to those who graduated (22%), and (e) a duplicate survey. Because a major concern was the validity of addresses to which the postcards and surveys would be sent, an introductory postcard was used to determine which addresses were valid and which were revised. Therefore, the postcard was mailed return receipt (First-Class).
In all, 377 addresses, which represent 6% of the pool of potential respondents, were updated. In addition, 1,137 undeliverable addresses were removed from the database, resulting in a database of 5,293 valid addresses (instead of the original 6,430). Thus, with 1,420 surveys returned, the true completion rate was 26.8% of 5,293, instead of the 22% which would have been assumed (1420/6430) if the undeliverable addresses had been retained for all steps of the survey.
Because there is limited information on the costs/benefits associated with sending an introductory postcard to collect undeliverable addresses, an important outcome of this project was to assess the financial implications of this approach. There were 1,514 returned postcards, of which 377 included updated addresses. As shown in Table 1, using the corrected addresses collected from the introductory postcard provided a survey cost savings of $516.40 because mailings subsequent to the introductory card were sent only to valid addresses. This cost savings represents a substantial 5.4% of the full cost of the traditional mail survey before address correction. These figures also understate the benefits of return receipt, because the 377 updated addresses do not lead to cost savings in this analysis, but do enhance completion rates. Since these addresses would not be updated in a traditional mail survey, the “wasted” mailings constitute $581.89, meaning that the true cost savings of using return receipt in this protocol are $1,108.29. The $1,100 represents a cost savings of 11.6% of the full cost of a traditional mail survey pre-correction.
Self-administered mail surveys of larger populations may see a savings in cost by using the return receipt method with the initial postcard mailing. Mail surveys (such as the one described above) which have a large number of “bad” addresses and several additional mailings may benefit from using the return receipt method (see appendix for specifics).
The current study demonstrates that, when conducting a mail survey with multiple, expensive contacts, an additional element to consider in the design methodology is sending an introductory return receipt postcard for the purpose of identifying invalid addresses. Return receipts are also an efficient means of improving completion rates in mail surveys. By correcting undeliverable addresses, researchers can boost the number of completes rather inexpensively, at least as compared to costly techniques used in RDD telephone interviewing (e.g. callbacks, refusal conversions, monetary incentives).
As the data from the graduate student survey research project at PSH demonstrated, the return receipt postcard strategy provided valuable information regarding relocation patterns of graduate students and changes in addresses. In turn, this effort yielded a substantial savings in overall costs associated with the present study, as well as providing the Alumni Office with updated addresses, thus producing a more valid population list. However, using return receipt may not be cost effective in all cases. Rather, when follow-up mailings are expensive, the subject population is mobile, or the contact list is unreliable, using return receipts can produce cost-savings for researchers.
This research was supported in part by a grant from the National Science Foundation ADVANCE Leadership Award (MRW).
The formulas below demonstrate the utility of return using a simple operations research model.
Let G = the number of “good” (deliverable) addresses
Let B = the number of “bad” (undeliverable) addresses
Let C = the cost of sending out mail beyond the introductory postcard
Let R = the cost of the return receipt
The return receipt will only be cost-effective if:
The left-hand side is the cost of sending out additional mail to both “good” and “bad” addresses, but not paying for return receipts. The right-hand side is the cost of only sending out additional mail to “good” addresses, but paying for return receipts for all addresses. Rearranging terms in equation , we get:
In other words, by equation , return receipts are only cost effective if the ratio of bad-to-good addresses exceeds the relative cost between the return receipt and the additional mailings. As C ® R, B/G must go to infinity for the inequality to hold, meaning that there is virtually no utility to using return receipts if their cost is about the same as the additional mailings.
Thus, this simple model demonstrates that it is only useful from a cost perspective to use return receipts when the mail survey has two properties: (1) there is a high ratio of bad-to-good addresses; and (2) the cost of the additional mailings is substantial relative to the cost of the return receipt.
In our example, B/G=0.215 and R/(C – R)=0.138, so the return receipt is effective since the first number is larger than the second. In other words, the ratio of bad-to-good addresses (B/G) is higher than the cost of the return receipt relative to the cost of additional mailings (R/(C – R)). The bad-to-good ratio is particularly large (perhaps because the population is mobile) and the cost of the additional four mailings is high (although in line with Dillman’s model of survey implementation). Hence, mail surveys with these features may benefit from using the return receipt method.