Over the years, a number of methods have been investigated to increase response rates to mail surveys. These methods include pre-notification letters, incentives, reminders, survey form design, endorsement letters, sending additional surveys, shortening survey length, and delivery method. These methods have been demonstrated to have a positive impact on the level of participation in surveys with the general population (Dillman 2000; Fowler, F.J., Jr. 1993) as well as with many specialized populations such as physicians (Kellerman and Herold 2001)). Despite these efforts, response rates for mail questionnaires, and in particular mail questionnaires of physicians have been declining over the past decade (Cull et al. 2005). As a result, researchers must continue to explore possible methods for increasing survey response.
Some research with physician populations has shown that the use of stamps can be effective compared to metered or business reply envelopes for return mail (Armstrong and Lusk 1987; Fox, Crask, and Kim 1988; Kellerman and Herold 2001; Streiff, Dundes, and Spivak 2001; Urban, Anderson, and Tseng 1993). However, there is no current research that examines the use of stamps on outgoing mail. Seven studies conducted in the 1970′s found no difference in response rates for mailings that used a stamped outgoing envelope versus a metered outgoing envelope (Edwards et al. 2007; Gullen and Garrison 1973). Yet, it is unclear what the effect would be today. It is generally recommended to use first-class mail over bulk-mail to improve response (Dillman 2000), but it is unknown whether the use of first-class stamps compared to first-class metered postage will have a similar effect on response rates. Armstrong and Lusk (1987) posit that stamps might be expected to increase response rates compared to metered mail because respondents perceive stamps as more personal. While this did not prove to be true several decades ago, it may have an effect now. In a letter to The Lancet in 2000, one physician noted that he received approximately 122 kg of medical junk mail in one year (Montauk 2000). The use of stamps may distinguish a survey request from what is perceived as unwanted medical junk mail. In addition to using stamps on the outgoing envelopes, there may be other ways to personalize the appearance of the envelope encouraging recipients to open and respond to the mail. In addition, most research comparing the results of the stamped return envelopes versus business reply mail were conducted on either the first mailing or the study overall. What impact on response rates, if any, do stamped envelopes have on nonresponse follow-up mailings?
In an effort to add to this body of literature, this paper discusses two experiments focused on improving physician survey cooperation by altering the appearance of the outgoing and return survey envelopes. The experiments were conducted within a national survey of board certified physicians by researchers at RTI International as part of the “America’s Best Hospitals” project conducted for U.S. News & World Report.
In the first survey mailing, the authors tested the effect of postage stamps compared to metered mail on the outgoing envelope. The authors also tested the effect of including a small sticker on the outgoing envelope as well. The sticker was intended to have the same personalizing effect of the stamp while also identifying the purpose of the envelope. The two methods were combined into a 2 by 2 factorial design. The stickers indicated that the material in the letter was from the “America’s Best Hospitals” project and highlighted the sampled physician’s medical specialty. The authors hypothesized that both the stamps and the stickers would be viewed as more personal by respondents. Therefore more respondents would open and ultimately respond to the survey.
In order to test the effect of stamped return envelopes on the follow-up mailings, an additional experiment was conducted on the third, fourth, and fifth nonresponse follow-up mailings to the physicians. The authors tested the effect of business reply mail compared to stamped return envelopes, and the use of two different types of stamps on the return envelopes: American flag stamps compared to breast cancer research stamps. Breast cancer research stamps were chosen because they donate six cents per stamp to breast cancer research and this might appeal to physicians. Previous research by Choi et al. (1990) showed that commemorative stamps on return envelopes increased response compared to regular stamps or business reply mail.
Methods
Since 1990, U.S. News & World Report has assessed the quality of hospitals in the United States annually in the form of lists collectively titled “America’s Best Hospitals.” Each year, the magazine identifies hospitals of exceptional quality from over 5,000 hospitals in the United States across a variety of medical specialties. Hospitals are assigned a composite score and ranked based on data from multiple sources. One of the primary sources of data is a survey of board-certified physicians asking them to nominate the “best hospitals” in their medical specialty.
The sample for the 2007 physician survey consisted of 3,400 board-certified physicians selected from the American Medical Association (AMA) Physician Masterfile. Stratifying by census region (Midwest, Northeast, South, West) and medical specialty, we selected a random probability sample of 200 physicians (50 from each region) from each of the 17 specialty areas. The 17 medical specialties represented in the sample included the following: Cancer; Digestive Disorders; Ear, Nose, and Throat; Geriatrics; Gynecology; Cardiology; Hormonal Disorders; Kidney Disease; Neurology and Neurosurgery; Ophthalmology; Orthopedics; Pediatrics; Psychiatry; Rehabilitation; Respiratory Disorders; Rheumatology; and Urology.
The physician survey mailings were conducted in stages during several weeks. The initial mailing was sent via United States Postal Service (USPS) first-class metered mail. It included a cover-letter, survey, business-reply envelope and a $2 bill as an incentive. Three weeks after the initial survey mailing, a thank-you/reminder letter was sent to the sampled physicians with a replacement survey and business reply envelope. Two weeks following the reminders, we sent a USPS Priority mailing to nonresponders, along with another copy of the questionnaire, a new cover letter, and a business reply envelope. Two weeks after the priority survey was sent, a fourth survey mailing was sent overnight via Federal Express to the remaining nonresponders; the packet included the questionnaire, a cover letter, and a business reply envelope. A final mailing was sent via USPS first-class mail approximately 4 weeks later. This mailing included the questionnaire and a personalized letter with a handwritten note and signature. (See Table 1 for a schedule of the physician survey mailing.)
Experimental Conditions
To evaluate methods for improving the overall level of participation in the survey, two experiments were conducted to see if subtle changes in the appearance of the outgoing and return envelopes would have any impact on physician participation in the survey. The experiments are described below.
Experiment 1: Appearance of outgoing envelope
In the first survey mailing, we tested the effect of using postage stamps on the outgoing envelopes compared to traditional metered mail. The authors also tested the use of stickers on the outgoing envelope. The stickers were bright blue and indicated that the material in the letter was from the “America’s Best Hospitals” project and highlighted the sampled physician’s medical specialty. The two alterations were combined into a 2 by 2 factorial design, where physicians in each region/specialty combination were randomly assigned to one of four experimental conditions such that 850 physicians or 25% of the sample fell into each of the 4 conditions (See Table 2). This allowed the authors to assess the individual and joint effects of the experimental conditions. The results were analyzed using an two-way analysis of variance test. The sticker-stamp experiment was not conducted on any of the follow-up mailings.
Experiment 2: Return envelope postage
On the third, fourth, and fifth nonresponse follow-up mailings to physicians, an additional experiment was conducted exploring different types of postage used on the return envelope. The authors compared three types of postage: (1) traditional postage-paid business reply mail, (2) a standard 39-cent first-class postage stamp of an American flag, and (3) a 45-cent breast cancer research stamp, which donated 6 cents per stamp to breast cancer research. Of the 2,450 nonresponders after the first and second mailing, 50% were randomly assigned to receive a postage-paid business reply envelope. The remaining nonresponders were randomly assigned to receive either the American flag stamp or the breast cancer research stamp. See Table 3 for approximate sample distributions. Chi-square tests were used to analyze the impact of these methods on response rates.
Results
Data for the physician survey were collected from September 15, 2006 to January 31, 2007. The survey achieved an overall response rate of 47.2% as calculated using American Association for Public Opinion Research (AAPOR) standard response rate 6 (standard definitions are located on the Web at www.aapor.org/pdfs/standarddefs_ver3.pdf), which treats undeliverables as ineligible cases. The results of the two experiments are reported below.
Experiment 1: Appearance of outgoing envelope
Chi-square tests show that the respondents in each of the four groups were not significantlydifferent with regard to several demographic variables that may have an effect on survey response age, region, gender, years since graduation (McFarlane, Olmsted, and Murphy 2007).
A total of 35 surveys were considered ineligible because they were undeliverable or because the sample member was deceased or no longer practicing medicine. Out of the 3,365 eligible surveys, 648 surveys were received during the first three weeks of data collection, before the second mailing was sent, and were included in the analyses for experiment 1. Table 4 shows the response rate for the four experimental conditions and for the use of stamps overall and the use of stickers overall.
The response rate data were analyzed to determine if the different treatment groups resulted in significant response rate differences and what, if any, are the main effects or interactions of using stickers.
An analysis of variance did not reveal a significant effect of treatment on response rate (F=0.38, df‘s=2 and, p=0.7679). The test results did not show statistically significant separation between groups that received stickers and groups that did not receive stickers (F=0.13, df=1, p=0.7218). The second factor analyzed the impact on survey response rates for groups that received stamped envelopes versus those that received metered envelopes. The test results do not show a significant main effect on response rates for stamps (F=0.10, df=, p=0.7483). Finally, the interaction between the two independent variables was not statistically significant (F=0.91, df=, p=0.3411). However, the group that received mailings with neither a sticker or stamp had the lowest response rate.
Experiment 2: Return envelope postage
Chi-square tests show that the respondents in each of the three groups were not significantlydifferent with regard to several demographic variables that may have an effect on survey response age, region, gender, years since graduation.
The analyses for the second experiment were limited to the 2,450 physicians who did not respond to the first or second mailings. The nonresponders were randomly assigned to one of the three groups. The three groups were not statistically different with respect to region, sex, age and years since graduation. Up to three additional mailings were sent to these nonresponders, which we will call mailings 3, 4, and 5. Table 5 shows the cumulative response rates for the three groups after each mailing. Chi-square analyses did not reveal a significant effect of treatment group on response rate for any of the mailings.
It was also hypothesized that the breast cancer stamps might have a larger impact with women compared to men and with oncologists compared to other specialists. To compare differences between men and women, we limited the analyses to only those sample members who received a stamped return envelope (n=1,224). An analysis of variance showed no differences between the groups for males versus females (F=0.21, df‘s=3 and, p=0.8925).
We then limited the analyses to sample members who were oncologists to determine if there was a difference in response for this group between the two types of stamps. However, only 74 oncologists were included in the follow-up mailings. Table 6 shows the cumulative response rate for the three follow-up mailings for oncologists by the two experimental groups.
There was a marginally significant difference between response rates for those who received the breast cancer stamps and those who received the American flag stamps. Response rates were considerably higher for physicians who received breast cancer stamps compared to American flag stamps.
Summary and discussion
This study found that surveys sent using first-class metered postage achieved a comparable response rate among physicians to surveys sent using first-class stamps. Tailored stickers applied to the outside of the envelope were also not effective in increasing response rates. While not statistically significant, the group that received neither a stamp nor a sticker on the outgoing envelope had the lowest response rate. These findings suggests that including something on the outgoing envelope may be better than nothing. Further research investigating different approaches for altering the appearance of the outgoing envelope is needed.
This study also found that the neither the use of stamped return envelopes nor the type of stamp (American Flag versus Breast Cancer research) affected the overall physician response rates in follow-up survey mailings. As a result, stamped return envelopes were also not cost-effective compared to business reply mail. However, this experiment was not conducted on the first survey mailing, so it is impossible to tell if stamped return envelopes are ineffective in this population or ineffective for follow-up mailings only. In the 2008 survey, the authors are going to continue the stamp experiment on the first mailing.
The use of breast cancer research stamps did not improve response rates over standard stamps or business reply mail for the entire sample of physicians. However when the analyses were limited to oncologists only, the results showed that breast cancer research stamps marginally improved response rates compared to American flag stamps. The results of this analysis are extremely limited, however, due to the small sample size of 74. The findings suggest that breast cancer stamps may be a promising approach to increasing response rates specifically among oncologists. Further research investigating the use of breast cancer research stamps with oncologists is needed.
Note
For more information on the “America’s Best Hospitals” project: Website: www.RTI.org/BestHospitals
Email: BestHospitals@RTI.org
A version of this article was presented at the Southern Association for Public Opinion Research (SAPOR) meeting on October 4, 2007.