Introduction
Researchers have turned to methods such as push-to-web and drop-off/pick-up (DOPU) to improve response rates (Dillman 2017; Lynn 2020; Steele et al. 2001; Trentelman et al. 2016; Jackson-Smith et al. 2016). The push-to-web approach typically involves mailing cards with a web link, while the DOPU method centers on getting a research team on the ground in direct contact with households, encouraging them to complete a survey booklet to be picked up by the research team later. We implemented a novel hybrid between the web survey and DOPU. Instead of dropping off an entire survey booklet and asking respondents to complete the survey after a brief conversation, we used a door hanger with a link to the survey. Our logic was that this approach had the advantages of the DOPU in that it placed local research teams in the field but also avoided two major costs of the method—printing the survey booklets and the travel and labor costs associated with the return visit to the respondent—but also placed researchers in the field, potentially conveying the importance and legitimacy of the research that could encourage response like DOPU (Steele et al. 2001; Trentelman et al. 2016).
Methods
In July 2021, we conducted a survey in Fort Collins, CO about an energy efficiency program sponsored by the city government. The selected sample was 6,035. The city provided contact information for 1,683 program participants and we contracted with Marketing Systems Group (MSG), a sampling and data services firm, for a list of mailing addresses linked to email addresses. MSG also provided 1,031 mailing addresses that were not linked to an email address. The email list from the city included 310 duplicates and 136 undeliverable addresses—we evaluated the undeliverable addresses for missing characters, obvious spelling errors, and other issues but were only able to fix three of them. MSG provided 3,321 email addresses tied to physical addresses, which included 89 duplicates and 269 un-deliverables; we could not correct the undeliverable email addresses. All respondents were screened for residence in Fort Collins and age over 18, and 13 respondents did not pass this screening. Thus, there were 4,190 valid email addresses for an invited sample of 5,221 (that is, valid email addresses combined with the mailing addresses that were not linked to an email address). Using up to four contact attempts, we collected 859 responses via email for a response rate of 20.5%. We asked the primary decision-maker regarding home upgrades to complete the survey.
After the initial efforts to collect data via email links, we organized eight trained field researchers to distribute the web survey door hangers over a long weekend. Over the course of a weekend, we delivered 632 door hangers, but within 3 days only 33 respondents had started the survey by following the web link on the door hanger, a response rate of 5.22% using AAPOR definition 1.
Given the cost and time involved with distributing door hangers, we pivoted to mailing 4,329 web survey cards (1,031 mailing addresses with no email address, and 3,298 mailing addresses tied to email addresses that had not responded to the survey), which resulted in 109 completions (a response rate of 2.5%). Although the web survey method produced a much lower response rate, it did so at significantly less cost per completion.
Conclusion
We implemented a novel strategy involving door hangers that provided a link to an online survey. Ultimately, our experimental mode was ineffective. DOPU is likely effective because it often involves direct contact with respondents; our poor results imply that this social exchange is key to its success (Steele et al. 2001; Trentelman et al. 2016; Jackson-Smith et al. 2016). Although the door hangers produced a response rate like a mail survey, the cost per completion was much higher, as distributing the hangers was much more expensive and time consuming than simply bulk mailing web survey cards. The lack of direct contact with respondents (or only sporadic contact) likely explains why the door hangers were so ineffective.
The door hanger approach should not be abandoned altogether. It was especially challenging for a probability sample, as the houses were selected at random and could be some distance from each other. However, a web survey door hanger approach might be useful for simpler sampling approaches when a sampling frame and enumerated probability sample is not available (i.e., selecting every other home in a densely populated area). Further, we encourage additional experimental work that hybridizes proven methods.
Corresponding author
Adam Mayer
adam.mayer@colostate.edu
