Loading [Contrib]/a11y/accessibility-menu.js
Survey Practice
  • Menu
  • Articles
    • Articles
    • Editor Notes
    • In-Brief Notes
    • Interview the Expert
    • Recent Books, Papers, and Presentations
    • All
  • For Authors
  • Editorial Board
  • About
  • Issues
  • search

    Sorry, something went wrong. Please try your search again.
    ×

    • Articles
    • Blog posts

RSS Feed

Enter the URL below into your favorite RSS reader.

https://www.surveypractice.org/feed
×
Articles
Vol. 15, Issue 1, 2022March 24, 2022 EDT

Experimental Effects of Advance Postcards, Survey Title, Questionnaire Length, and Questionnaire Content on Response Rates and Incentive Costs in a Mail Non-Response Follow-Up Survey

Matthew DeBell,
advance mailing response rate mail survey
• https://doi.org/10.29115/SP-2022-0002
Photo by Becky Phan on Unsplash
Survey Practice
DeBell, Matthew. 2022. “Experimental Effects of Advance Postcards, Survey Title, Questionnaire Length, and Questionnaire Content on Response Rates and Incentive Costs in a Mail Non-Response Follow-Up Survey.” Survey Practice 15 (1). https://doi.org/10.29115/SP-2022-0002.
Save article as...▾
  • PDF
  • XML
  • Citation (BibTeX)
Data Sets/Files (3)
Download all (3)
  • Table 1. Response rates by advance card group.
    Download
  • Table 2. Response rates by study title group.
    Download
  • Table 3. Response rates by questionnaire length and content.
    Download

Sorry, something went wrong. Please try again.

If this problem reoccurs, please contact Scholastica Support

Error message:

undefined
×

View more stats

Abstract

A non-response follow-up study by mail in a national sample of U.S. households had five embedded experiments to test the effects of an advance mailing, alternate survey titles, 1- or 2-page questionnaire length, the inclusion or exclusion of political questions on the 1-page questionnaire, and the position of political content on the first or second page of the 2-page questionnaire. None of these design elements affected the payout of escalated postpaid incentives. Advance mailings had no effect on response rate. A short title (National Survey of Households) had a slightly higher response rate than a longer, more descriptive one (National Survey of Households, Families, and Covid-19). Political question content, whether by inclusion, exclusion, or position, had no discernable effect on response, even among prior-study non-respondents. Questionnaire length was inversely related to response: the 2-page questionnaire depressed the overall response rate by 3.7 points (58.5 compared to 54.8 percent, weighted) and depressed response for the critical sample group of prior non-respondents by 6.9 points (36.9 compared to 29.9).

Introduction and Research Questions

Non-response follow-up (NRFU) studies are used to gather data on sample units that fail to respond to a survey. Unit non-response bias is then estimated by comparing the original survey’s respondents to non-respondents who respond to the NRFU. Minimizing non-response is a top priority for NRFUs, since their purpose is to obtain responses from people who previously did not respond.

Several aspects of NRFU study design are important for promoting response. The mode of data collection should differ from the main study to avoid correlated non-response, which makes paper mail-in questionnaires a fitting NRFU mode for studies originally conducted online, by telephone, or in person. For the same reason, the title or topic and sponsorship of the study as presented to participants may be differentiated from the main study. Meta-analysis of many studies indicates that a survey’s topic can be particularly impactful if respondents find it interesting (response odds ratio 2.4; Edwards et al. 2002), but this may be harmful because what is interesting to one sample subgroup may discourage another from participating. The same meta-analytic evidence also indicates that advance letters should be used to announce the study because they have a substantial effect on response (odds ratio 1.5), as does a short questionnaire (odds ratio 1.9). To further quantify the tradeoff between questionnaire length and response rate, more than 40 years ago it was estimated that each additional question reduced the response rate to mail surveys by half a percentage point, and each page reduced the response rate by 5 points (Heberlein and Baumgartner 1978), though obviously this relationship must be bounded. Somewhat more recently a systematic review of 292 randomized controlled trials estimated that the odds of response for a 3-page questionnaire were half the odds for a 1-page questionnaire (Edwards et al. 2002).

These are powerful effects. Yet these lessons do not always translate as directly into practice as we might wish, for two connected reasons. First, the effects of particular design elements may vary across contexts, creating uncertainty about the benefits of a particular design choice for a particular study. Indeed, meta-analysis shows considerable heterogeneity in the effects of advance letters (De Leeuw et al. 2007) and other design elements (Edwards et al. 2009). Second, study goals such as a high response rate and a long questionnaire are in direct tension. Using a 1-page questionnaire constitutes such an enormous opportunity cost that researchers may reasonably ask whether the effect would be as large in the specific application to their study as it appears to be more generally.

The motivation for the current study was to test the effects of advance mailings, questionnaire length, and study topic (in study title and questionnaire content) on the response rate to a mail survey, in the context of a non-response follow-up to the American National Election Studies (ANES) Time Series Study. The research questions (RQ) were as follows:

RQ1: Questionnaire content. The ANES is a study of adult U.S. citizens. Its questionnaire content is heavily political. For this study’s purposes, non-response correlated with political behavior, opinions, or attitudes is a serious concern. For the NRFU, this raises the question of how strongly political questionnaire content is associated with non-response and whether more ANES non-respondents can be included by using a non-political NRFU questionnaire.

RQ2: Questionnaire length. The ANES NRFU, unlike some NRFUs, includes both ANES respondents and ANES non-respondents, raising the question of whether effects of study design on response rates differ for ANES respondents (whose NRFU response propensity is very high) and ANES non-respondents (whose NRFU response propensity is relatively low). It is the lower-propensity non-respondents who are critical to reach with the NRFU. Thus, though we anticipate that questionnaire length will be inversely related to response rate, we ask if this relationship is different for ANES respondents and non-respondents.

RQ3: Study title. What is the effect of alternate titles on the response rate? Topics of interest promote response, suggesting that a study title about matters that are timely, such as Covid-19, or important to most people, such as families, could be higher than a study that is more generic, such as a National Study of Households.

RQ4: Advance mailing. Does an advance postcard increase the response rate to a brief mail study, for prior respondents or non-respondents? Advance letters promote the study’s legitimacy, which is beneficial for many studies, but is it still beneficial for a study that already uses numerous design elements known to promote response: a low-burden study (short questionnaire) with a pre-paid monetary incentive, multiple follow-up mailings, questionnaire re-sent after non-response, sent by first-class mail, and sponsored by a university?

RQ5: Costs. Do these design elements affect the financial cost of administering the study? The ANES NRFU used a $5 prepaid incentive and, after initial non-response, made a $20 post-paid incentive offer. Thus, promoting initial response has the potential to save money by reducing the number of participants who receive the escalated incentive.

Methods and Data

Data come from the ANES 2020 Non-Response Follow-Up Study (NRFU; American National Election Studies 2021). The study was conducted by mail between January 28 and June 1, 2021. The sample included both respondents and non-respondents to the ANES 2020 Time Series Study (ANES; American National Election Studies 2020). The ANES was a survey of adult U.S. citizens selected using address-based sampling, interviewed using a mail push-to-web design. The ANES selected one adult U.S. citizen per sampled household after a brief initial screening questionnaire, and the minimum overall response rate was 37 percent. The NRFU sample consisted of 4,000 individuals who responded to the ANES, 1173 individuals whose household was screened and who were selected for ANES but did not respond, and 2827 households that were selected for ANES but did not respond at the screening stage (total 8,000). The individuals who completed or who were screened for ANES were invited to NRFU by name; at the non-responding households, invitations used the Hagan and Collier (1983) method, randomly requesting a response from the oldest or youngest male or female. The NRFU study’s nominal sponsor was Duke University. A prepaid $5 incentive was enclosed, and up to 6 invitation or reminder mailings were sent. The last two mailings offered a $20 postpaid incentive. There were 3,779 responses to the NRFU, for an unweighted response rate of 47 percent overall and a weighted response rate of 57 percent overall, 83 percent among ANES respondents, and 34 percent among ANES non-respondents. The weighted NRFU data represent the full ANES sample.[1]

Independently randomized methodological experiments were integrated in the NRFU study to test questionnaire content, length, title, and advance mailing effects. Half the sample (n=4,001) was sent an advance postcard shortly before the initial questionnaire invitation, while half (n=3,999) omitted this mailing. Half the sample (n=4,001—again, randomized independently) was invited to the National Study of Households while half (n=3,999) were invited to the National Study of Households, Families & Covid-19. Questionnaires were randomized for length and content into four types (each n = 1,943 to 2,056): a one-page non-political questionnaire, a one-page questionnaire with political questions, a 2-page questionnaire with political questions on the second page, and a two-page questionnaire with political questions on the first page. The questionnaires are available on the ANES website at www.electionstudies.org.

Results

The advance postcard had no discernable effect on the response rate for the NRFU overall or among ANES respondents or non-respondents. These results are shown in Table 1, where the response rates to NRFU were about 57 percent overall, 83 percent for ANES respondents, and 34 percent for ANES non-respondents, regardless of the advance mailing condition; differences for the mailing conditions were 0.2 percentage points or less.

Table 1.Response rates by advance card group.
Response rate n Sampling error
Sample group Card No card Difference p Card No card Card No card Diff z
All sample 56.5 56.7 -0.2 .884 3906 3911 0.87 0.87 1.23 0.146
ANES Rs 83.4 83.3 0.1 .945 1970 1977 0.71 0.71 1.01 0.069
ANES NRs 33.5 33.5 0.0 .980 1936 1933 1.38 1.37 1.94 0.025

Note: “Rs” are respondents; “NRs” are non-respondents; “Diff” is the sampling error for the Difference.

The survey title experiment (see Table 2) found that the shorter, simpler title—National Study of Households—yielded a higher response rate than the longer title, but no differences reached statistical significance. Overall, the response rate was 2.2 points higher with the short title (p = .080) and 3.2 points higher for ANES non-respondents (p = .095).

Table 2.Response rates by study title group.
Response rate n Sampling error
Sample group Short title Long title Difference p Short Long Short Long Diff. z
All sample 57.7 55.5 2.2 .080 3912 3904 0.87 0.87 1.23 1.751
ANES Rs 83.8 82.8 1.0 .331 1979 1966 0.70 0.73 1.01 0.972
ANES NRs 35.1 31.9 3.2 .095 1932 1936 1.39 1.35 1.94 1.672

Note: “Rs” are respondents; “NRs” are non-respondents; “Diff” is the sampling error for the Difference.

Questionnaire length showed large effects overall and for ANES non-respondents, as shown in Table 3. Overall, the response rate was 3.7 points higher for the one-page questionnaire than the two-page questionnaire (p=.003) and 6.9 points higher for the one-page questionnaire among non-respondents (36.9 compared to 29.9 percent, p<.001). For ANES respondents, the difference of 1.5 was in the expected direction but not significant.

Questionnaire content showed no detectable effects (also Table 3). Response rate differences between the political and non-political one-page questionnaires were about 1 point or less and were nonsignificant. For the two-page questionnaire, response rate differences associated with the position of political content on the first or second page were 0.8 to 1.7 points and were nonsignificant.

Table 3.Response rates by questionnaire length and content.
Questionnaire length n Sampling error
Sample group 1 page 2 pages Difference p 1 page 2 pages 1 page 2 pages Diff z
All sample 58.5 54.8 3.7 .003 3924 3892 0.86 0.88 1.23 3.001
ANES Rs 84.1 82.6 1.5 .137 1977 1970 0.70 0.73 1.01 1.488
ANES NRs 36.9 29.9 6.9 .000 1948 1922 1.39 1.35 1.94 3.582
1-page questionnaire
Political Non-pol. Political Non-pol. Political Non-pol.
All sample 58.0 58.9 -1.0 .581 2016 1909 1.20 1.23 1.72 0.553
ANES Rs 83.5 84.7 -1.2 .373 1003 974 1.00 0.97 1.39 0.891
ANES NRs 36.9 36.8 0.1 .977 1013 935 1.91 2.01 2.77 0.029
2 page questionnaire
Pol. p. 1 Pol. p. 2 Pol. p. 1 Pol. p. 2 Pol. p. 1 Pol. p. 2
All sample 55.6 54.0 1.7 .342 1885 2007 1.25 1.23 1.76 0.951
ANES Rs 83.0 82.1 0.9 .534 969 1001 1.02 1.04 1.46 0.621
ANES NRs 30.4 29.5 0.8 .759 916 1006 1.94 1.89 2.70 0.307

Note: “Rs” are respondents; “NRs” are non-respondents; “Diff” is the sampling error for the Difference. “Pol.” abbreviates
“Political” and “Non-pol.” abbreviates “Non-political,” referring to questionnaire content.

There were no significant differences in total incentive payout costs. Between one- and two-page conditions, where the response rate difference reported above was significant, there was an average cost difference of three cents (not shown in tables).

Conclusions

This study found advance postcards were not effective in promoting response to the NRFU, suggesting similar studies in the future can omit this mailing. The short title had a slightly higher response rate than the one also referencing families and Covid-19, suggesting the short, plain title was better. Political content, whether by inclusion, position, or exclusion, had no discernable effect on survey response, even among ANES non-respondents, indicating that such content need not be avoided to promote response. However, questionnaire length was strongly and inversely related to response, especially for the critical sample group of prior non-respondents: for this group, the 2-page questionnaire depressed the response rate by almost 7 points compared to the 1 page questionnaire, indicating the price of more questions is high for the sample group where NRFU response is most important.


  1. The NRFU data represent the full ANES sample when weighted using the weight WIHHNRFUWT. The dataset includes jackknife replicate weights for design-consistent estimates of variance and sampling error (WIHHNRFUWT1 through WIHHNRFUWT100). These weights will be used for analysis unless otherwise noted.

Submitted: December 13, 2021 EDT

Accepted: February 23, 2022 EDT

References

American National Election Studies. 2020. “ANES 2020 Time Series Study.” Dataset and documentation. July 19, 2021 version. http://www.electionstudies.org.
———. 2021. “ANES 2020 Non-Response Follow-Up Study.” Dataset. October 28, 2021 version. http://www.electionstudies.org.
De Leeuw, Edith, Mario Callegaro, Joop Hox, Elly Korendijk, and Gerty Lensvelt-Mulders. 2007. “The Influence of Advance Letters on Response Rates in Telephone Surveys: A Meta-Analysis.” Public Opinion Quarterly 71: 413–43.
Google Scholar
Edwards, Philip James, Ian Roberts, Mike J. Clarke, Carolyn DiGuiseppi, Sarah Pratap, Reinhard Wentz, and Irene Kwan. 2002. “Increasing Response Rates to Postal Questionnaires: Systematic Review.” British Medical Journal 324 (7347): 1183. https://doi.org/10.1136/bmj.324.7347.1183.
Google ScholarPubMed CentralPubMed
Edwards, Philip James, Ian Roberts, Mike J. Clarke, Carolyn DiGuiseppi, Reinhard Wentz, Irene Kwan, Rachel Cooper, Lambert M. Felix, and Sarah Pratap. 2009. “Methods to Increase Response to Postal and Electronic Questionnaires.” Edited by Cochrane Methodology Review Group. Cochrane Database of Systematic Reviews, no. 3 (July): MR000008. https://doi.org/10.1002/14651858.mr000008.pub4.
Google Scholar
Hagan, Dan E., and Charlotte Meier Collier. 1983. “Must Respondent Selection Procedures for Telephone Surveys Be Invasive?” Public Opinion Quarterly 47 (4): 547–56. https://doi.org/10.1086/268811.
Google Scholar
Heberlein, Thomas A., and Robert Baumgartner. 1978. “Factors Affecting Response Rates to Mailed Questionnaires: A Quantitative Analysis of the Published Literature.” American Sociological Review 43 (4): 447–62. https://doi.org/10.2307/2094771.
Google Scholar

Powered by Scholastica, the modern academic journal management system