Introduction

The University of Michigan’s Surveys of Consumers (SCA) tested the effect of a set of design variations on response rates in address-based sampling (ABS) mail and Web-Mail surveys in monthly experiments between March 2011 and April 2012. In this paper, we report findings from a selected set of experiments focusing on the effect of prepaid cash incentives in the ABS Web-Mail surveys for a general population. The selected set of findings includes response rates, proportion of completed interviews by web (web completion rates), and demographic and socioeconomic characteristics of respondents by the experimental groups.

Methods

Sample

The SCA is a monthly telephone survey that follows a rotating panel survey design (Curtin 1982). Each monthly sample is composed of 300 fresh and 200 recontact cases. The SCA targets household heads 18 years and older in the contiguous United States. In addition to the random digit dialing (RDD) sampling telephone interviews, postal addresses from the U.S. Postal Service Computerized Delivery Sequence File were contacted in a set of monthly mail survey experiments. Further information on the SCA can be found on https://data.sca.isr.umich.edu/.

In September and October 2011, monthly fresh samples of 1,500 postal addresses were randomized to concurrent and web-intensive contact strategies. In these two Web-Mail contact strategies, the respondents were allowed to complete either a web or a paper questionnaire. In addition to the fresh samples, a total of 507 respondents who completed the interview six months ago were randomly assigned to be recontacted by one of the three strategies: mail, Web-Mail concurrent, or Web-Mail web-intensive. Under the assumption that monthly differences are not related to the outcomes of interest and contact strategies, prepaid incentive and no incentive conditions were assigned to September and October samples, respectively (see Figure 1).

Figure 1  Study flow diagram.

In both the mail only and the Web-Mail, the mailing protocol consisted of five mailings which are seven days apart as follows: (1) advance letter, (2) paper questionnaire package, (3) nonresponse follow-up: postcard reminder, (4) nonresponse follow-up: replacement paper questionnaire package, and (5) nonresponse follow-up: second postcard reminder. In the Web-Mail design, the invitation package included letters with a Uniform Resource Locator (URL) and a custom id for the sample units to use to complete the web survey if they choose to do so. While in the concurrent design, the web survey invitation was included concurrently with the paper questionnaire package, the web response option was included in the advance letter in the web-intensive design. When incentives were offered to the prepaid cash incentive group, a 5 USD cash incentive was included when a web survey and/or paper questionnaire invitation was included the first time. The advance letter in the web-intensive contact strategy mentioned a forthcoming paper questionnaire.

Analysis Plan

Between the incentive and no incentive groups, we compared the following: response rates, web completion rates, and demographic and socioeconomic characteristics. The comparisons included the differences by the contact strategy for both fresh and recontact cases. In addition, we explored the differences between the incentive and no incentive groups, specifically for the recontact cases, in terms of the SCA baseline scores.

Results

Table 1 reports the response rates by the experimental groups. The incentives yielded higher response rates (Wald chi-square [df=1]=40.1, p<0.0001) controlling for the fresh and recontact case status. The differences in response rates by fresh and recontact case status were not significant by the incentive group. In addition to the fresh and recontact case status, we also tested the effect of incentives and contact strategy combined in each group (results not shown here), and these effects were not significant.

Table 1  Response rates (RR2) by randomized group, 2011 SCA ABS Web-Mail survey data.
Sample Incentive No incentive
Fresh 34.5% 17.4%
Recontact 70.1% 50.3%

The web completion rates by sample and contact strategies and incentive groups are shown in Table 2. For the fresh cases, incentives increased the web completion rate significantly in the web-intensive contact strategy (Wald chi-square [df=1]=6.8066, p=0.01). That is, when the incentives were offered, respondents were more likely to respond by web in the fresh sample. Although the direction of the difference was same for the recontact cases, it was not significant significantly due to small sample size. For the concurrent contact strategy, when the incentives were offered, respondents were less likely to respond by web in the recontact sample (Wald chi-square [df=1]=4.3730, p=0.04).

Table 2  Web completion rates by randomized group, 2011 SCA ABS Web-Mail survey data.
Sample and contact strategies Incentive No incentive
Fresh
 Concurrent 22.8% 21.8%
 Web-intensive 45.5% 31.1%
Recontact
 Concurrent 23.3% 36.8%
 Web-intensive 34.3% 30.3%

As shown in Tables 3 and 4, the characteristics of the respondents were similar by the incentive group in both fresh and recontact samples except for the stock ownership in the recontact group. In the recontact sample, percentage of stock owners was significantly higher in the no incentive group.

Table 3  Characteristics of respondents by randomized group, fresh cases.
Characteristic Incentive
 No incentive
Counts % Counts %
Age
 18–24 6 1.4% 3 1.3%
 25–34 54 12.5% 22 9.6%
 35–44 54 12.5% 22 9.6%
 45–54 84 19.4% 41 17.8%
 55–64 97 22.4% 72 31.3%
 65–97 138 31.9% 70 30.4%
 Missing 48 10.0% 11 4.6%
Homeownership
 Owns or is buying 385 82.4% 195 82.3%
Stock ownership
 Yes 305 66.6% 156 69.0%
Region
 West 107 22.2% 39 16.2%
 Midwest 129 26.8% 66 27.4%
 Northeast 88 18.3% 35 14.5%
 South 157 32.6% 101 41.9%
Income
 Under $10,000 19 4.2% 11 5.0%
 $10,000–$14,999 24 5.3% 10 4.6%
 $15,000–$19,999 15 3.3% 11 5.0%
 $20,000–$29,999 33 7.3% 27 12.3%
 $30,000–$39,999 46 10.2% 18 8.2%
 $40,000–$49,999 44 9.8% 18 8.2%
 $50,000–$59,999 42 9.3% 25 11.4%
 $60,000–$74,999 62 13.7% 26 11.9%
 $75,000–$99,999 56 12.4% 22 10.0%
 $100,000–$124,999 42 9.3% 21 9.6%
 $125,000–$149,999 16 3.5% 8 3.7%
 $150,000–$174,999 20 4.4% 6 2.7%
 $175,000 or more 32 7.1% 16 7.3%
 Missing 30 6.2% 22 9.1%
Table 4  Characteristics of respondents by randomized group, recontact cases.
Characteristic Incentive
 No incentive
Counts % Counts %
Age
 18–24 0 0.0% 1 0.4%
 25–34 14 4.2% 11 4.6%
 35–44 41 12.2% 24 10.1%
 45–54 71 21.1% 49 20.6%
 55–64 82 24.4% 68 28.6%
 65–97 128 38.1% 85 35.7%
 Missing 14 4.0% 9 3.6%
Homeownership
 Owns or is buying 296 86.8% 213 87.3%
Stock ownership*
 Yes 209 63.5% 171 72.5%
Region
 West 66 18.9% 50 20.2%
 Midwest 85 24.3% 70 28.3%
 Northeast 78 22.3% 50 20.2%
 South 121 34.6% 77 31.2%
Income
 Under $10,000 14 4.3% 7 3.0%
 $10,000–$14,999 9 2.8% 7 3.0%
 $15,000–$19,999 23 7.1% 15 6.4%
 $20,000–$29,999 34 10.5% 25 10.7%
 $30,000–$39,999 24 7.4% 26 11.2%
 $40,000–$49,999 37 11.4% 24 10.3%
 $50,000–$59,999 27 8.3% 21 9.0%
 $60,000–$74,999 41 12.6% 27 11.6%
 $75,000–$99,999 39 12.0% 20 8.6%
 $100,000–$124,999 36 11.1% 19 8.2%
 $125,000–$149,999 11 3.4% 22 9.4%
 $150,000–$174,999 8 2.5% 6 2.6%
 $175,000 or more 22 6.8% 14 6.0%
 Missing 25 7.1% 14 5.7%

*Significant at p=0.05.

Table 5 shows the baseline scores from the first interview for the recontact cases. Table 5 includes three leading indices that the SCA reports monthly. The Index of Consumer Sentiment is composed of five survey items: (1) current personal finances, (2) expected personal finances, (3) expected business conditions in 12 months, (4) expected business conditions in 5 years, and (5) current buying conditions. While the current index includes items 1 and 5, the expected index includes items 2, 3, and 4. The pairwise comparisons were not significant in the index scores by the incentive group.

Table 5  Baseline measures by randomized group, recontact cases.
Index Incentive No incentive
Index of Consumer Sentiment 69.8 73.9
Current index 76.8 81.5
Expected index 65.3 69.0

Discussion

These experiments allowed the principal investigator to test the mailing protocol and the package for a general population in addition to testing the effect of the incentives. Overall, the prepaid cash incentive increased the response rates in this study. This is in line with the previous literature (Church 1993). Additional analysis offered further information on the potential magnitude of the possible change in nonresponse bias for a general population survey on the economic attitudes.

According to the leverage-saliency theory of survey participation, the survey participation decision is based on an equation including the importance and the saliency of the feature and the direction of the influence (Groves, Singer, and Corning 2000). Furthermore, the saliency and the importance of a survey feature may differ by people in addition to the direction of the influence. That is, incentives could motivate people with certain characteristics to respond disproportionately at a higher rate. The higher the covariance between the survey measures of these people and their response propensities, the higher the nonresponse bias (Groves, Presser, and Dipko 2004; Groves et al. 2006, 2009). For example, incentives attracting poorer people disproportionately could be a concern as this could imply a higher covariance between the economic attitudes and the response propensities, consequently higher nonresponse bias. In particular, this study used a set of socioeconomic demographics to understand the differences in the respondent characteristics by the incentive group. For the fresh sample, socioeconomic demographics were comparable. For the recontact cases, further research is required on the significant difference by the stock ownership, but the baseline scores did not suggest any differences in the nonresponse bias by the incentive group. Overall, we concluded that using the incentives increased the average response propensity without changing the covariance component of the nonresponse bias substantially.

For the fresh cases, the incentives did increase the web completion rates in the web-intensive contact strategy. On the other hand, the concurrent contact strategy yielded lower web completion rate in the incentive group for the recontact cases. This finding should be interpreted with caution due to small sample sizes and future research should investigate its replicability. One possible explanation is that incentives could be urging the participants to respond and to choose the mode at hand.