The National Longitudinal Transition Study 2 (NLTS2), funded by the U.S. Department of Education, is a nationwide longitudinal study designed to collect information on the education, activities, and development of youth with disabilities. NLTS2 follows approximately 11,000 youth with disabilities and their parents over a 10-year period as the youth transition from high school into their young adult lives; interviews began in 2001 and are conducted by telephone every two years with both parents and youth. In 2007, we conducted an experiment to test the effectiveness of using a prepaid token incentive (rather than a post-survey incentive) and a mail questionnaire (as an alternative to the telephone interview). The results showed that parents who were provided the option of completing a mail survey were significantly more likely to respond than those who were provided only a letter reminding them of the telephone interview (with or without a prepaid incentive).
The study has faced an increasing challenge to meet response rate goals from wave to wave. The high degree of emotion associated with the questions as reported by interviewers and the parents themselves, and the length of the interview—often 40 minutes or more—has contributed to a lower response rate with each succeeding wave. Participation, especially from parents, has declined.
Wave 1 of NLTS2 (in 2001) achieved an 82 percent response rate, but as early as the second wave, response rates among parents began to drop. To counteract the decline, the study team implemented advance and refusal conversion letters, intensive tracing efforts to locate parents and youth, and a $20 post-paid incentive for completed interviews. With each wave, these measures were repeated and expanded, yielding parent response rates of 70 percent in Wave 2 (2003) and 68.6 percent in Wave 3 (2005).
With indications that the parent response rate would drop more steeply in the fourth wave (2007), the study team designed an experiment to test the effectiveness of offering parents a pre-paid token incentive and a mail questionnaire. Employing a mixed-mode approach to survey data collection—offering one mode of response and then switching to another mode—has been identified as an effective way to improve response rates (e.g., Converse et al. 2008; Dillman et al. 2001; Fowler et al. 2002; Shettle and Mooney 1999), and the literature also supports the use of pre-paid versus post-paid incentives in both telephone and mail surveys (Edwards et al. 2005; Singer, Hoewyk, and Maher 2000).
NLTS2 has always offered youth who are hearing or speech impaired the option to complete a mail questionnaire; therefore, the decision to expand this option to parents seemed reasonable. The parent mail questionnaire was an abbreviated version of the telephone interview, designed to capture critical items such as the youth’s postsecondary education enrollment, employment, and residential independence. Although the mail questionnaire does not yield as much data as the telephone interview, the team hoped that it would encourage parents to continue participation in the current wave.
Wave 4 interviews began in April 2007 with a sample of 8,573 parents; the experiment took place 17 weeks after the start of Wave 4. We created three experimental groups of 200 parents each from the 3,296 parents remaining in the sample at the time of the experiment. Parents were selected into each group in the same proportion from among pending nonrefusal cases, initial and final refusals, and cases for which we had no telephone number.
Table 1 illustrates the three experimental groups and the treatments they received.
Parents in the three experimental groups did not differ significantly in their demographics (youth’s disability category, gender, household income, race/ethnicity) when compared with parents in the remaining NLTS2 sample. Parents in each of the three experimental groups also did not significantly differ from each other.
This experiment focused on the following research questions:
- What is the impact on response rates of a prepaid token incentive and a reminder of the full incentive (Group 2), compared with only a reminder of the full incentive (Group 1)?
- What is the impact of including a mail questionnaire, a prepaid token incentive, and a reminder of the full incentive (Group 3) versus a prepaid token incentive and a reminder of the full incentive (Group 2)?
- What is the impact of including a mail questionnaire, a prepaid token incentive, and a reminder of the full incentive (Group 3) versus no prepaid incentive, no questionnaire, and just a reminder of the full incentive (Group 1)?
- To what extent did the respondents differ from nonrespondents?
- To what extent did the different approaches reach different types of respondents?
These questions were addressed using univariate and bivariate descriptive statistics (e.g., percentages, means, and cross-tabulations). Rather than test for differences between all independent subgroups (e.g., responders in the different experimental groups) simultaneously, the statistical significance of differences between selected pairs of independent subgroups was tested using the F statistic.
Figure 1 presents response rates for each of the three experimental groups. Statistical results are presented in Table 2. Key findings are summarized below.
Group 3 Response Rate Was Highest
Parents who were given the option of completing a mail survey along with a letter and $5 (Group 3) were significantly more likely to respond to the mailing than were those who only received a letter (Group 1) or a letter with $5 (Group 2). Within 6 weeks of the mailing, 23 percent of parents in Group 3 had responded, compared with 10 percent in Group 2 (p<0.01) and 7 percent in Group 1 (p<0.001). Response rates for Groups 1 and 2 did not differ significantly.
No Difference in Percent Completing the Telephone Interview
Parents in the three experimental groups did not differ significantly in terms of the percentage choosing to complete the telephone interview—7 percent, 10 percent, and 10 percent completed telephone interviews in Groups 1 through 3, respectively. The additional 14 percent of parents in Group 3 who completed a mail survey accounted for the significant differences in the overall response rates between this group and the other two experimental groups.
Mail Survey Option Was as Important as a Larger Incentive
Providing the mail survey option appeared to have a similar impact as the size of the overall incentive. Parents in Group 3 were offered $20 to complete a telephone interview and only $15 to complete the mail questionnaire; however, 14 percent of Group 3 parents opted to complete the mail questionnaire, compared with 10 percent who chose to complete the telephone interview.
No Significant Differences Between Responders and Nonresponders
Responders and nonresponders in the full experimental sample did not differ significantly from each other, nor did responders to the three types of mailings differ significantly in their demographics, satisfaction, or youth’s experiences (Table 3). The lack of significant differences between respondents to the different approaches might be due in part to the small number of respondents in each of the groups and the large standard errors.
Our results show that for parents in this study, who may be tired of a long telephone interview each wave, offering a mail questionnaire to nonrespondents is a viable option. When the experiment was over, NTLS2 expanded this option to all remaining nonrespondents, and almost 400 parents ultimately chose to respond by mail.
Although computer assisted telephone interviewing yields robust data, respondents may be more inclined to complete a mail questionnaire for reasons such as the ability to choose the time and place to complete it, not having to interact with an interviewer, and being able to see the length and types of questions. Disadvantages to using a mail questionnaire do not in this instance seem to outweigh the advantage of actually obtaining data from otherwise nonresponding parents.
We conclude that where feasible in longitudinal telephone surveys, it is better to offer an alternative mode to nonrespondents than to potentially lose data from those individuals in a given wave. However, more research is needed to understand whether individuals who choose to respond by mail are significantly different from those who choose to respond by telephone. Similarly, this experiment did not provide a clear sense of the benefit of including a $5 bill with the mailing. The study team is currently in the field with the final wave of data collection and is offering a mail questionnaire to parents without a prepaid incentive. Early results show that an even greater percentage of parents in Wave 5 are choosing to respond by mail.