As the use of web-based surveys grows primarily due to lower costs, concerns of selection bias are increasing, as some people are still not computer or internet literate. The Teacher Follow-up Survey (TFS), a widely used survey of teachers, has undergone similar changes moving from paper to an internet survey. Sponsored by the National Center for Education Statistics (NCES), the TFS is a component of the Schools and Staffing Survey (SASS) designed to determine attrition and mobility rates of U.S. teachers. Designed as a one-year follow-up, the TFS has been administered after each SASS administration since school year 1988–89. Until the 2004–05 TFS, the survey was administered only through paper instruments. Because the 2004–05 TFS was collected by both web and paper instruments and the 2008–09 was primarily collected by web instrument, the authors investigated the possibility that trends could be compromised due to a change in collection mode.
Previous studies suggest that the survey mode could affect responses, even when controlling for demographic characteristics. Response mode differences were observed, particularly for more demanding items such as open-ended questions (Berrens et al. 2003; Fricker et al. 2005) or highly sensitive questions (Heerwegh and Loosveldt 2008; Rookey, Hanway, and Dillman 2008). In addition, prior investigations indicate that there are differences in web respondents’ demographic characteristics such as age and education, compared to those of respondents of other survey modes such as traditional mail or telephone interviews (Fleming and Bowden 2009). Ignoring these differences could lead to erroneous conclusions regarding the existence of actual mode effects.
Unique to this study, compared to previous investigations, is the use of an instrumental variable regression model technique. This regression technique is critical for modeling the two separate components of the survey response process. This is, first, the decision to choose between the paper-based or the web-based instrument, affected by the teacher’s characteristics and second, the responses themselves, affected by the survey instrument chosen.
Using the 2004–05 TFS, this study explores the potential effects of survey mode on the long term consistency of TFS estimates. The existence of response mode effects could jeopardize the quality of future survey data collections and cause an artificial break in the trend with previous TFS administrations. As web tools are increasingly being used for data collection purposes, the need to explore these potential issues becomes critical.
Description of the analyses
The main purpose of this study is to understand potential differences between web-based and paper-based responses. Two main hypotheses are tested in this study, which follow the survey response process (i.e, the selection of the survey mode to be used and then the act of responding itself). First, teachers’ characteristics are expected to affect their choice of responding to the paper-based instrument over the web-based instrument. These analyses provide information regarding response differences linked to the characteristics of the teachers rather than the instrument itself. Second, once teachers’ self-selection of one survey instrument over the other is considered, no differences are expected in the type and quality of the responses. This component investigates whether or not response differences are solely linked to the mode of data collection utilized.
This study utilizes measures of differentiation and regression analyses to explore differences in teachers’ characteristics as well as the type and quality of responses between web-based and paper-based surveys. Table 1 shows the weighted number of total former and current teachers, by response mode. The total weighted number of former teachers is 333,000 and 3,350,000 for current teachers. Standard errors are reported in parentheses.
The 2004–05 TFS collection design incorporated a unique web-based collection experiment, which randomly assigned teachers to receive either the paper or web instrument. Given that this assignment was random, these two groups are expected to have similar characteristics. Those receiving the web instrument could then choose to respond to the survey using the web-based or paper-based questionnaire. These two groups are likely to be different, because teachers self-selected themselves into one group or the other.
In addition, among teachers who received the internet survey, two randomly selected groups were sub-divided, one group who knew about the availability of a paper survey and a second group who did not know about the availability of the paper instrument when they first received the internet questionnaires. All these teachers were aware of the existence of the paper interview by the end of the data collection process.
Several survey questions were selected to explore mode differences. Questions were chosen based on their location on the survey and the type of question, such as dichotomous, multiple item, rating categories, and open ended questions. In addition, questions were also selected based on their inclusion in the 2007–08 Schools and Staffing Survey (SASS) and the 2008–09 TFS.
Selected questions with multiple items and multiple rating categories were transformed to be analyzed as estimated measures of differentiation (P). The P is used to examine potential issues in the quality of data responses. This constructed measure captures, for example, if teachers who used the web instrument are systematically and illogically responding to all question items checking the middle rating category and thus jeopardizing the quality of these data responses. The measure P ranges from 0, if the respondent gave all items the same rating (indicating low quality responses), to 1, if the respondent answered all items differently. P represents a simple yet comprehensive indicator to evaluate response mode differences between the web and paper questionnaire.
The main component of this study’s analyses uses a two-stage Heckman-type instrumental variable (IV) regression model. This regression model deals with potential selection bias issues, such as those observed in this study. The main source of potential selection bias would come from not considering the two steps of the survey response process (i.e., first, teachers’ decisions to use the paper-based over the web-based survey instrument, and second, the response itself, potentially affected by the survey instrument used).
Teachers’ choices are largely determined by their demographic characteristics, education, work experience, or access to internet. As a result, if the actual observed web-choice variable is to be used to explain the type and quality of the responses, it would be largely correlated with this regression’s estimated unobserved error term, providing erroneous estimates. The IV regression model corrects for this issue, using an estimated instrumental web-choice variable that will be uncorrelated with the error term.
Using a hypothetical example, we could assume that younger and less educated teachers were more likely to choose the web instrument but also more likely to respond poorly to any survey, such as misreporting or exaggerating. Utilizing the observed web-choice variable to explore mode effects could lead to the erroneous conclusion that using the web-based questionnaire would provide lower quality or different responses from those collected with the paper-based instrument. In fact, these differences in responses would be due to systematic lower quality responses of younger and less educated teachers.
The first stage of the regression analysis explores the first hypothesis, modeling whether or not teachers with certain characteristics were more prone to choose the web instrument than the paper instrument. From this regression, an instrumental web-based choice variable, henceforth referred to as the instrumental variable (IV) web choice, is estimated. To test the second hypothesis, the IV web-choice is used in the second stage of the model to determine whether the sole use of the web-based or the paper-based survey instrument had an effect on the type and quality of the survey responses. The absence of statistically significant coefficients would suggest no mode effects in the 2004–05 TFS.
The sample for the regression analyses is limited to teachers who received the internet option only, excluding teachers who only were sent the paper questionnaire. By analyzing only those who were randomly assigned to receive the web-instrument, the two main groups of analysis (i.e., web-respondents and paper-respondents) will share the same sampling experimental conditions (i.e., having received a web-based questionnaire in addition to having the possibility of opting for a paper-based questionnaire). Former and current teachers are analyzed separately. Exhibit 1 summarizes the specific regression models used for each dependent variable, the dependent variables, and the unweighted sample sizes.
Findings
Regression analyses tested the first and second hypotheses indicating non-rejection of both hypotheses. These findings indicate that teachers’ characteristics affect their decisions to opt for using a paper over a web-based survey instrument. In addition, responding to the survey using the paper-based instrument over the web-based instrument does not lead to significant systematic differences in the type and quality of the teachers’ responses.
The first stage of the regression model indicates that teachers’ age, education, teaching experience, not knowing about the paper option, and access to the internet had an effect on their decision to respond to the web instrument. Younger teachers (i.e., those 18–29 and 30–49 among former teachers and those 18–29, 30–49, and 50–64 years old among current teachers) were more likely to choose the web questionnaire when compared to those 65 and older. Among former teachers, more educated teachers (i.e., teachers with Bachelor’s as highest degree and those with Master’s as highest) were more likely to respond to the web survey, compared to those with less than a Bachelor’s degree as their highest level of education.
Also, former teachers with zero to three years of experience were less likely to respond to the web survey. Knowing about the paper option had a positive effect on the web-choice among current teachers. Access to the internet had a significant and positive effect on using the web instrument among former teachers.
The second stage of the regression model suggests the absence of mode effects on the estimates. Only the estimated IV web-choice coefficients, their standard errors, and F tests are presented in table 2, to be concise. Results reveal that none of the estimated IV web-choice coefficients are statistically significant. These findings indicate that using the web-instrument to respond to the 2004–05 TFS did not affect teachers’ responses compared to those of teachers responding to the paper-based questionnaire. F tests provide information on the overall fit of the estimated model. Regression estimations show that F tests are statistically significant in five of the six former teacher regressions and statistically significant in four of the six current teacher regressions.
Conclusions
Unique to this study is the use of an instrumental variable regression model technique, which is critical for differentiating the two separate components of the survey response process. Findings indicate that even though teachers’ characteristics determine their choice of paper over web instruments, the type and quality of the responses collected using the 2004–05 TFS web-based instrument do not differ from those collected using the paper instrument. These results lead to the conclusion that no mode effects are observed for the 2004–05 TFS. As a result, the use of web surveys in future TFS administrations would not bias the consistency of estimates over time.
Nevertheless, caution should be used when interpreting these findings. The non-significant overall F tests indicate that the independent variables included in the regressions do not fully explain the model significantly, increasing the potential error of the estimations. One explanation for this result is that, for comparison and consistency purposes, all dependent variable regressions use the same group of explanatory variables. As a consequence, unobserved error could be affecting the actual explanatory power of some regression models. More ad-hoc regression models could improve the significance of these F tests. Using more ad-hoc models goes beyond the scope of this investigation.
Additionally, web respondents represent a small proportion of all sampled teachers who received the internet questionnaire in the 2004–05 TFS. Based on unweighted counts, web respondents represent 22 and 21 percent of the total sample of former and current teachers respectively who received the internet questionnaire (359 out of 1,601 former teachers and 651 out of 3,041 current teachers). As web-based instrument response rates increase, there is a chance, though small, that results might vary.
Outcomes also suggest that as the use of web-based instruments rapidly increases, certain groups would need special attention to avoid significant issues of low response rates or sample bias. Finally, the sample utilized in this study comes from a very specific adult, more educated, and potentially more computer and internet literate population than the average individual. These results should not be generalized to the general population.
Acknowledgment
The authors would like to acknowledge the National Center for Education Statistics, whose support made this work possible.