Introduction
In the era of big data, survey researchers are exploring new means to gather information. An example of such means is data collection through wearable devices such as smartphones and smartwatches. These devices can, for example, be used for data collection through passive measurement of location and movements or for active capturing of images, audio, or video. Another example of new types of data collection is the possibility to merge different types of register databases with survey data. These techniques are not only expected to generate objective data, but they are also thought to reduce respondent burden. However, little research has been done to find out how respondents think about being engaged in such new ways of research.
At the same time, researchers should not forget the more traditional modes, such as mail and telephone interviews. The willingness to participate in such modes has been studied widely before, but society keeps changing, as do preferences for research modes. This study investigates the willingness to participate in face-to-face surveys, mail surveys (paper and pencil), telephone interviews, web surveys, app based surveys, passive smartphone and smartwatch data tracking, and research using register data. Our focus lies in the implementation of different data collection techniques as additional tasks for online panelists. We want to know how willing online respondents are to carry out such tasks and whether this is influenced by respondent characteristics or the research topic.
Background
Different data collection modes may differentially affect nonresponse bias. Whereas one person finds it more attractive to fill out a paper questionnaire, another person finds a personal interview less burdensome.
Earlier research has shown that responses among demographic subgroups may differ by mode. De Leeuw (1992) compared mail, face-to-face, and telephone interviews. She investigated whether respondents differed in background variables gender, age, education, marital status, and having children. The modes differed by gender and marital status. For example, men were more present in the mail condition compared to the face-to-face interview mode, in which more women were present. Compared to the population distributions, women and divorcees were overrepresented for the face-to-face interviews. None of the modes differed in age or education.
Many studies have found that young adults prefer to respond via web while older individuals prefer non-web modes. For example, Millar et al. (2009) found that respondents to mail surveys were older than web respondents, had lower levels of education and lower incomes. Mail respondents were also more often female and married. Diment and Garrett-Jones (2007) report web respondents to be more likely “young, male, middle ranking, and working in information technology–related sectors,” when given a choice between web- or paper-based versions of a questionnaire. Smyth et al. (2014) found the young to prefer the web mode, and older individuals to be more likely to prefer interviewer-administered modes over the self-administered modes of mail and web. In one of the two studies in their research, women preferred mail over interviewer-administered modes.
Regarding newer techniques, Revilla et al. (2018b) studied the willingness of online respondents to participate in additional tasks such as Global Positioning System capture; installing a research app; or collecting passive tracking data, accelerometer data, or biomarkers. The researchers provided an overview of earlier literature on these topics. The stated willingness varied greatly, from 72% to 9%, but they found that people are more likely to participate in tasks where they have control over the reporting of the results. The main reason not to participate was due to privacy concerns. They also found diverse attitudinal variables, such as sharing: the more frequently respondents reported posting on Facebook, the more willing they were to accept a variety of additional research tasks, safety: greater trust in the anonymity of their data means they were more willing to accept additional tasks, and attitude toward surveys: those who liked answering the survey showed higher levels of willingness. Of the demographic variables that were included (age, gender, and education), the willingness increased (overall) if the respondent was male or had lower education. Interestingly, age was not a significant predictor.
In another study where they investigated the willingness to use voice input in web surveys, age had a marginally significant effect but related behavioral factors (such as general use of voice input) were the most robust predictors (Revilla, Couper, and Ochoa 2018a).
In an earlier study, Revilla et al. (2017) investigated the use of tracking applications which registered respondents’ online behavior (i.e. URLs, use of apps, time and length of visit, ad exposure). Compared to the general population (in Spain), the net sample was underrepresented for the oldest age group as of 65–74 and middle/lower social class. People aged 25–43, people with higher education and people living in cities were overrepresented. Gender distribution did not deviate from the general population.
Pinter (2015) found only a weak effect by demographic variables in a Hungarian online panel, in which the respondents were asked to download a research app. About 42% of smartphone owners were willing to download the app. General familiarity with downloading apps had the largest impact on the willingness rate, while respondents’ (young) age only had a weak effect, and gender, education and settlement type, such as living in a city, had no connection with the willingness rate.
In this study, we aim to add to the existing literature by investigating the willingness of online panelists to carry out tasks in different data collection modes. By examining both traditional and new data collection modes, we aim to contribute to a more complete picture of the current willingness to participate in research.
Method
Online surveys have become one of the most used survey modes. However, some studies require a mixed-mode approach. In this study, we investigate the stated willingness to participate in different data collection modes in addition to online surveys. Our target population is an online probability-based panel.
We administered a survey on the attitudes toward different ways of data collection among the Dutch CentERpanel[1] This panel was established in 1991 and was originally based on a random sample of addresses drawn from the pool of national postal delivery addresses. The panel comprises over 2,000 households, forming a representative cross-section of the Dutch speaking population. The respondents participate through online surveys, but prior internet access or computer usage is not necessary in order to participate. If a recruited household does not have a computer with an Internet connection, it is provided with a so-called SimPC and/or Internet connection to be able to complete the questionnaires.
Since the survey was conducted in an online panel, the sample is naturally biased toward preferring online surveys above other modes. In fact, we assume a ‘status quo bias’ (Samuelson and Zeckhauser 1988) and do not attempt to make a fair comparison between the online and other modes. The results reveal, however, which alternative modes are relatively attractive to online respondents and which less so.
The questionnaire was fielded in week 31 (August 5–8) of 2016. It was presented to all panel members aged 16 years and older, at that moment 2,505 panelists, of which 1.821 persons fully completed the survey. The response rate was 73%.
The questionnaire included questions about the likeliness to participate in research using different data collection techniques in general, and likeliness to participate when a specific research topic was provided. The order of the techniques and topics was randomized.
During the data analysis we identified 40 cases (2.2%) which committed to straightlining, giving the same answer within all the three analyzed item batteries, respectively. These respondents were removed from the data. The analyses were conducted based on the remaining 1,781 responses.
Results
We asked the respondents how likely they would be to participate in a research study when it was conducted either through the means of an online survey, paper survey via regular mail, telephone interview, personal interview, via an app installed on a smartphone or tablet, or automatic data collection via a smartphone or smartwatch. Willingness to participate was also asked for research in which external register data, such as the community register, are appended to the respondents’ answers. The respondents could provide their answer on a scale from 1 = very unlikely to 7 = very likely. The item order was randomized.
A one way repeated measures analysis of variance, with Greenhouse-Geisser correction, showed that there is a statistically significant difference in the willingness to participate per data collection mode (F(4.492, 7995.386) = 413.998, p < 0.001). Post hoc pairwise comparisons with Bonferroni correction shows us each of the pairs differ significantly from each other in willingness. The exception to this is the pair linkage to register data and passive data collection through a smartwatch. These two modes are equally unpopular ways of alternative data collection.
The panel sample proved to have a relatively high willingness to participate in self-administered modes (Figure 1). The traditional paper and pencil interviews scored relatively high (M = 4.15, SD = 2.10). Also, app-based surveys on smartphones or tablets were attractive to many (M = 3.62, SD = 2.29). However, the respondents were still more likely to participate in personal (M = 3.00, SD = 1.97) or telephone interviews (M = 3.22, SD = 1.94) than in studies where they would be asked to take pictures (M = 2.63, SD = 1.85) or record sound (M = 2.37, SD = 1.74) with their smartphone. Passive data tracking via smartphones (M = 2.20, SD = 1.69) or smartwatches (M = 1.98, SD = 1.60) or linkage to register data (M = 1.96, SD = 1.48) seem to be the least popular data collection modes.
Best fit data collection mode per respondent type
We further investigated whether different demographic subgroups hold different likelihoods to participate in data collection modes. When comparing the willingness to participate by gender (Figure 2), each sex shows the same pattern as the total sample. But what stands out is that women (M = 4.39, SD = 2.01) are somewhat more likely to participate in mail surveys than men (M = 3.93, SD = 2.06). Men (M = 3.10, SD = 1.99) are, on the other hand, slightly more willing to participate in face-to-face surveys than women (M = 2.88, SD = 1.94. Also, the acceptance for passive data capture via smartwatch is a little higher among men (M = 2.08, SD = 1.68) than women (M = 1.88, SD = 1.50) as well as for register data (men: M = 2.08, SD = 1.56; women: M = 1.82, SD = 1.37). Yet, compared to other modes, these are still the least popular.
A mixed between-within subjects analysis of variance with Greenhouse-Geisser correction shows us there is a statistically significant interaction effect between gender and willingness to participate in the different data collection modes (F(4.486, 7981.274) = 8.817, p < 0.001). This tells us willingness to participate in different data collection modes depends on whether you ask men or women, which can be observed in Figure 2.
After Bonferroni correction, these gender differences are only significant for the mode where women (M = 4.39, SD = 2.01) are more likely to participate in paper and pencil mail surveys than men (M = 3.93, SD = 2.06), t (1779) = -4.72, p < .001 and the mode where men (M = 2.08, SD = 1.56) are more willing to linkage to register data than women (M = 1.82, SD = 1.37), t (1778.8) = 3.76, p < .001.
Willingness to participate in different modes differs clearly per age group. Among the younger age groups, up to the age of 44, app-based research is the most popular of the compared modes. The results are given in Figure 3, where also the online survey mode is shown as a reference. The willingness to participate in app-based research starts to decline as of the age group of 45–54 years old. The same applies to other forms of data collection via smartphone or smartwatch that were included in the study. As of this age group (45–54 years), paper and pencil survey is the most popular mode of data collection.
The youngest respondents are more likely to respond to telephone interviews than to face-to-face interviews, a difference that declines by age and reverses as of the age of 65. The older the respondents, the more likely they are to participate in in-person surveys.
A mixed between-within analysis with Greenhouse-Geisser correction shows us there is a significant interaction effect between age and willingness to participate in different data collection modes (F(27.951, 8264.143) = 14.683, p < 0.001). Willingness to participate to a data collection mode depends highly on which age group you ask, as shown in Figure 3.
We also find a small effect by education level: the higher the education, the higher the overall willingness to participate in research. In Figure 4, this is shown by the linear trend line (Linear, Mean of all modes), which represents the mean of all modes (r = .119, n = 1781, p < .001). Two modes with interesting results are featured in Figure 4. What stands out is that the high willingness to participate in paper and pencil mode holds for all levels of education. Furthermore, it is interesting to see there seems to be a positive correlation between level of education and willingness to participate in app-based surveys (r = .193, n = 1,781, p < .001).
Willingness by research topic
The willingness to participate using different modes was also investigated per research topic. We included a number of research topics (see Figure 5) for which one could theoretically collect the data either through surveys (mode undefined), passive data capture via smartphones, or using register data.
We find a similar effect of the research topics for all three data collection techniques. It stands out that the respondents are relatively unwilling to participate in surveys about their spending habits. A similar but smaller effect is identified in research on the respondent’s personality type. The more general topics, such as the level of noise in living areas, use of public transport, or choice of supermarket gain a higher participation rate.
Conclusions
Overall, this research indicates that people in online panels are relatively likely to participate in self-administered modes such as paper and pencil questionnaires and surveys through an app. However, they find traditional modes such as personal interviews or telephone interviews more attractive than passive data tracking via smartphones or smartwatches, or linkage to register data. In line with earlier results e.g. (Revilla et al. 2017), it is still a challenge to collect data via new data collection techniques such as data tracking, when a representative sample is aimed at.
Respondents in this research are used to filling out web surveys on a regular basis. This may influence their willingness to participate in other self-administered modes due to mode preference or, as discussed in the “Method” section, due to a ‘status quo bias’ (Samuelson and Zeckhauser 1988). Therefore, the results from this research can be applied to other online panelists, but potential interaction effects should be taken into account when applying the results to different types of populations.
We found systematic differences in the willingness to participate by demographic characteristics. While we could find some influence by sex and education, age came out as the most determining background characteristic for participation. Around the age of 50, the willingness to participate in app-based surveys and research using new techniques via smartphones or smartwatches, starts to decline. The older the respondents, the more likely they are to participate in personal interviews.
Finally, our research suggests that people in online panels are less willing to participate in research about personal, sensitive topics, no matter which data collection technique is used. The more intrusive the topic, the more intrusive they regard the research, even if their personal burden would be lower.
Contact information:
Joris Mulder
CentERdata – Tilburg University
Room I 309 (Intermezzo Building)
PO Box 90153, 5000 LE Tilburg,
The Netherlands
E-mail: j.b.mulder@uvt.nl
Phone: +31-13-466 3046 / 8325
For more information about the CentERpanel, see Teppa and Vis (2012).