A Pew Internet and American Life Project report indicated 42 percent of American adult cell phone users own smartphones (Smith 2011). Also, 25 percent of these individuals go online mostly from their phones, rather than using a computer. As more people adopt smartphones and rely on them for Internet access, Web surveyors must consider how their use affects the survey response process.
Research has demonstrated methods for increasing response to Web surveys. Millar and Dillman (2011) found that combining postal mail contacts (with a cash incentive) with follow-up e-mail messages containing a link to the survey website resulted in higher response rates than using only one contact mode. This technique, called “e-mail augmentation,” produced response rates more than double those achieved using only e-mails (without the incentive). This highlights the appeal of using postal and e-mail messages in Web survey implementation.
However, the value of e-mail messages for easing the response task and encouraging immediate response may change as more people rely on phones for Internet access. Individuals who receive survey e-mails on their smartphones must be able to respond with their phones, or else they may delete the message before having the opportunity to respond on a computer. Therefore, it is clear we must find ways to persuade people to respond to surveys via smartphones.
This paper presents an experiment designed to build upon the e-mail augmentation strategy by delivering messages encouraging response using smartphones. We compare the overall response rate as well as the proportion of responses received via smartphones when using this strategy to those of two other treatments that used e-mail augmentation but did not mention the mobile device option. Our goal was to learn whether it is possible to drive respondents not only to the Web but also to the smartphone, as an alternative to desktops and laptops.
Background and Research Questions
Surveyors’ interest in the possibilities of mobile data collection is rapidly increasing. The 2011 and 2012 meetings of the American Association for Public Opinion Research offered short courses and other sessions dedicated to a variety of topics relating to mobile devices. Also, a recent Survey Practice article by Buskirk and Andrus (2012) outlined the advantages and disadvantages of several approaches to implementing surveys for smartphones. Thus far, there has not been much success in persuading a large proportion of respondents to complete traditional web surveys via their phones (Callegaro and Macer 2011). Several studies of smartphone surveying suggest typical survey websites create a variety of usability and visual problems for those who attempt to respond via mobile devices (Callegaro 2010; Peytchev and Hill 2010). This suggests there are still many obstacles that need to be addressed.
Using Callegaro and Macer’s suggestions for adapting Web questionnaires for mobile devices, we designed a smartphone-friendly website, in addition to a standard Web questionnaire for a study conducted on a sample of college undergraduates, a population in which smartphone use is relatively common. By highlighting the presence of a smartphone response option and urging smartphone use, we aimed to encourage a greater number of students to not only participate in the survey, but also to respond via mobile devices.
The study was conducted in 2011 on a random sample of 1800 undergraduate students from the main campus of Washington State University (WSU). This sample consisted of three treatment groups of 600 students. The first treatment, a web control group, asked students to respond to an online questionnaire. The second treatment also requested Web response, but each correspondence sent to this group contained a prominent message that highlighted and encouraged the option of responding via phones using the smartphone-optimized website. The third treatment group offered a choice of responding to the Web questionnaire or filling out a paper copy. Anyone who accessed the survey website could opt for either the standard or the smartphone-optimized questionnaire, but only the second treatment group’s messages mentioned this option.
The questionnaire contained 20 items to gauge students’ experiences and satisfaction with WSU. We designed a paper questionnaire and two online questionnaires - one for use with desktop/laptop computers, and another compatible with smaller smartphone displays and mobile browsers. We attempted to standardize the appearance of the paper and online versions to reduce measurement differences across modes. However, the mobile Web questionnaire was simpler in appearance in order to work with the small touch screen displays of most smartphones.
Following the implementation strategies discussed by Millar and Dillman (2011), all treatment groups used five contacts and a $2 cash incentive. We alternated between postal letters and e-mail messages to promote high response rates and make it easier for students to respond. The final postal contact for the two Web-only groups provided the option of completing a paper questionnaire (an option available to the third treatment throughout all contacts).
The messages for the smartphone-emphasis treatment differed from the other treatments by informing students of the smartphone-optimized website and encouraging those with smartphones to use them. For example, the initial letter contained the following message: “Do you have a smartphone or other mobile device? We have designed the survey so it can be answered either on mobile or standard browsers. If you prefer to use a smartphone, we encourage you to do so.”
To assess whether this treatment received a greater proportion of smartphone responses, we collected user agent strings for each individual who accessed the survey website. This allowed us to determine which type of Internet browser and device each respondent used. We calculated response rates and the percent of responses obtained via smartphones for each treatment group.
Table 1 displays the survey response rates by treatment group. The smartphone-emphasis treatment achieved a similar response rate to the standard Web group (49.7 and 50.0 percent, respectively). Thus, encouraging the smartphone option did not persuade a greater number of students to respond. The third group (offering a choice of Web or mail) received the highest overall response rate. This is consistent with prior research that showed the e-mail augmentation strategy increases response when offering a choice of mail or Web (Millar and Dillman 2011).
Paradata collected on each Web respondent revealed that there were only 40 students in the entire study who responded using a smartphone. This equates to 4.3 percent of all respondents, or 5.5 percent of Web respondents. Table 2 displays the percentage of smartphone Web responses for each treatment group.
The percentage of Web responses delivered via smartphones for treatment two was significantly higher than the proportion of smartphone responses to treatment one (7.0 vs. 1.9 percent). This suggests our efforts to encourage students to use their phones were somewhat successful. However, this increase was still relatively low. Also, it is of interest that the third treatment also received a significantly larger proportion of smartphone responders than the first (7.8 vs. 1.9 percent), since it did not mention the option of responding via a mobile device. The difference in the proportions of smartphone responses between groups two and three was not statistically significant.
The questionnaire asked respondents what type of cell phone, if any, they have, as well as what method they prefer for responding to surveys. These items provide some clues as to why the choice treatment received a similar proportion of smartphone responders as the smartphone treatment. The third treatment did not contain a higher number of respondents who owned smartphones. However, for both the smartphone and the choice groups, a significantly higher percentage of respondents (compared to the Web control group) reported they prefer to respond to surveys using smartphones. Thus, students’ preferences for using/not using their phones to complete Web questionnaires may matter more than our encouragements in determining whether respondents will use their smartphones. It is also possible that the choice group’s higher smartphone response rate was the result of other features of this treatment. In particular, the presence of a paper questionnaire may have helped to illustrate the simplicity of the task and thereby encouraged more individuals to attempt to respond via their phones when they received the follow-up e-mail message.
This paper has shown that encouraging respondents to use smartphones only slightly increases the proportion of responses delivered via mobile devices. Although it may be possible to push some respondents to use smartphones, more work is needed to improve upon these results. We obtained more mobile responses when encouraging smartphone response, but the overall number of smartphone responses we received was still rather low. This suggests that relying on the standard approaches to improving web response do not necessarily translate to encouraging response via smartphones. Indeed, some argue that we need to consider more ways of enticing respondents to use their phones (Link and Buskirk 2011). On the other hand, the results indicated that offering and encouraging mobile response did not have a negative impact on the overall response rate (relative to a standard web treatment), a tendency observed in other research when a choice of two different modes is offered (Millar and Dillman 2011).
The questionnaire data indicated that 46.4 percent of all respondents reported owning a smartphone. Clearly, a majority of students who could have used a smartphone to respond did not, even when we advertised the mobile-friendly website and encouraged smartphone response. In group two, 45.5 percent of Web responders reported owing a smartphone, but only 7.0 percent used them. Students’ preferences for how to respond to surveys provide some additional insight. Only 6.6 percent of all respondents indicated they prefer to complete surveys using their smartphones. Cognitive interviews may be helpful for determining why people do not want to complete surveys on their phones.
This experiment and the preference data gathered from the questionnaire suggest individuals may not yet be ready to transition to mobile survey response, at least for this type of survey. However, the Pew study on smartphone usage implies that this may change as more people replace traditional computers with phones for accessing the Internet. If and when this change occurs, we may need to cater to smartphone users to maintain survey response rates. However, we then must consider the implications that mobile responses will have for Web surveys’ data quality. For example, other researchers are beginning to explore if smartphone use increases break-offs or item nonresponse rates. If smartphone users provide lower quality data, this must be taken into account when encouraging mobile device use.
This research was conducted by Washington State University’s Social and Economic Sciences Research Center with support from the U.S. Department of Agriculture - National Agricultural Statistics Service and the National Science Foundation - National Center for Science and Engineering Statistics under cooperative agreement number 43-3AEU-5-80039 (Don A. Dillman, Principal Investigator). Morgan M. Millar is a Postdoctoral Research Associate in the Social and Economic Sciences Research at Washington State University. Don A. Dillman is Regents’ Professor and Deputy Director in the Social and Economic Sciences Research Center at Washington State University. This paper is a revised version of a presentation given at the 2012 annual meeting of the American Association for Public Opinion Research in Orlando, FL. The authors would like to acknowledge with thanks the contributions of Thom Allen, Nikolay Ponomarev, Bianca Garza, and other SESRC staff members in the design and implementation of this experiment.