The Smart(Phone) Way to Collect Survey Data

The use of mobile technologies has grown immensely over the past decade. This increase has not gone unnoticed by survey researchers who see mobile technology as a new way to engage with potential respondents. However, survey design for mobile technologies has imported many features directly from web- and paper-based methodologies without considering the potential impact on data quality and the respondent experience.

Traditionally, when new modes of survey data collection are introduced, they borrow best practices in design from similar existing modes before rigorous testing on the new mode has been conducted (Peytchev and Hill 2010). This is where the current state of mobile survey design finds itself with very little information about what best practices are for this emerging survey platform.

This article addresses the need for more research on data collection with mobile devices by presenting results from two separate studies designed to uncover best practices for mobile technology surveys. The focus of these experiments was on two key topics: what type of survey inputs work best on mobile technology and what is the appropriate length of a survey administered on a mobile device.

Study Design

Both studies were conducted in 2010 using respondents taking a customer satisfaction survey for one of two large, multi-unit restaurant brands. Respondents could choose to take the survey through mobile technology or through a traditional computer. The first study examined the impact on data quality of employing horizontal radio buttons in surveys taken with mobile technologies. Based on the findings from this study, a second study was conducted to begin developing best practices in mobile survey design.

Use of Horizontal Radio Buttons in Mobile Surveys

The most obvious point of differentiation between mobile technology and traditional desktop or laptop computers is the screen size. Even the largest mobile phone screen available in the United States is considerably smaller than even the smallest computer monitor. These smaller screens on mobile phones can impact multiple features of survey design. Response options are closer together, which may cause respondents to unknowingly select the wrong response option (Jones et al. 1999; Parush and Yuviler-Gavish 2004). Also, when a survey employs horizontal radio button inputs, not every response option will be visible on a mobile screen. This could lead to an increase in satisficing as the task difficulty of responding accurately increases. Although Peytchev and Hill (2010) found respondents were willing to navigate to response options that were not initially visible, they also reported that 23 percent of respondents indicated they did not notice there were additional response options or that it was too much of an effort to scroll to see them.

If respondents are more likely to select the response option that is immediately visible, we should see more mobile respondents selecting the very first response option in the horizontal radio button list when not all options are on screen. To test this hypothesis, respondents were assigned to one of two test conditions. The first test condition listed a satisfaction scale with the positive scale point, Highly Satisfied, first. The second test condition listed the same satisfaction scale with the negative scale point, Highly Dissatisfied, first.

Results of this initial test are displayed in Figure 1 and show that when respondents are taking surveys with mobile technology, they are more likely to select response options that are immediately visible than response options that are not. When the Highly Satisfied scale point is displayed first, respondents using mobile technology are more likely to select the Highly Satisfied scale point (81.7 percent) versus computer respondents (78.8 percent). Conversely, when the negative scale point is displayed first, respondents using mobile technology were less likely to select the Highly Satisfied scale point (70.1 percent) versus computer respondents (73.1 percent). The interaction effect between the scale order and the survey mode is significant at p < 0.05, which supports the hypothesis that respondents using mobile technology are more likely to select the visible scale points with a horizontal radio button input. While the differences are not extremely large, these results do show that the bias introduced by not having all scale points visible for mobile respondents can significantly impact survey responses.

Figure 1  Impact of horizontal radio buttons on responses.

Developing Best Practices for Mobile Survey Design

The second study was conducted to begin developing a more comprehensive understanding of best practices in mobile survey design. Best practices focused on three key areas, although only two will be discussed in this article:

1. Input Types: Vertical Radio Buttons or Dropdown Buttons
2. Survey Duration: Same Amount of Questions as Computer Survey or Fewer
3. Question Paging: Same Amount of Questions per Page as Computer or More per Page (not discussed)

Study Design

Respondents were randomly assigned to one of four test scenarios with approximately 2000 completes per test scenario:

1. Vertical Radio Buttons, Full Survey Content (average 44 questions), and Computer Paging
2. Dropdown Buttons, Full Survey Content (average 44 questions), and Reduced Paging
3. Vertical Radio Buttons, Reduced Survey Content (average 26 questions), and Reduced Paging
4. Dropdown Buttons, Reduced Survey Content (average 26 questions), and Computer Paging

For this test, respondents saw a survey optimized for the platform they used. This included stripping out unnecessary graphics and pictures for respondents using mobile technology to allow pages to load more quickly for respondents.

Input Types

Based on the findings from the first experiment, both tested input types are vertically oriented to remove the bias of horizontal radio buttons. The limited research on scale orientation suggests vertical and horizontal radio buttons produce similar results, meaning mobile survey data could be merged with computer survey data without concerns over potential mode effects (Thomas, Uldall, and Krosnick 2004). This leads to the first hypothesis tested:

Hypothesis 1: The percentage of questions rated Highly Satisfied will be the same for mobile and computer respondents.

Survey Duration

Preliminary research into mobile survey abandonment rates shows they are significantly higher for mobile respondents than computer respondents (Callegaro and Macer 2011). This is likely due to increased task difficulty when interacting with smaller screens (Hardwick, Pulido, and Adelson 2007). In addition, mobile surveys that are not optimized for mobile phones contain unnecessary graphics that increase page load time. This leads to the next hypothesis:

Hypothesis 2: Reducing the number of questions on a mobile survey will reduce the higher abandonment rates associated with mobile surveys relative to computer surveys.

Results and Conclusions

The first hypothesis tested was that the percentage of respondents using the Highly Satisfied scale point would be more similar between the mobile and the computer surveys than using horizontal radio buttons. The percentage of Highly Satisfied responses given by respondents on 5-point fully labeled satisfaction scale questions were compared across the four test scenarios. As shown in Figure 2, the first test scenario produced results that were most similar to those of computer respondents, with the average percentage of Highly Satisfied selections being 63.5 percent on mobile vs. 61.5 percent on computer. Although the smallest, this difference was statistically significant at a 95 percent confidence level. The other three test scenarios had significantly higher percentages of respondents selecting the Highly Satisfied scale point.

Figure 2  Use of highly satisfied scale point by mobile test group.

However, there are demographic differences between respondents who use mobile technology and those who do not. To control for these differences, a linear regression model was built to predict the percentage of Highly Satisfied responses on each survey. These results are displayed in Table 1. Results suggest that, after controlling for demographics, the test scenarios with vertical radio buttons showed no significant differences in the percentage of Highly Satisfied responses from computer respondents. Mobile surveys with dropdown inputs had a higher percentage of Highly Satisfied responses than their computer counterparts.

The goal of the second test in this study was to examine the impact of questionnaire length on abandonment rates. Two of the four test groups had reduced content, which asked 26 questions, versus the full-length content for the other two test groups and computer respondents, which asked 44 questions. Results for the second test are displayed in Figure 3 which shows the abandonment rates for each of the four test groups, as well as the initial abandonment rate for the unoptimized version of the mobile survey. By optimizing the survey for mobile platforms, the abandonment rate declined from 15.9 percent for the unoptimized survey to between 6.3 percent and 10.2 percent for optimized surveys with different input formats. Abandonment rates were the lowest in the optimized mobile survey with reduced content, single paging, and vertical radio buttons. However, computer respondents still abandoned the survey less frequently than all test groups at 3.3 percent. This shows that in order to match abandonment rates across platforms, even more questions need to be cut from mobile surveys. Given the cost and difficulty in acquiring survey respondents, researchers need to consider if the additional abandonment rates associated with longer mobile surveys can be tolerated. However, one thing that should always be considered when designing survey questionnaires is the trade-off between potential abandonment rates and the information that is required from your respondents. Longer surveys will almost always results in higher abandonment rates, so it is up to the individual researcher to decide where he or she is comfortable.

Figure 3  Abandonment rate comparing computer respondents with mobile respondents.

Conclusion and Recommendations

The recommendations from these studies are to remove horizontal radio buttons from all mobile surveys because respondents will choose the visible options significantly more than those off screen. Vertical radio buttons are the recommended input type for all questions on mobile devices. This input type leads to less biased data and is displayed consistently on mobile devices, unlike dropdown inputs, which are handled differently by different mobile devices. Respondents are less likely to tolerate long surveys on mobile technologies, so researchers need to weigh costs associated with acquiring survey respondents.

As more survey respondents are “voting with their feet” by taking more surveys on their mobile devices, survey methodologists must start optimizing their online surveys for these emerging platforms. By not optimizing for mobile technology, surveys taken on mobile devices will be biased and not comparable to surveys taken on traditional computer platforms.