Matching Data Collection Method to Purpose: In the Moment Data Collection with Mobile Devices for Occasioned Based Analysis

Introduction

Helping clients understand and influence consumer buying behavior is one primary focus of today’s market researchers. Occasion-based marketing uniquely accomplishes this goal by finding specific occasions tied to the consumer’s habits and needs. Clients can then create products to address the needs of specific occasions. In one example, Kraft was able to develop new products that would better meet specific consumer occasions in retail stores (Hartman Group and Kraft Foods 2010). One significant obstacle to overcome in the data collection process for occasion-based marketing involves the respondent’s natural memory processes in accurately reporting his or her own retrospective behavior at different occasions (Stone 2000). Although there are many approaches to this obstacle including personal interviews, behavioral observation, and field trials, cost restraints and feasibility call for a more effective self-reporting methodology to fuel occasion-based marketing (McDonald 2008).

Luckily, new data collection techniques allow us to overcome this obstacle by collecting data in the moment of each occasion. Mobile devices have geolocating apps with around-the-clock instant internet access to submit survey data, thus allowing us to capture consumer actions, thoughts and motivations as they are happening (Bhaskaran 2011). This capability matches the exact need occasion-based marketing has: overcoming potentially faulty memories. For innovators, combing in-the-moment mobile data collection with occasion-based sampling could provide more realistic and detailed data on consumer behaviors and motivations, leading to better products and marketing.

Methodology

Olivetree Research and SSI conducted a test to explore the feasibility of mobile data collection in occasion-based research. To avoid selection bias only panelists who owned and reported using a mobile web and SMS-capable smartphone were allowed to participate in the study. Additionally, all panelists had previously agreed to be contacted for surveys using mobile technology as the access point. We randomly assigned 400 panelists to one of the four following test groups:

1. In the moment (ITM) short message system (SMS)
2. In the moment (ITM) mobile web (smartphone or other mobile web device)
3. Retrospective end of day (EOD) mobile web
4. Retrospective end of day (EOD) traditional web

The topic of the study was snacking-specifically when respondents thought about having a snack, whether or not they actually ate the snack, and what they considered or actually consumed. The survey only consisted of three questions. Due to character limits on SMS texts we used minimal question text and solicited open-ended data which was later coded into product categories. Each day all participants received an email reminding them to participate in their assigned role throughout the day. Both ITM groups were asked to take the survey in their respective modes every time they considered having a snack. Both EOD groups were asked to take a single survey at the end of each day through their respective modes. Following the week of data collection, all participants were asked to participate in an exit survey assessing the participant experience and self-reported accuracy rating for their responses.

Results

The participation rates for each of the groups are shown in Figure 1. Almost all of the differences here are significant at the 5 percent level (see Appendix A). Over half of the respondents participated for at least one day in the seven-day data collection phase. Unsurprisingly, the EOD web group had the highest rate of completion as it mimicked the familiar format of surveys that these panelists had already experienced. The mobile and SMS modes of contacts although available would be less familiar as part of their previous panel experience. In the exit survey, both the ITM and the EOD mobile groups (the ones with the lowest participation rates) primarily cited technical difficulties with the phone or their phone not being accessible at the time they had a snack. The SMS mobile participants were less likely to cite technical or availability problems with their phone and had a corresponding higher participation rate. Lack of interest in the survey was not an issue as over 85 percent of respondents in each groups agreed that the survey was very or somewhat interesting.

Figure 1  Days of dairy participation by group.

While the ITM groups had slightly lower participation rates, the drop in participation might be offset by more accurate data. While we were not able to verify against their actual behavior, all participants were asked to self-report if the study “fully captured all the times you had a snack or felt like a snack” (Figure 2). Interestingly, the ITM groups self-reported lower accuracy rates (63 percent mobile and 79 percent SMS) compared to the EOD groups (94 percent mobile and 93 percent web). All the differences were statistically significant expect between the two EOD groups (see Appendix A). Some might argue that this is a participation effect and that the ITM groups weeded out those who are likely to be more accurate, but even with the SMS ITM group with had over a 80 percent recontact rate we see lower accuracy results than we see in the mobile EOD group which had a much lower recontact rate.

Figure 2  Self-reported accuracy of dairy results by group.

We also found that the method of reporting (ITM versus EOD) actually changed the respondent’s behavior (Figures 3 and 4). While only 40 percent stated that the survey actually changed their thinking or habits, the mode effects were very important. Using an ITM format seems more likely to decrease their normal snacking behavior while the EOD format seems more likely to increase or not change their snacking behavior (p=0.04 See Appendix A). This finding matches those documented by weight loss experts who cite that simply tracking food intake “encourages people to consume fewer calories” (Paddock 2008).

Figure 3  Survey effect on number of times they thought about snacking.

Figure 4  Survey effect on actual snacking habits.

Implications

In the moment data collection has a large potential to change the way occasion-based marketing is done. High participation rates and high levels of interest in the survey from the ITM groups indicate that in the moment data collection is not only feasible, but desirable. However, there are still barriers to full adoption. ITM respondents needed to insert a new action (completing a survey) into their daily habits (having or thinking of snacks). This disruption to their normal routine could be hard to implement as indicated by the lower self-reported accuracy scores. Because this study did not have information on their actual snacking behavior, we could not definitely answer which mode of data collection is more accurate. It could be that the ITM respondents were not able to disrupt their routine to take a survey while snacking (for example eating in a car). There was no mechanism for a respondent to correct for missing occasions like there was in the EOD modes. However, the ITM respondents could also potentially be more aware of missing data than the EOD respondents as they had to pay attention throughout the whole day of what snacks they had. Further research is needed in this area. We would also suggest a more robust approach of a hybrid between the ITM and EOD modes. If you added an EOD feedback mechanism to allow respondents to report times when they missed reporting ITM you could potentially get the best of both worlds: occasion-specific data with no memory obstacle and at least the standard EOD diary data when it was not convenient to provide the survey responses in the moment.

While ITM data might provide important insights into consumer behavior, it could also potentially skew their “normal” behavior patterns. Each time an ITM respondent wanted a snack, the act of taking the survey seemed to cause them to question whether or not they really wanted/needed the snack. Many ITM respondents appear to have chosen to resist the temptation reporting that the process changed their behavior. This finding matches those documented by weight loss experts cited earlier and should be considered when adopting this mode of data collection.

It is also interesting to consider how these methods of data collection will mature as the audience they rely upon grows increasingly familiar with and reliant on mobile technology. While advancements are made every day in the mobile technology space, only some early adopters have integrated their mobile devices into every aspect of their lives. The average respondent is not as accustomed to near constant interaction with his or her phone or to translate thoughts and behaviors into instantaneous survey responses. As applications develop to passively collect the data that surveys now collect (GPS signal to answer where they are, language analysis of phone calls and text messages around the time of purchase to analyze mood) the active intrusion into the average respondent’s life will decrease as well. We believe that as the mobile market matures the frequency of use and level of comfort with this kind of data collection will also increase, giving researchers another important tool.

Appendix A