Survey Practice sent a query to the Association of Academic Survey Research Organization’s listserv to ask the members to describe their experiences with cell phone samples. Four organizations provided responses. Feel free to ask for more information in the comments section at the end.
Gene M. Lutz
Center for Social and Behavioral Research
University of Northern Iowa
Our center conducts data collection for the Iowa Behavioral Risk Factor Surveillance System survey. For the period of February through September of 2008, both a traditional landline version and a cell phone pilot version of the survey were in the field, using samples of telephone numbers provided by CDC/Genesys. During this period there were 3766 landline completions and 245 cell completions. The landline questionnaire contained 156 items and the cell questionnaire contained 101 items. Cell respondents were offered a $10 compensation; no compensation was offered for landline respondents. Some of the first findings are the following:
*More dials were required to achieve completions by cell. The mean number of dials per complete were 87 for cell and 24 for landline; a ratio of 3.6 cell dials to every 1 landline dial to achieve a completion.
*All AAPOR cooperation rates slightly favored cell over landline. E.g., Estimated AAPOR Cooperation Rate 2 was 0.825 for cell and 0.747 for landline.
*Response rates were similar for cell and landline. Estimated AAPOR Response Rate 3 was 0.445 for cell and 0.443 for landline.
*Comparably fewer refusals than completes occurred by cell for eligible, selected respondents. The ratio of completes to refusals was 4.6:1 for cell and 3.8:1 for landline. Estimated AAPOR Refusal Rate 2 for cell was 0.091 and 0.140 for landline.
*Nine of ten cell respondents accepted the compensation offer. 75% accepted an electronic gift card, 14% accepted a physical store gift card, and 11% declined any compensation.
*However, eight of ten cell respondents claimed they would have completed the interview without the compensation offer. 79% said “yes” and 21% said “no” to this post interview question.
*Most cell respondents (16% strongly agree and 69% agree) said they had a positive experience using a cell phone for the interview.
*Seven of ten cell respondents were multi-tasking while doing the interview. Only one-fourth (26%) were doing nothing else, but 22% were watching TV, 14% were watching children, 11% were using the computer, 11% were doing household chores, and lesser percentages were doing other things.
*Most cell respondents (76%) said being able to do other things while completing the interview by cell made them more likely to do the survey.
*Overall, the cell survey required more resources (sample pieces and dial attempts) for locating eligible respondents than the landline, but the actual cell respondents were generally positive about using a cell phone to be interviewed.
Thomas M. Guterbock
Center for Survey Research
University of Virginia
*If you want to start doing cell phone interviewing, plan to involve your interviewers, supervisors, and technical staff in the process from the start. Expect to be revising your training materials, adding and defining new disposition (call outcome) codes, checking your partially collected data in new ways, and helping the entire staff adjust to something that is quite novel in some ways. Treat this transition as a learning moment for your entire organization.
*We ran a small experiment here using cash incentives for those who agree to be interviewed via cell phone. One group was offered $5, another group was offered $10 for the interview. The rate of completions per hour was better with the $10 incentive, so much better, in fact, that it more than pays for the $5 difference in the incentive cost.
*The people we reach by cell phone, even those who have a landline, are very different demographically from those we reach by landline. This suggests that it is probably not wise to ‘screen out’ dual-phone households from the cell phone sample. Better to interview all cell phone users you can reach, whether or not they have a landline phone in the household.
*Costs for interviewing cell phone numbers are certainly higher. The time it takes per completion is a little more than double that needed for a conventional landline interview, even when the cell phone respondents are offered a cash incentive. Add in the incentive cost and the cost of a cell phone completion is about 2-1/2 times more.
Survey Research Center
University of Arkansas
The following were found on 1,587 completed surveys of the general population of a single state conducted during October 2008.
*Cell-phone interviews are slightly shorter, on average, than are landline interviews.
*The number of calls to complete cell versus landline interviews are the same.
*Hours per completion for cell are about one-third longer than for landline interviews.
University of Wisconsin Survey Center
In Wisconsin, the state Department of Health Services conducts the Behavioral Risk Factor Surveillance System (BRFSS) survey. In 2008, a pilot project attempted to extend the BRFSS survey to cell-phone-only adults who would never be reached under the usual landline protocol. The BRFSS cell pilot (BCP) differed from the BRFSS landline (BLL) protocol in several ways: sampling, eligibility, compensation, content, and calling strategy (both overall and for refusals).
The BCP sample was pre-screened with the intention to include only cell numbers; the BLL sample was pre-screened to exclude them. BCP respondents were asked whether they also had a residential landline, and those that did were screened out. BCP included respondents living outside Wisconsin, as long as they had an in-state phone number and lived in one of several other states also involved in the pilot study. BCP treated any adult answering the cell phone as the selected respondent; BLL enumerated the household and selected one adult at random. BCP respondents were offered ten dollars as compensation for their airtime cost; BLL respondents were not compensated.
BCP used a shorter questionnaire averaging 12 minutes long; BLL interviews averaged 19 minutes and included more potentially sensitive questions about sexual behavior and orientation, binge drinking, and children. Calling protocols differed. BCP required 5 or 8 calls (depending on the month) before closing out unresolved cases, and no case received more than 15 calls; BLL used a 15-call minimum and a 25-call ceiling. BCP employed more experienced interviewers who had previously worked on BLL for several months.
Refusal conversion protocol differed significantly. BLL called back to convert all first refusals (except abusive ones). BCP did not call back refusals if the respondent referred to their cell phone in the process of refusing, although interviewers did try to convert those respondents during the initial refusal. Both studies ceased calling at the second refusal.
BLL and BCP were conducted by the University of Wisconsin Survey Center during the months of February through August 2008. Although the protocol variations make it difficult to draw rigorous conclusions about cell phones vs. landline effects, a few intriguing differences emerged.
*The yield of completed interviews per cases called was much lower for BCP (4%) than for BLL (30%) (Table 1).
*Protocol differences for refusals and number of calls meant BCP refusal conversion attempts (96%) were less likely than BLL conversion attempts (99%) (Table 2).
*In those cases where refusal conversion was attempted, BLL was more successful at obtaining a completed interview (14%) than was BCP (1%) (Table 3).
*However, the percentage of respondents cooperating with the conversion attempt long enough to either complete the interview or establish ineligibility was very similar (BLL 14% complete, 1% ineligible; BCP 1% complete, 12% ineligible).
*Conversion attempts on BCP helped improve the response rate mainly by identifying those ineligible cases, rather than by obtaining completed interviews.