With the well-documented drop in response rates over the past quarter century, survey researchers continue to explore new and innovative ways to boost cooperation. Strategic use of caller ID has been shown in the past to potentially help participation. This paper reports on five experiments manipulating the caller ID text and phone number in a variety of contexts.
Customized caller IDs are increasingly becoming a prerequisite for high quality research. A host of scholars have argued that caller IDs can (and will) be used against us as survey researchers [as a method of screening out calls (see Battaglia et al. 2007; Couper 2005; Lepkowski 1999; Tuckel 2001)]. However, recent articles suggest that it can also be strategically used for, rather than against, survey research [as a method of fostering participation (see Callegaro, McCutcheon, and Ludwig 2010; Trussell and Lavrakas 2005)]. Still others find inconclusive results in any clear direction (Curtin, Presser, and Singer 2005; Link and Oldenick 1999).
There have been to date a number of caller ID experiments noted in prior literature. Trussell and Lavrakas (2005) found about a 1.5 percent increase in responses rates when using “Nielsen Ratings” compared to “Unknown” or “Out of area.” Okon, Moore, and Bates (2008) found increased response rates using “Census Bureau” compared to “Unknown Caller.” For the National Immunization Survey, Barron and Khare (2008) found about a 3 percent jump in response rate when using “NORC U Chicago” instead of “Toll Free.” And Callegaro, McCutcheon, and Ludwig (2010) found the use of “Gallup” to help by 3 percent. However, not all experiments were positively efficacious. Callegaro et al. also found some of their experimental conditions (varying by certain target populations) actually depressed response rate. And Fernandez and Hannah (2007) found no difference in a number of BRFSS-related experiments.
It is important to note that all of the above research generally leaves two major questions untouched with regard to caller ID. First, can a specific caller ID help if you are not a “brand name?” That is to say, how effective is a caller ID when it does not include commonly recognizable names such as Gallup, Census, Nielsen, or a prominent University? And more importantly, the above experiments only deal with half the message, and to some degree, the less important one. That is, they manipulate the caller ID text, but not the number itself. What if some numbers foster greater participation than others?
This paper details five experiments regarding both text and number. The first largely replicates what has been done before by comparing the use of “SSRS” (for the survey vendor, Social Science Research Solutions) with “Univ MN” in a study within the state of Minnesota. A second experiment looks at the potential efficacy of using a Spanish caller ID compared to an English ID in a national survey of Latinos. Third, we explore the use of the term “Jewish” in surveys of Jews. Fourth, we compare the use of a local 617 number for a study conducted in the state of Massachusetts, compared to the use of a 1-800 number. And finally, we similarly compare a “local” (484) number compared to a 1-877 number in a national omnibus survey context.
Before delving into these experiments, it is important to provide a brief primer on the technicalities of caller ID. Caller ID is not owned by everyone. The most recent published estimates are about five years old, and find that just under 60 percent of landline households have caller ID (Glaser 2006). Since caller ID comes standard today with many telephone contracts, this figure could be somewhat higher. It is important to remember however that a quarter of households do not own a landline telephone at all (Blumberg and Luke 2011). Cell phone owners all have caller ID; however, they will not see any text unless the person calling is already in their contact list. Presumably, this will not be the case for survey organizations, and as such, cell phone recipients will see an incoming phone number, but no text. Overall, then, if 60 percent of landline owners have caller ID and 25 percent of households are cell phone only, then only about 45 percent of all households have a caller ID that will show both number and text.
It is also critical to note that caller ID is dependent upon the network service provider of both the sender and receiver. To attain customized caller ID text, survey organizations have to request that their telephone provider change the text on record. Changing the number itself requires purchasing the number of choice, which must route to a physical location in the local area associated with the number. When calls go through digital providers, the receiver’s telephone provider will pull the number and text from the sender’s database. But this is not guaranteed, as the receiver’s provider does not have to be digital, and/or does not have to do lookup. Therefore, the receiver may not end up getting the ID you are trying to display. But largely, it is assumed that the text does “get through” and is displayed correctly.
The Experiments
Given that three of our five tests were concerned with caller ID text, only landline sample was utilized, since again text will not show on cell phones. The Massachusetts survey utilized an address-based design and therefore did not have outbound cell phone dialing. The final experiment, however, comparing a 484 number to a 1-877 number, was explored on both landlines and cell phones.
Our first experiment was associated with the Minnesota Health Access Survey of 2009. While 90 percent of all landline sample was routed through a local number with the text “UNIV MN” displayed, a random 10 percent used the ID “SSRS SURVEY RSRC.” Overall, response rate (RR3) for the SSRS text was in fact significantly (p = 0.03) higher than it was for UNIV MN (48.9% to 47.7%). We have been concerned that the use of a local University text may in fact be less effective if respondents assume that the call is going to solicit donations, but of course we have no way to determine if this is the case.
Our second experiment utilized two texts, with near identical 1-800 numbers: “Pew Survey” and “Estudio de Pew,” during the 2009 Pew Hispanic Center’s National Survey of Latinos. It was thought that Spanish text would be more efficacious in areas of high Hispanic household incidence. Overall, the English text secured a 31.4 percent response rate, compared to 31.6 percent for the Spanish text. There was no effect on response rates, cooperation rates, and other measures by language of the text, both overall and when interacted with Hispanic density. Though not statistically significant, the Spanish text actually attained a higher response rate in areas of low Hispanic household incidence compared to the English text (36.1 versus 32.9).
The third experiment yielded quite interesting results. We randomly assigned one of two texts to the Baltimore Jewish Population Survey: “BALTJEWISHSURV” and “BALTSURVSSRS.” Our initial hypothesis was that the use of the term “Jewish” would foster participation in a listed sample source of likely Jewish households (a compilation of synagogue and other lists), but depress participation in RDD sample, where the vast majority of respondents would be non-Jewish (over 95 percent non-Jewish). In fact, we found no significant effect in the Listed sample, and a significantly positive effect of the term “Jewish” in RDD sample. We found not only a 6.1% increase in response rate in the RDD sample when using the term “Jewish,” but as well, we found that the source of this increase was due to both a lower refusal rate and answering machine rate (p < 0.01). In short, we surmise that the use of “Jewish” suppressed the tendency to outwardly refuse and increased the tendency to pick up the phone rather than allow the call to be screened by an answering machine. The results are interesting when one considers that 19 times out of 20, the person deciding whether to refuse or not and/or pick up the phone or screen is not Jewish. Importantly, the difference in ID text did not impact the incidence attained in the data, since one goal of the study was Jewish population estimation.
The final two experiments underscore our most important finding, as they are concerned with the efficacy of the telephone number itself. Again, we consider telephone number to be the most important part of the caller ID since it is the only type of identification that we can reliably assume will be seen by the majority of respondents on landlines and on cell phones. In the first of these two tests, we randomly assigned either a 617 (Boston area code) or a 1-800 number during the 2010 Massachusetts Health Insurance Survey [1]. We found, as expected, that the 617 number attained a higher response rate compared to the 1-800 number (23% versus 19.5%, p < 0.01). The source of the difference was not in a lower refusal rate or a lower answering machine rate, as these data were nearly identical across the two numbers. Rather, the only difference was in the percent of sample that completed the interview.
A final comparison replicated the Massachusetts experiment on a national scale. This is an interesting test, since any local number will in fact only be “local” for those who live in its service area. Californians will not and should not consider a “484” number local: They will nevertheless recognize it as not being a 1-800/888/877 number, and therefore perhaps be more willing to pick up the phone. In the end, we found a significant effect in the landline sample, but not in the cell phone sample. Response rate increased from 9 to 10 percent on landlines (p < 0.01). It is possible that this difference would increase with effort, such that a study with response rates in the 30s or 40s could realize a 3 to 4 percentage point difference overall. The non-finding on the cell phones is interesting in that we found slightly lower rates of voicemail using the local number, but then a higher percentage of non-completed callbacks. One possibility is that the local number piqued people to answer the phone, but then upon realizing that the call was a request for a survey, asked to be called at a later time. The net result was a near identical response rate for both conditions.
Overall, we note that consistent with past research, the efficacy of caller ID varied by context, and even where differences were significant, the effect was modest at best. It is likely that in many instances a particular caller ID has a slightly positive impact on some segments of the population but conversely a negative effect on others. Our key findings suggest that avoiding 1-800 numbers seems to be the most promising manipulation of caller ID in order to increase cooperation and response rate. We believe that further research on number variation is paramount, given the promise that local numbers can foster significantly greater response compared to 1-800 numbers, and the fact that cell phone respondents can only see the caller ID number and not the text.
The MA HIS final response rates approached 50%. However, the data here are based on the dispositions after only a maximum of 6 call attempts since the caller ID’s were varied starting at the 7th attempt in an effort to maximize response. Sample here is approximately half phone-listed address based sample and half landline RDD.