RDD cell phone samples contain a substantial proportion of non-working numbers, making cell phone surveys relatively expensive. We tested a technology which allows users to automatically identify non-working numbers in a sample (Home Location Register Lookup or short HLR-Lookup). HLR-Lookup is available virtually in all countries where the GSM-standard is used – in about 200 countries [1]. Though the GSM-standard coexists with other standards in many countries, it is estimated that about 80% of cell phone subscribers use it [2].
The results presented here are especially useful for conducting surveys in European countries, since in Europe the GSM-standard is used exclusively, as well as in world regions where GSM tends to dominate other standards (Africa, Americas, Asia Pacific, and Middle East). Our findings will be of limited use for the US, since to our knowledge the proportion of GSM-subscribers in the US is not as high as in other countries [3] and technological standards used in the US are incompatible with each other, which does not allow screening of all sampled numbers via HLR-Lookup. In the meantime while the evolving technology may become more compatible in the future, the possibility of using HLR-Lookup to screen cell phone samples and check numbers which belong to operators of the GSM-network should be considered by survey researchers in the US.
Dual Telephone Sample Frame and Challenges of Cell-Phone Samples
Survey organizations increasingly use dual telephone sample frames in response to the problem of a growing cell-phone-only population (Blumberg and Luke 2010; Ehlen and Ehlen 2007). The cell-phone component of the dual frame poses certain challenges.
First, about 40% of cell phone numbers in RDD samples are non-working [4]. Since this problem cannot be solved by the use of autodialers, all numbers have to be attempted for contact by interviewers. This results in higher costs as more call attempts are needed per realized interview [5] and causes possible “burnout” of interviewers due to the “demoralizing effect” of lower productivity that often arises in cell phone surveys (AAPOR 2010).
Second, it is not always possible to identify a number as definitely non-eligible because of ambiguous operator messages like “the subscriber is not accepting calls at this time,” “this service/service attribute is not available,” “this number is barred at the moment,” etc. (AAPOR 2010). Therefore, the unknown eligibility component in final disposition codes is higher for cell phone surveys than for landline surveys affecting the calculation of response rates.
Possible Solution: HLR-Lookup
A possible solution is to identify non-working numbers without calling. This can be done via HLR-Lookup, a service available in the GSM-networks, standard in most countries. Some other names for the service include network query and number lookup.
Information about the status of the number is obtained from the Home Location Register (HLR) database, which stores profiles of cell phone subscribers for billing, charging and other administrative purposes. The access to this database is available for service providers (mediators between mobile network operators and enterprises), who have been granted access to the GSM-network by one of the operators [6]. Currently the HLR-Lookup service is largely used for commercial purposes including messaging or call-center services.
Possible outcomes of the status query and the meanings of those messages are shown in Table 1.
Costs of the service vary starting from 0.06 eurocent per number-lookup when performed by a service-provider. It is also possible to screen a number right before making the call, though this option is more expensive.
Efficiency of the Service
To test the reliability of the service, the outcomes of interviewer contact attempts to cell phone numbers are compared to the results of the HLR-Lookup outcomes. These cell phone numbers were fielded as a part of an RDD dual frame survey (n=24,691, Fieldwork June–August 2010) [7]. The results are shown in Table 2.
The comparison presented in Table 2 shows that HLR-Lookup successfully identifies non-working numbers. From the 10,835 numbers that were classified as non-working by the interviewers 10,081 (93.1 percent) were correctly identified as non-working by HLR-Lookup. Had we used HLR-Lookup before calling, we would have dropped those numbers from the original sample. In that case we could have lost 49 potential respondents, since those working numbers were erroneously identified by HLR-Lookup as non-working [8]. This error, however, only amounts to 0.2% of all numbers and from a practical perspective might be seen as negligible.
Cost Savings: an Estimation
Assuming that it takes an interviewer 1.5 minutes to identify a non-working number and the interviewer’s wage is €15 per hour, it would take about 250 hours of interviewer time to identify 10,000 non-working numbers. The costs add up to €3,750 for making those calls. Compared to the costs of €180 for HLR-Lookup, cost savings of 95 percent are evident in this example. One should keep in mind that this is a rough estimation, since it is necessary to consider additional costs in cell phone surveys. For instance, in some cases it takes more than one call attempt to identify a non-working number. The estimation also does not account for supervisor and other administrative costs. Therefore, the cost-saving presented here is likely to be underestimated. In addition to direct cost savings, reduced interviewer burnout should be considered as time saved for more completed interviews.
Response Rate Calculation: Higher Precision in Cases of Unknown Eligibility
The numbers with unclear status sum to about 34 percent of all numbers to which calls have been made (8,364 out of 24,691 calls, see Table 2). In part these unclear outcomes can be attributed to ambiguous operator messages. Table 3 differentiates between unclear outcomes due to ambiguous operator messages and other unclear outcomes. Ambiguous operator messages account for about 23 percent of numbers with unclear status (1,895 out of 8,364 calls). When contrasted with the outcomes of the HLR-Lookup, about 14 percent of those numbers with ambiguous operator messages are identified as non-working. For all numbers called which result in an unclear status, HLR-Lookup resolves about 8 percent as non-working.
According to AAPOR (2011) numbers with unclear outcomes are classified as “unknown eligibility,” whereas non-working numbers fall under the disposition “non-eligible.” Non-eligible numbers are excluded when calculating the response rate. Therefore, when HLR-Lookup is used the unknown-eligibility component of nonresponse can be reduced for about 8 percent of numbers with unclear status, making the response rate more accurate.
Possible Drawbacks and Privacy Issues
Cell phone numbers may have higher fluctuation rates than landline telephone numbers, making the results of the HLR-Lookup less reliable unless it is performed right before the call. Though performing a real-time lookup may be too expensive for small survey organizations since it requires additional costs (software, maintenance). It has been shown that the error when performing single-time HLR-Lookup tends to be relatively small (see “efficiency of the service”). Nevertheless, to achieve accurate results, one should minimize the time lag between the screening and call attempts. If a real-time lookup is not available, it is advisable to divide the sample into several packages and screen those as close to calling as possible.
It needs to be pointed out, that data received as a result of HLR-Lookup only concerns information about numbers; no personal information about subscribers is available to the research organization. No automatic calls are made to the numbers and no contacts of any nature are made to subscribers. Numbers are protected from sharing with third parties by respective disclaimer policies of service providers.
Summary
HLR-Lookup, the technology of identifying non-working numbers without calling proves to be efficient in reducing costs of cell phone surveys. About 40 percent of non-working numbers can be excluded from the original sample, contributing to 95 percent cost savings in identifying non-working numbers.
About 8 percent of numbers with unclear status can be resolved as non-eligible, making the response rate calculation more accurate. For call outcomes of ambiguous operator messages this number is as high as 13 percent.
Further research should investigate the possibilities of HLR-Lookup for calculating “e”—the estimated proportion of numbers with unknown eligibility which are treated as eligible. This would allow for more accurate calculation of AAPOR response rates with formulae 3 and 4 and may contribute to the spectrum of methods using auxiliary information for estimating “e”.
Acknowledgement
Neither of the authors endorses or is personally involved in any specific provider of the technology evaluated here. An earlier draft of this article was presented at the General Online Research Conference GOR, Duesseldorf, Germany, March 2011 and at the AAPOR conference, Phoenix, AZ, May 2011.
According to GSMA (Global System for Mobile Communications Association, www.gsmworld.com).
GSMA Market Data Summary as of 2009 (http://www.gsmworld.com/newsroom/market-data/market_data_summary.htm).
4G Americas (http://www.4gamericas.org/index.cfm?fuseaction=page&pageid=849).
Estimation for Germany based on telephone studies conducted by GESIS—Leibniz Institute for the Social Sciences.
For costs of cell phone surveys see Guterbock et al. (2011).
Note that since GSM-network is a cross-border system, service providers can perform queries not only for the countries they are located in.
CELLA-2 Study, sponsored by German Science Foundation (DFG), Michael Haeder (Technical University of Dresden), Sabine Haeder, Siegfried Gabler (GESIS—Leibniz Institute for the Social Sciences).
This error is most likely caused by not-timely lookup: the numbers were screened at the end of September 2010. We can only speculate about the reasons why those numbers were identified as non-working, most likely being the end of contract by the subscriber. Looking at the data more closely we see that only 5 out of these 49 cases were realized interviews. We could not identify any bias associated with either of these groups.