Processing math: 100%
Skip to main content
Survey Practice
  • Menu
  • Articles
    • Articles
    • Editor Notes
    • In-Brief Notes
    • Interview the Expert
    • Recent Books, Papers, and Presentations
    • All
  • For Authors
  • Editorial Board
  • About
  • Issues
  • Blog
  • Subscribe
  • search

RSS Feed

Enter the URL below into your favorite RSS reader.

http://localhost:22663/feed
In-Brief Notes
Vol. 11, Issue 2, 2018April 30, 2018 EDT

Identifying Non-Working Phone Numbers for Response Rate Calculations in Africa

Charles Lau, Nicolas di Tada,
africaresponse ratenonworking numbermobile phone
https://doi.org/10.29115/SP-2018-0020
Photo by Quino Al on Unsplash
Survey Practice
Lau, Charles, and Nicolas di Tada. 2018. “Identifying Non-Working Phone Numbers for Response Rate Calculations in Africa.” Survey Practice 11 (2). https:/​/​doi.org/​10.29115/​SP-2018-0020.
Save article as...▾

View more stats

Abstract

Identifying Non-Working Phone Numbers for Response Rate Calculations in Africa

Mobile phone surveys are increasingly common in low- and middle-income countries, but the methodology for these surveys is still emerging. This brief addresses a critical question: How should researchers handle nonworking phone numbers in response rate calculations?

A common formula for response rates (American Association for Public Opinion Research 2016) is:

AAPOR Response Rate1=CompleteComplete+Parital+Refusal+Noncontact+Other+Unknown Eligibility

Nonworking numbers are not assigned to any person; they are “not eligible” (AAPOR 2016: 18) and should be excluded from the calculation.

Identifying nonworking numbers in lower income countries can be challenging. There are internationally standardized call outcome codes (see examples here) for voice modes (e.g., interactive voice response). However, mobile network operators oftentimes do not assign codes correctly. For example, a mobile network operator we worked with assigned integrated services digital network (ISDN) code 19 (no answer) to nonworking numbers instead of ISDN code 1 (unallocated or unassigned). As a result, we could not distinguish numbers that were working and nonworking. Text-based surveys (e.g., short message service or SMS) are even more challenging because there are not standardized codes, as there are for voice. Furthermore, information about call outcomes varies by mobile network operator and country.

If nonworking numbers cannot be identified, researchers may tend to lump nonworking and working numbers in an “unknown eligibility” category, which is in the denominator in response rate calculations. As a result, response rates are understated — i.e., lower than they should be. If nonworking numbers could be excluded from the denominator, the response rate would increase.

We recommend the following:

  1. Researchers conducting mobile phone surveys should be transparent about their identification and treatment of nonworking numbers. When in doubt, include numbers in the denominator to provide a conservative response rate.

  2. Test error codes empirically — by attempting to contact obviously nonworking numbers such as “999999999999”) and to review the error codes that operators return.

  3. Be wary of third parties that claim to “validate” phone numbers: In our experience in Ghana and other sub-Saharan Africa countries, these third parties have a high rate of false negatives (i.e., they return impossibly high working number rates).

  4. If resources allow, conduct a special methods study using manual telephone calls to estimate the proportion of “unknown eligibility” numbers that are working and nonworking (Kennedy, Keeter, and Dimock 2008). In a survey in Nigeria, the first author (Lau) found that humans could code numbers as nonworking better than the computer-assisted telephone interviewing (CATI) system. That information was then used to adjust the response rate using AAPOR Response Rate 3.

The telecommunications landscape in low- and middle-income countries is changing rapidly. We hope that in the future, better methods will exist to identify nonworking numbers. Until then, we recommend transparency and conservative approaches outlined here when calculating response rates.

References

American Association for Public Opinion Research. 2016. “Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (9th Ed).” 2016. http:/​/​www.aapor.org/​AAPOR_Main/​media/​publications/​Standard-Definitions20169theditionfinal.pdf.
Kennedy, C., S. Keeter, and M. Dimock. 2008. “A ‘Brute Force’ Estimation of the Residency Rate for Undetermined Telephone Numbers in an RDD Survey.” Public Opinion Quarterly 72 (1): 28–39.
Google Scholar

This website uses cookies

We use cookies to enhance your experience and support COUNTER Metrics for transparent reporting of readership statistics. Cookie data is not sold to third parties or used for marketing purposes.

Powered by Scholastica, the modern academic journal management system