Both the Fahimi et al. and Boyle et al. papers provide reasonable approaches to measuring the same thing – landline coverage in the listed residential 100-banks from Telcordia. The fact that the two papers reach different conclusions is puzzling since both appear to start from about the same place. In order to sort out this controversy, we need to look at each component in the process separately.
Although the two sets of authors stratify the frame in different ways, both end up with relatively equal counts of listed 100-banks (just under 3 million). When the 100-banks from the other strata in the frame are added to those counts, both papers report totals of about 9 million or so 100-banks. Moreover, after doing a little checking, I have concluded that both groups of researchers have counts of total households (about 112 million) and non-landline households (those cell-only or with no phone service) that are roughly the same (close to 20% of households or 22 million). Since they both start equal, why don’t they end up that way? I believe the problem, as concluded by Boyle et al., is a non-statistical one, it is all in the administrative process that goes into identifying a residential number.
Let’s begin with the listed residential banks. Fahimi et al. report a residential hit rate of 30.8%, and Boyle et al. have the rate at 29.4%. Based on the strata size in each paper, Fahimi et al. estimate the number of residential numbers in this part of the frame to be about 89 million and Boyle et al. get 82 million. This seems to result from the different hit rates each estimates and the slightly different count in listed 100-banks. Splitting the difference on the hit rates gives about 0.30 which produces a residential number estimate of 85.5 million in that part of the frame. Neither paper deals explicitly with the number of households with multiple voice landlines, where the range seems to be between 3% and 7%. However, whichever line is reached, it counts as a household hit in terms of raw (not weighted) counts.
Fahimi et al. and Boyle et al. approach the estimation of residential numbers in the other part of the frame differently. The former uses reverse matching to commercial databases for undetermined numbers, and the latter estimates the number by extrapolating from their empirical results to the undetermined in that part of the frame. The results are quite different in the two papers. Fahimi et al. conclude that almost 20% of residential lines are not in listed banks, while Boyle et al. estimates only 5% are outside listed 100-banks. This is quite a difference. How do we explain it?
I think we have to look beyond the pure statistics to the actual survey procedures. Given that there are now cell-mostly households that rarely use their landline and call screening devices which reinforce concerns about privacy, more burden is put on survey researchers to identify residential phones. Boyle et al. discuss their procedures for making call attempts and verifying residences once the phone is answered, but Fahimi et al. are relatively silent on that score. In any case, identifying residential numbers is difficult no matter what procedure is followed; and, as response rates continue to decline, the exact nature of these procedures will take on more importance. Yet, there are other problems to overcome. With the expansion of the number of telephone service companies, getting information on individual numbers is almost impossible. This is clearly a greater problem in the non-listed banks, but it is still a problem in listed banks where the residential hit rate has declined with the expansion of area codes and the less systematic assignment of numbers (estimates of residential hit rates obviously vary even here). Although making many call attempts and verifying if the number is at a residence after someone answers is a step in the right direction, what about all those numbers that are ring-no-answer? The jury is still out on the adequacy of matching to commercial or other databases as Fahimi et al. do to resolve the problem. Of course, all of these issues do not even consider the rules used by a survey organization to assign final dispositions.
The only remedy is for the different survey organizations to start directly talking to one another. Joint studies should be conducted where all procedures used are transparent and state-of-the-art. The results of that research should be presented jointly in journals and at conferences. This does not ensure we have the right estimates, but it is a big step in the right direction.