Geographic Inaccuracy of Cellphone Samples

INTRODUCTION

Researchers frequently employ sample stratification by geographic region when conducting a telephone study using a dual-frame random digit dialing (RDD) design. For some surveys, the population may be restricted to a state, county, or other area. In these studies, researchers must assign each number to a geographic area at the sampling stage. In RDD cellphone samples, the geographic location for a cellphone number is typically based on the cellphone rate center—the location in which the cellphone was purchased. Therefore, the information derived from a cellphone number may be inaccurate if the cellphone is purchased at a location different from where the cellphone owner resides or if the owner moves after the cellphone purchase. These inaccuracies have a negative impact on survey bias, variance, and costs (Skalland and Khare 2013).

Christian et al. (2009) estimated cellphone sample inaccuracy rates were 6% at the census region, 10% at the state level, and 41% at the county level for all adult cellphone users. For cell-only adults, the estimates were 6%, 12% and 43% respectively. Montgomery et al. (2011) inaccuracy rate estimates among all cellphones are 14% at the state and 65.4% at the county level. Benford et al. (2012) reported inaccuracy rates among all cellphone respondents as 5% at the regional, 9% at the state, and 44% at the county level. The cellphone sample inaccuracy rate reported by Skalland and Khare (2013) at the state level for cell-only adults was 11.5%. Dost and McGeeney (2016) estimated cellphone sample geographic inaccuracy rates among all cellphone adults as around 5% at the census region and 10% at the state level. For cell-only adults, the corresponding estimates were 7 and 14%, respectively.

Skalland and Khare (2013) presented overcoverage and undercoverage rates for cell-only adults by each state’s cellphone sampling frame and discussed the implications on bias and variance of dual-frame RDD estimates. Overcoverage was defined as the percentage of cellphone-only adults sampled from the state’s cellphone frame that reside in a different state. Undercoverage was defined as the percentage of cell-only adults that reside in the state whose cellphone number appears on a different state’s cellphone sampling frame. This paper builds upon this research by generating overcoverage and undercoverage estimates by each state’s cellphone sampling frame for ‘cell-only users’ and ‘dual-users,’ and ‘all-cell users’ (‘cell-only’ and ‘dual-users’ combined).

METHODS

The data presented in this paper are from three 6-month datasets of the Gallup Daily Tracking survey. The survey includes daily interviews with approximately 500 U.S. adults, and utilizes a dual-frame RDD design. For the landline frame, a list-assisted telephone sampling method and a random, within-household selection method are used. For the cellphone frame, the cellphone numbers are randomly generated among all exchanges dedicated to cellphone numbers.

Table 1 presents time-periods and the corresponding sample sizes.

The survey weights for each dataset were calculated following Kennedy (2007).

The estimates presented relate to cellphone sampling frames only. While talking about inaccuracy rates for ‘all-cell,’ ‘cell-only,’ or ‘dual-user’ populations, the analysis is restricted to telephone numbers sampled from the cellphone frames. Completed surveys belonging to those groups that were originally sampled from landline frames are excluded from the inaccuracy rates. The error rate is significantly lower (around 1%) for landline sample frames, and the focus in this paper is to examine the issue of geographic inaccuracy related to cellphone frames only. All estimates presented in this paper are for U.S. adults only.

ESTIMATES OF CELLPHONE SAMPLE GEOGRAPHIC INACCURACY

Table 2 presents inaccuracy rates at different geographic levels for different time periods and for ‘all-cell,’ ‘cell-only,’ and ‘dual-user’. The inaccuracy rates at the census region are relatively low for all time periods and subpopulations. Based on the location designated on the sample file and current residence, the state level error rate, for example, is 12.5% for ‘all-cell users’ (‘cell-only’ and ‘dual-users’ combined), 13.4% for ‘cell-only,’ and 9% for ‘dual-users’ for January–June 2017.

The inaccuracy rates increase as the area gets smaller. The inaccuracy rates are lower for ‘dual- users’ as compared with ‘cell-only users.’ The estimates and corresponding confidence intervals for the other time periods are very similar. The estimates are stable over time with a slight increasing trend. These estimates are consistent with those reported in the past by Christian et al. (2009), Benford et al. (2012), Skalland and Khare (2013) and Dost and McGeeney (2016).

Table 3 presents overcoverage (OC) and undercoverage (UC) estimates by each state’s cellphone sampling frame of ‘all-cell users’ (‘cell-only’ and ‘dual-users’ combined). Following Skalland and Khare (2013), the overcoverage (OC) rate was defined as the percentage of ‘all-cell users’ sampled from the state’s cellphone frame that reside in a different state. The undercoverage (UC) rate was defined as the percentage of all cell users that reside in the state whose cellphone number appears on a different state’s cellphone sampling frame.

The overcoverage and the undercoverage percentages vary by state. High overcoverage (20% or more) are found in smaller states and small overcoverage (10% or less) are found in relatively bigger states. Relatively higher (lower) undercoverage are observed for larger states. The overcoverage and undercoverage percentages did not change much over time except for a few smaller states.

Table 4 presents the overcoverage (OC) and undercoverage (UC) rates of ‘all-cell,’ ‘cell-only,’ and ‘dual-users’ separately for the latest time period (January–June 2017).

The overcoverage and undercoverage percentages for all three populations vary by state. For ‘all-cell’ and ‘cell-only’ users, the overcoverage is lowest for Texas (7.2% and 7.3%, respectively) and highest for the District of Columbia (DC) (59.7% and 59.5%, respectively). For ‘dual users,’ it was lowest for Maine (3.9%) and highest for DC (60.4%). The undercoverage is lowest for California for all three populations. For the ‘all-cell’ and ‘cell-only’ users, the undercoverage is highest for DC (42.5% and 47.1%, respectively). For ‘dual-users,’ the undercoverage is highest for Delaware (22.8%). The smaller states had relatively higher overcoverage and undercoverage percentages. For ‘dual-users’, the overcoverage and the undercoverage percentages are generally lower compared to ‘all-cell’ and ‘cell-only’ populations. The confidence intervals associated with ‘cell-only’ estimates are generally narrower than those for ‘dual-users’ because of larger sample sizes and are asymmetric because CROSSTAB, the SUDAAN procedure employed to construct the confidence intervals, uses logit transformation.

CHARACTERISTICS OF RESPONDENTS BY MATCH STATUS

Table 5 presents selected characteristics of ‘all-cell users’ for January–June 2017 by match status. A match happens when the sampling state is the same as the state of residence. The bias, if any, due to undercoverage is a function of the difference between matched and unmatched groups and the amount of undercoverage. The bivariate relationship between match status and demographic variables was significant for all variables (based on the chi-square test), except gender, indicating a risk of potential bias. Those with a nonmatch status are more likely to be younger adults, non-Hispanic, more educated, belong to households without children, and residents of metropolitan statistical areas (MSAs). Nonmatches are also more common in Southern states as compared with states in Midwest region.

A logistic regression model was also fit using the sample match status as a 0-1 binary dependent variable and the demographic variables in Table 5 as independent variables. The goal was to see if any of these bivariate relationships are nonsignificant after controlling for the other variables. The regression coefficients are presented in Table 6 for ‘all-cell,’ ‘cell-only,’ and ‘dual-users.’

All demographic variables included in the model (Table 6) were statistically significant with the exception of gender for both ‘all-cell’ and ‘cell-only’ user populations. For ‘dual-users,’ the variables marital status, race/ethnicity, and in MSA were nonsignificant. The differences in the demographics of matches and nonmatches were more pronounced for the ‘cell-only users’ as compared with ‘dual-users.’ These findings are generally consistent with those of Christian et al. (2009) and Skalland and Khare (2013). The analysis included large sample sizes which may be why more demographic variables were statistically significant. The analysis was carried out using the software SUDAAN (https://www.rti.org/impact/sudaan-statistical-software-analyzing-correlated-data) to account for the complex sample design and the resulting weights. SUDAAN functions CROSSTAB and RLOGIST were used to run statistical tests and fit logistic regression models.

For ‘all-cell users,’ the response categories of selected survey questions are included in Table 7 along with estimates and corresponding 95% confidence intervals. The differences in the distribution of these variables between matches and nonmatches were statistically significant at 1%. When these variables are added to the logistic regression model (of Table 6) as independent variables, all variables except gender, marital status, MSA, and employment status are significant.

The findings from the same analyses for ‘cell-only’ and ‘dual-users’ were similar.

IMPLICATIONS FOR DUAL-FRAME STATE SURVEYS

For single state dual-frame telephone surveys, the geographic inaccuracy of cellphone samples has operational, cost, and bias implications (Skalland and Khare 2013). For a state survey, cellphones are sampled from that state’s cellphone frame only. The resulting undercoverage of residents living in that state who are present on another state’s frame may introduce bias in estimates. Residents whose cellphone number appears on the state’s cellphone frame but actually live in a different state (overcoverage) may be screened out. Therefore, the overcoverage willnot affect estimates but will negatively impact cost.

Following Skalland and Khare (2013), alternative state level estimates were derived using a modified sample (all landline sample respondents and only those cellphone sample respondents whose self-reported state matched the sampling state). Those estimates were compared with the full sample estimates for each state to get a measure of potential noncoverage bias due to geographic inaccuracy of cellphone samples.

Each of the variables were coded 0-1 and two estimates (based on full sample and modified sample) were derived for the mean (or proportion) of each variable. Table 8 presents, for each state, the difference between the estimates. None of these differences were found statistically significant (at 5% level of significance). However, some differences were relatively large, suggesting potential noncoverage bias due to geographic inaccuracy of cellphone samples.

DISCUSSION

The mobile nature of cellphones poses a significant problem for geographic sampling, and particularly for small areas. For landline numbers, the inaccuracy rate (around 1%) is considerably smaller. The estimates presented in this paper show that about 12.5% of U.S. cell users do not live in their sampling state and that there is variation across states. For ‘cell-only’ users, the corresponding estimate is 13.4%, whereas it is 9% for ‘dual-users’. These estimates increase over time except for Midwest region where they are generally stable. The inaccuracy rates for ‘dual-users’ is lower than for ‘cell-only users.’ Given the percentage of ‘cell-only’ population is growing within cell users, the cellphone sample geographic inaccuracy will continue to be a concern for all areas and in particular for smaller geographic levels.

For a given area, respondents may be identified and screened out of the survey if they do not live there, but eligible respondents who live in the area with cellphone numbers from a different area will not be covered. Screening out respondents from other areas will not cause coverage problems but will lead to higher costs. The undercoverage will result in noncoverage bias. If most of those individuals with inaccurate sample states live in contiguous states, sampling from adjacent states may reduce undercoverage. However, that will increase the overcoverage percentage significantly. For example, if Virginia and DC cellphone sampling frames are also used to sample for Maryland, the undercoverage percentage for all Maryland users will drop from 21.5 to 14.1%. However, the overcoverage will increase from 19.3 to 66.3%, leading to significantly higher survey costs and lead to inefficient use of all resources including time and people. Sampling from selected counties and zip codes from adjoining states may be more effective.

Efforts are being made to improve sample accuracy. Sample providers can now append the billing zip code of the cellphone owner’s address, although this is available for only about 60% of all cellphone numbers (Dutwin 2014). Moreover, this information is appended after the sample is drawn and cannot be used while sampling to improve sample efficiency and reduce sample costs. Address based sampling (ABS) is another alternative to RDD telephone surveys. However, for conducting telephone surveys starting with an ABS sample, telephone numbers will have to be appended to sampled addresses and only about 50–60% of the addresses can be appended with telephone numbers at the national level. Another option to improve estimates may be to employ additional weighting steps to account for correlates of mobility, the characteristic that may be most related to sampling inaccuracy. Additional research is needed to evaluate if any such strategy will be effective for specific states. Finally, this problem of geographic inaccuracy may worsen over time as more and more households switch from landline to cell phones. The impact of geographic inaccuracy will depend on several factors including the geographic level, the target population, and the topic of the survey.

Author information

Stephanie Marken, 901 F Street NW Suite 400, Washington, DC 20004, stephanie_marken@gallup.com, 202-715-3084

Manas Chattopadhyay, 901 F Street NW Suite 400, Washington, DC 20004, manas_chattopadhyay@gallup.com, 202-715-3179

Anna Cahn, 901 F Street NW Suite 400, Washington, DC 20004, anna_chan@gallup.com, 402-818-1868

Appendix A-Questionnaire Wording

The following includes complete question wording from the Gallup Daily Tracking survey for all relevant questions included in this analysis. All parenthetical response categories were not read to the respondent but were accepted after one probe by the interviewer.

Is religion an important part of your daily life?

1. Yes

2. No

3. (DK)

4. (Refused)

In politics, as of today, do you consider yourself [(If code 1, read, A Republican, A Democrat)/(If code 2, read, a Democrat, a Republican)], or an Independent?

1. Democrat

2. Republican

3. Independent

4. (DK)

5. (Refused)

Have you, or has anyone currently living in your household, ever served in the U.S. military?

Do you approve or disapprove of the way [name of the president] is handling his job as president?

1. Approve

2. Disapprove

3. (DK)

4. (Refused)

Employment status

WP10200. Thinking about your WORK SITUATION over the past 7 days, have you been employed by an employer—even minimally like for an hour or more—from whom you receive money or goods? (This could be for one or more employers.)

(Interviewer: PAYCHECKS ARE CONSIDERED “MONEY OR GOODS”)

(Interviewer: If the respondent has trouble deciding if they are employed for an organization, probe by asking:) Do you receive money or a paycheck from them?

(Interviewer: If the respondent says they are “self-employed,” code as NO, and ask next question about self-employment.)

1. Yes

2. No

3. (DK)

4. (Refused)

Skip: (If code 1 in WP10200, Continue;

Otherwise, Skip to WP10202)

WP10215. In a typical week (7 days), how many hours do you work for an employer.

(Programmer: Open ended and code)

(Interviewer: This could be for one or more employers.)

(Interviewer: If respondent says “Don’t know” or has difficulty determining the number of hours worked in the last 7 days, you can help him/her calculate the ‘approximate’ number of hours TYPICALLY worked each week.)

(Interviewer: For example, you could ask him/her:) About how many days do you work in a typical week? AND about how many hours do you work in a typical day? (From this, you can approximate the number of hours typically worked in a week.)

(Interviewer: If the respondent is on leave from their job and not working normal hours, re-ask the question stressing the word typical.)

1. 30 hours per week or more

2. 15 to 29 hours per week

3. 5 to 14 hours per week

4. Less than 5 hours per week

5. (DK)

6. (Refused)

Skip: (If code 1 in WP10215, Skip to WP102CK2;
Otherwise, Continue)

WP10202. Again thinking about the last 7 days, were you self-employed, even minimally like for an hour or more? This means working for yourself, freelancing, or doing contract work, OR working for your own or your family’s business?

Self-employment also includes fishing, doing farm work, or raising livestock for either your own or your family’s farm or ranch.

(Interviewer: If the respondent says YES after you read the first sentence, (again, thinking about the last 7 days, were you self-employed, even minimally like for an hour or more) OR has already told you they are self-employed, you may record as YES and you do not need to read the rest of the question. If the respondent does not volunteer a “yes” while reading the first part of the question or says NO during the first sentence you MUST read the remainder of the question.)

1. Yes

2. No

3. (DK)

4. (Refused)

Skip: (If code 2-4 in WP10200 AND code 2-4 in WP10202, Skip to WP10208;

If code 2-6 in WP10215 AND code 2-4 in WP10202, Skip to WP10229; Otherwise, Continue)

(Programmer: If code 1 in WP10202, ask:)

WP10216. In a typical week (7 days), how many hours do you work as a self-employed individual?

(Programmer: Open ended and code)

(Interviewer: If respondent says “Don’t know” or has difficulty determining the number of hours worked in the last 7 days, you can help him/her calculate the ‘approximate’ number of hours TYPICALLY worked each week.)

(Interviewer: For example, you could ask him/her:) About how many days do you work in a typical week? AND about how many hours do you work in a typical day? (From this, you can approximate the number of hours typically worked in a week.)

(Interviewer: If the respondent is on leave from their job and not working normal hours, re-ask the question stressing the word typical.)**

1. 30 hours per week or more

2. 15 to 29 hours per week

3. 5 to 14 hours per week

4. Less than 5 hours per week

5. (DK)

6. (Refused)

Skip: (If code 2-6 in WP10216, Continue;

Otherwise, Skip to Note before WP10208)

(Programmer: If [code 2-6 in WP10216] OR [code 2-6 in WP10215 AND code 2-4 in WP10202], ask:)

WP10229

Do you want to work 30 hours or more per week?

1. Yes

2. No

3. (DK)

4. (Refused)

Skip: (If code 2, 3, or 4 in WP10200 AND WP10202, Continue;

Otherwise, Skip to WP102CK2)

WP10208. In the past four weeks, have you actively been looking for employment? “Actively looking” means applying for jobs, searching for jobs, and the like.

1. Yes

2. No

3. (DK)

4. (Refused)

WP10209. Would you have been able to begin work had you been offered a job within the last four weeks?

1. Yes

2. No

3. (DK)

4. (Refused)

For details about how Gallup calculates employment status categories, see http://news.gallup.com/poll/128192/gallup-employment-tracking-work.aspx