It is widely recognized that response rates of landline household random digital dialing (RDD) surveys are declining (Galea and Tracy 2007; Montaquila et al. 2007). Many factors contribute to the decline of response rates, among them new technologies which allow potential respondents to screen callers using call blocking and caller ID. Estimates for 2009 (Blumberg and Luke 2009) indicate that nearly one fourth of all U.S. households were wireless only. An additional one in eight households rely on cell phones for the majority of telephone calls. This could also be a source of both nonresponse and under coverage in landline RDD samples. As a result, researchers are adopting mixed modes to compensate for both under coverage and nonresponse (Dillman et al. 2009).
Although trends toward cell phone usage might simply direct researchers to incorporate cell phone RDD samples, there are barriers to the adoption of cell phone samples (Brick et al. 2007; Pew Foundation 2006; Steeh 2004). Cell respondents are more difficult to reach than are landline respondents, due to safety concerns (such as driving), technologies which allow potential respondents to screen calls before answering, including caller ID; call blocking; and the proportion of phones which are used exclusively by minor children, who are not eligible respondents in many surveys. This produces a larger number of unproductive calls. Finally, the Telephone Consumer Protection Act (2010) requires survey centers to manually dial cell phone numbers rather than using the automated dialing systems employed in most RDD landline surveys. Recent meta-analysis and earlier studies show that cell phone surveys are more expensive than similar surveys conducted using landline (Guterbock et al. 2010; Keeter 2007).
The Behavioral Risk Factor Surveillance System (BRFSS) is a state based RDD for non-institutionalized adults over 18 years, conducted by state health departments in collaboration with the Centers for Disease Control and Prevention (CDC). The survey has been a unique source of data for health behaviors, chronic disease conditions, and the use of preventive health services for states since 1984 (http://www.cdc.gov/brfss). Now active in all 50 states, three U.S. territories, and the District of Columbia (D.C.), the BRFSS is the largest telephone health surveillance system in the world with over 400,000 interviews completed each year. Traditionally conducted as an RDD landline survey, the BRFSS started collecting data by cell phone from 2008. In 2009, cell phone RDD samples were included in 48 states, D.C., Guam, Puerto Rico, and the Virgin Islands.
Many surveys begin by using screening questions to ascertain whether persons reached by telephone are eligible respondents. Screening questions may also be used to select from among eligible adults within households. Introductory scripts which describe the survey and introduce the respondent to the survey topic have been shown to have effects on response rates (Currivan et al. 2007). Both landline and cell phone questionnaires of the BRFSS include screening questions at the outset of the survey to ensure that respondents meet eligibility criteria. The BRFSS screens landline households by ensuring that the number reached is a landline phone within a residence. Respondents are then selected at random from the number of adults who reside within the household. Cell phone respondents are screened to ensure that the interviewer has reached an adult who lives within a private residence. Cell phone respondents are further screened to eliminate persons who could also be within the landline sample. Therefore, if a potential respondent reached by cell phone indicates that he or she also maintains a landline telephone in his or her residence, he or she is screened out of the sample. As a result of this additional screening question, many potential respondents reached by cell phone do not qualify to be interviewed.
The literature suggests that cell phone respondents will be more difficult to reach and that they will be less cooperative than respondents reached by landline phones (Link et al. 2007). Although much is known to support the proposition that cell phone respondents are difficult to reach, little is known concerning their propensity to cooperate in survey research once they have been successfully contacted. This research describes the overall and state level response and cooperation rates of the landline and cell phone only samples of the BRFSS.
Data presented here are taken from the 2009 BRFSS. All states, U.S. territories and the District of Columbia were included in the landline sample, but two states (Tennessee and South Dakota) did not collect data from their cell phone only household residents. Both the landline and cell phone samples are supplied by Marketing Systems Group (2010). A sampling of 100 number banks is generated from all area codes and telephone exchanges within each state. The 100 number banks with at least one published residential number (one-plus) are disproportionately sampled with number banks which do not have at least one published residential number at a ratio of 1.5:1, resulting in a disproportionate stratified sample (DSS). The cell phone samples are selected randomly from dedicated 1,000 cell phone banks. The sampled numbers are dialed manually.
States conduct the BRFSS separately and submit data in a standard format to the CDC. Standardized disposition and data files for landline and cell phone surveys are submitted separately by the states. BRFSS uses the American Association for Public Opinion Research (AAPOR) standards for reporting interview outcomes.
Using the landline and cell phone disposition files for the 48 states that collected both landline and cell phone data, we calculated several response measures: response rate4 (RR4), cooperation rate2 (COOP2), differences between landline and cell phone response and cooperation rates, and percent of completed interviews of the total sample for the U.S. In addition, we calculated the percent refusal/break-offs (selected respondents who did not answer the first core question or who answered at least the first core question); percent of contacts not selected or without determining their eligibility; percent reached through answering device/ voicemail and percent with telecommunication barriers to be reached.
We also calculated percent of the sample that completed the screening questions. Among those who completed the screening questions, we calculated the percent of interviews completed, refusal/break-off rates and percent of respondents who did not complete the interview by state in both surveys.
As indicated in Figure 1, rates of completion for cell phone numbers were a little less than half of those for landline phone numbers (4.3% and 10.1%, respectively). Calls to cell phone numbers were more likely to reach answering devices/voicemails (14% for cell phone and 4.2% for landlines), and refusal/break-off rates were also higher among cell phone numbers compared to landline (17% vs. 11.8%), respectively. Telecommunication barriers for the landline and cell phone only household samples were 0.4% and 3.3%, respectively. Landline numbers were also more likely to be non-working.
As Figure 2 illustrates, the sample for landline households included over 4.2 million phone numbers. The cell phone sample included over 400,000 cell phone numbers. Although only 5% of cell phone numbers were successfully screened, compared to 15% of landline numbers, 75% of all screened cell phone numbers produced completed interviews. This number compares favorably with the completion rate of 66% among landline numbers that successfully completed screening questions. Moreover, the refusal/break-off rate among cell phone respondents was also lower after screening, with 15% refusing or breaking off the interview, compared to 20% of landline numbers. The percent of eligible respondents who did not complete the interview, referred as others, was also lower in cell phone compared to landline respondents (10% and 14%, respectively).
Because the BRFSS is completed in each state individually, median state level cooperation and response rates are provided (Table 1). Overall, the median RR4 among the states was 52% for landlines and 39% for cell phones.
State Level Differences
Nationwide, landline respondents had higher levels of completion of screening questions (Figure 2). This remained true for all states, as illustrated in Figure 3. Table 1 also illustrates the differences in cooperation rates between cell phone and landline samples by state, with some states showing stronger cooperation for landline respondents (Alabama, Florida, New Jersey, Kentucky, Minnesota, and South Carolina), while the majority of states show higher levels of cooperation for cell phone respondents (as indicated by the negative value of the differences between the landline and the cell phone). Thus, the trend noted in the nationwide data is also observed in the majority of states. Contrary to the findings in the cooperation rates, landlines had higher RR4 than cell phones for all states, except Wyoming.
Table 2 shows the percentage of completes, refusal/break-offs and eligible respondents not completing the interviews for all screened respondents in both cell phone and landline samples in 2009. Although the data indicate that differences between completion rates and refusal/break-off rates for screened respondents may vary among states, the findings indicate higher completions in cell phones than landline in all but nine states. The refusal/break-offs and eligible respondents not completed the interview also show that landline numbers had higher percentages than cell phones numbers in the majority of states, which contributes to lower completion rate in the landline among screened respondents.
The findings agree with the current literature that cell phone populations have lower rates of response than do landline populations (Brick et al. 2007; Link et al. 2007; Steeh and Piekarski 2008). As the data shows, cell phone only numbers respond less often to telephone calls and in completing the screening questions and have higher refusal and break-off (selected respondents who did not answer the first core question or who answered at least the first core question) rates relative to landline samples. The findings here are comparable to those reported in the AAPOR Cell Phone Task Force Report in 2010 (A.A.P.O.R. Cell Phone Task Force 2010) which estimated cell phone refusals and break-offs at five percentage points higher than landline refusals and break-offs.
Contrary to the findings above, the completion rates, cell phone refusal and break-offs are lower after the respondent completes the screening questions. This partially explains the higher cooperation rate of the cell phone respondents than the landline, 78% and 75%, respectively. Because the proportion among the cell phone samples that completes the screening questions was lower than that of the landline sample (5% vs. 15%), the overall contribution to the response rate is trivial. Recent research (Frasier and Morrison 2011) also indicated that cell phone respondents who complete eligibility screening were more cooperative than the landline respondents.
The landline response rates were higher than cell phone in all states, except Wyoming. However, the cooperation rates in the cell phone sample were higher in the majority of the states, which is attributed to the reasons mentioned earlier. The consistency of higher response rates in almost all states assures the favorability of the landline mode over cell phone, despite the looming coverage bias in landline.
In addition, the higher completion rates in the cell phone, compared to landline, in the majority of states among respondents who completed the screening questions indicates the cell phone respondents’ tendency to complete the interview when conditions are likely to be favorable to complete. This phenomenon was consistent across the majority of states, which shows that similar conditions among the states are probably driving this favorable situation and merits further investigation. Survey organizations could benefit from such situation in the cell phone if common conditions are found across the states.
The results indicate that the landline survey has higher response rate than cell phone, whereas the cell phone show higher cooperation rate. There are three major challenges that the cell phone survey has to reverse to achieve a response rate comparable to that of the landline: (1) higher rate of calls reaching voicemails, (2) higher rate of contacted individuals not initiating the interview, and (3) higher rate of refusal/break-offs. Despite these challenges and having lower completion of screening questions in the cell phone, the cell phone showed favorable completion rate than the landline among respondents who completed the screening questions. Thus, the success with cell phone populations is dependent on finding the factors responsible for the lower screening completion and the higher completion of the interview among those who completed the screening questions.
Disclosure: The BRFSS survey was reviewed by the Human Research Protection Office at the Centers for Disease Control and Prevention and determined to be exempt from human subject guidelines. Authors have no conflict of interest to disclose.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.