Polling on police use of body cameras: The effect of response option order on distribution and concurrent validity

The United States is engaged in a national conversation on policing. Debates on key features of policing are far from settled and are highly politicized. However, one point seems to have some convergence among the public and police: a majority of each favor the use of body cameras—66% of officers and 93% of the public—when asked whether they favor or oppose their use (Parker 2017).

Knowing that “favoring” overall might encompass several perspectives, we sought to document how frequently the public thought police should use body cameras: all the time with no exceptions, all the time with minimal exceptions, some of the time, or never. We were then confronted with a perennial survey design issue—how to order these response options in our web survey?

Beginning with the least desirable response option increases the likelihood that respondents consider a range of response options (Bradburn, Sudman, and Wansink 2004; Sudman and Bradburn 1982). Given the overwhelming public support favoring police use of body cameras, this suggests starting with “never.” However, we wanted to empirically examine whether doing so improved data quality. Thus, we randomly assigned respondents to receive one of two response option orderings: from “never” to “all the time with no exceptions” and the reverse. Importantly, because we asked respondents about their support for or attitudes about items related to policing, we had the opportunity to examine the concurrent validity of each of the two versions of the item by examining their correlation with the related items.

We fielded our survey from February through March 2021 using an online panel of respondents from Dynata, achieving a 65% participation rate (N=1,195). The population was adult residents of Cook County, IL, which includes Chicago and its immediate suburbs. The survey was administered in English, and individual questions were administered page-by-page.

Table 1 shows that response distributions varied by scale direction. Respondents were more likely to choose “all the time with no exceptions” when it was the first option listed compared to the last, and respondents were more likely to choose “some of the time” when “never” was presented first compared to last. This is consistent with the literature on response option order effects: options near the beginning of the scale, particularly the first response option respondents perceive as acceptable, are more likely to be chosen (Garbarski, Schaeffer, and Dykema 2019; Krosnick 1991; Yan and Keusch 2015; Yan, Keusch, and He 2018). We also confirmed that the difference in the distribution by response option order is largely similar across device type (desktop/laptop vs. smartphone/tablet, not shown) (Krebs and Höhne 2021; Leon, Aizpurua, and van der Valk, Forthcoming).

Table 2 shows the correlations between each version of the body camera question and criteria of interest. When the response options for body camera are ordered from “never” to “all the time with no exceptions,” the correlation is larger between body camera use and the following items: support for prosecuting police for use of excessive force and how well the police in the suburbs are doing using the right amount of force for each situation and treating people fairly, regardless of race or ethnicity.

Overall, the results suggest when polling about police use of body cameras in terms of relative frequencies, the response options should be ordered starting with “never” or the lowest frequency.