Adapting Clinical Instruments for Population Mental Health Surveillance: Should an Explicit “Don’t Know” Response Option Be Given?

Rachel Suss; Tashema Bholanath; Tenzin Yangchen Dongchung; Amber Levanon Seligson; Christina C. Norman; Sarah E. Dumas

doi:10.29115/SP-2023-0003

Suss, Rachel, Tashema Bholanath, Tenzin Yangchen Dongchung, Amber Levanon Seligson, Christina C. Norman, and Sarah E. Dumas. 2023. “Adapting Clinical Instruments for Population Mental Health Surveillance: Should an Explicit ‘Don’t Know’ Response Option Be Given?” Survey Practice 16 (1). https://doi.org/10.29115/SP-2023-0003.

View more stats

Abstract

When adapting clinical instruments for population surveys, the inclusion of an explicit “Don’t Know” (DK) response option is an important consideration. DK options for sensitive questions may reduce survey breakoff but may also increase item non-response. Data from the New York City Department of Health and Mental Hygiene indicate that offering an explicit DK response option to a screening instrument for probable anxiety and depression can significantly alter prevalence estimates of these mental health outcomes. Researchers should carefully consider whether including a DK option will yield the most accurate results.

In-Brief Note

Amid concerns of rising anxiety and depression driven by the COVID-19 pandemic (Vahratian et al. 2021), reliable and validated screening instruments to track population mental health are crucial. The 2-item Patient Health Questionnaire (PHQ-2) and 2-item Generalized Anxiety Disorder Scale (GAD-2) are widely used short screeners for probable depression and anxiety, respectively. While developed for use in clinical settings, these instruments have been validated for use in population surveys (Löwe et al. 2010). They begin with the question: “Over the last 2 weeks, how often have you been bothered by the following problems?” Explicit response options typically include “not at all,” “several days,” “more than half the days,” and “nearly every day.” When used in clinical settings these questions typically do not include a “Don’t Know” (DK) response option (Sapra et al. 2020). However, when used in surveys, some instruments have included DK (for example, National Health Interview Survey Questionnaire [Centers for Disease Control and Prevention 2019]), while others have not (for example, the Household Pulse Survey [United States Census Bureau 2022]). When adapting clinical instruments for surveys, does the inclusion of an explicit DK option impact population mental health prevalence estimates? We aimed to explore this question, considering two conflicting concepts in survey practice: (1) DK options in sensitive questions reduce survey breakoff, and (2) DK options increase item nonresponse (Alwin and Krosnick 1991; Schuman and Presser 1996).

During the COVID-19 pandemic, the New York City Department of Health and Mental Hygiene (NYC DOHMH) has conducted Health Opinion Polls (HOPs) using probability-based samples to assess health and health-related opinions among NYC adults. HOPs are primarily self-administered online; some responses are collected over the phone with an interviewer. From March 2021 to November 2021, four HOPs were conducted that included the GAD-2 and PHQ-2 scales with a DK option to mirror the response options of the other questions on the largely opinion-based survey. DK responses were recoded as missing and excluded from analysis. In August 2022, a HOP was conducted without a DK option, and probable depression and probable anxiety decreased relative to previous waves. To investigate if this change was due to the altered response options or a true change over time, in the October 2022 HOP, respondents were randomized to receive (n=683) or not receive (n=687) an explicit DK option on the GAD-2 and PHQ-2 scales.

The two groups did not differ significantly (α = 0.05) on major demographic characteristics or mode of survey completion (phone vs. web). The mental health questions were located at the end of the survey, and the two groups had similarly low breakoff numbers. However, consistent with previous literature (Schuman and Presser 1996), nonresponse (including DK responses) was higher among the group with the explicit DK option (67 vs. 4). The unweighted prevalence of probable anxiety was higher among the group with the explicit DK option (18.94% vs. 14.72%, χ²=4.24, p-value=0.0395), but probable depression was not significantly different between the two groups (15.03% vs. 11.99%, χ²=2.61, p-value=0.1060). Among individual items, scores on one item on the GAD-2 (not being able to stop or control worrying) differed significantly between the two groups (χ²=8.25, p-value=0.0412).

While further research comparing with a “gold standard” measure is needed to establish whether including a DK option yields estimates closer to the true prevalence, these results have implications for survey practice. Including an explicit DK option can significantly alter prevalence estimates of some mental health outcomes. When adapting clinical instruments for surveys, researchers should consider whether including a DK option will yield the most accurate results.

Funding

This work was supported by the Centers for Disease Control and Prevention Grants: Epidemiology and Laboratory Capacity for infectious Diseases SUPPLEMENTAL COVID-19 CARES: NU50CK00517, Epidemiology and Laboratory Capacity for infectious Diseases SUPPLEMENTAL COVID-19 PPPHCE NU50CK00517, ELC COVID Enhancing Detection Expansion SUPPLEMENTAL: NU50CK00517, Immunization & VFC COVID-3 CYCLE VACCINATION SUPPLEMENTAL: 6 NH23IP922636, CDC COVID-19 Crisis Response Workforce supplemental grant: 6 NU90TP922148, and CDC Public Health Infrastructure grant: 1NE11OE000057.

Competing interests

The authors have no competing interests to report.

Submitted: February 16, 2023 EDT

Accepted: March 23, 2023 EDT

References

Alwin, Duane F., and Jon A. Krosnick. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods & Research 20 (1): 139–81. https://doi.org/10.1177/0049124191020001005.

Google Scholar

Centers for Disease Control and Prevention. 2019. “2019 National Health Interview Survey (NHIS) Questionnaire.” https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Survey_Questionnaires/NHIS/2019/EnglishQuest.pdf.

Löwe, Bernd, Inka Wahl, Matthias Rose, Carsten Spitzer, Heide Glaesmer, Katja Wingenfeld, Antonius Schneider, and Elmar Brähler. 2010. “A 4-Item Measure of Depression and Anxiety: Validation and Standardization of the Patient Health Questionnaire-4 (PHQ-4) in the General Population.” Journal of Affective Disorders 122 (1–2): 86–95. https://doi.org/10.1016/j.jad.2009.06.019.

Google Scholar

Sapra, Amit, Priyanka Bhandari, Shivani Sharma, Trupesh Chanpura, and Lauri Lopp. 2020. “Using Generalized Anxiety Disorder-2 (GAD-2) and GAD-7 in a Primary Care Setting.” Cureus 12 (5). https://doi.org/10.7759/cureus.8224.

Google Scholar PubMed Central PubMed

Schuman, Howard, and Stanley Presser. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Thousand Oaks, CA: SAGE Publications, Inc.

Google Scholar

United States Census Bureau. 2022. “Phase 3.7 Household Pulse Survey.” https://www2.census.gov/programs-surveys/demo/technical-documentation/hhp/Phase_3-7_Household_Pulse_Survey_ENGLISH.pdf.

Vahratian, Anjel, Stephen J. Blumberg, Emily P. Terlizzi, and Jeannine S. Schiller. 2021. “Symptoms of Anxiety or Depressive Disorder and Use of Mental Health Care among Adults during the COVID-19 Pandemic — United States, August 2020–February 2021.” MMWR. Morbidity and Mortality Weekly Report 70 (13): 490–94. https://doi.org/10.15585/mmwr.mm7013e2.

Google Scholar PubMed Central PubMed