During the first wave of the COVID-19 pandemic in New York City (NYC), SARS-CoV-2 diagnostic tests were generally limited to persons with severe COVID-19. As a result, laboratory test-based surveillance included few mild COVID-19 cases, resulting in an underestimation of the burden of COVID-19 (Wu et al. 2020). While availability of SARS-CoV-2 detection tests increased following the first wave, subsequent case surges, including the Omicron variant-driven wave, caused periods of constrained test availability. Furthermore, even in times of widespread test availability, surveillance that relies on laboratory confirmation of COVID-19 alone will not be representative of the population due to selection bias in who receives testing (Lieberman-Cribbin et al. 2020).
In recognition of these limitations, the NYC Health Department incorporated questions about symptoms consistent with COVID-19 (Alroy et al. 2021) into the Community Health Survey (CHS), an annual population-based survey (Levanon Seligson et al. 2021). This population-based surveillance of symptoms is an important complement to test-based surveillance, together providing more complete estimates of the burden of disease and a more representative picture of which NYC populations suffered a disproportionate burden of disease. However, because CHS collects data only during select months of the year, symptom questions were also added to a second Health Department survey, Healthy NYC, to provide coverage during the remaining months of the year. Additionally, symptom data were collected from a Department of Transportation-based citywide survey.
These three surveys each had different sampling frames and modes of administration, which could result in estimates of the burden of COVID-19 that differed by survey. We tested whether estimates of period prevalence and cumulative monthly incidence of COVID-19-like illness and COVID-19 differed across the three surveys during the two months in which the surveys were administered concurrently in order to support the use of symptom-based surveillance to monitor COVID-19 and thereby understand the full burden of illness caused by the COVID-19 pandemic.
Survey design and characteristics
We analyzed data collected in three surveys of adult residents of NYC during July and August 2020 (Table 1). The Community Health Survey (CHS) is an annual population-based survey that monitors the health conditions, behaviors, and healthcare utilization of NYC adult residents living in non-group quarters (New York City Department of Health and Mental Hygiene 2022). In 2020, participants were sampled via random digit dialing and interviews were completed over the phone from March through August. Because data for CHS were collected only during select months of each year, the Health Department also included questions about COVID-19-consistent symptoms in monthly surveys conducted among Healthy NYC members during the months when CHS did not collect data (September through December 2020), as well as one overlapping month (August 2020) to assess comparability of results. Healthy NYC is a probabilistically based panel established in 2020 in which NYC adult residents are recruited via address-based sampling to participate in periodic health-related surveys (Levanon Seligson et al. 2021). By August 2020, the panel included 4,474 persons who could be invited to participate in individual surveys via stratified random sampling. Respondents had the option to complete surveys online or by phone with most participants opting to complete surveys online. Lastly, the Citywide Mobility Survey (CMS), a population-based survey conducted by the NYC Department of Transportation to assess travel-related behavior, preferences, and attitudes of NYC residents (Department of Transportation 2022), added questions about COVID-19-consistent symptoms. The 2020 CMS focused on travel impacts of the COVID-19 pandemic. Respondents in 2020 were part of a panel formed by sampling 2019 CMS respondents (adult NYC residents recruited through an address-based sample) who consented to be recontacted. All CMS surveys were completed online and were administered in May, July, and October 2020. A summary of key attributes for the three surveys is shown in Table 1. COVID-19-related variables were collected from both CHS and CMS in July 2020 and from both CHS and Healthy NYC in August 2020.
In all three surveys, respondents were asked if they had experienced select COVID-19-consistent symptoms during the past 30 days, including cough, shortness of breath, difficulty breathing, loss of taste or sense of smell, fever, chills, muscle aches, headache, sore throat, vomiting, diarrhea, and nasal congestion. An individual with any one of cough, shortness of breath, difficulty breathing, or loss of taste or smell, or any two of the remaining symptoms was identified as a case of COVID-19-like illness (CLI) (Council of State and Territorial Epidemiologists 2020).
In addition to questions about COVID-19-consistent symptoms, additional COVID-19-related questions were asked in both the CHS and Healthy NYC surveys that could be compared between the two surveys, including the date of onset of symptoms; suspected COVID-19 since February 2020; experience of select COVID-19 symptoms since February 2020; and the extent of engaging in physical distancing during the previous 14 days. Question wording and response options were identical for most items across the three surveys (Table S1).
Respondent data from CHS were weighted to be representative of the NYC adult population living in nongroup quarters per American Community Survey population control totals for 2019. Healthy NYC data were weighted to the American Community Survey population control totals for 2019 to be representative of the NYC adult population excluding persons living in noninstitutional group quarters. CMS data were weighted to the American Community Survey population control totals for 2013–2017 (United States Census Bureau 2018), including all persons living in group quarters.
Period prevalence of CLI in the prior 30 days among respondents of a given month was calculated as the number of people with CLI out of all respondents with a nonmissing response for CLI. Respondents with incident CLI were identified as respondents with CLI with symptom onset in the month of interest. (Symptom onset was assessed only in CHS and Healthy NYC.) To estimate cumulative monthly CLI incidence for a given month, each respondent’s weight was multiplied by the proportion of the prior 30 days that fell within the month of interest. For example, when estimating cumulative CLI incidence in August, a respondent interviewed on August 5 would have a weight adjustment of 1/6 because only five days of the 30-day look-back period fell in the month of August. This adjustment accounts for variation among respondents in opportunities for ascertainment of incident CLI given variation in survey administration date.
For the purposes of COVID-19 surveillance, estimates of CLI period prevalence and cumulative monthly incidence for a given month were multiplied by the percentage of persons with CLI in NYC hospital emergency departments who tested positive for SARS-CoV-2 in the corresponding month (Lall et al. 2017) to calculate COVID-19 period prevalence and cumulative monthly incidence (Alroy et al. 2021). This percentage is applied equally across all three surveys. We compared estimates of CLI period prevalence in July 2020 between CHS and CMS and estimates of CLI period prevalence and cumulative monthly incidence in August 2020 between CHS and Healthy NYC. We also compared estimates of CLI period prevalence and cumulative monthly incidence among CHS respondents who did and did not consent to be contacted for future research as a proxy to better understand potential differences between respondents who were or were not sampled from a panel.
All analyses were completed with SAS Enterprise Guide Version 7.15 (SAS Institute Inc., Cary, North Carolina) and SUDAAN Version 11.0.03 (RTI International, Research Triangle Park, North Carolina). All estimates presented here are weighted; we tested differences across surveys with design-based t-tests and corresponding p-values. We considered two-sided p-values less than 0.05 to be statistically significant.
The New York City Department of Health and Mental Hygiene Institutional Review Board reviewed study procedures and determined them to be exempt pursuant to 45 CFR §46.104(d)(4)(ii) and (iii). This activity was also reviewed by the Centers for Disease Control and Prevention (CDC) and was conducted consistent with applicable federal law and CDC policy.
Across the three surveys in July and August 2020, between 805 and 1,144 respondents each month per survey were included in analyses. Weighted demographic characteristics were generally similar across all three surveys (Table 2). There were significant differences in the distribution of race/ethnicity and educational attainment between CHS and CMS, and in household poverty between CHS and Healthy NYC (Table 2).
In July 2020, an estimated 12.1% of adult NYC residents had prevalent CLI in the past 30 days per CHS, compared to an estimated period prevalence of 9.9% per CMS (p=0.511) (Table 3). Of citywide emergency department visits in which a patient had CLI and was tested for SARS-CoV-2, 13.6% were positive for SARS-CoV-2, translating to adult COVID-19 period prevalence of 1.6% as estimated via CHS and 1.3% as estimated via CMS. The period prevalence of individual COVID-19 symptoms was generally also similar across the two surveys, with the exception of chills, which was more common in CHS than CMS (2.7% vs. 2.0%, respectively, p=0.002) (Table 3).
In August 2020, an estimated 11.3% of adult NYC residents had prevalent CLI in the past 30 days as estimated in CHS, compared to 18.1% of adults per Healthy NYC (p=0.014). Among adults attending emergency departments with CLI who were tested for SARS-CoV-2 in August 2020, 3.7% were positive for SARS-CoV-2, suggesting that approximately 0.4% of adult NYC residents had prevalent COVID-19 in August per CHS and 0.7% per Healthy NYC. Period prevalence of most symptoms was similar across the two surveys, though fever was more common as estimated by CHS than by Healthy NYC and several other symptoms were more common per Healthy NYC than CHS estimates, including muscle aches, nasal congestion, and cough (Table 3). Estimated cumulative monthly incidence of CLI in August was not significantly different in CHS (4.0%) and Healthy NYC (5.7%; p=0.246), with no statistically significant differences in incidence of individual symptoms (Table 3). Among CHS respondents only, estimates of CLI period prevalence and cumulative monthly incidence among those who consented and those who declined to be contacted for future research were similar (period prevalence of 11.5% versus 11.0%, respectively, p=0.896; cumulative monthly incidence of 4.0% in both groups, p=0.956).
When asked about select symptoms, data from CHS collected in August 2020 show that 20.9% of adult NYC residents reported experiencing a fever, cough, shortness of breath, sore throat, or loss of taste or sense of smell since February 2020, similar to the proportion in Healthy NYC (17.4%, p=0.209) (Table 3). About one in 10 adult NYC residents reported thinking they may have had COVID-19 since February 2020 (9.8% per CHS and 10.1% per Healthy NYC, p=0.886). Finally, physical distancing none or some of the time was more common as estimated in CHS than Healthy NYC (both p<0.05), while physical distancing most of the time was more common per Healthy NYC than CHS (p<0.001) (Table 3).
In our comparison of three population-based surveys conducted during the summer of 2020, estimates of CLI period prevalence were similar between CHS and CMS, yet differed between CHS and Healthy NYC. In contrast, estimates of CLI cumulative monthly incidence in CHS and Healthy NYC were similar. We considered several differences in the sampling frame and mode of administration between the CHS and Healthy NYC surveys that might have contributed to the observed difference in estimated period prevalence across the two surveys.
First, CHS and Healthy NYC survey respondents in August 2020 were selected from different sampling frames. Persons completing CHS were selected from among all NYC adult residents with a telephone, whereas those completing the Healthy NYC survey were members of a survey panel. Panelists may differ from one-time respondents due to panel conditioning and attrition bias (Das, Toepoel, and van Soest 2007). However, when these data were collected, Healthy NYC panelists had been newly recruited to participate in the panel, with the August survey being the second they completed as panelists and the first they completed related to COVID-19, which may have limited conditioning and attrition effects. There may also be selection bias associated with panel participation if persons who participate in long-term research differ from the general population in unmeasured ways that are associated with COVID-19 symptom experience, recognition, or disclosure. To isolate any selection bias associated with panel participation, we also tested for any difference in estimates of CLI symptom period prevalence and cumulative monthly incidence among CHS respondents only, comparing those who did and did not consent to future research (as a proxy for persons who might or might not join a panel). In this comparison, estimates were similar between those who did and those who did not consent to be contacted for future research. Additionally, all 2020 CMS respondents were also members of a panel, yet estimates of COVID-19 period prevalence were similar between CHS and CMS.
Second, with respect to mode of administration, most Healthy NYC respondents completed their surveys online, while CHS respondents completed surveys via a telephone interview. Observable COVID-19-consistent symptoms, such as cough, have been stigmatized since the pandemic began (Williams and Dienes 2020), and respondents may have felt more comfortable disclosing symptoms online than over the phone. Additionally, at the time of survey administration, COVID-19 messaging encouraged sick individuals to stay home and contact tracers routinely reached out to persons with COVID-19 and their close contacts. CHS respondents may have associated a phone call asking about COVID-19 symptoms with phone calls for case and contact tracing; if so, this may have deterred them from disclosing symptoms if they thought doing so would lead to a quarantine requirement. This type of social desirability bias by survey mode has also been observed in prior studies (Christensen et al. 2014; Kreuter, Presser, and Tourangeau 2009). However, CHS respondents were more likely than Healthy NYC respondents to report physical distancing none or some of the time in the previous two weeks. If social desirability bias by mode played a role in the observed difference in CLI period prevalence estimates, we would have expected the opposite pattern: that Healthy NYC respondents completing surveys by the relatively more private online platform would feel more comfortable reporting a lack of physical distancing than CHS respondents.
Phone and web modes may also have produced different amounts of satisficing or multitasking, as has been observed in previous research (Aizpurua et al. 2018; Chang and Krosnick 2009). Completing the survey online may have afforded Healthy NYC respondents more time to consider the symptoms they had experienced over the past 30 days than CHS respondents, while CHS respondents may have been more likely to satisfice and multitask during a phone survey. However, CMS respondents also completed surveys online, and we observed similar estimates of CLI period prevalence in CHS and CMS in July 2020.
Finally, differences in the unweighted and weighted demographic composition of survey respondents may have affected results. Both CMS and Healthy NYC allowed participation from residents of congregate residential facilities (who were at higher risk of COVID-19 illness), while CHS did not. Despite this difference in population, though, differences in CLI period prevalence were observed only between CHS and Healthy NYC and not between CHS and CMS. The weighted Healthy NYC population had a significantly higher proportion of low-income respondents than the weighted CHS population, and point estimates suggest a weighted older age distribution in Healthy NYC than CHS, as well. Given that older persons are more likely to have symptomatic COVID-19 than younger persons (Wang et al. 2023) and that low-income persons were at higher risk of SARS-CoV-2 infection (Kim et al. 2021; Masterson et al. 2023), it is possible that these differences in the weighted populations may have contributed to observed differences in CLI period prevalence between CHS and Healthy NYC. Future analyses may consider alternative weighting approaches that include demographic characteristics associated with SARS-CoV-2 risk and symptoms when using data collected from multiple surveys.
This comparison of CLI period prevalence and cumulative monthly incidence estimates across multiple surveys benefitted from at least two strengths. First, the inclusion of three surveys with different combinations of similarities and differences allowed us to assess if observed differences in estimates were consistent by survey attribute to better understand what may contribute to different estimated CLI period prevalence. Additionally, comparison across surveys of responses to items beyond CLI symptoms, such as extent of physical distancing, provided a way to support or refute hypothesized social desirability bias. However, these analyses are restricted to a single month of overlap each between CHS with CMS and Healthy NYC, limiting our ability to assess if differences are consistent over time. The three surveys also differed in their inclusion of congregate residents and weighting approaches, which may have led to differences in both the unweighted and weighted distribution of demographic characteristics across surveys.
Across three population-based surveys, estimated CLI period prevalence significantly differed between two of the surveys, while other estimates of period prevalence and cumulative monthly incidence did not; differences were not consistent by survey mode nor sampling frame. Additional research to understand the potential sources of differences between CHS and Healthy NYC can support use of symptom-based surveillance to monitor COVID-19 and thereby understand the full burden of illness caused by the COVID-19 pandemic.
The findings and conclusions of this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention
Parts of this work were supported by the Centers for Disease Control and Prevention (CDC; Epidemiology and Laboratory Capacity grant 6 NU50CK000517-01-06).
Many thanks to William Carry, Ann McGrane, and others at the Department of Transportation for their contributions to and support of this work. Many thanks also to Tony He for his help with SAS programming. We also thank Jessica Sell, Sungwoo Lim, and Shu Meir Wang for their contributions to the symptom-based surveys implemented via the Community Health Survey and Healthy NYC.
§ See e.g., 45 C.F.R. part 46.102(l)(2), 21 C.F.R. part 56; 42 U.S.C. §241(d); 5 U.S.C. §552a; 44 U.S.C. §3501 et seq.