Introduction

During the first wave of the COVID-19 pandemic in New York City (NYC), SARS-CoV-2 diagnostic tests were generally limited to persons with severe COVID-19. As a result, laboratory test-based surveillance included few mild COVID-19 cases, resulting in an underestimation of the burden of COVID-19 (Wu et al. 2020). While availability of SARS-CoV-2 detection tests increased following the first wave, subsequent case surges, including the Omicron variant-driven wave, caused periods of constrained test availability. Furthermore, even in times of widespread test availability, surveillance that relies on laboratory confirmation of COVID-19 alone will not be representative of the population due to selection bias in who receives testing (Lieberman-Cribbin et al. 2020).

In recognition of these limitations, the NYC Health Department incorporated questions about symptoms consistent with COVID-19 (Alroy et al. 2021) into the Community Health Survey (CHS), an annual population-based survey (Levanon Seligson et al. 2021). This population-based surveillance of symptoms is an important complement to test-based surveillance, together providing more complete estimates of the burden of disease and a more representative picture of which NYC populations suffered a disproportionate burden of disease. However, because CHS collects data only during select months of the year, symptom questions were also added to a second Health Department survey, Healthy NYC, to provide coverage during the remaining months of the year. Additionally, symptom data were collected from a Department of Transportation-based citywide survey.

These three surveys each had different sampling frames and modes of administration, which could result in estimates of the burden of COVID-19 that differed by survey. We tested whether estimates of period prevalence and cumulative monthly incidence of COVID-19-like illness and COVID-19 differed across the three surveys during the two months in which the surveys were administered concurrently in order to support the use of symptom-based surveillance to monitor COVID-19 and thereby understand the full burden of illness caused by the COVID-19 pandemic.

Methods

Survey design and characteristics

We analyzed data collected in three surveys of adult residents of NYC during July and August 2020 (Table 1). The Community Health Survey (CHS) is an annual population-based survey that monitors the health conditions, behaviors, and healthcare utilization of NYC adult residents living in non-group quarters (New York City Department of Health and Mental Hygiene 2022). In 2020, participants were sampled via random digit dialing and interviews were completed over the phone from March through August. Because data for CHS were collected only during select months of each year, the Health Department also included questions about COVID-19-consistent symptoms in monthly surveys conducted among Healthy NYC members during the months when CHS did not collect data (September through December 2020), as well as one overlapping month (August 2020) to assess comparability of results. Healthy NYC is a probabilistically based panel established in 2020 in which NYC adult residents are recruited via address-based sampling to participate in periodic health-related surveys (Levanon Seligson et al. 2021). By August 2020, the panel included 4,474 persons who could be invited to participate in individual surveys via stratified random sampling. Respondents had the option to complete surveys online or by phone with most participants opting to complete surveys online. Lastly, the Citywide Mobility Survey (CMS), a population-based survey conducted by the NYC Department of Transportation to assess travel-related behavior, preferences, and attitudes of NYC residents (Department of Transportation 2022), added questions about COVID-19-consistent symptoms. The 2020 CMS focused on travel impacts of the COVID-19 pandemic. Respondents in 2020 were part of a panel formed by sampling 2019 CMS respondents (adult NYC residents recruited through an address-based sample) who consented to be recontacted. All CMS surveys were completed online and were administered in May, July, and October 2020. A summary of key attributes for the three surveys is shown in Table 1. COVID-19-related variables were collected from both CHS and CMS in July 2020 and from both CHS and Healthy NYC in August 2020.

Table 1.Summary of key methodological attributes of three population-based surveys administered in July and August 2020.
Attribute Community Health Survey Citywide Mobility Survey Healthy NYC
Sampling frame and sample selection Random digit dial with dual frame of mobile and landline telephones. Respondents were selected via a random sample stratified by 34 neighborhood groups. The 2020 CMS panel was formed from respondents to the 2019 CMS who agreed to be contacted for future surveys. 2019 CMS respondents were sampled from an address-based sample of all addresses in New York City. 2019 respondents were selected via stratified random sampling from ten DOT-defined zones, with oversampling in hard-to-reach (defined as ≥90th percentile of proportion of households with limited English-speaking or income below $35,000) block groups. All members of the 2020 CMS panel were invited to complete the COVID-19 survey. The Healthy NYC panel was formed primarily from an address-based sample of New York City addresses and additionally from respondents to prior health department probability-based surveys who consented to future contact. COVID-19 survey respondents were selected via stratified random sampling from among Healthy NYC panelists.
Inclusion criteria Adults aged 18 years or older living in non-group quarters in New York City who speak English, Spanish, Russian, Chinese, Bengali, or Haitian Creole. Adults aged 18 years or older living in New York City who speak English, Chinese, or Spanish. Adults aged 18 years or older living in New York City who speak English, Spanish, Chinese, or Russian.
Mode of survey completion Computer-assisted telephone interview Online Computer-assisted telephone interview (n=49) and online (n=756)
Weighting approach Initial weights were assigned as the inverse probability of selecting a telephone number within the respondent’s given sampling stratum. This weight was adjusted for nonresponse and by the number of landlines and adults in each household and further adjusted using a weight calibration algorithm so that weight sums would match select household-level (number of adults, presence of children, borough of residence) and person-level (sex, age, race/ethnicity, educational attainment) population totals derived from the 2019 American Community Survey, excluding persons living in group quarters. Initial weights were assigned according to the respondent’s probability of selection. An iterative proportional fit algorithm adjusted initial weights to match household-level (number of household members, number of workers, presence of children, age of head of household, number of vehicles, income) and person-level (sex, age, employment status, student status, commute method, race/ethnicity) characteristics in the 2013-2017 American Community Survey population control totals. Initial weights were assigned as the inverse probability of selection from the sampling frame from which the respondent was selected. Among respondents recruited via a previous probability-based survey, weights were adjusted for non-response bias using household-level predictors. Weights were then adjusted with an iterative proportional fit algorithm so that the distribution of select characteristics of included households matched the distribution of the same characteristics of each respective sampling frame. A pooling factor was then applied to account for overlap in the sampling frames. Finally, weights were adjusted with an iterative proportional fit algorithm to match person-level (sex, age, race/ethnicity, educational attainment) characteristics in the 2018 American Community Survey population control totals excluding persons living in non-institutional group quarters.
Placement of COVID-19 items Middle of survey End of survey COVID-19 items were administered as a standalone survey.
Treatment of non-substantive responses "Don't know" and "refused" response options were not explicitly offered to respondents, but were selected if offered by respondent. "Prefer not to answer" was explicitly offered as a response option for COVID-19 symptom items. For surveys conducted over the phone, "don't know" and "refused" response options were not explicitly offered to respondents, but were selected if offered by respondent. For surveys completed online, non-response to questions was coded as "refused".
Participation rates a 7.4% (response rate); 74.4% (cooperation rate) 50.2% 56.1%

a Response and cooperation rates for monthly surveys are not available for Community Health Survey. Instead, response and cooperation rates for the 2020 annual Community Health Survey are provided. Participation rates are provided for the July Community Mobility Survey and the August Healthy NYC survey.

In all three surveys, respondents were asked if they had experienced select COVID-19-consistent symptoms during the past 30 days, including cough, shortness of breath, difficulty breathing, loss of taste or sense of smell, fever, chills, muscle aches, headache, sore throat, vomiting, diarrhea, and nasal congestion. An individual with any one of cough, shortness of breath, difficulty breathing, or loss of taste or smell, or any two of the remaining symptoms was identified as a case of COVID-19-like illness (CLI) (Council of State and Territorial Epidemiologists 2020).

In addition to questions about COVID-19-consistent symptoms, additional COVID-19-related questions were asked in both the CHS and Healthy NYC surveys that could be compared between the two surveys, including the date of onset of symptoms; suspected COVID-19 since February 2020; experience of select COVID-19 symptoms since February 2020; and the extent of engaging in physical distancing during the previous 14 days. Question wording and response options were identical for most items across the three surveys (Table S1).

Respondent data from CHS were weighted to be representative of the NYC adult population living in nongroup quarters per American Community Survey population control totals for 2019. Healthy NYC data were weighted to the American Community Survey population control totals for 2019 to be representative of the NYC adult population excluding persons living in noninstitutional group quarters. CMS data were weighted to the American Community Survey population control totals for 2013–2017 (United States Census Bureau 2018), including all persons living in group quarters.

Analysis

Period prevalence of CLI in the prior 30 days among respondents of a given month was calculated as the number of people with CLI out of all respondents with a nonmissing response for CLI. Respondents with incident CLI were identified as respondents with CLI with symptom onset in the month of interest. (Symptom onset was assessed only in CHS and Healthy NYC.) To estimate cumulative monthly CLI incidence for a given month, each respondent’s weight was multiplied by the proportion of the prior 30 days that fell within the month of interest. For example, when estimating cumulative CLI incidence in August, a respondent interviewed on August 5 would have a weight adjustment of 1/6 because only five days of the 30-day look-back period fell in the month of August. This adjustment accounts for variation among respondents in opportunities for ascertainment of incident CLI given variation in survey administration date.

For the purposes of COVID-19 surveillance, estimates of CLI period prevalence and cumulative monthly incidence for a given month were multiplied by the percentage of persons with CLI in NYC hospital emergency departments who tested positive for SARS-CoV-2 in the corresponding month (Lall et al. 2017) to calculate COVID-19 period prevalence and cumulative monthly incidence (Alroy et al. 2021). This percentage is applied equally across all three surveys. We compared estimates of CLI period prevalence in July 2020 between CHS and CMS and estimates of CLI period prevalence and cumulative monthly incidence in August 2020 between CHS and Healthy NYC. We also compared estimates of CLI period prevalence and cumulative monthly incidence among CHS respondents who did and did not consent to be contacted for future research as a proxy to better understand potential differences between respondents who were or were not sampled from a panel.

All analyses were completed with SAS Enterprise Guide Version 7.15 (SAS Institute Inc., Cary, North Carolina) and SUDAAN Version 11.0.03 (RTI International, Research Triangle Park, North Carolina). All estimates presented here are weighted; we tested differences across surveys with design-based t-tests and corresponding p-values. We considered two-sided p-values less than 0.05 to be statistically significant.

The New York City Department of Health and Mental Hygiene Institutional Review Board reviewed study procedures and determined them to be exempt pursuant to 45 CFR §46.104(d)(4)(ii) and (iii). This activity was also reviewed by the Centers for Disease Control and Prevention (CDC) and was conducted consistent with applicable federal law and CDC policy.[1]

Results

Across the three surveys in July and August 2020, between 805 and 1,144 respondents each month per survey were included in analyses. Weighted demographic characteristics were generally similar across all three surveys (Table 2). There were significant differences in the distribution of race/ethnicity and educational attainment between CHS and CMS, and in household poverty between CHS and Healthy NYC (Table 2).

Table 2.Weighteda distribution of demographic characteristics of respondents from three population-based surveys of adult New York City residents, July and August 2020.
July 2020 August 2020
Community Health Survey (CHS)b,
n = 857		 Citywide Mobility Survey (CMS),
n = 939		 Community Health Survey (CHS),
n = 1,144		 Healthy NYC (HNYC),
n = 805
% % pc % % pc
Age, in years
18-24 13 10.1 0.442 12.9 8 0.058
25-44 40.2 41 0.868 40.4 41.9 0.683
45-64 31.5 30.7 0.868 31.5 30.6 0.795
65+ 15.3 18.2 0.5 15.3 19.6 0.152
Sex at birth
Female -- -- -- 53.6 53.2 0.921
Male -- -- -- 46.4 46.8 0.921
Race/ethnicity
Asian/Pacific Islander 13.5 20.8 0.068 13.3 15.1 0.478
Non-Hispanic Black 22 13.8 0.027 22 21 0.778
Hispanic 26.9 28.8 0.702 26.9 28 0.758
Non-Hispanic White 35.7 34.4 0.775 35.6 33 0.454
Multi-Racial or Other 1.9 2.2 0.752 2.3 2.9 0.582
Country of birth
United States -- -- -- 56.5 61.9 0.152
Outside of U.S. -- -- -- 43.5 38.1 0.152
Marital status
Never married -- -- -- 32.4 33.3 0.819
Married or partnered -- -- -- 50.4 50.1 0.932
Divorced or separated -- -- -- 11.5 13.1 0.553
Widowed -- -- -- 5.7 3.5 0.162
Sexual orientation
Gay, lesbian, bisexual, or other orientation -- -- -- 10.2 12.2 0.406
Straight -- -- -- 89.8 87.8 0.406
Educational attainment
High school degree or less 41.2 7.8 <0.001 41.7 40.7 0.805
Some college 22.6 24.9 0.627 22.2 21.9 0.923
College degree or more 36.2 67.3 <0.001 36.1 37.4 0.707
Household povertyd
Low income -- -- -- 37.2 46.2 0.01
Medium-high income -- -- -- 62.8 53.8 0.01
Borough
The Bronx 15.6 17.6 0.663 15.9 16.7 0.741
Brooklyn 30.7 21.8 0.053 30.2 29.9 0.918
Manhattan 21.1 23.8 0.496 20.3 19.9 0.904
Queens 27.3 29.9 0.609 28.1 27.7 0.911
Staten Island 5.4 7 0.505 5.5 5.8 0.852

a Percentages are weighted per the descriptions in Table 1.
b Select items were not available in the Citywide Mobility Survey data; the comparison of the distribution of these items with Community Health Survey data is omitted in these instances.
c p-value obtained from a Student’s t-test.
d Low income defined as household with income less than 200% of the federal poverty level. Medium-high income defined as households with income greater than or equal to 200% of the federal poverty level.

In July 2020, an estimated 12.1% of adult NYC residents had prevalent CLI in the past 30 days per CHS, compared to an estimated period prevalence of 9.9% per CMS (p=0.511) (Table 3). Of citywide emergency department visits in which a patient had CLI and was tested for SARS-CoV-2, 13.6% were positive for SARS-CoV-2, translating to adult COVID-19 period prevalence of 1.6% as estimated via CHS and 1.3% as estimated via CMS. The period prevalence of individual COVID-19 symptoms was generally also similar across the two surveys, with the exception of chills, which was more common in CHS than CMS (2.7% vs. 2.0%, respectively, p=0.002) (Table 3).

Table 3.Comparison of COVID-19 outcomes and physical distancing behaviors estimated from three population-based surveys of New York City residents, July and August 2020.
July 2020 August 2020
Prevalence % Prevalence, % Cumulative monthly incidence, %
Outcome CHS CMS pa CHS HNYC pa CHS HNYC pa
n = 857 n = 939 n = 1,144 n = 805 n = 1,144 n = 805
COVID-19-like illness 12.1 9.9 0.511 11.3 18.1 0.014 4 5.7 0.246
COVID-19 1.6 1.3 -- 0.4 0.7 -- 0.1 0.2 --
Symptoms
Cough 6.3 5.7 0.769 4.7 9.9 0.01 2.3 4.7 0.07
Shortness of breath or difficulty breathing 3 2.3 0.575 3.6 4 0.786 0.4 0.7 0.626
Loss of sense of smell 1.6 0.7 0.315 1.3 1.9 0.571 0.1 0 0.541
Loss of sense of taste 1.7 0.8 0.34 1.9 1.8 0.955 0.4 0.3 0.782
Fever 3.5 1.6 0.072 3.5 1.2 0.035 0.8 0.1 0.055
Chills 2.7 0.2 0.002 1.9 1.9 0.994 0.3 1.1 0.109
Muscle aches 4.9 4.5 0.862 4.2 9.9 0.011 1.3 2.3 0.312
Headaches 6.2 4.9 0.676 5.1 5.2 0.91 2.1 1.6 0.473
Sore throat 2.6 1.4 0.3 3.4 3.4 0.981 0.7 1.7 0.154
Vomiting 0.9 1.5 0.58 1.3 0.5 0.144 0.8 0.1 0.1
Diarrhea 5.9 4.5 0.634 7.1 8.5 0.476 4.4 2.1 0.084
Nasal congestion 1.7 2 0.83 1.3 4.2 0.011 0.7 2.3 0.069
Proportion reporting select COVID-19 symptomsb since February 2020 -- -- -- 20.9 17.4 0.209 -- -- --
Proportion reporting suspected SARS-CoV-2 infection or COVID-19 since February 2020 -- -- -- 9.8 10.1 0.886 -- -- --
Extent of staying at home and avoiding interactions with others outside the home except for essential needs in past two weeksc
None of the time -- -- -- 13 8.6 0.045 -- -- --
Some of the time -- -- -- 23.6 17.1 0.035 -- -- --
Most of the time -- -- -- 37.9 53.4 <0.001 -- -- --
All of the time -- -- -- 25.6 20.9 0.142 -- -- --

a p-value obtained from Student’s t-test.
b Symptoms included fever, cough, shortness of breath, sore throat, or loss of taste or loss of smell.
c Essential needs were defined for survey respondents to “include getting groceries, prescriptions filled, doing laundry, etc.”
CHS = Community Health Survey; CMS = Community Mobility Survey; HNYC = Healthy New York City;

In August 2020, an estimated 11.3% of adult NYC residents had prevalent CLI in the past 30 days as estimated in CHS, compared to 18.1% of adults per Healthy NYC (p=0.014). Among adults attending emergency departments with CLI who were tested for SARS-CoV-2 in August 2020, 3.7% were positive for SARS-CoV-2, suggesting that approximately 0.4% of adult NYC residents had prevalent COVID-19 in August per CHS and 0.7% per Healthy NYC. Period prevalence of most symptoms was similar across the two surveys, though fever was more common as estimated by CHS than by Healthy NYC and several other symptoms were more common per Healthy NYC than CHS estimates, including muscle aches, nasal congestion, and cough (Table 3). Estimated cumulative monthly incidence of CLI in August was not significantly different in CHS (4.0%) and Healthy NYC (5.7%; p=0.246), with no statistically significant differences in incidence of individual symptoms (Table 3). Among CHS respondents only, estimates of CLI period prevalence and cumulative monthly incidence among those who consented and those who declined to be contacted for future research were similar (period prevalence of 11.5% versus 11.0%, respectively, p=0.896; cumulative monthly incidence of 4.0% in both groups, p=0.956).

When asked about select symptoms, data from CHS collected in August 2020 show that 20.9% of adult NYC residents reported experiencing a fever, cough, shortness of breath, sore throat, or loss of taste or sense of smell since February 2020, similar to the proportion in Healthy NYC (17.4%, p=0.209) (Table 3). About one in 10 adult NYC residents reported thinking they may have had COVID-19 since February 2020 (9.8% per CHS and 10.1% per Healthy NYC, p=0.886). Finally, physical distancing none or some of the time was more common as estimated in CHS than Healthy NYC (both p<0.05), while physical distancing most of the time was more common per Healthy NYC than CHS (p<0.001) (Table 3).

Discussion

In our comparison of three population-based surveys conducted during the summer of 2020, estimates of CLI period prevalence were similar between CHS and CMS, yet differed between CHS and Healthy NYC. In contrast, estimates of CLI cumulative monthly incidence in CHS and Healthy NYC were similar. We considered several differences in the sampling frame and mode of administration between the CHS and Healthy NYC surveys that might have contributed to the observed difference in estimated period prevalence across the two surveys.

First, CHS and Healthy NYC survey respondents in August 2020 were selected from different sampling frames. Persons completing CHS were selected from among all NYC adult residents with a telephone, whereas those completing the Healthy NYC survey were members of a survey panel. Panelists may differ from one-time respondents due to panel conditioning and attrition bias (Das, Toepoel, and van Soest 2007). However, when these data were collected, Healthy NYC panelists had been newly recruited to participate in the panel, with the August survey being the second they completed as panelists and the first they completed related to COVID-19, which may have limited conditioning and attrition effects. There may also be selection bias associated with panel participation if persons who participate in long-term research differ from the general population in unmeasured ways that are associated with COVID-19 symptom experience, recognition, or disclosure. To isolate any selection bias associated with panel participation, we also tested for any difference in estimates of CLI symptom period prevalence and cumulative monthly incidence among CHS respondents only, comparing those who did and did not consent to future research (as a proxy for persons who might or might not join a panel). In this comparison, estimates were similar between those who did and those who did not consent to be contacted for future research. Additionally, all 2020 CMS respondents were also members of a panel, yet estimates of COVID-19 period prevalence were similar between CHS and CMS.

Second, with respect to mode of administration, most Healthy NYC respondents completed their surveys online, while CHS respondents completed surveys via a telephone interview. Observable COVID-19-consistent symptoms, such as cough, have been stigmatized since the pandemic began (Williams and Dienes 2020), and respondents may have felt more comfortable disclosing symptoms online than over the phone. Additionally, at the time of survey administration, COVID-19 messaging encouraged sick individuals to stay home and contact tracers routinely reached out to persons with COVID-19 and their close contacts. CHS respondents may have associated a phone call asking about COVID-19 symptoms with phone calls for case and contact tracing; if so, this may have deterred them from disclosing symptoms if they thought doing so would lead to a quarantine requirement. This type of social desirability bias by survey mode has also been observed in prior studies (Christensen et al. 2014; Kreuter, Presser, and Tourangeau 2009). However, CHS respondents were more likely than Healthy NYC respondents to report physical distancing none or some of the time in the previous two weeks. If social desirability bias by mode played a role in the observed difference in CLI period prevalence estimates, we would have expected the opposite pattern: that Healthy NYC respondents completing surveys by the relatively more private online platform would feel more comfortable reporting a lack of physical distancing than CHS respondents.

Phone and web modes may also have produced different amounts of satisficing or multitasking, as has been observed in previous research (Aizpurua et al. 2018; Chang and Krosnick 2009). Completing the survey online may have afforded Healthy NYC respondents more time to consider the symptoms they had experienced over the past 30 days than CHS respondents, while CHS respondents may have been more likely to satisfice and multitask during a phone survey. However, CMS respondents also completed surveys online, and we observed similar estimates of CLI period prevalence in CHS and CMS in July 2020.

Finally, differences in the unweighted and weighted demographic composition of survey respondents may have affected results. Both CMS and Healthy NYC allowed participation from residents of congregate residential facilities (who were at higher risk of COVID-19 illness), while CHS did not. Despite this difference in population, though, differences in CLI period prevalence were observed only between CHS and Healthy NYC and not between CHS and CMS. The weighted Healthy NYC population had a significantly higher proportion of low-income respondents than the weighted CHS population, and point estimates suggest a weighted older age distribution in Healthy NYC than CHS, as well. Given that older persons are more likely to have symptomatic COVID-19 than younger persons (Wang et al. 2023) and that low-income persons were at higher risk of SARS-CoV-2 infection (Kim et al. 2021; Masterson et al. 2023), it is possible that these differences in the weighted populations may have contributed to observed differences in CLI period prevalence between CHS and Healthy NYC. Future analyses may consider alternative weighting approaches that include demographic characteristics associated with SARS-CoV-2 risk and symptoms when using data collected from multiple surveys.

This comparison of CLI period prevalence and cumulative monthly incidence estimates across multiple surveys benefitted from at least two strengths. First, the inclusion of three surveys with different combinations of similarities and differences allowed us to assess if observed differences in estimates were consistent by survey attribute to better understand what may contribute to different estimated CLI period prevalence. Additionally, comparison across surveys of responses to items beyond CLI symptoms, such as extent of physical distancing, provided a way to support or refute hypothesized social desirability bias. However, these analyses are restricted to a single month of overlap each between CHS with CMS and Healthy NYC, limiting our ability to assess if differences are consistent over time. The three surveys also differed in their inclusion of congregate residents and weighting approaches, which may have led to differences in both the unweighted and weighted distribution of demographic characteristics across surveys.

Conclusion

Across three population-based surveys, estimated CLI period prevalence significantly differed between two of the surveys, while other estimates of period prevalence and cumulative monthly incidence did not; differences were not consistent by survey mode nor sampling frame. Additional research to understand the potential sources of differences between CHS and Healthy NYC can support use of symptom-based surveillance to monitor COVID-19 and thereby understand the full burden of illness caused by the COVID-19 pandemic.

Disclaimer

The findings and conclusions of this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention

Acknowledgments

Parts of this work were supported by the Centers for Disease Control and Prevention (CDC; Epidemiology and Laboratory Capacity grant 6 NU50CK000517-01-06).

Many thanks to William Carry, Ann McGrane, and others at the Department of Transportation for their contributions to and support of this work. Many thanks also to Tony He for his help with SAS programming. We also thank Jessica Sell, Sungwoo Lim, and Shu Meir Wang for their contributions to the symptom-based surveys implemented via the Community Health Survey and Healthy NYC.

  1. § See e.g., 45 C.F.R. part 46.102(l)(2), 21 C.F.R. part 56; 42 U.S.C. §241(d); 5 U.S.C. §552a; 44 U.S.C. §3501 et seq.