Introduction

The COVID-19 pandemic, which within a few months resulted in close to a million deaths worldwide (WHO 2020), presented a serious challenge for public health officials in the United States and around the world. The public’s willingness to receive the COVID-19 vaccines was critical. However, decades of vaccine misinformation promoting vaccine hesitancy, decreased trust in healthcare systems and decreased trust in general, including efforts to politicize vaccine misinformation during the pandemic presented a new challenge for successful vaccination efforts (Betsch et al. 2018; Bradshaw et al. 2020; Dubé et al. 2013; Elgar, Stefaniak, and Wohl 2020; Fisher et al. 2020; Goldenberg 2019; Kye and Hwang 2020; Kreps et al. 2020; Larson et al. 2014, 2015, 2018; Larson 2020; MacDonald and Dubé 2015; Momplaisir et al. 2021; Rönnerstrand 2013; Verger and Dubé 2020; Glaeser et al. 2000; Putnam 2001). The public’s willingness to vaccinate against COVID-19 and vaccination rates after vaccines became available have been of great interest to public health officials. This paper reports on survey data, collected in Boston, that asked about plans to vaccinate against COVID-19 before the vaccines became available, and then compares those survey estimates to the actual neighborhood-level vaccination rates after the vaccines became available.

To evaluate whether the willingness to vaccinate can help predict actual vaccination rates, we compare planned and actual rates by neighborhood and over time. According to the theory of planned behavior (TPB), developed by Ajzen (Ajzen and Fishbein 1977; Ajzen 1991, 2011; Shmueli 2021), one’s behavior (action) is determined by intention to perform the behavior (operationalized by asking people whether they intend to engage in the behavior), which is a function of one’s attitude toward performing the behavior and one’s subjective norms (Ajzen and Fishbein 1977). Ajzen examined correlations between intentions and behaviors and found mixed results, with low, partial, or high correlation in 84 out of 109 studies (Ajzen and Fishbein 1977). The intentions and behaviors are defined by the action itself, target, time, and context of a given action (Ajzen and Fishbein 1977; Ajzen 2011). Despite some criticisms, TPB has found applications in models predicting behaviors, including vaccination (Ajzen 2011; Shmueli 2021; S. Wang et al. 2020; Barattucci et al. 2022; Canova, Bobbio, and Manganelli 2020; Carfora et al. 2021; Asif et al. 2022; Saeri et al. 2014).

Our survey estimates were collected from a sample of residents from Boston neighborhoods, while the actual rates are based on the vaccination records for those neighborhoods. Our study provides an opportunity to compare survey estimates with actual vaccination rates to assess the ability of a self-reported planned vaccination measure to predict vaccination behaviors at the neighborhood level.

Methods

The Living in Boston during COVID Survey[1] employed a stratified random sample to ensure representation of Boston residents. Residential addresses were sampled randomly from 25 distinct neighborhoods and three neighborhoods with a high proportion of Black or Latinx populations were oversampled. The survey was conducted by mail with an option to complete it online. In addition, we supplemented this survey by inviting participants from an existing online panel of Boston residents sampled in 2018 from the same neighborhoods using only a web survey.

The survey was conducted in two parts. First, sampled residents received a mail survey in June 2020 and were able to complete it by mail or online. In September 2020, an invitation to the second part was sent only to respondents from Part 1 – via email to 47% of respondents who provided an email address and via USPS to those households who did not provide an email address. Part 1 achieved an overall response rate of 26.9% (AAPOR RR3), with 1370 respondents; of these, 53.8% (n=737) completed Part 2. Demographic questions were asked in Part 1 (June 2020), while the question about plans to vaccinate was asked in Part 2 (September 2020), so only those respondents who completed both parts of the survey are included in this analysis.[2] In addition, 195 members of the previously constituted online survey panel completed the Part 2 web survey and are included in this analysis.[3] The analytic dataset thus includes all survey respondents who completed both surveys (n=932).

To compare potential vaccination willingness and actual vaccination rates, we used the survey data and data from two Boston Public Health Commission (BPHC) COVID-19 vaccination reports from September 21, 2021 and July 5, 2022.[4] Although the BPHC COVID-19 vaccination reports were published weekly from March 2021 through 2023, we selected the September 21, 2021 and July 5, 2022 reports because they were published about 1 or 2 years after the survey, respectively. Since the BPHC reports presented vaccination rates based on dividing Boston into 15 neighborhoods, rather than the more detailed 25 neighborhoods used for the stratified sampling in the Living in Boston during COVID Survey, we added a variable indicating in which of the 15 BPHC-defined Boston neighborhoods respondents lived.[5]

Outcome variable

To measure plans to vaccinate against COVID-19, the survey asked the following question in September 2020: If a vaccine against the coronavirus becomes available, do you plan to get vaccinated - definitely, probably, probably not, definitely not? Table 1 shows both unweighted counts and weighted percentages for plans to vaccinate. Since a majority reported planning to vaccinate (78.9%), this variable was dichotomized into “plan to vaccinate” (definitely or probably) and “hesitant to vaccinate” (probably not or definitely not).

Table 1.Table of measures.
Outcome variable N Unweighted* % Weighted 95% CI
Plan to vaccinate if a vaccine against coronavirus becomes available
Definitely } Plan to vaccinate 461 41.4% 35.8% -⁠ 47.2%
Probably 321 37.5% 31.7% - 43.7%
Probably not } Hesitant to vaccinate 98 12.4% 8.8% - 17.2%
Definitely not 41 8.7% 5.4% - 13.7%
↓
Plan to vaccinate 782 78.9% 73.1% - 83.8%
Hesitant to vaccinate 139 21.1% 16.2% - 26.9%

* Total Unweighted Ns vary slightly between variables due to missing data

Analyses

The survey data were weighted to account for the probability of selection within the neighborhood and survey nonresponse. Post-stratification weights were calculated to match these data to the American Community Survey estimates for age, gender, race and Hispanic origin, and level of education.

To assess the ability of the intention to vaccinate to predict actual vaccination, we ran the variable “plan to vaccinate” by neighborhood (categorized into 15 neighborhoods), using IBM SPSS 28 Complex Samples that accounts for the stratified survey design effects on estimated variances. We then compared the percentages of those who planned to vaccinate in each neighborhood with actual vaccination rates of those who received at least one dose and those fully vaccinated, reported for the City of Boston by neighborhood in September 2021 and July 2022. We rank-ordered neighborhoods based on survey estimates for planning to vaccinate, from lowest to highest percentage, and grouped them into three ranges: “low” rates (below 75%), “medium” (75%-89.9%), and “high” (90% or greater). We compared neighborhood-level survey estimates with the actual rates for having received at least one dose or being fully vaccinated at two specified times. To assess the association between neighborhood-level vaccination plans and actual rates, we calculated Spearman’s rank-order correlation coefficients.

Results

The following analyses aimed to validate the use of a “plan to vaccinate” measure and its ability to predict actual vaccination rates. We have individual reports of plans to vaccinate but not of respondents getting vaccinated; therefore, the analyses were conducted at the neighborhood level.

The reported neighborhood-level planned vaccination rates ranged from 49.3% (Hyde Park) to 94.5% (Charlestown). We also calculated average planned vaccination rates for each of the three neighborhood groups (see Table 2, column 1).

Table 2
Table 2.Survey estimates for vaccination plans and actual neighborhood vaccination rates.

Next, we compared our survey estimates with the actual vaccination rates. Table 2 (columns 2-5) shows the percentages of actual vaccination by neighborhood in September 2021 and July 2022 for two measures (at least one dose and fully vaccinated). We found that the two neighborhoods with the lowest planned vaccination rates (Hyde Park and Dorchester 21-25) had actual vaccination rates that already surpassed planned rates by September 2021 for both measures. For Charlestown and West Roxbury, the two neighborhoods with the highest planned rates, the actual vaccination rates never reached or surpassed the reported planned vaccination rates.

Next, we compared the average planned rates for three groups (low, medium, and high) with the average actual rates (Table 2, columns 1-5, rates in bold). For the “low” group, the average actual rate for at least one dose in September 2021 already surpassed the group’s planned average rate (66.0% vs. 63.4%), while the average actual rate for fully vaccinated in September 2021 lagged (58.8% vs. 63.4%). In July 2022, both the average actual rate for at least one dose (82.6%) and for fully vaccinated (69.5%), surpassed the group’s average planned rate (63.4%). For the “medium” group, only the average actual rate for at least one dose in July 2022 surpassed the average planned rate (90.5% vs. 81.3%). For the “high” group, the average actual rates never reached or surpassed the reported average planned rate (92.5%).

Next, we examined the differences between the actual and planned rates to learn which neighborhoods had actual rates that were lower than planned rates by more than 3%, lower but within 3%, equal or surpassed by up to 9.9%, and which surpassed by 10.0% or more (Table 2, columns 6-9). When comparing the actual rates for at least one dose and planned rates, we found that in the “low” group, in September 2021, three of five neighborhoods surpassed the planned rates, and as a group, the actual average rate surpassed the average planned rate. By July 2022, all five neighborhoods had actual rates that surpassed the planned rates, and as a group, they were above expectation. For the “medium” group, in September 2021, no neighborhood was within 3%, and as a group, the actual average rate was below the average planned rate. By July 2022, the actual rates for all five “medium” neighborhoods surpassed the planned rates, and as a group, they were above expectation. For the “high” group, no neighborhood was within 3% of expectation in September 2021, the group’s average actual rate was 15.3% below expectation, but by July 2022, three of five neighborhoods surpassed the planned rate, although as a group they were still below expectation. Across neighborhoods, by July 2022, all but two surpassed the planned rates (Table 2, column 8).

Finally, we calculated Spearman’s rank-order correlation coefficients between neighborhood-level survey estimates for vaccination plans and actual vaccination rates at two times and for two measures. The neighborhood-level correlation between planned rates (asked in September 2020) and actual rates in September 2021 was significant (r=0.56, p<0.05 for at least one dose, and r=0.59, p<0.05 for being fully vaccinated), while in July 2022 it decreased (r=0.50, p=0.058 for at least one dose, and r=0.33, p=0.231 for being fully vaccinated)[6].

Discussion

Our findings show that a single survey question about vaccination plans can help predict actual vaccination rates. This ability of survey estimates to predict future behavior can be valuable to public health practitioners, notwithstanding the accuracy of those predictions and moderate correlation. Additionally, the time it took for the actual rates to be within 3% or to surpass our survey estimates was conceivably longer than public health officials might have hoped. Neighborhoods with the lowest planned vaccination rates were first to surpass those estimates, already in September 2021 for both measures, whereas the neighborhoods with the highest planned rates never reached nor surpassed the survey estimates. This suggests a type of floor and ceiling effect, as neighborhoods with the lowest initial rates were likely to increase, while those with the highest initial rates had little room for improvement. Furthermore, in neighborhoods with majority Black residents (e.g., Hyde Park and Dorchester), compared to majority White residents (e.g., Charlestown and West Roxbury) (Boston Planning & Development Agency Research Division 2021), vaccination rates varied in terms of the speed of uptake. According to the KFF COVID-19 Vaccine Monitor from March 2021, conducted after the vaccine rollout, Black adults had the biggest increase in reporting getting at least one dose or saying they want to get it as soon as possible (Hamel et al. 2021), and this trend is reflected in the actual vaccination already surpassing the planned rates in September 2021. We also found a significant correlation between the planned and actual rates in September 2021, yet a decreased correlation with the actual rates in July 2022. Per TPB (Ajzen and Fishbein 1977; Ajzen 2011), context and time impact the correlations, and in this case, by July 2022 the vaccination rate was already very high (the average rate for at least one dose was 88.4%), which may suggest that most neighborhoods reached a saturation point,[7] which in turn reduced the correlations; that is, there was no more significant variance to predict.

Another important finding is that by July 2022, most neighborhoods reached or surpassed our survey predictions for receiving at least one dose of vaccine. To better understand the time element, it is helpful to refer to the trends in the COVID-19 cases for Boston[8] which showed that cases were relatively low until January 2022, which might help explain why the actual rates in September 2021 were considerably lagging our survey estimates. Although we do not know the reason(s) for the lagging vaccination rates, we can posit that until the Omicron wave in the winter 2021–2022, most Boston residents did not personally know many people in their community who had contracted COVID-19, and therefore did not feel the urgency to vaccinate. However, by July 2022, the big spike in cases due to Omicron variants might help explain why the actual rates in July 2022 were much closer to, or even surpassed our survey estimates. Also, reports during the Omicron wave showed that people who were vaccinated had milder symptoms, which might have encouraged those who were hesitant and weighing the risks from COVID-19 and vaccines to ultimately decide to vaccinate. Our survey estimates came close to or surpassed the actual rates for at least one dose, not so much for fully vaccinated. In September 2020, when the survey took place, the general public knowledge was that scientists were working on developing a vaccine to help fight against COVID-19, not that people would need two doses to be “fully vaccinated,” or booster doses later on. It is therefore possible that some people felt that by getting at least one dose they got some (necessary) protection, and never got the second dose because they felt it was not necessary for them to be “fully” vaccinated.

Our findings have important limitations. First, our data are limited to adult Boston residents and cannot be generalized to populations in other parts of the country. Next, although we weighted our survey data to account for the probability of selection within the neighborhood and survey nonresponse, and we used post-stratification weight adjustments, it is possible that those who did not respond to our survey were different in significant ways from those who responded, and therefore our response rate presents an important limitation. Next, although we combined the data from the original 25 Boston neighborhoods into 15 neighborhoods, our sample for certain neighborhoods was still relatively small, resulting in a limited ability to analyze neighborhood-level differences more closely. Also, our survey estimates were based on the responses from adults 18 years of age and older, whereas the actual rates were for the Boston residents eligible to receive the vaccine, which corresponded to 12 years of age and older in September 2021 and 5 years of age and older in July 2022. This might help explain why the actual rates for at least one dose in most neighborhoods surpassed our survey estimates. However, according to the American Community Survey (Boston Planning & Development Agency Research Division 2020), 70% of Boston households in 2018 were without children, which is within the 95% confidence interval of our June 2020 survey estimate and just outside the interval for September 2020. Therefore, we believe that the effect of only surveying adults 18 years or older on our estimates would not have been large and could have been in either direction. Notwithstanding these limitations, we believe our findings are informative and important. Our analyses seized an opportunity to examine the ability of a survey question to predict vaccination rates and showed that survey estimates can be used to inform public health efforts. Our findings also provide important information on the limitations of such estimates. For public health practitioners who want to use a single survey question in future studies, it is important to know that both the value of the estimate and the time since the question was asked must be considered to properly interpret the meaning of the results.

More research is needed to better understand the individual and social factors contributing to vaccine hesitancy, even in unprecedented circumstances such as the COVID-19 pandemic. Future research should further examine the ability of survey estimates to predict actual rates, including vaccination rates as well as other measures of compliance with health recommendations, to help inform public health efforts, especially in extraordinary health circumstances such as the global pandemic.

Lead author contact information

Dragana Bolcic-Jankovic, PhD
Senior Research Fellow
Center for Survey Research
University of Massachusetts Boston
100 Morrissey Blvd. Boston, MA 02125
Phone: 617-287-6692
E-mail: dragana.bjankovic@umb.edu

Acknowledgments

We thank Daniel T. O’Brien (Professor of Public Policy and Urban Affairs and Criminology and Criminal Justice at Northeastern University; Director, Boston Area Research Initiative) and Hannah Grabowski (University of Massachusetts Boston) for their contributions to the design and implementation of the Living in Boston during COVID Survey.

  1. The study was conducted by the Boston Area Research Initiative (BARI), the Center for Survey Research at UMass Boston, and the Boston Public Health Commission, with funding from a National Science Foundation RAPID grant, Geography and Spatial Sciences program (BCS-2032384).

  2. Methodological details about the Living in Boston during COVID Survey, are presented elsewhere (https://www.umb.edu/csr/covid19-in-boston/methodolgy-nsf). We factored response to both parts in the response rate calculation.

  3. The panel was a stratified probability sample of Boston which used the same 25-neighborhood stratification scheme as the Living in Boston during COVID Survey. The panel data was originally weighted using the same methodology as was applied to the survey. To merge the panel and the survey data, new post-stratification adjustments were applied to the combined weighted data.

  4. Available at https://www.boston.gov/government/cabinets/boston-public-health-commission/covid-19-boston.

  5. The 15 BPHC-defined neighborhoods are Allston/Brighton, Back Bay/Beacon Hill/Downtown/North End/West End, Charlestown, Dorchester 21-25, Dorchester 22-24, East Boston, Fenway, Hyde Park, Jamaica Plain, Mattapan, Roslindale, Roxbury, South Boston, South End, and West Roxbury.

  6. Correlations are similar to those in a simulation analysis that combined our survey data with data from two other surveys (Y. Wang et al. 2021).

  7. We posit that saturation is associated with the herd immunity threshold, which typically requires 75–85% of the population to be vaccinated (Suryawanshi and Biswas 2023). Some argue that the saturation point can be subjective, depend on the discipline, context, time, and that it would be “impractical” for researchers to predict how many comparisons might be needed to reach the saturation point (Tight 2023). Our findings suggest that a reasonable timeframe for the comparisons would be between 1 year and 2 years after the survey. We believe that more frequent comparisons would be costly and possibly not practical or warranted.

  8. https://www.mass.gov/info-details/covid-19-response-reporting#covid-19-interactive-data-dashboard- accessed 10/5/2022.