Non-probability online panels for Non-Communicable Diseases (NCDs) surveillance: evidence from a population-based comparison in Brazil

INTRODUCTION

Health surveys are essential for monitoring behavioral risk factors and guiding prevention strategies at national and international levels (Brasil 2021). However, declining response rates and rising operational costs have raised concerns about their sustainability, especially in low- and middle-income countries (Brasil 2023).

Alternative strategies have been adopted to address these challenges, such as web-based surveys (De Leeuw 2018; Dillman 2017). They have gained prominence due to their lower cost, flexibility, and faster data collection, and are often used either as stand-alone methods or combined with traditional methods to address non-response (De Leeuw 2018). They can be administered using probability or non-probability samples across a range of platforms. Among non-probability approaches, online opt-in panels have emerged as platforms through which individuals volunteer to participate in surveys, enabling rapid data collection from a pre-recruited pool of respondents (Baker et al. 2010). However, they rely on internet access and self-enrollment, and use non-probabilistic samples, making them vulnerable to coverage gaps and selection bias, which may result in large and potentially unpredictable errors in population estimates, even with substantial sample sizes (McPhee et al. 2022; Meng 2018).

In Brazil, annual telephone surveys, particularly the “Surveillance System for Risk and Protective Factors for Chronic Diseases by Telephone Survey” – Vigitel - have been used since 2006 to monitor non-communicable diseases (NCD) and related risk factors (Brasil 2023). These surveys have faced increasing costs and declining response rates, raising concerns about their sustainability (Sousa et al. 2025). However, the exploration of alternative methodological approaches, particularly web-based data collection, remains limited.

Although web-based surveys have become more widespread globally, doubts remain about their suitability for population health surveillance. Most evidence comes from high-income countries and focuses on isolated applications rather than routine systems. In middle-income settings, there is limited evidence on whether non-probability web panels can produce comparable estimates, and little is known about how differences in data collection method and sampling design influence the measurement of behavior. Thus, this study aims to provide evidence from Brazil to assess whether non-probability web panels can produce estimates comparable to routine probability-based monitoring systems.

METHODS

Study Design

We conducted a methodological study comparing data from a non-probability online panel with those from a probability-based telephone survey. The analysis focused on differences related to both the data collection mode (online vs. telephone) and sampling design (non-probability vs. probability), capturing their effects on prevalence estimates of risk and protective factors for NCD. These mode and sampling differences are described collectively as ‘survey approach’ because the study design does not allow mode and sample effects to be distinguished.

Sampling and data collection

A minimum sample size of 500 individuals per survey was defined to ensure a maximum margin of error of ±4.4 percentage points at a 5% significance level, assuming maximum variability. Both surveys targeted adults (≥18 years) residing in Belo Horizonte, a Brazilian urban center with approximately 2.3 million inhabitants and a high Human Development Index (HDI = 0.810) (Instituto Brasileiro de Geografia e Estatística, n.d.), reflecting relatively high telephone and internet coverage.

Online panel. Data collection took place between February and March 2023 using an opt-in, non-probability online panel managed by a market research company. The panel comprises over 150,000 registered users who voluntarily enroll through online recruitment, including website, social media campaigns, and partnerships with digital platforms. The sociodemographic profile of panelists broadly reflects Brazil’s internet connected population. Panelists’ sociodemographic information is periodically updated, and basic quality control procedures are applied by the provider (e.g., duplicate detection and response consistency checks) (Opinion Box 2022). For this study, approximately 20,000 panel members residing in Belo Horizonte constituted the eligible pool. Invitations were sent to 5,259 individuals, of whom 540 completed the questionnaire (Table 1). Invitations were randomly distributed based on predefined sociodemographic quotas to limit extreme imbalances in sample composition: at least 20% of the total required interviews should be completed by each age group, 30% by each education level, and a minimum of 150 interviews by sex. These quotas established minimum thresholds across categories but did not ensure representativeness. When necessary, additional participants were recruited through targeted digital advertisements to meet quota requirements.

Participants received small incentives upon completion (e.g., mobile phone credits or reward points), consistent with standard panel practices. A reminder was sent five days after the initial invitation to non-respondents.

Telephone survey. The telephone survey was conducted during the same period using a probability-based design based on Random Digit Dialing, including both landline and mobile phone numbers. A total of 14,900 telephone numbers were generated, of which 1,951 were classified as eligible, resulting in 504 completed interviews (Table 1). For landline interviews, one adult was randomly selected among household residents using a computerized within-household roster selection. For mobile phone interviews, the respondent was the phone user (an individual device). Numbers corresponding to businesses, inactive lines, or non-existent numbers were classified as ineligible. Numbers with no response after six attempts at different days and times were classified as unknown eligibility. These cases with unknown eligibility were incorporated into the calculation of the American Association for Public Opinion Research (AAPOR) Response Rate 3. The refusal rate was calculated using the AAPOR Refusal Rate 2, in accordance with AAPOR guidelines (American Association for Public Opinion Research 2023).

Both surveys used a questionnaire based on Vigitel, the Brazilian surveillance survey on NCD (Brasil 2023), with minor adaptations for the self-administered web format (Supplementary Materials).

Data organization

We analyzed lifestyle and health indicators, all of which are included in the Brazilian action plan to address NCD (Brasil 2021): consumption of fruits and vegetables (≥5 servings/day), sweetened beverages (≥5 days/week), and ultra-processed foods (≥5 items on the day before the interview), leisure-time physical activity (≥150 minutes of moderate-intensity or ≥75 minutes of vigorous-intensity activity/week), heavy episodic drinking, smoking, and health conditions, including overweight and obesity based on Body Mass Index (≥25 kg/m² and ≥30 kg/m², respectively), as well as self-reported arterial hypertension, diabetes, and depression.

Sociodemographic variables included were standardized by sex (male and female), age group (18-34, 35-54, and ≥ 55 years), and education level (no formal education to incomplete university degree and completed university degree or higher).

Weighting procedures were applied to both surveys to improve comparability. For the telephone survey, base weights reflecting selection probabilities were first computed, accounting for the number of telephone lines and the number of adults in the household. These base weights were subsequently adjusted through calibration using raking to align the sample with known population distributions by sex, age group, and education level. The population parameters used for calibration were based on the 2010 Brazilian Demographic Census and updated using intercensal population projections. For the online panel, no base weights were available due to the non-probability design. Therefore, only calibration weights were applied using the same raking procedure to align the sample distribution with population margins. No adjustments were made for non-response, as the primary objective was to align the sample distribution with known population parameters.

Data Collection Performance

The telephone survey showed an eligibility of 23.2%, a response rate of 6.3%, which was higher in mobile telephones, and a refusal rate of 1.5%. For the online panel, the completion rate was 40.2%, and the break-off rate was approximately 60%. Telephone interviews lasted an average of 18 minutes, with landlines taking longer (20 minutes) than mobile interviews (16 minutes), whereas the online survey averaged 10 minutes. The cost per completed interview was 2.3 times lower for the online panel (Table 1).

Data Analysis

We described the study populations using unweighted and weighted prevalence estimates with 95% confidence intervals (CI). For the telephone survey, variance estimation accounted for the sampling design and weights. For the online panel, confidence intervals were calculated using weighted estimates combined with model-based variance estimators (McPhee et al. 2022).

Relative differences between methods were calculated, treating estimates from the telephone survey as the reference (standard method in Brazil) (Brasil 2023). Positive differences indicated higher estimates in the online panel than in the telephone survey, while negative differences indicated lower online estimates, reflecting the potential impact of methodological effects on precision and accuracy. This allows the reliability of a new survey to be evaluated when prior population data are unavailable, while acknowledging potential loss of precision (Kalton 1983). The calculation is expressed as:

$$\small{\text{a) } Relative\ Difference = \frac{(\%\ online\ panel) - (\%\ telephone\ survey)}{\%\ telephone\ survey} \times 100}$$

We evaluated the variations in indicators based on the method of data collection (telephone or online) with Poisson regression models with robust variance, which is suitable for modeling binary outcomes with higher prevalence, such as NCD-related indicators (Barros and Hirakata 2003). The model estimating the association between the survey approach and health outcomes was:

$$\text{b) } \log\left( \mu_{i} \right) = \beta 0 + \beta 1 \times {Method\ of\ Data\ Collection}_{i}$$

Where μ_i is the expected probability of the outcome for individuals, i. Method of Data Collection_i indicates which method of data collection was used: 0= telephone (reference) and 1= online. β₀ is the intercept, and β₁ the log difference in outcome probability associated with the online method. The association measure was the Prevalence Ratio (PR) calculated as:

$$PR = exp\ (\beta 1)$$

Lifestyle and health condition indicators were analyzed as dichotomic and expressed as proportions (%). We estimated independent models for each outcome using the weights and reported the parameters for each collection method.

The data were organized and analyzed using the Stata application, version 16.1 (StataCorp LP, College Station, USA). The studies were approved by the Research Ethics Committee of the Nursing School, Federal University of Minas Gerais (UFMG) (CAAE No. 58329722.9.0000.5149). Participants in the online panel provided electronic informed consent before completing the questionnaire, while consent for the telephone survey was obtained orally at the start of the interview.

RESULTS

The sex distribution was similar across surveys. However, important differences were observed in age and education. The online panel overrepresented individuals aged 35–54 years (58.7% vs. 36.7%) and underrepresented those aged ≥55 years (11.9% vs. 39.3%) compared to the telephone survey. In addition, the online panel included a higher proportion of individuals with a university degree or higher (59.8% vs. 32.9%) (Table 2).

After calibration, most lifestyle indicators and health conditions showed similar prevalence estimates between the two survey approaches. For example, the prevalence of fruit and vegetable consumption, leisure-time physical activity, and most health conditions differed only marginally between surveys. The smoking and obesity indicators were similar across the surveys, but the difference between them increased after adjustment (Table 3).

Notable discrepancies remained for specific indicators. The prevalence of ultra-processed food consumption was substantially higher in the online panel (37.5%; 95% CI: 32.1–43.2) compared to the telephone (18.3%; 95% CI: 14.2–23.2), representing more than a twofold difference. Similarly, heavy episodic drinking was more frequent in the online panel (34.8%; 95% CI: 29.4–40.5) than in the telephone (26.2%; 95% CI: 21.8–31.1) (Table 3).

Poisson regression analysis confirmed significant differences between the survey approaches in estimates for both ultra-processed food consumption and heavy episodic drinking, suggesting that these indicators may be more sensitive to methodological effects. The ultra-processed food consumption prevalence was 105% higher in the online panel compared to the telephone (PR: 2.05), and the latter prevalence was 33% higher among participants in the online panel (PR: 1.33) (Table 3).

DISCUSSION

This study characterized the performance of a non-probability online panel for monitoring risk and protective factors for NCD in a middle-income country, providing a first empirical step toward understanding how web-based data collection behaves in the Brazilian health surveillance context. After calibration, most indicators showed comparable estimates between the online panel and the telephone survey. Sociodemographic quotas and weighting reduced discrepancies in sample composition and health estimates between the two survey approaches. Data collection was faster and more cost-efficient than the telephone survey, highlighting the operational advantages of web-based methods.

In our study, both surveys required adjustments to approximate the target population, but the probability of sampling remained closer. The sociodemographic quotas used online were a strategy to reduce large discrepancies and mitigate selection biases inherent in a non-probability sample (Cornesse et al. 2020). Although we cannot directly attribute the reduction of bias to quotas alone, they likely contributed to a more stable weighting process, which in turn helped to reduce residual imbalances and improved the comparability of prevalence estimates (McPhee et al. 2022). Despite these measures, the online sample remained skewed towards younger and more educated individuals.

Calibration using sociodemographic variables reduced discrepancies for most indicators, suggesting that statistical adjustment improved the comparability between the online panel and the target population, although residual selection bias may remain (McPhee et al. 2022). The similarity in the prevalence of most variables reinforces the adequacy and reliability of the questionnaire, even in a self-administered format (Brasil 2023). Remaining discrepancies, such as higher estimates for ultra-processed foods and heavy episodic drinking, likely reflect both data collection method and selection effects (Baker et al. 2010; Elliott and Valliant 2017). Firstly, the weights were constructed solely using sociodemographic variables, without any auxiliary information related to NCDs in the target population. This could limit their capacity to correct selection bias for these outcomes. Furthermore, the substantial biases suggest that residual errors in the point estimates may persist even after weighting and model-based adjustments, potentially affecting variance estimation and compromising the 95% coverage of CI (Cochran 1977). We also believe that measurement effects related to the data collection method may have influenced the reports. Web surveys provide greater privacy and reduce interviewer presence (Dillman 2017), which can lead to increased reporting of sensitive or stigmatized behaviors. Furthermore, the self-administered format may also allow more time to reflect and report consumption, potentially increasing reported prevalence. Alternatively, this discrepancy may reflect differences in how individuals engaged with the questionnaire across methods or unmeasured characteristics associated with participation in online panels (Dillman 2017).

The use of online opt-in panels in health research has expanded due to efficiency and scalability (Baker et al. 2010; Elliott and Valliant 2017). During the Covid-19 pandemic, online surveys, particularly those using non-probability methods, played a crucial role in enabling large-scale data collection at a lower cost (De Boni 2020). However, non-probability samples are limited by the lack of random selection (Elliott and Valliant 2017), and by the reliance on registration and incentives (Cornesse et al. 2020). These issues can introduce self-selection bias and errors that statistical adjustments cannot fully anticipate or correct, particularly in opt-in samples (MacInnis et al. 2018; Meng 2018). Although weighting procedures mitigate residual imbalances (McPhee et al. 2022), their effectiveness varies with sample size and the specific variables of interest, and substantial errors may persist even in higher-quality opt-in samples (Haddad et al. 2022; Pruett 2025).

Therefore, probability-based sampling remains the preferred method (Baker et al. 2010), especially for its more accurate representation of the population and for providing more reliable statistical inferences (Cornesse et al. 2020). Nevertheless, implementing probabilistic online methods remains challenging in large and socioeconomically diverse countries like Brazil, due to the lack of standardized sampling frames (De Leeuw 2018). In this context, multimodal surveys are advantageous, as online data collection can complement the primary method by offering it sequentially (typically beginning with the least costly) or by allowing respondents to select their preferred method (De Leeuw 2018; Dillman 2017). This approach helps reduce costs and address declining coverage and interest in voice calls (American Association for Public Opinion Research 2023), making it a path forward for health surveys in contexts where fully probabilistic designs are not yet feasible, such as Brazil.

The decline in the use of telephone calls across Brazil, even in regions with higher HDI, restricts the sustainability of relying exclusively on this method for NCD surveillance. With this scenario, our study explores the potential for integrating online methodologies into the current telephone-based surveillance system and provides the first evidence about online data collection into Brazilian NCD surveillance. The results showed that, even when based on a non-probabilistic panel, online data collection produced estimates broadly consistent with those from a probability survey for most indicators after calibration. The differences appear to be driven mainly by mechanisms inherent to online participation and by differences in sampling design.

Some limitations should be acknowledged. The study did not include adjustments for nonresponse beyond calibration to population margins, and the online panel lacked a probabilistic sampling frame. Additionally, potential overlap between landline and mobile phone ownership in the telephone survey was not explicitly addressed. Finally, the analysis focused on a single city, which may limit generalizability to other contexts.

In conclusion, the non-probabilistic online panel produced prevalence estimates broadly comparable to those of a probabilistic telephone survey for several NCD indicators after demographic calibration. These findings provide initial empirical evidence on the performance of an online environment when applied to NCD indicators in Brazil. The similarities and differences observed between the two survey approaches offer insights into participation and response patterns in online surveys and may guide the development of probability-based online surveys in the future. However, significant differences persisted for specific outcomes, which do not support the use of voluntary-participation online panels as a substitute for probability-based surveillance or as an ongoing complement for prevalence estimation. Further methodological research is needed to assess the validity of these approaches and identify the conditions under which online surveys can reliably contribute to population health monitoring.

CORRESPONDING AUTHOR CONTACT INFORMATION

Izabella Paula Araújo Veiga
E-mail: i.araujoveiga@gmail.com
Address: Federal University of Minas Gerais, 190 Professor Alfredo Balena Avenue, Belo Horizonte (MG), Brazil, Postal Code 30130-100.