Survey Practice in Non-Survey-Literate Populations: Lessons Learned from a Cognitive Interview Study in Brazil

Introduction

Surveys are a fundamental tool in social science research, enabling the collection of data on a range of topics, from health to political opinions. The quality of survey data hinges on respondents’ clear understanding of the questions and their ability to engage meaningfully in the survey process (Lenzner 2012). As structured social interactions, interviewer-administered surveys are governed by norms and expectations that shape how both respondents and interviewers contribute toward the survey’s objectives (Bradburn 2016; Groves 1990; Hines 1993). Successful survey outcomes therefore rely on respondents’ ability to participate and their awareness of their role within this social exchange (Biemer and Lyberg 2003; Bradburn 2016; Tourangeau, Rips, and Rasinski 2000).

Survey literacy

Researchers such as Gerber and Harkness have noted that respondents’ familiarity with the expectations of the survey process is crucial for successful data collection (Gerber and Wellens 1997; J. Harkness 1999; Houtkoop-Steenstra 2000; Hudler and Richter 2001). The term survey literacy is sometimes used to describe respondents’ understanding of these expectations and their familiarity with the dynamics of survey participation (J. Harkness 1999; Hudler and Richter 2001; Willis and Miller 2011). Survey-literate respondents typically approach the survey with an understanding of what is expected from the interaction. They recognize that their primary task is to respond to questions by selecting response options that most closely align with their experience (Groves 1990). While they understand that capturing their experience is important, they are also aware that their responses will be aggregated, contributing to a broader research objective. Additionally, they possess enough background knowledge and familiarity with basic concepts to effectively map their experiences onto the broader concepts explored in the survey (J. Harkness 1999; Houtkoop-Steenstra 2000).

Although “survey literacy” is not always explicitly named in academic literature, studies have documented its impact on survey error, manifesting as inaccurate, incomplete or missing data (Biemer and Lyberg 2003; Miller 2003; Pan, Craig, and Scollon 2005; Pan 2008; Chan and Pan 2011). For example, Miller (2003) describes challenges faced by respondents from rural Mississippi who, with limited exposure to surveys, struggled with the survey interaction. While Miller does not label this phenomenon as “survey illiteracy,” her findings illustrate how unfamiliarity with survey practices can compromise the respondent’s ability to effectively engage in the survey task. Other researchers, such as Chan and Pan (2011), directly refer to this concept, emphasizing the importance of survey literacy in fostering respondent engagement and improving measurement accuracy. These studies illustrate how a lack of survey literacy often manifests in low-income, low-education settings, with difficulties further amplified in cross-cultural contexts. Nevertheless, research into the practical effects of survey illiteracy remains sparce.

Overview of Cognitive interviewing and cross-cultural contexts

Cognitive interviewing is a qualitative method used to examine how respondents interpret and respond to survey questions, providing insights into the question-response process that help refine items and offer context for data users (Willis 2015). This approach is particularly valuable in cross-cultural research, where language, norms, and experiences may lead to varied interpretations of questions (Miller et al. 2011; Tourangeau 1984; Willis 2015). For example, cognitive interviews reveal how cultural norms shape understanding of survey concepts. (J. A. Harkness et al. 2010; Massey 2018; Miller et al. 2011). While conventional cognitive interview methods focus primarily on identifying and quantifying problems and error typologies with questions, an interpretive approach to cognitive interviewing, grounded in social theory, provides critical insights into how respondents’ response processes are linked to their social location and lived experiences (Gerber and Wellens 1997; Miller 2003). This interpretive approach not only ensures that questions are understood across sub-groups but also aligns them with the lived realities of respondents, thereby enhancing the reliability and validity of the data collected (Miller et al. 2014).

Drawing on findings from an interpretive cognitive interview study, this article focuses on how survey illiteracy contributes to response error and item nonresponse, underscoring the need to account for this issue in survey design and implementation. Findings from this study, conducted with caregivers in low-income favelas in Rio de Janeiro, Brazil, highlight the particular challenges encountered by this survey-illiterate population and demonstrate how these challenges lead to response error—inaccurate, incomplete, and missing data.

Method

Project background

Data for this study were drawn from 80 cognitive interviews conducted in three low-income favelas in Rio de Janeiro, Brazil, in August 2023. These interviews represented the final phase of a larger question evaluation study conducted by the National Center for Health Statistics’ Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) ^[1] to support the development of a survey on Inclusive Education^[2] for UNICEF. This survey, intended as a companion to a previously developed module on child functioning (UNICEF, n.d.-a), was designed to measure factors related to school participation among children with and without disabilities (UNICEF, n.d.-b). As tested, the module consisted of four sections covering school type, school environment, out-of-school children, attendance determinants, and attitudes to inclusive education. The instrument had previously undergone extensive testing and revision in various countries including the United States, India, Jamaica, and Cambodia. This final round of testing focused principally on questions related to out-of-school children, and Brazil was selected as the testing location due to its relatively high population of children not attending school. While individual question performance is beyond this article’s scope, a forthcoming report will detail findings and revisions (Massey, forthcoming).

Sample

A purposive sample of 80 caregivers of children, both with and without disabilities, was recruited by partner organizations within each community through outreach and word of mouth. The majority of respondents were mothers, though some were grandmothers, aunts, or other caretakers. Most respondents had very low education levels and resided in extremely low-income communities characterized by high levels of police and gang-related violence. Table 1 details the demographic characteristics of the sample. Additionally, due to pervasive violence and poverty, government officials and researchers have extremely limited access to favela communities, and as a result, the respondents in this study were largely unfamiliar with the norms of survey research.

Data collection and analysis

The study employed interpretive cognitive interview methods entailing hour long semi-structured interviews with respondents. The interviews were conducted in Portuguese by a team of social scientists from the Brazilian Institute of Geography and Statistics, who were trained by researchers from CCQDER. The interviews were observed by CCQDER researchers employing a real-time translation from Portuguese to English. During the interviews, respondents were first asked all of the survey items, followed by semi-structured probing where they were invited to explain their answers and how they interpreted the questions within their lived context. The semi-structured format allowed the interviewer to explore how the questions related to the respondents’ lives while also focusing on identifying potential sources of measurement error within the instrument. Respondents were remunerated R$100 (Brazilian Reais) for their time.

The analysis of interviews involved a systematic process of data reduction and theory building, using a grounded theory approach (Glaser and Strauss 1967; Miller et al. 2014). Initially, detailed interview notes, based on audio recordings of the interviews, were generated by summarizing each interview. These summary notes detailed how each respondent answered every survey question, documenting their interpretations of questions and key terms, the activities and experiences they considered, as well as any response difficulties or errors encountered. These notes were translated into English for further analysis by CCQDER researchers. In the next phase, comparisons across all interviews identified interpretive patterns, including common types of response errors. Findings from this second level of analysis depict the phenomena captured by each question and enables the assessment of construct validity.

Results

The Survey Task

Despite being briefed on the research goals and process during recruitment and again upon arrival for participation, many remained uncertain about what was expected of them. One respondent expressed her unfamiliarity with the survey process by explaining, “I never participated in something like that. [The interviewer] asked questions that we would never imagine. I would never imagine someone asking me a question like that.”

Some respondents misunderstood the purpose of the interaction, believing that the researchers were there to offer assistance, which distracted them from fully engaging with the survey. For example, notes from one interview describe a respondent who was not focused on the survey, saying, “[This respondent] was just released from prison. She asked us all the time for a job and could not think about the questions. The conversation was relatively shallow/superficial.” Similarly, due to the nature of the questions, other respondents hoped the research team could help them secure better school placements for their children.

Even those respondents who remained focused on the survey questions often exhibited hesitation and insecurity in providing their answers. Though most questions on the survey focused on basic information from their own lives, many respondents were concerned about providing the “right” answer. Notes from one interview state, “The interviewee states that she felt very nervous when answering, even though it was explained that there was no right or wrong answer… She still remained nervous and gave her last answers without thinking about the question.”

This hesitation became even more evident when respondents encountered the final set of opinion-based questions which asked about attitudes toward inclusive education. In one interview, a respondent hesitated to answer any of the opinion questions. This respondent expressed embarrassment over her lack of knowledge, admitting, “I’m too bad for these things.” The notes from this interview continue, “She was nervous, worried if she was answering right because she had never participated in an interview.” Such apprehension indicates that respondents’ anxiety extended beyond concerns about factual accuracy, affecting their perceived ability to contribute meaningfully to the survey interaction even on subjective, opinion-based topics.

Terms and Concepts

Respondents often lacked knowledge of basic terminology and concepts relevant to the survey domain. For example, respondents had difficulty with terms such as social skills, academic skills, teaching methods, or special school. When asked about her understanding of the term “academic skills,” one respondent seemed very unsure and said, “Having more resources? Is that it?” Further discussion with the respondent indicated that she was unfamiliar with the overall idea of academic skills not just the term.

Indeed, extensive probing by the interviewers revealed that respondents’ difficulties with these terms stemmed more from a lack of understanding of the underlying concepts than from the vocabulary itself. Since all respondents had children—some with disabilities, some without—who had participated in some way in the local education system, it was expected that they would have the basic background knowledge necessary to engage with the survey. However, many were unable to translate their experiences of their child’s education into related concepts that they could think about or articulate. Because these concepts were fundamental to understanding the questions, respondents often struggled to answer accurately or, in some cases, to answer at all.

This difficulty engaging with the survey’s system of knowledge was further exemplified when one respondent was presented with a question asking her to indicate how strongly she agreed with the statement, “Having children with disabilities in regular school requires extensive adaptation of the school.” Notes from this interview state, “She didn’t understand; she asked to have the question repeated. Even after the repetition, she didn’t understand. I tried to rephrase the question with simpler terms and examples, but she still didn’t understand. She responded by saying, ‘I think a special school is better… a room with 40 children, two handicapped, how will you have patience?’ I asked again, trying to rephrase again, but she still didn’t understand the question.” In this case, the interviewer repeatedly tried using different words to help the respondent understand the question, but the respondent’s lack of conceptual understanding of the foundational constructs of the question prevented her from offering a meaningful response, even with the interviewer’s assistance.

Response Options

Respondents often had difficulty choosing from the available response options, especially when those options did not perfectly align with their expectations or experiences. In one notable example, a respondent was asked to answer the question, “What type of classroom does [name] currently attend?” by selecting from the options: “a. regular classroom, b. special classroom for children with disabilities, or c. both regular and special classroom.” The following exchange illustrates her struggle:

Respondent: He goes to the resource room alone.
Interviewer: Can you choose regular classroom or special classroom?
Respondent: It’s called the resource room.
Interviewer: Is it a regular classroom or a special classroom?
Respondent: Special. It only has special children.

Although the respondent stated that she understood that her child’s classroom was a special classroom and not a regular classroom, she was reluctant to choose that response option because it did not exactly match the name of her child’s classroom. With the interviewer’s help, the respondent was eventually able to choose from the given options, but without this guidance, she struggled to relate her experience to the provided categories, even though a suitable choice was available and even though she recognized that her child’s classroom fit within those categories.

Response scales are a particular type of response option that survey-literate respondents understand as being ordered and mutually exclusive. The attitudes section of the instrument asked respondents to state their level of agreement with opinion statements using the options “a. strongly agree, b. agree, c. disagree, d. strongly disagree.” This scale was chosen because previous testing with “yes/no” response options indicated a preference for a non-binary choice that allowed for more subtle degrees of agreement. However, the respondents in the current study struggled to respond using the scale.

One respondent struggled to grasp the relationships among the response options. Notes from her interview read, “She didn’t seem to quite understand the scale, always asking to repeat. She showed a lot of confusion. First, she chose the option [strongly agree]. Then she wanted to add [disagree]. I told her it could not be both. At the end, she declared, ‘I don’t know.’” Ultimately, the respondent could not provide a response because she found the format of the question and the response scale confusing.

Other respondents had difficulty distinguishing between “agree/disagree” and “strongly agree/strongly disagree.” One respondent saw the options as binary, saying, “If I say agree, it can’t be more agree. Either I agree or disagree.” The notes from another interview state, “The respondent sees no difference between ‘agree’ and ‘strongly agree’. To her, it has the same meaning.” These examples indicate that the response scale posed significant challenges for some respondents.

Expressing Salience

Respondents also expressed a strong desire to share the details of their lives. Given the unusual opportunity to reflect on and answer questions about their life experiences, they were eager to share their full stories rather than just the specific information requested by the survey. This eagerness often made it challenging to efficiently proceed through the survey. The following notes from one interview summarize this difficulty:

The interview was quite time-consuming because the mother did not directly answer the questions and ended up narrating her life story. Most of the questions I had to filter through the whole story and direct her to the answer. She talked a lot about herself and not about the boy. She rambled on about the answers. It was very difficult to get to the point.

This tendency to share more than was asked reflects a deeper need for respondents to have their broader experiences acknowledged, which can complicate data collection. At times, this desire led respondents to choose more expansive response options rather than more specific ones. For example, when answering the question, “Why does [name] need help to get to school?” one respondent did not choose the response option “d. it’s not safe for [him/her] to go alone.” Instead, she chose “e. for some other reason” so that she could explain the details of why it was not safe for her child to travel to school. She said, “The path is not safe. Sometimes there are police operations against drug trafficking and it’s not possible to leave the house. Shootings.” For respondents living in an atmosphere of pervasive danger and violence, sharing these details was not just about answering the survey; it was an opportunity to voice the gravity of their daily realities, taking every opportunity to talk about the dangers they faced.

Impact on Response Error

These manifestations of a lack of survey literacy, had substantial impacts on response error, leading to satisficing, nonresponse, inaccurate responses, and drop-off due to respondent fatigue. For example, the notes from one interview describe a respondent’s satisficing, “The interviewee had difficulty thinking about the survey questions. (For example, sometimes when asked if she had doubts about a question, she simply repeated the answer she had given in the question, instead of saying whether she had understood it or not.) Often, during the survey questions, the interviewee was embarrassed that she might not understand, so she gave a simple answer.”

Another respondent repeatedly refused to answer questions. The notes from this interview read, “She could not respond. When I asked her why, the respondent repeated again ‘I don’t know’ and could not explain. It was clear she did not understand and didn’t want to say this.” Response error was frequently observed when respondents misunderstood the questions. One interviewer explained, “She answered, but I realized that she believed she understood the statements even when she answered the opposite of what she thought.”

The stress of engaging in a new and unfamiliar activity also caused respondents to quickly become tired. This respondent fatigue led to drop-off when respondents stopped answering questions, forcing the survey to be ended prematurely. The notes from one interview describe what happened when the respondent gave up, “At that moment, the interviewee was already quite mentally tired and was only answering ‘agree,’ not being possible to work the survey. She only shook her head until we stopped.”

These effects were less noticeable in fact-based questions centered on respondents’ personal lives and more evident in abstract, opinion-based questions. Nonetheless, response errors were observed throughout the survey and across the entire sample.

Conclusion

An interpretive approach to cognitive interviewing, with an emphasis on the sociocultural factors that influence question response, provides valuable insights into the challenges of Brazilian respondents from underserved and marginalized communities, particularly in how they understand and engage with the survey process. Findings from this study are congruent with previous studies that find that non-survey-literate respondents have difficulty understanding the survey task, comprehending fundamental concepts within the survey’s sphere of knowledge, choosing among available response options, intuiting scale relationships and limiting personal commentary (Agans, Deeb-Sossa, and Kalsbeek 2006; Bernal, Wooley, and Schensul 1997; Flaskerud 1988; Miller 2003; Pan, Craig, and Scollon 2005; Chan and Pan 2011). Further, this study explicitly ties these difficulties to specific types of response error.

While survey-literate groups may also encounter any of the difficulties described to a greater or lesser degree, these challenges are compounded among survey-illiterate populations, creating substantial barriers to meaningful survey participation. The current findings suggest practical measures to make participating in surveys accessible to a broader range of respondents by addressing the specific obstacles and limitations they face. First, testing questions in the target context is essential to ensure that respondents can engage fully and provide accurate responses. Other recommended measures include: 1) maximizing opportunities to orient respondents to the survey task prior to administration, ideally using community members, 2) interviewer training on this population’s special needs, 3) avoiding abstract or unfamiliar terminology, 4) ensuring response options are relevant and remain as consistent as possible throughout the instrument, and 5) focusing questions on respondents’ direct experiences.

In summary, this research provides further evidence that fully understanding the relationship between sociocultural factors and the response process is crucial for conducting effective survey research. Respondents’ social and cultural backgrounds profoundly shape how they interpret and respond to survey questions and the survey task, directly influencing data quality and the overall effectiveness of the research. The complexities of survey interaction revealed in this study underscore the importance of inclusive survey design for marginalized populations.

Lead author contact information

Meredith Massey mmassey2@cdc.gov