Introduction
Survey instrument pretesting has long relied on mixed-methods approaches that combine quantitative pilot testing with qualitative interviews to improve questionnaire design. In the early 1980s, a systematic focus on the cognitive processes involved in responding to surveys emerged (Cognitive Aspects of Survey Methodology, CASM; see, e.g., Schwarz 2007). With the development of cognitive interviews (CIs; see e.g., Beatty and Willis 2007; Collins 2015; Lenzner et al. 2024; Miller et al. 2014; Willis 2005), this approach became dominant in in-depth investigations into respondents’ difficulties in answering questionnaires and attempts to mitigate subsequent measurement error. More recently, emphasis has been placed on respondents’ lived experiences and social contexts for a sufficient understanding of how they answer questions and how these answers can be improved (Miller 2014; Tourangeau 2018). This has led to a shift in the conceptualization of the response process, incorporating a more sociological perspective and adapting qualitative methodology for in-depth pretest interviews.
The Qualitative Pretest Interview (QPI) is a recent approach to pretesting questionnaires and other survey materials, based on well-established qualitative interpretive methodology (Bethmann et al. 2019; Buschle et al. 2022). It provides survey practitioners with concrete guidelines for conducting and analyzing pretest interviews (Bethmann, Buschle, and Reiter forthcoming; Buschle and Reiter forthcoming) and bridges the gap between standardized survey research and qualitative methodology. QPIs leverage the benefits of qualitative methodology to reduce measurement error and improve the validity of quantitative analyses based on these data. They provide a deeper understanding of respondents’ interpretations of survey questions and issues with questionnaires within their social context, which is particularly valuable for complex topics where meaning is shaped by lived experience – such as digital engagement.
While QPIs have been used successfully in several smaller studies (e.g., Hatem et al. 2021; McAlpine et al. 2024; Reynolds et al. 2024; Schönauer et al. 2025), to our knowledge, they have not previously been applied in large-scale survey projects. In this paper, we discuss the findings of a recent project that employed this approach for the first time in such a context: the Survey of Health, Ageing and Retirement in Europe (SHARE; Börsch-Supan et al. 2013), a long-running, cross-European panel study for studying the effects of health, social, economic and environmental policies over the life-course (see Hunsicker et al. 2025 for the project report). The project objectives were to
-
Improve the IT module: Use QPIs to identify and address ambiguities, cultural nuances, and practical challenges in the questionnaire, ensuring that it accurately captures the digital engagement of older Europeans.
-
Test QPIs in a large-scale survey setting: Evaluate the feasibility and effectiveness of QPIs in a multinational context, where qualitative pretesting is rarely applied due to resource constraints and methodological complexity.
IT usage is an important topic in SHARE, as the digital transformation of European societies presents opportunities and challenges, particularly for older adults. As the European Commission (2023) states, the EU’s Digital Decade vision aims to ensure that digital technologies enhance the well-being and quality of life of all Europeans, respect their rights and freedoms, and promote democracy and equality. In order to inform these policy ambitions with reliable evidence, it is essential to collect high-quality data on internet access, usage patterns, and digital proficiency across different population groups.
However, the original SHARE IT module, introduced in Wave 5 (2013), only included four questions on basic computer use and internet access. Due to the rapid pace of technological development and the widespread adoption of internet usage in daily life over the past few decades (ITU 2024), this was deemed insufficient for providing detailed data. Thus, the questionnaire module required a thorough revision for Wave 10 (2024/25). The goals were to
-
Close knowledge gaps about internet use, digital literacy, and device access among older Europeans, providing a more differentiated understanding of how this population engages with digital technologies.
-
Prepare for methodological transitions within SHARE, most importantly, the potential implementation of web-based data collection modes.
New questions were added to capture access to and use of internet-enabled devices, frequency of use, online activities, support patterns, proxy internet use, and reasons for limited or non-use. Given the complexity of these new questions – and the critical importance of their validity for both research and policy – we employed QPIs to pretest the revised module.
In this paper, we discuss the implementation of QPIs in SHARE, the revisions to the IT module informed by qualitative insights, and the methodological lessons learned. We employed a sequential mixed-methods design, augmenting SHARE’s standard pretesting process,
-
an internal, technical questionnaire Pretest,
-
a Field Rehearsal to simulate real-world data collection, and
-
the Main Fieldwork, which produces the final scientific-use-file,
by adding a preceding round of QPIs. Our findings provide practical guidance for researchers seeking to integrate qualitative pretesting into large-scale survey processes, illustrating how QPIs can complement quantitative methods to enhance questionnaire validity and inform survey methodology.
Methodology & Implementation
Qualitative Pretest Interviews (QPIs) focus on ensuring that survey questions are understood and answered as intended by treating QPI-participants as co-experts in a dialogical process (Bethmann et al. 2019; Buschle et al. 2022). Unlike traditional cognitive interviews, QPIs emphasize the intersubjective understanding of meanings and connotations of, e.g., survey questions within the interviewer’s and the interview partner’s individual frames of reference, scientific and everyday knowledge, respectively.
QPIs draw on qualitative interpretive methodologies, such as problem-centered interviews (Witzel and Reiter 2012), in which interviewers engage in active listening and active understanding, switching between exploration techniques to uncover both explicit and implicit misunderstandings. In practice, QPIs require interviewers to build rapport with participants, framing them as co-experts and reinforcing this role throughout the interview. This approach allows for a rich contextualization of respondents’ interpretations – such as how social relationships, cultural norms, or life circumstances shape their understanding of concepts like “access” or “help” – and provides actionable insights for improving the questionnaire. These contextual layers are particularly important for topics like digital engagement, where meaning is deeply embedded in respondents’ daily lives.
While specific methods of analysis tailored to QPIs are still evolving (e.g., Buschle and Reiter, forthcoming), recommendations from elaborated, qualitative interpretive approaches can be applied (e.g., Flick 2014). This project used a pragmatic, collaborative approach to fit SHARE’s strict timeline and resource constraints.
Ideally, QPIs are iterative, with findings from early interviews informing subsequent rounds. However, due to time limitations in SHARE’s survey preparation schedule, only one round of QPIs was conducted before the revised module underwent technical and quantitative pretesting, as per SHARE’s standard procedures. It should be noted that the interviews were conducted only within the German sub-study and that the results of the analyses were incorporated into the generic English questionnaire.
Interviewer Training
In preparation for fieldwork, a team of interviewers was recruited from within SHARE and its project partner, Hochschule Kempten, and trained in QPI methodology. The training consisted of three virtual sessions. The first session introduced the principles of qualitative interviewing and focused on intersubjective understanding and indexicality (Buschle et al. 2022). The interviewers learned communication strategies for achieving mutual understanding and observed these strategies in a live mock interview. The second session was practical, with the interviewers developing their own QPI interview plans and conducting mock interviews with each other. Feedback from trainers and peers helped them refine their techniques. The final session, held shortly before fieldwork began, served as a refresher and addressed logistical details, such as participant recruitment and the use of interview materials. This training prepared the interviewers to engage respondents effectively and leverage their in-house knowledge of SHARE to explore the questionnaire’s topical and methodological aspects.
Qualitative Fieldwork
Participants were recruited through interviewers’ family and social networks. Purposive sampling ensured variation in gender, age, retirement/working status, and internet literacy among adults aged 50 and older in order to capture a diverse range of perspectives (see Schreier 2018). Recruiting individuals with little or no internet experience proved particularly challenging, reflecting broader difficulties in reaching “true offline users” (zur Kammer, Hudelmayer, et al. 2025).
A total of 17 German-speaking adults participated in the QPIs, which were mainly conducted in May and June 2023, with one test interview in January 2023. Although the interviewers rotated the participants to some extent, about half of the interviews involved individuals with whom the interviewers were already acquainted. While this familiarity may have facilitated rapport and contextual understanding, it also carried the risk of introducing unintended bias, as preconceived notions about the interviewees’ situations may not have been fully revealed during the interview. Most interviews were conducted in person at respondents’ homes, with two sessions held via Zoom. Interview duration ranged from 20 to 120 minutes, averaging 45 minutes. All interviews were audio-recorded and supplemented with interviewer postscripts, creating a comprehensive dataset for analysis.
Collaborative Analysis Workshops
Three collaborative workshops were conducted to analyze transcripts, recordings, and field notes (see Cornish et al. 2014). Interviewers and QPI experts examined the materials together. The workshops focused on identifying recurring themes and patterns of misunderstanding, as well as potential sources of error in the respondents’ interpretations. Due to time constraints, priority was given to the most challenging questions.
A key principle of the workshops was accurately representing the respondents’ voices. Original comments and examples were carefully reviewed to ensure that the analysis reflected the respondents’ lived experiences. The process was iterative, moving between the interviewers’ impressions, collective interpretations, and close examination of the original material. The workshops were recorded to preserve the discussions and insights for further refinement.
The findings and proposals for adjusting the questionnaire were communicated to the SHARE coordination team responsible for the IT module. The revisions were incorporated into the generic English version, which was then translated into each country’s language by the respective SHARE Country Teams, following standard translation protocols. This process ensured that the QPI results directly informed the refinement of the IT module, aligning it more closely with respondents’ realities and improving the validity of the items, while allowing for some local linguistic and cultural adaptations where necessary.
Results
Analysis of the QPIs revealed that the IT module items required different treatments. Some questions were revised, some were retained for longitudinal comparability, and a few were omitted due to their complexity. Due to time and personnel constraints, we prioritized the most salient items – those that caused respondents the most confusion or irritation – while summarizing the findings for questions that were not analyzed in depth. The version of the questionnaire module used in the QPIs is included in the appendix.
The device access question (IT007) was refined to clarify how respondents interpreted “access.” While most respondents understood the question, they struggled with the distinction between “use” and “access.” For instance, one respondent was confused because they owned a smartphone but did not use its internet function. Interpretations of “access to internet-enabled devices” also varied. Some respondents included workplace or public devices, while others limited it to household devices. Additionally, respondents had difficulty distinguishing between device types (e.g., desktop versus laptop) and were unfamiliar with terms such as “smartwatch” and “smart TV.” To address these issues, the question was rephrased to emphasize household ownership: “Which of these devices do you own or have access to at home?” The interviewer instructions were expanded to provide clearer definitions (e.g., smartwatches may include GPS or biometric monitoring capabilities) and examples (e.g., “Kindle” for “e-reader”). The category “Internet-enabled gaming device” was removed due to confusion. “TV” was simplified to avoid distinctions between “smart TV” and traditional television.
The online activities question (IT010) proved problematic due to its lengthy list of options, which confused respondents. Key issues included ambiguity in the “calling or chatting” category, where the example “Zoom” (the video call service) was confused with zooming into a photo. The term “online forum” was often misunderstood, and categories like “videos” and “radio” were interpreted as physical media rather than online activities. The distinction between health-related categories was also unclear. To address these issues, “chatting” was clarified as “writing/reading messages in apps or making video calls (e.g., WhatsApp, Telegram).” Examples were added for media activities (e.g., “watching videos online (YouTube, Netflix)”), and redundant categories like “online dating” were removed. New options for health/well-being apps and mobile gaming were introduced to better reflect actual internet use.
Respondents interpreted the help frequency question (ITX) in many different ways, revealing a discrepancy between the predefined response options and the diversity of real-life support dynamics. Some respondents included situations in which they received general technical assistance, such as setting up a device, rather than internet-specific support. Others found that the categories did not accurately reflect their experiences. Due to the complexity of these dynamics, the question was omitted from the Wave 10 questionnaire and identified as needing further development.
The question about years of internet usage (IT008) was revised to better align with how respondents naturally recalled their usage. Many respondents anchored their answers to life events, such as buying their first computer or starting a new job. They also expressed a preference for specifying an exact year rather than selecting from predefined categories. Therefore, the original question, “For how many years have you been using the internet?” was changed to “Since when have you been using the internet?” An open entry field was added to allow respondents to input a specific year between 1989 and 2025.
Other questions were retained without changes. Though challenging for some respondents and difficult for data users to interpret conceptually, the PC skills question (IT003) was kept to maintain comparability with earlier SHARE waves. See zur Kammer et al. (2025) for a more detailed analysis of this question. Questions on computer use at work (IT001 and IT002) and internet use in the past seven days and frequency of use (IT004 and IT009) did not reveal significant problems during the QPIs and were thus kept unchanged. Two questions addressing reasons for not using the internet were merged into a single item to streamline the response process.
In summary, the QPIs provided a nuanced understanding of how respondents interpreted and answered the IT module questions. The revisions, ranging from rephrasing and adding examples to omitting problematic items, were designed to improve clarity, more accurately reflect respondents’ experiences, and balance the need for longitudinal comparability.
Conclusion
This study had two primary goals. First, it aimed to improve the validity of the IT questionnaire module in SHARE by incorporating respondent feedback. Second, the study sought to test the feasibility of conducting QPIs in a large-scale, multinational survey.
The QPIs successfully identified several key issues, resulting in targeted revisions that enhanced the questionnaire’s clarity and relevance. These changes demonstrate how qualitative pretesting can complement quantitative methods to improve survey instruments. By embedding QPIs as an initial step in SHARE’s established pretesting workflow, preceding the technical Pretest, Field Rehearsal, and Main Fieldwork, we augmented the traditional process and provided a qualitative foundation for subsequent quantitative evaluation. The subsequent pretesting steps did not identify additional issues requiring questionnaire revision (see Fabel et al., forthcoming for some quantitative analyses of SHARE Wave 10 Field Rehearsal data). The main fieldwork was completed in late 2025 and included the revised IT module. Quantitative data to analyze the impact of the questionnaire changes will become available around the end of 2026.
More broadly, we found that this sequential mixed-methods approach should be considered for reasons beyond improving the quality of subsequent quantitative analyses. First, the experiences and interpretations of potential respondents can inform conceptual and theoretical reflections on substantive topics. Second, even if items are not adapted based on this feedback, it can still provide important context for interpreting the resulting quantitative analyses, particularly if the results seem unintuitive. Therefore, findings and recommendations, even if they are not considered, should be published as part of the survey’s data documentation.
The study showed that QPIs could be effectively integrated into SHARE’s workflow, suggesting potential for application in other large-scale surveys. This could be achieved by providing in-house interviewers with structured training and by holding collaborative analysis workshops. While using in-house interviewers had advantages, such as their familiarity with SHARE’s objectives enriching the interviews, it also posed risks. For instance, their dual roles as developers and interviewers could introduce unintended biases. Additionally, the lack of qualitative training among survey staff underscores the importance of targeted hiring or comprehensive training for future applications.
Regarding the sample, two challenges emerged. First, the sample lacked “true offline users,” limiting insights into digital exclusion. Second, pre-existing relationships between interviewers and participants introduced potential biases.
A more significant limitation was the geographical focus on Germany, which restricted the generalizability of the findings in the 28-country SHARE context. While the QPI-informed revisions were incorporated into the generic English version of the IT module and subsequently translated by Country Teams, the absence of qualitative pretesting in other languages has likely left some cross-cultural nuances unaddressed. Although the revisions were designed to minimize ambiguity and improve clarity in the source language, future studies would benefit from parallel qualitative pretesting in multiple languages to ensure that the module’s validity is maintained across diverse linguistic and cultural settings. Language barriers further complicated communication with the international Area Coordination team, emphasizing the need for multilingual or cross-cultural adaptations in future studies.
Therefore, future research should explore ways to adapt QPIs for use in multinational contexts, possibly drawing from work on cross-cultural cognitive interviewing and survey translation (e.g., Behr 2023; de Jong et al. 2020; Willis and Miller 2011). Implementing qualitative interviews in these settings is particularly challenging due to the interpretive nature of qualitative methodology. This is particularly evident when compared to quantitative testing and cognitive interviewing. Qualitative methodology relies heavily on the interviewer’s and interview partner’s embeddedness within their language and socio-cultural context. To our knowledge, this issue has not yet been fully resolved, and addressing it will be critical for enhancing the cross-national applicability of QPIs.
Another promising area of future research is the application of QPIs to sensitive topics. The method’s explicitly empathic and dialogical approach may be particularly well-suited to contexts where emotional or socio-cultural factors significantly influence responses. By prioritizing rapport-building and a deep understanding of respondents’ lived experiences, QPIs could help reduce measurement error and social desirability bias in such settings.
Corresponding author contact information
Arne Bethmann
abethmann@share-berlin.eu
Chausseestraße 111
10115 Berlin, Germany
Acknowledgements
The SHARE data collection has been funded by the European Commission, DG RTD through FP5 (QLK6-CT-2001-00360), FP6 (SHARE-I3: RII-CT-2006-062193, COMPARE: CIT5-CT-2005-028857, SHARELIFE: CIT4-CT-2006-028812), FP7 (SHARE-PREP: GA N°211909, SHARE-LEAP: GA N°227822, SHARE M4: GA N°261982, DASISH: GA N°283646) and Horizon 2020 (SHARE-DEV3: GA N°676536, SHARE-COHESION: GA N°870628, SERISS: GA N°654221, SSHOC: GA N°823782, SHARE-COVID19: GA N°101015924) and by DG Employment, Social Affairs & Inclusion through VS 2015/0195, VS 2016/0135, VS 2018/0285, VS 2019/0332, VS 2020/0313, SHARE-EUCOV: GA N°101052589 and EUCOVII: GA N°101102412. Additional funding from the German Federal Ministry of Research, Technology and Space (01UW1301, 01UW1801, 01UW2202), the Max Planck Society for the Advancement of Science, the U.S. National Institute on Aging (U01_AG09740-13S2, P01_AG005842, P01_AG08291, P30_AG12815, R21_AG025169, Y1-AG-4553-01, IAG_BSR06-11, OGHA_04-064, BSR12-04, R01_AG052527-02, R01_AG056329-02, R01_AG063944, HHSN271201300071C, RAG052527A) and from various national funding sources is gratefully acknowledged (see www.share-eric.eu).
The authors would also like to thank Christina Buschle, Theresa Fabel, Magdalena Hecher, Imke Herold, Annika Hudelmayer, Herwig Reiter, Silvia Strutinsky, Franziska Schäfer, Johanna Schütz, Barbara Thumann, Claudia Weileder and Kenneth zur Kammer for their invaluable contributions to the project.