Processes and Challenges Associated with Conducting Survey Research in Conflict Zones

1. Why is Conflict Zone Research Important

Survey research is much more challenging in conflict zones, particularly in less developed parts of the world. Still, policy makers, humanitarian and development organizations, donor and health organizations, and other non-governmental organizations have a need for timely and reliable data in order to make decisions. Conflict zone research poses several specific challenges, as it can be hard to meet the usual standards for reliable and generalizable data in such a context. In addition, given the urgent need for data, the question may be raised, is gathering data that does not meet the usual standards still better than having no data at all?

While conducting research in real time might be difficult or risky, without current data, policy makers run risks when they base decisions on data from pre-conflict perspectives. Goodhand (2000) suggests that delaying research until conflicts cool provides a weak basis for policy makers in real time. Urgent policy decisions may hinge on an assessment of how conditions are, whether programs implemented in the area seem to be working and understanding the degree and direction of political support for political leaders and/or the regime. These assessments are complicated by all the challenges that may impede the collection of unbiased data.

2. Who Conducts Conflict Zone Research and Why?

A variety of policy makers and scholars, both domestically and internationally, need reliable survey data to make assessments and/or policy decisions. U.S. based international research organizations such as Pew and Gallup along with their regional partners, conduct research world-wide and in conflict zones. They work to engage reliable regional partners to help select subcontractors who are familiar with the issues and situations affecting research in conflict zones to provide information to policy makers and scholars. Many of these international partners have regional or local experience that is invaluable in addressing threats to both internal and external validity.

In addition, the U.S. State Department, who is in need of reliable survey data to make assessments and/or policy decisions uses its intelligence gathered by the Bureau of Intelligence and Research through the Office of Opinion Research (INR/OPN) to formulate their policy and decision making. However, because the data is part of the intelligence communities data collection, their information is generally restricted.

Recent research in Ukraine by the Kyiv International Institute of Sociology (KIIS 2023) demonstrates one distinctive approach to working in conflict zones. Unlike many conflict zone surveys done by international organizations, which use their own survey staff, the KIIS survey was undertaken by a domestic Ukrainian survey organization using their own internal staff, measuring opinions in their country. Their omnibus survey addressed a variety of issues, including political leadership, approval, and trust. Given their situation, KIIS researchers simply excluded from their sample those areas not under the control of the Kyiv government, so it is essentially not a measure of all of Ukraine. They did responsibly report their methodology, a transparent approach that allowed others to judge the generalizability of their research.

Unlike the KIIS example, many international organizations conduct conflict zone studies as external entities for their organization’s purposes, while sometimes using local researchers.

3. Problems Encountered with Conflict Zone Research

The problems often associated with conflict zone research concern both internal and external validity. Researchers must be clear about the goals of a study and create a research design that can best meet their goals. This determines the mode of the study and where to place one’s emphasis in terms of internal and external validity. Researchers must weigh constraints including cost, speed, and accuracy as well as the omnipresent concern for the security of the researchers and respondents. Unlike in Western democracies, huge challenges often emerge due to a lack of infrastructure, access to samples, physical displacement due to refugee migration, danger to respondents, or simply ongoing wars and violence in conflict zones.

Policy makers need to feel comfortable about the reliability and the generalizability of the findings they receive or develop. In other words, are they measuring what they think they are measuring (internal validity) and how generalizable are their results (external validity)?

Part of the challenge with conflict zone research is addressing external validity. Varughese (2006) described his large-scale data collection in Afghanistan as having several challenges, including “widespread illiteracy, hostility to research, severe cultural restrictions on access to households, and especially to women, unfavorable physiographic conditions.” Still, he states that “survey research to international standards is possible in difficult contexts like Afghanistan, with patience, understanding of local context, and a willingness to make and accept incremental improvement in fieldwork. Building and using Afghan capacity for doing survey research is the key.”

Additionally, while constructing a sampling frame is difficult, the actual collection of the data is even harder. Data collection must adapt to the constraints of the security environment. Therefore, coordination with facilitators and local partners who undertake data collection is essential, as is the protection of both the researchers and residents who participate.

Himelein et al. (2016) described the challenges of attempting traditional two-stage sample designs for face-to-face interviews in developing countries such as those in the Sub-Sahara Africa region. In this design, one first identifies a census enumeration area as the primary sampling unit, with a probability proportional to its size. Subsequently, a listing process is carried out to create a frame of households from which a sample is selected. This design may simply not be feasible in conflict areas. First stage updated counts may not be available since the region does not maintain good listings of their population. The second stage requires the survey staff to canvass all the selected areas, which may prove to be too dangerous. This may lead to more qualitative work or non-probability research designs.

Sometimes the “random walk” procedure is implemented or a uniform grid process, which instructs the enumerators to choose households or ask them to start at a random point in a cluster and walk in a set direction until the interviewer encounters a structure. In other cases, satellite mapping allows for the possibility of a probability sample when working with primary sampling units. These flexible adaptations are all very different from a random digit dialing (RDD) sample in a developed stable environment. Undeveloped areas in conflict zones often may not have household listings, postal addresses, or phone numbers. Himelein and her colleagues note that second stage sampling in conflict zones often present trade-offs between cost, precision, and tolerance for bias.

The quality and reliability of surveys in conflict zones has long been considered a controversial issue. Barakat et al. (2002) notes that the development of a scientific sample to create a truly representative population sample may not always be possible in conflict zones, while traditional structured interviews might not always be the appropriate process to collect data.

Problems associated with internal validity, which can occur anywhere, are exacerbated in conflict zones. For example, due to fears of retribution in conflict zones, respondents may not answer honestly. This issue is addressed in the solutions section of this work.

In the end, researchers must decide whether the chief goal is representing a population accurately (generalizability or external validity) or accurately determining why some segment of the population thinks as it does (internal validity), even if it does not represent the population as a whole.

While solutions to problems of internal validity are not a substitute for solutions to problems of external validity, researchers must be adaptable and take what they can get.

Haer and Becher (2012) likewise argue that more qualitative approaches, i.e., focus groups or participant observations by researchers or journalists, can provide invaluable information about internal dynamics even if the conclusions do not have the same degree of external validity expected in research conducted in peaceful contexts. They also point out unavoidable challenges may occur where government census data is sketchy or unavailable due to conflicts and where phone service is not universal. Deb and Baudais (2022) note that using outdated census data leads to a non-representative sample. In their research, they sought other sources of data for sample construction, including the United Nations Office for the Coordination of Humanitarian Affairs.

Jacobsen and Landau (2003) found that a range of data collection methods are employed, including focus groups, open-ended questions, semi-structured interviews, and in-depth life history interviews.

4. Potential Solutions to Problems

A sound research design depends on the clarity of goals. This includes clarity about the relative emphasis on internal and external validity. The design must also consider the constraints articulated above including cost, speed, accuracy and particularly the safety of the researcher and respondent.

Romano (2006) pointed out several useful considerations from his research in Middle East conflict zones that would also be applicable elsewhere. His process included careful planning to determine the information that needed to be gathered before setting out on the research. He also noted the importance of recognizing where to draw a clear line about acceptable risk to researchers. Additionally, he prepared for worst-case scenarios, always setting up emergency contacts for the research staff. He also found it necessary to develop a diplomatic approach to engage not just with respondents but also with the community and governmental leaders. He also pointed out that one must decide on the degree to which one should mask the research to protect the study and those involved. He also recommended that one utilize flexible research designs such as snowball sampling, recognizing that the communities being investigated may not be structured in a stable manner. He also recommended working in teams rather than as a single researcher in the field. In his summation, he noted that he often kept two somewhat contradictory thoughts in mind when conducting conflict zone research: “Nothing risked nothing gained” and “dead researchers tell no tales.”

Haer and Becher (2012) note that quantitative research in conflict zones requires methodological flexibility. They specifically focus on two important aspects of research, the sampling process and data collection. They also suggest that their research may take a more ethnographic or non-probabilistic approach. They present a review of what sampling techniques are best suited for various environments. While desirable, a traditional RDD sample or even a probability sample may not be possible given the constraints found in some locations. Respondents may have no phones, internet or even a stable residential address, which makes telephone, web, or mail surveys ineffective. Additionally, the lack of census data to determine housing units in an unstable situation with massive destruction and refugee movement is a common obstacle. Bulmer (1993) reported that using census data in less developed areas may only be appropriate for a year after the census is completed.

Haer and Becher (2012) also noted that non-probability sampling, including chain-referral sampling and convenience sampling are often used techniques in conflict zones, even though they may not have the same value as probability samples found in more stable environments. They allow for the research to be undertaken even though it may increase the risk of bias.

Shaver and Zhou (2015) also have suggested that one should carefully evaluate the refusal rate, which in their Afghan study was more than 50 percent. High refusal rates may serve as a sign of a potentially flawed set of data. This may also be related to potential measures of truthfulness.

Jacobsen and Landau (2003) found that interviews in a conflict zone study may also vary in terms of the mode of data collection, with focus groups, open-ended questions, semi-structured interviews, or in-depth life history interviews being employed. While some may be conducted by the “researcher,” some may be dependent on translators or a local research team who may have different degrees of training. This may impact the quality and rigor of the study.

When getting access to respondents is a major challenge, sometimes other techniques are used to gather data that may foster external validity but do not necessarily have high internal validity. This may include a reliance on experts to provide information about a population, rather than a traditional survey using a high-quality sample. Other scenarios may involve the use of focus groups. This allows one to understand the question but not necessarily the proportion for or against something.

Ford et al. (2009) advise that researchers consider ethical concerns about the design and conduct of research in conflict zones. While some research organizations seek to apply their designs without any adaptation, some may be forced to recognize the unique environments and choose simply to do the best they can. Ford et al. (2009) suggest that research conducted in conflict zones must be scientific in nature, protect the subjects who are often vulnerable, and undertake research that benefits the study population.

Mneimneh et al. (2014) provide five guiding principles for research in conflict zones. They include (1) maintaining a flexible approach to all phases of the research; (2) using a mixed methods approach to maximize flexibility and minimize weaknesses in the survey collection design to capitalize on the strengths of supplementary methods; (3) recruiting and training interviewers in a manner promoting political neutrality through training, addressing the process to minimize refusals and non-contacts, securing interview privacy, and promoting unconventional monitoring; (4) tailoring data collection methods to deal with specific armed conflict challenges; and (5) recognizing that ethical considerations must be addressed.

In order to respond to bias in quantitative surveys in conflict zones, statistical approaches may also be applied to address non-sampling errors. Dixon and Tucker (2010) provide a review of important non-sampling errors for researchers. These may include non-response errors and response errors associated with interviewer effects. Axinn, Ghimire, and Williams (2011) recognized that even when using traditional “scientific” procedures associated with survey data collection, one must acknowledge the constraints researchers operate under. Researchers may use the total survey error perspective developed by Groves (1987) and Groves and Lyberg (2010). These address both sampling and non-sampling errors as important for total survey quality. In this approach, non-sampling errors include measurement errors due to the respondent, the interviewer, and the questionnaire.

While a RDD survey may often be considered the gold standard for data collection in Western industrialized locations, sometimes alternative proxy measures such as estimates of crowd sizes at rallies or demonstrations, estimates or counts of voter turnout, or population measurements at refugee or displacement camps may provide useful data for policy makers. These types of proxy measures might be used when traditional quantitative surveys are not practical or safe. In the end, making decisions about trade-offs may relate to what is possible.

Other approaches to address threats to internal validity related to the fear of respondents when responding to sensitive issues could include survey design issues which provide confidentiality. Researchers could consider using coin flip techniques as proposed by Warner (1965) in large sample size studies where the goal is to maximize internal validity. This approach asks them to provide an untruthful reply if they get heads. This approach may reduce the fear of respondents in terms of their response. However, its effectiveness become apparent when you do not specifically care about the individual respondent’s response but wish to gain an estimated population response. Another technique to reduce respondent fears when dealing with politically sensitive items would be to ask the respondent to select as many items as they agree with. This would be used when the researcher is trying to maximize internal validity, and respondents are perhaps unwilling to engage with certain topics otherwise.

5. Conclusions

Timely surveys in conflict zones are often essential for policy makers. Much of the work done in conflict zones is done collaboratively. While this data is essential for planning and evaluation, various challenges are presented in this work. Sampling, along with survey design issues, seem to be an issue of concern in many of the studies of surveys in conflict zones. Challenges to both internal and external validity are often addressed as trade-offs for researchers and end users. In addition, researchers must consider the role of adaptation and flexibility, when necessary. This is a recognition that the best laid plans must often be adapted to meet the challenges presented.

In the end, researchers must practice their work with caution, regardless of the locale. However, the challenges may be greater in conflict zones. Common sense along with experience need to serve as a guide to identify and address bias that may occur. It is essential that the researcher documents as many of the challenges they confront, which will permit others to evaluate the context of the study. Often, even when the challenges are documented, there may not be a simple means to evaluate the true impact of the methodological challenges. Policy makers will have to depend on reasoned judgment to appreciate the influence of the potential biases and consider the findings not always cast in stone. However, in the end, it may be better to have some data even if there are challenges than no data at all. Issues present in less developed parts of the world engaged in conflict are particularly challenging as has been discussed. Simply recognizing the challenges alone is not enough. The data collected in conflict zones needs to strive to develop techniques to quantify the data and provide clarity on how to evaluate the results.

Author contact information

William L. Rosenberg, PhD
Department of Politics
Drexel University
9-3021D MacAlister Hall
3141 Chestnut Street
Philadelphia, PA 19104
rosenberg@drexel.edu