Changing survey measures and measuring change: A call for religion measurement experiments

Conrad Hackett; Matthew Conrad

doi:10.29115/SP-2026-0015

Hackett, Conrad, and Matthew Conrad. 2026. “Changing Survey Measures and Measuring Change: A Call for Religion Measurement Experiments.” Survey Practice 20 (April). https://doi.org/10.29115/SP-2026-0015.

View more stats

Abstract

If we want to measure change, we shouldn’t change our measures. Yet measurement changes do happen. Some result from global circumstances such as mode switches necessitated by the COVID-19 pandemic. Others are related to changes in the presentation and prevalence of a topic.
The number of people who identify with no religion, also known as religious “nones,” has grown rapidly in many countries (Hackett et al. 2015; Stolz et al. 2025). We recently published an extended discussion of many measurement issues that may exaggerate the growth of religious “nones” (Conrad and Hackett 2026). In this short note, we highlight three common changes in the measurement of religious “nones” and call for survey and census organizations that make these changes to concurrently study how they affect the apparent prevalence of the “no religion” population.

The number of people who identify with no religion, also known as religious “nones,” has grown rapidly in many countries (Hackett et al. 2015; Stolz et al. 2025). We recently published an extended discussion of many measurement issues that may exaggerate the growth of religious “nones” (Conrad and Hackett 2026). In this short note, we highlight three common changes in the measurement of religious “nones” and call for survey and census organizations that make these changes to concurrently study how they affect the apparent prevalence of the “no religion” population.

Change 1: Making it easier to choose “no religion”

As the share of religious “nones” grows, censuses that measure religion are increasingly making “no religion” the first response category in religious identity questions rather than the last. Some respondents may choose this option simply because it appears first, a primacy effect.

In 2011, “no religion” was the last religion response category in Australia. In 2016, Australia made “no religion” the first option and the religiously unaffiliated share rose 9 percentage points from 24% to 33%. Reporting on Australia’s census results, CNN noted that the “nones” outnumbered Catholics for the first time. However, the report didn’t mention that there was a change in response categories that may have contributed to this shift (Berlinger 2017). During this period, the rise of religious “nones” was lower—about three points—on International Social Survey Programme surveys that kept their measure constant.

Other questionnaire design decisions also have made it easiest to choose “no religion.” In 1991, Canada’s census introduced a write-in box for respondents to spell out their religious identity. However, identifying with “no religion” only required filling in a bubble. New Zealand and Poland introduced a similar change in recent censuses. The relative ease of choosing “no religion” may contribute, at least in part, to the rise of the “nones” in these countries.

Change 2: Shifting between one-step and two-step measures of religious identity

One-step measures of religious identity ask a question like, “What is your religion, if any?” and typically offer “no religion” as a response category. By contrast, a two-step question first asks a filter question like “Do you have a religion?” and if respondents say they do, they are invited to specify it. Outside Muslim-majority countries, these two types of questions tend to produce different results (Brenner et al. 2024; Hackett 2014; Voas and Bruce 2004; Voas 2015).

The share of religious “nones” is higher in surveys that use a two-step measure. People with low levels of religious commitment who might volunteer a religious identity in a one-step question tend to fall into the “no religion” bucket in a two-step question.

Slovakia changed from a one-step to a two-step question between 2011 and 2021. In 2021, the unaffiliated share of the population was 25%, up from 15% in the 2011 census. It is possible, however, that much of the apparent change between census waves in Slovakia may have been a measurement artifact. Interestingly, the European Social Survey in 2012 used a two-step question and found an identical unaffiliated share, 25%, as the 2021 Slovakia census, which used a two-step question.

The opposite change could create an illusion of decline in the share of religious nones. For example, as Lithuania switched from a two-step question in 2001 to a one-step question in 2011, the unaffiliated share of the population dropped from 10% to 7%.

These types of changes are not limited to censuses. We observed that the unaffiliated share in Sweden dropped about 10 points between the 2010 ISSP, which used a two-step measure, and the 2011 wave, which used a one-step measure.

Change 3: Changes in survey mode

Changes in survey mode may affect social desirability and coverage biases.

When monthly surveys carried out in Spain by the Center for Sociological Research changed from in-person to phone interviews in April 2020, there was an immediate jump of 5 percentage points in the share of respondents who said they were atheist, agnostic or indifferent toward religion (González and Cabrera 2023).

Many censuses and surveys have transitioned from in-person interviews to self-administered questionnaires. For example, in Hungary, the option to complete the census online or by mail in 2011 may have contributed significantly to the 9-point rise in the unaffiliated share of the population from the 2001 face-to-face census. By contrast, with the same options for completing the census in place in 2021, the unaffiliated share rose only 2 additional points.

A change from primarily face-to-face interviews to a mail-to-web format saw the “no religion” share of U.S. General Social Survey respondents rise 5 points from 2018 to 2021. Some of this increase may have been the result of more religious Americans, including older adults, being less willing to take the survey online (Schnabel et al. 2024).

The need to test the impact of measurement changes

When media coverage overlooks methodological issues, it may mislead the public (Hackett 2013; 2023; Hackett and Tong 2025; Hackett 2026). Reporters may emphasize what appears to be a large change while overlooking, omitting or being unaware of measurement artifacts that exaggerate the change (Berlinger 2017).

Survey and census organizations should research how changes in the way religion is measured affect results. Without such study, it is difficult to distinguish real social change from methodological artifacts. For all who seek to understand religious change, including journalists, religious leaders, policymakers, researchers and the public, it’s vital to disentangle the two (Hackett 2020). Organizations should publish the results of experiments to measure the impact of measurement change. Approaches may include:

Experiments with split-samples: When changing question format, respondents can be randomly assigned to the new and old conditions. Differences between the groups can be used to quantify the measurement effect.

Experiments with two modes: When survey modes are changing, a mode experiment may use the old and new mode of data collection (Pew Research Center 2021).

External data comparisons: While it would be ideal for organizations to conduct their own experiments, they may also evaluate the extent to which religious change has occurred on other high-quality surveys that have maintained consistent methodology during the period of interest.

Organizations should thoroughly describe their methodological changes and draw attention to how changes may affect trend data (Sullivan et al. 2012). Researchers, journalists, and the public need help understanding whether apparent change is primarily the result of measurement change.

Conclusion

The overall growth of religious nones is real (Hackett 2025; Voas 2025). However, during periods when measurement changes occur, the magnitude of the rise of the nones has possibly been exaggerated. There are often good reasons for changes in survey questions and modes, but such changes should be accompanied by published studies about how the changes are expected to affect results, for religion as well as other topics.

Methodological changes create research opportunities. Experiments during periods of transition can quantify measurement effects, which may be large for religious and non-religious identity.

Corresponding author contact information

Conrad Hackett
chackett@pewresearch.org
901 E St. NW Suite 300
Washington, DC 20004

Submitted: February 10, 2026 EDT

Accepted: March 17, 2026 EDT

References

Berlinger, Joshua. 2017. “Australians Ditch Religion at Rapid Rate, Becoming More Diverse.” CNN, June 27. https://www.cnn.com/2017/06/27/asia/australia-census-2016/index.html.

Brenner, Philip S., Jill LaPlante, and Tracy L. Reed. 2024. “Sources of Inconsistency in the Measurement of Religious Affiliation: Evidence from a Survey Experiment and Cognitive Interviews.” Sociology of Religion 85 (4): 404–28. https://doi.org/10.1093/socrel/srad048.

Google Scholar

Conrad, Matthew, and Conrad Hackett. 2026. “How Measurement Changes Can Exaggerate the Growth of Religious ‘Nones.’” Sociological Science 13: 89–108. https://doi.org/10.15195/v13.a5.

Google Scholar

González, Felipe Manuel Rosa, and Leopoldo Cabrera. 2023. “A Sociological Perspective on Religious Identification in Spain: A Multidimensional Analysis Based on Empirical Data (Over 467,187 Individuals).” Secularism and Nonreligion 12 (1). https://doi.org/10.5334/snr.176.

Google Scholar

Hackett, Conrad. 2013. “No Clear ‘Pope Francis Effect’ among U.S. Catholics.” Pew Research Center. https://www.pewresearch.org/short-reads/2013/11/25/no-clear-pope-francis-effect-among-u-s-catholics/.

Hackett, Conrad. 2014. “Seven Things to Consider When Measuring Religious Identity.” Religion 44 (3): 396–413. https://doi.org/10.1080/0048721X.2014.903647.

Google Scholar

Hackett, Conrad. 2020. “Demography of Religion.” In The Sage Encyclopedia of Sociology of Religion, edited by Adam Possamai and Anthony J. Blasi. Sage. https://doi.org/10.4135/9781529714401.n117.

Google Scholar

Hackett, Conrad. 2023. “China’s Christian Population Appears to Have Stopped Growing after Rising Rapidly in the 1980s and ’90s.” Pew Research Center. https://www.pewresearch.org/short-reads/2023/12/12/chinas-christian-population-appears-to-have-stopped-growing-after-rising-rapidly-in-the-1980s-and-90s/.

Hackett, Conrad. 2025. “How Religion Declines around the World.” Pew Research Center. https://doi.org/10.58094/hkw0-6309.

Hackett, Conrad. 2026. “Has There Been a Christian Revival among Young Adults in the U.K.? Recent Surveys May Be Misleading.” Pew Research Center. https://doi.org/10.58094/k9vn-k647.

Hackett, Conrad, Marcin Stonawski, Michaela Potančoková, Brian Grim, and Vegard Skirbekk. 2015. “The Future Size of Religiously Affiliated and Unaffiliated Populations.” Demographic Research 32: 829–42. https://doi.org/10.4054/DemRes.2015.32.27.

Google Scholar

Hackett, Conrad, and Yunping Tong. 2025. “The Growth of Christianity in China May Have Come to an End.” Socius: Sociological Research for a Dynamic World 11. https://doi.org/10.1177/23780231241310469.

Google Scholar

Pew Research Center. 2021. “Measuring Religion in Pew Research Center’s American Trends Panel.” https://www.pewresearch.org/religion/2021/01/14/measuring-religion-in-pew-research-centers-american-trends-panel/.

Schnabel, Landon, Sean Bock, and Michael Hout. 2024. “Switch to Web-Based Surveys During COVID-19 Pandemic Left Out the Most Religious, Creating a False Impression of Rapid Religious Decline.” Sociology of Religion 85 (4): 429–53. https://doi.org/10.1093/socrel/srad061.

Google Scholar

Stolz, Jörg, Nan Dirk de Graaf, Conrad Hackett, and Jean-Philippe Antonietti. 2025. “The Three Stages of Religious Decline around the World.” Nature Communications 16: 7202. https://doi.org/10.1038/s41467-025-62452-z.

Google Scholar

Sullivan, Alice, David Voas, and Matt Brown. 2012. “The Art of Asking Questions about Religion.” Center for Longitudinal Studies.

Google Scholar

Voas, David. 2015. “Religious Involvement over the Life Course: Problems of Measurement and Classification.” Longitudinal and Life Course Studies 6 (2): 212–27. https://doi.org/10.14301/llcs.v6i2.311.

Google Scholar

Voas, David. 2025. “Invisible Secularity: American Theism beyond Belief.” Social Forces 104 (1): 366–85. https://doi.org/10.1093/sf/soaf018.

Google Scholar

Voas, David, and Steve Bruce. 2004. “Research Note: The 2001 Census and Christian Identification in Britain.” Journal of Contemporary Religion 19 (1): 23–28. https://doi.org/10.1080/1353790032000165087.

Google Scholar