Loading [Contrib]/a11y/accessibility-menu.js
Skip to main content
Survey Practice
  • Menu
  • Articles
    • Articles
    • Editor Notes
    • In-Brief Notes
    • Interview the Expert
    • Recent Books, Papers, and Presentations
    • All
  • For Authors
  • Editorial Board
  • About
  • Issues
  • Blog
  • Subscribe
  • search

RSS Feed

Enter the URL below into your favorite RSS reader.

http://localhost:28157/feed
Articles
Vol. 9, Issue 4, 2016September 30, 2016 EDT

Gay Marriage Tends to be More Popular than Same-Sex Marriage

Kenneth E. Fernandez, Jason A. Husser,
public opinionsame-sex marriagegay marriagequestion wording
https://doi.org/10.29115/SP-2016-0024
Survey Practice
Fernandez, Kenneth E., and Jason A. Husser. 2016. “Gay Marriage Tends to Be More Popular than Same-Sex Marriage.” Survey Practice 9 (4). https:/​/​doi.org/​10.29115/​SP-2016-0024.
Save article as...▾

View more stats

Abstract

Gay marriage and same-sex marriage are often used synonymously in society. However, not all individuals may react identically to these synonyms when making policy choices. The choice between “gay marriage” and “same-sex marriage” is a methodological decision with potentially significant implications for social scientists. In this short study, we present results of nine experiments over a three year period (2013-2016) in which we randomly rotated “gay marriage” and “same sex marriage” in a telephone survey question. We find that “gay marriage” tends to be slightly, but significantly, more popular than “same-sex marriage.”

Introduction

Academic researchers almost universally recognize that responses to a survey item are influenced by the choice of question wording (Schuman and Presser 1977). However, researchers are often unsure about how or when question wording will specifically matter. Recent studies have found sizeable differences in attitudes toward climate change depending on the terminology used (Schuldt, Konrath, and Schwarz 2011). Similarly, questions which refer to the Affordable Care Act as “Obamacare” systematically received less support from the public than when the “Obamacare” label is not used (Newport 2013). Yet, scholars do not always find effects from changes in question wording. Singer and Couper (2014) found no significant differences in policy preferences on abortion when a survey item contained the term “fetus” instead of “baby.”

This study examines whether the terminology used in describing gay marriage could influence a respondent’s answer to a survey item. Gay marriage, as a morality issue, is a both highly-salient issue and nontechnical in nature and therefore relatively easy to understand (Haider-Markel 2001). Like abortion, members of the public may hold strong opinions on gay marriage, making it less susceptible to question wording effects. On the other hand, the debate over gay marriage is relatively new compared to some issues, and therefore, opinions on the subject may not have crystalized. Lewis and Gossett (2008) suggest that gay marriage did not become a mainstream issue on the political agenda until the Supreme Court of Hawaii ruled prohibiting same-sex couples from marrying was a violation of the state’s constitution (Baehr v. Lewin 1993). This ruling caused Congress to respond with the Defense of Marriage Act (DOMA) in 1996 placing the issue firmly in the national agenda. The same authors found just over 50 percent of those surveyed in California in 2006 said their opinion on gay marriage had changed over time, while Baunach (2011) estimated that much of the change in support for gay marriage nationally came from individuals changing their mind (rather than cohort replacement or demographic changes). These findings suggest that attitudes on gay marriage are evolving and therefore more susceptible to framing effects produced by question wording. As a result, the choice between using “gay marriage” or “same-sex marriage” in a survey is a methodological decision with potentially significant implications for social scientists. In this short study, we present results of nine experiments over a three-year period (2013–2016) in which we randomly rotated “gay marriage” and “same sex marriage” in a telephone survey question.

Gay Marriage or Same-Sex Marriage?

Two prominent themes within the literature on survey research have emerged to help explain question wording effects: (1) how salient an issue is to a respondent and (2) how the survey item frames the issue. The level of salience of an issue is critical because answering a survey question requires a certain level of cognitive effort, and a respondent may be more willing to exert this effort when an issue is more relevant to his or her life or interests (Krosnick 1999; Schuman, Ludwig, and Krosnick 1986). When a respondent does exert this effort, he or she is said to be optimizing their response; when a respondent does not, he or she is said to be satisficing (Krosnick 1999). This notion of satisficing is similar to Converse’s concept of “nonattitudes” where a respondent provides an answer to a question using a cognitive process that may approximate flipping a mental coin. Respondents who are knowledgeable about an issue and feel the issue is important are less likely to be influence by variation in question wording. A respondent who is less interested in the issue and has yet to consider the issue beforehand is more likely to be affected by question wording.

How question wording differs between two survey items is also critical. Minor wording differences often have no measurable effects. But if one set of terms or descriptors frames causes a respondent to recall different information or expand or restrict the scope of the issue, then it is expected that a proportion of the public will adjust or change their response. Question wording may frame an issue in such a way that it directs a person in how to understand or make sense of a problem and to “locate, perceive, identify, and label” his or her past experiences (Goffman 1974). Smith (1987, 82) suggests the term “‘welfare’ clearly describes a policy or program that provides assistance to the poor, but it also can conjure up a second concept of waste and perhaps an antibureacratic image as well.” Rasinski (1989) found that using “assistance to the poor” produced significantly different results compared to “welfare.”

The empirical evidence on the effects of question wording on gay marriage is mixed. Using online survey data from 2009, McCabe and Heerwig (2012) found support for gay marriage was lowest when the term “homosexual couple” was used. When “same-sex couple” was used support increased, and support increased even further when “gay and lesbian couple” was used. McCabe and Heerwig (2012) suggest that the term “same-sex” causes respondents to focus on the sexual behavior of gay or lesbian couples, therefore reducing support for gay marriage. Yet, using telephone survey data from 2009, Pizmony-Levy and Ponce (2013) found no significant difference between questions using the “homosexual” or “same-sex.” Furthermore, Flores’ (2015) meta-analysis of 138 surveys and 36 different survey items uncovered no systematic differences between the surveys using “homosexual,” “same-sex,” or “gay.”

Data and Methods

To test the hypothesis that wording differences between “same-sex marriage” and “gay marriage” will be associated with differing levels of support, we analyze a simple question wording rotation. Our experiments were embedded within the question, “Do you support or oppose [same-sex|gay] marriage?”[1] Approximately half of respondents heard “gay” while another random half heard “same-sex.” The order of “support” and “oppose” were also randomly and independently rotated. We excluded the term “homosexual” for testing in part because the term is less frequently used by organizations and because prior studies have typically found that the term produces more negative responses (Brewer 2008).[2]

We conducted this experiment across nine surveys between April 2013 and April 2016. These surveys were live-caller dual frame (landline and cell) samples of adults living in North Carolina. We restrict results below to registered voters for consistency across the surveys, but all resident samples yield virtually identical findings. While the samples were of North Carolina, not the entire United States, we expect the North Carolina findings to be generalizable to the country as a whole.

Results

Our results appear in Table 1. When all nine surveys are pooled to create 6,921 cases, support for “gay marriage” is 47.0 percent while support for “same-sex marriage is 42.1 percent. This 4.8 percent difference is statistically significant (t=4.03; p=0.0001). To put this wording effect into context, this is slightly less than the equivalent difference in support between males and females (41.5 percent and 47.1 percent pooled sample support, respectively). Similarly, the difference between April 2013 and April 2016 is 3.6 percent. The wording effect is greater than three years of change during a time of historic levels of opinion shift towards support of same-sex marriage.

Table 1  Gay marriage vs. same sex marriage- mean differences across nine surveys.
Date of survey Same-sex wording Gay marriage wording Difference
Overall 0.421 0.470 −0.048
(0.008) (0.009) (0.012)
n=3,504 n=3,417 t=−4.03
p=0.0001
April 2013 0.416 0.474 −0.0584
(0.027) (0.027) (0.039)
n=327 n=331 t=−1.51
p=0.1321
September 2013 0.360 0.478 −0.118
(0.027) (0.028) (0.039)
n=322 n=316 t=−3.03
p=0.0026
February 2014 0.415 0.418 −0.003
(0.023) (0.024) (0.033)
n=451 n=426 t=−0.096
p=0.9234
April 2014 0.394 0.500 −0.106
(0.028) (0.029) (0.040)
n=315 n=290 t=−2.64
p=0.0085
September 2014 0.498 0.464 0.034
(0.023) (0.023) (0.033)
n=464 n=455 t=1.03
p=0.3013
October 2014 0.391 0.479 −0.088
(0.023) (0.023) (0.032)
n=470 n=463 t=−2.72
p=0.0067
February 2015 0.410 0.481 −0.071
(0.026) (0.026) (0.037)
n=371 n=366 t=−1.95
p=0.0520
September 2015 0.439 0.456 −0.016
(0.022) (0.022) (0.031)
n=503 n=498 t=−0.52
p=0.6010
April 2016 0.452 0.511 −0.059
(0.030) (0.030) (0.0425)
n=281 n=272 t=−1.390
p=0.1651

Independent sample t-tests. Std. errors in parentheses. Registered voters in North Carolina.

Support for Gay/Same Sex Marriage – 0=Oppose, 1=Support.

When we limit analysis to individual surveys, the pattern is somewhat inconsistent. In seven of the nine surveys, we find gay marriage having at least one percent higher mean support than same-sex marriage. The highest observed advantage for gay marriage was 11.8 percent in September 2013. In only one of the nine surveys (September 2014) was same-sex marriage more supported than gay marriage. In February 2014, the term “gay marriage” had less than half a percent more support than “same-sex”; a statistically insignificant advantage. Though the pooled survey analysis shows high levels of statistical significance, only three (9/2013, 4/2014, 10/2014) of the nine surveys reach conventional (p<0.05, two-tailed) significance levels. However, an additional three waves approach conventional levels of significance for between-subject experiments (4/2013, 2/2015, 4/2016).

Which Groups Prefer Gay Marriage to Same-Sex Marriage?

Our data do not tell a clear story about the reason why gay marriage is often more popular than same-sex marriage. Table 2 presents t-tests for pooled surveys split by several demographic groups. No clear pattern emerges about which subgroups are most susceptible to wording differences. Wording differences exist between both liberals and conservatives, between both those with high school education and those with college degrees, between both Democrats and Republicans, between both frequent church goers and those who never attend church and between both men and women. Effect sizes and significance levels are not uniformly consistent, but the direction of effect shows most demographic subgroups have an overall preference for gay marriage over same-sex marriage.[3]

Table 2  Subgroup reactions to gay vs. same-sex marriage.
Difference Std. error t p-Value n
Gender
 Female 0.057 0.016 3.49 0.00 3,689
 Male 0.037 0.017 2.14 0.03 3,205
Party
 Democrat 0.039 0.024 1.62 0.11 1,632
 Independent+leaners 0.081 0.023 3.46 0.00 1,835
 Republican 0.037 0.022 1.73 0.08 1,401
Ideology
 Liberal 0.042 0.023 1.80 0.07 913
 Moderate 0.058 0.022 2.65 0.01 2,035
 Conservative 0.054 0.017 3.18 0.00 1,815
Education
 Less than college 0.055 0.016 3.41 0.00 3,421
 College more 0.040 0.017 2.32 0.02 3,389
Age
 Younger than 65 0.050 0.014 3.45 0.00 4,830
 65 or older 0.043 0.022 1.98 0.05 1,962
Church attendance
 Never 0.036 0.025 1.46 0.14 1,267
 A few times a year to almost every week 0.059 0.022 2.65 0.01 1,978
 Every week 0.064 0.020 3.22 0.00 1,731

Independent sample t-tests (two-tailed). Higher difference indicates higher mean support for “gay marriage” over “same-sex marriage.” n is for combined “gay” and “same-sex” marriage treatment groups for the subgroup identified on the left.

Conclusion

Many social phenomena are known by multiple names. When a phenomenon is known in the general public by multiple names, researchers often feel the need to choose among those names. Public opinion researchers, in particular, must make difficult decisions when measuring attitudes about concepts with multiple accepted names. The choice of employing an instrument with a particular name has consequences for academics and policy actors when that name choice influences politically important results. Previous research has shown this with “Obamacare” versus the “Affordable Care Act” (Newport 2013).

Our study is fairly limited in scope, leading to opportunities for future research. We only test two synonyms and not wordings such as “LGBT” or “homosexual.” We only tested one survey mode, telephones. Though we expect our findings to be generalizable to other parts of the United States, we applied the experiment to only one state. We also only tested one policy area, marriage, and not any of the many other gay rights policy areas. Future studies could also employ more sophisticated tests of causal mechanisms behind why gay marriage tends to be more popular than same sex marriages.

We avoid arriving at a conclusion that researchers should use one term instead of the other. McCabe and Heerwig (2012, 434) noted that “the phrasing gay and lesbian is now the official preferred terminology of advocacy organizations, including the Gay and Lesbian Alliance against Defamation (GLAAD)” citing the GLAAD media reference guide. Yet, this has apparently changed. The most current GLAAD media reference guide (2014) states that same-sex marriage is “sometimes preferred [over gay marriage] to make clear that the expression covers both gay men and lesbians.” Furthermore, the term “same-sex” has grown in popularity among polling organizations and academics in the last decade.

Whichever term is used we offer caution that direct comparisons of data about support for “gay marriage” and “same-sex marriage” may show differences not because of fluctuations in actual support but because of question wording. That we did not identify a specific demographic subgroup driving the differences between “gay” and “same-sex” marriage does suggest that multivariate models of “gay marriage” and “same sex marriage” will likely yield similar estimates for right-hand side variables regardless of the wording used to construct the left-hand side variable. Taken as a whole, our results advise researchers studying “gay marriage” to be aware of these wording consequences when studying this important topic.


  1. Though we refer to this randomization as an “experiment” for the context of this paper, we initially included the randomization as a best practice for data collection and not as an explicit test of a pre-existing theory.

  2. Gallup changed their question wording from “homosexual” to “same-sex”; however, General Social Survey does still use the term “homosexual” (see Flores 2015 for a list of 36 different survey items addressing the issue of gay marriage).

  3. Multivariate models with interactions yield estimates with similarly insignificant interaction terms.

References

Baunach, D.M. 2011. “Decomposing Trends in Attitudes toward Gay Marriage, 1988–2006.” Social Science Quarterly 92 (2): 346–63.
Google Scholar
Brewer, Paul R. 2008. Value War: Opinion and the Politics of Gay Rights. Lanham, MD: Rowman & Littlefield Publishers.
Google Scholar
Flores, A.R. 2015. “Examining Variation in Surveying Attitudes on Same-Sex Marriage: A Meta-Analysis.” Public Opinion Quarterly 79 (2): 580–93.
Google Scholar
GLAAD. 2014. “GLAAD Media Reference Guide (9th Ed.).” 2014. http:/​/​www.glaad.org/​publications/​reference.
Goffman, E. 1974. Frame Analysis: An Essay on the Organization of Experience. New York: Harper and Row.
Google Scholar
Haider-Markel, D.P. 2001. “Policy Diffusion as a Geographical Expansion of the Scope of Political Conflict: Same-Sex Marriage Mans in the 1990s.” State Politics & Policy Quarterly 1 (1): 5–26.
Google Scholar
Krosnick, J. 1999. “Survey Research.” Annual Review of Psychology 50 (1): 537–67.
Google Scholar
Lewis, G.B., and C.W. Gossett. 2008. “Changing Public Opinion on Same-Sex Marriage: The Case of California.” Politics & Policy 36 (1): 4–30.
Google Scholar
McCabe, B.J., and J.A. Heerwig. 2012. “Reframing the Marriage Debate: Wording, Context, and Intensity of Support for Marriage and Civil Unions.” International Journal of Public Opinion Research 24 (4): 429–49.
Google Scholar
Newport, F. 2013. “What’s in a Name? Affordable Care Act vs Obamacare.” The Gallup Organization. 2013. http:/​/​pollingmatters.gallup.com/​2013/​11/​whats-in-name-affordable-care-act-vs.html.
Pizmony-Levy, O., and A. Ponce. 2013. “Framing Strategies and Public Support for the Legalization of Marriage between Two People of the Same Sex.” Sociological Perspectives 56 (2): 169–90.
Google Scholar
Rasinski, K.A. 1989. “The Effect of Question Wording on Public Support for Government Spending.” Public Opinion Quarterly 53 (3): 388–94.
Google Scholar
Schuldt, J.P., S.H. Konrath, and N. Schwarz. 2011. “Global Warming” or ‘Climate Change’?: Whether the Planet Is Warming Depends on Question Wording.” Public Opinion Quarterly 75 (1): 115–24.
Google Scholar
Schuman, H., J. Ludwig, and J.A. Krosnick. 1986. “The Perceived Threat of Nuclear War, Salience, and Open Questions.” Public Opinion Quarterly 50 (4): 519–36.
Google Scholar
Schuman, H., and S. Presser. 1977. “Question Wording as an Independent Variable in Survey Analysis.” Sociological Methods & Research 6 (2): 151–70.
Google Scholar
Singer, E., and M.P. Couper. 2014. “The Effect of Question Wording on Attitudes toward Prenatal Testing and Abortion.” Public Opinion Quarterly 78 (3): 751–60.
Google Scholar
Smith, T.W. 1987. “That Which We Call Welfare by Any Other Name Would Smell Sweeter an Analysis of the Impact of Question Wording on Response Patterns.” Public Opinion Quarterly 51 (1): 75–83.
Google Scholar

This website uses cookies

We use cookies to enhance your experience and support COUNTER Metrics for transparent reporting of readership statistics. Cookie data is not sold to third parties or used for marketing purposes.

Powered by Scholastica, the modern academic journal management system