Introduction

Academic researchers almost universally recognize that responses to a survey item are influenced by the choice of question wording (Schuman and Presser 1977). However, researchers are often unsure about how or when question wording will specifically matter. Recent studies have found sizeable differences in attitudes toward climate change depending on the terminology used (Schuldt, Konrath, and Schwarz 2011). Similarly, questions which refer to the Affordable Care Act as “Obamacare” systematically received less support from the public than when the “Obamacare” label is not used (Newport 2013). Yet, scholars do not always find effects from changes in question wording. Singer and Couper (2014) found no significant differences in policy preferences on abortion when a survey item contained the term “fetus” instead of “baby.”

This study examines whether the terminology used in describing gay marriage could influence a respondent’s answer to a survey item. Gay marriage, as a morality issue, is a both highly-salient issue and nontechnical in nature and therefore relatively easy to understand (Haider-Markel 2001). Like abortion, members of the public may hold strong opinions on gay marriage, making it less susceptible to question wording effects. On the other hand, the debate over gay marriage is relatively new compared to some issues, and therefore, opinions on the subject may not have crystalized. Lewis and Gossett (2008) suggest that gay marriage did not become a mainstream issue on the political agenda until the Supreme Court of Hawaii ruled prohibiting same-sex couples from marrying was a violation of the state’s constitution (Baehr v. Lewin 1993). This ruling caused Congress to respond with the Defense of Marriage Act (DOMA) in 1996 placing the issue firmly in the national agenda. The same authors found just over 50 percent of those surveyed in California in 2006 said their opinion on gay marriage had changed over time, while Baunach (2011) estimated that much of the change in support for gay marriage nationally came from individuals changing their mind (rather than cohort replacement or demographic changes). These findings suggest that attitudes on gay marriage are evolving and therefore more susceptible to framing effects produced by question wording. As a result, the choice between using “gay marriage” or “same-sex marriage” in a survey is a methodological decision with potentially significant implications for social scientists. In this short study, we present results of nine experiments over a three-year period (2013–2016) in which we randomly rotated “gay marriage” and “same sex marriage” in a telephone survey question.

Gay Marriage or Same-Sex Marriage?

Two prominent themes within the literature on survey research have emerged to help explain question wording effects: (1) how salient an issue is to a respondent and (2) how the survey item frames the issue. The level of salience of an issue is critical because answering a survey question requires a certain level of cognitive effort, and a respondent may be more willing to exert this effort when an issue is more relevant to his or her life or interests (Krosnick 1999; Schuman, Ludwig, and Krosnick 1986). When a respondent does exert this effort, he or she is said to be optimizing their response; when a respondent does not, he or she is said to be satisficing (Krosnick 1999). This notion of satisficing is similar to Converse’s concept of “nonattitudes” where a respondent provides an answer to a question using a cognitive process that may approximate flipping a mental coin. Respondents who are knowledgeable about an issue and feel the issue is important are less likely to be influence by variation in question wording. A respondent who is less interested in the issue and has yet to consider the issue beforehand is more likely to be affected by question wording.

How question wording differs between two survey items is also critical. Minor wording differences often have no measurable effects. But if one set of terms or descriptors frames causes a respondent to recall different information or expand or restrict the scope of the issue, then it is expected that a proportion of the public will adjust or change their response. Question wording may frame an issue in such a way that it directs a person in how to understand or make sense of a problem and to “locate, perceive, identify, and label” his or her past experiences (Goffman 1974). Smith (1987, 82) suggests the term “‘welfare’ clearly describes a policy or program that provides assistance to the poor, but it also can conjure up a second concept of waste and perhaps an antibureacratic image as well.” Rasinski (1989) found that using “assistance to the poor” produced significantly different results compared to “welfare.”

The empirical evidence on the effects of question wording on gay marriage is mixed. Using online survey data from 2009, McCabe and Heerwig (2012) found support for gay marriage was lowest when the term “homosexual couple” was used. When “same-sex couple” was used support increased, and support increased even further when “gay and lesbian couple” was used. McCabe and Heerwig (2012) suggest that the term “same-sex” causes respondents to focus on the sexual behavior of gay or lesbian couples, therefore reducing support for gay marriage. Yet, using telephone survey data from 2009, Pizmony-Levy and Ponce (2013) found no significant difference between questions using the “homosexual” or “same-sex.” Furthermore, Flores’ (2015) meta-analysis of 138 surveys and 36 different survey items uncovered no systematic differences between the surveys using “homosexual,” “same-sex,” or “gay.”

Data and Methods

To test the hypothesis that wording differences between “same-sex marriage” and “gay marriage” will be associated with differing levels of support, we analyze a simple question wording rotation. Our experiments were embedded within the question, “Do you support or oppose [same-sex|gay] marriage?”^[1] Approximately half of respondents heard “gay” while another random half heard “same-sex.” The order of “support” and “oppose” were also randomly and independently rotated. We excluded the term “homosexual” for testing in part because the term is less frequently used by organizations and because prior studies have typically found that the term produces more negative responses (Brewer 2008).^[2]

We conducted this experiment across nine surveys between April 2013 and April 2016. These surveys were live-caller dual frame (landline and cell) samples of adults living in North Carolina. We restrict results below to registered voters for consistency across the surveys, but all resident samples yield virtually identical findings. While the samples were of North Carolina, not the entire United States, we expect the North Carolina findings to be generalizable to the country as a whole.

Results

Our results appear in Table 1. When all nine surveys are pooled to create 6,921 cases, support for “gay marriage” is 47.0 percent while support for “same-sex marriage is 42.1 percent. This 4.8 percent difference is statistically significant (t=4.03; p=0.0001). To put this wording effect into context, this is slightly less than the equivalent difference in support between males and females (41.5 percent and 47.1 percent pooled sample support, respectively). Similarly, the difference between April 2013 and April 2016 is 3.6 percent. The wording effect is greater than three years of change during a time of historic levels of opinion shift towards support of same-sex marriage.

When we limit analysis to individual surveys, the pattern is somewhat inconsistent. In seven of the nine surveys, we find gay marriage having at least one percent higher mean support than same-sex marriage. The highest observed advantage for gay marriage was 11.8 percent in September 2013. In only one of the nine surveys (September 2014) was same-sex marriage more supported than gay marriage. In February 2014, the term “gay marriage” had less than half a percent more support than “same-sex”; a statistically insignificant advantage. Though the pooled survey analysis shows high levels of statistical significance, only three (9/2013, 4/2014, 10/2014) of the nine surveys reach conventional (p<0.05, two-tailed) significance levels. However, an additional three waves approach conventional levels of significance for between-subject experiments (4/2013, 2/2015, 4/2016).

Which Groups Prefer Gay Marriage to Same-Sex Marriage?

Our data do not tell a clear story about the reason why gay marriage is often more popular than same-sex marriage. Table 2 presents t-tests for pooled surveys split by several demographic groups. No clear pattern emerges about which subgroups are most susceptible to wording differences. Wording differences exist between both liberals and conservatives, between both those with high school education and those with college degrees, between both Democrats and Republicans, between both frequent church goers and those who never attend church and between both men and women. Effect sizes and significance levels are not uniformly consistent, but the direction of effect shows most demographic subgroups have an overall preference for gay marriage over same-sex marriage.^[3]

Conclusion

Many social phenomena are known by multiple names. When a phenomenon is known in the general public by multiple names, researchers often feel the need to choose among those names. Public opinion researchers, in particular, must make difficult decisions when measuring attitudes about concepts with multiple accepted names. The choice of employing an instrument with a particular name has consequences for academics and policy actors when that name choice influences politically important results. Previous research has shown this with “Obamacare” versus the “Affordable Care Act” (Newport 2013).

Our study is fairly limited in scope, leading to opportunities for future research. We only test two synonyms and not wordings such as “LGBT” or “homosexual.” We only tested one survey mode, telephones. Though we expect our findings to be generalizable to other parts of the United States, we applied the experiment to only one state. We also only tested one policy area, marriage, and not any of the many other gay rights policy areas. Future studies could also employ more sophisticated tests of causal mechanisms behind why gay marriage tends to be more popular than same sex marriages.

We avoid arriving at a conclusion that researchers should use one term instead of the other. McCabe and Heerwig (2012, 434) noted that “the phrasing gay and lesbian is now the official preferred terminology of advocacy organizations, including the Gay and Lesbian Alliance against Defamation (GLAAD)” citing the GLAAD media reference guide. Yet, this has apparently changed. The most current GLAAD media reference guide (2014) states that same-sex marriage is “sometimes preferred [over gay marriage] to make clear that the expression covers both gay men and lesbians.” Furthermore, the term “same-sex” has grown in popularity among polling organizations and academics in the last decade.

Whichever term is used we offer caution that direct comparisons of data about support for “gay marriage” and “same-sex marriage” may show differences not because of fluctuations in actual support but because of question wording. That we did not identify a specific demographic subgroup driving the differences between “gay” and “same-sex” marriage does suggest that multivariate models of “gay marriage” and “same sex marriage” will likely yield similar estimates for right-hand side variables regardless of the wording used to construct the left-hand side variable. Taken as a whole, our results advise researchers studying “gay marriage” to be aware of these wording consequences when studying this important topic.