Here’s the situation: You’re a consultant to a public policy polling organization, which wants to measure public attitudes toward a recent expansion of bicycle lanes in New York City. The press, politicians, pundits and the public will all use the results of this single poll, the marginals, to characterize public opinion. Thus, the pollster wants to obtain the most realistic picture of the public that is possible.
As the consultant, do you:
- Recommend that a neutral wording question is better than a tendentious one, or would you say it doesn’t matter either way?
- Recommend measuring non-opinion, or doesn’t it matter whether non-opinion is acknowledged?
- Recommend measuring intensity (or strength) of opinion, or doesn’t it matter whether intensity is accounted for?
These are essentially the three questions that were posed in the August 2011 issue of Survey Practice. The article elicited commentaries about such problems, and they were published in the December 2011 special issue of Survey Practice. All but one commentary suggested ways in which pollsters can deal with the issues of question wording, non-opinion, and opinion intensity, so that the marginals can provide as realistic an assessment of the public’s views as possible.
Howard Schuman, by contrast, appears to say that as far as marginal results go, it doesn’t matter whether question wording is tendentious, or whether non-opinion and intensity are measured – that such results don’t really tell us much “no matter the wording.”
In his Survey Practice article, he writes that treating marginals as though they give us insight into what the public is thinking
“verges on what I have written about elsewhere and called ‘survey fundamentalism’: the belief that some polls tell us the literal truth about public opinion. But ‘public opinion’ is a nebulous concept, highly important but difficult to pin down. And apart from attempting to predict a very simple object, for example, a specific and clear referendum to be voted on, or between two candidates in an election, univariate poll results cannot tell us such a truth…> “Study of change over time or of the differences between educational levels, can provide a plausible basis for a judgment about public opinion, but the marginals in any simple sense should almost never be taken literally, no matter the wording. Serious analysis of the data and then human judgment are essential.” (emphasis added)
This argument is not a new one for Schuman, who wrote with Jacqueline Scott in Science (1987) that the solution to the problem of understanding survey responses
“requires giving up the hope that a question, or even a set of questions, can be used to assess preferences in an absolute sense or even the absolute ranking of preferences, and relies instead on describing changes over time and differences across social categories. The same applies to all survey questions, including those that seem on their face to provide a picture of public opinion.”
As Eleanor Singer (1988:576) later writes,
“A similar point was made earlier by Schuman in his presidential address to the American Association for Public Opinion Research (1986), but the Science article goes far beyond the presidential address in rejecting the usefulness of marginals.”
Her “quarrel” with Schuman’s conclusion, she writes (1988:578),
“hinges on a suspicion that we are not helped very much, in our defense of surveys, by fleeing to trends. True, there may be a change in unemployment, or in tolerance, over time; and if question wording (and all other elements of the research design) have been held constant—an unlikely feat—then the change will be net of those sources of error. But what does the change signify? It is still anchored in a particular measurement procedure….> “For these reasons, I believe that the intellectual defense of surveys is helped in only a limited way by measuring change over time, and that it’s a short way from retreating from marginals to trends, to being blown out of the water altogether.”
Schuman’s (1988:580) counterargument notes that
“the results of hundreds of experiments on question form, wording, and context indicate that relationships are a great deal more robust than marginals—a point emphasized years ago by one of the pioneers of survey research, Samuel Stouffer (Stouffer et al., 1949:168). Thus for practical purposes, analytic results are much safer to present and interpret than are marginals.”
Yes, perhaps “safer,” as Schuman says. But that option is really not available most of the time to public policy pollsters. From a practical point of view, almost every day we read about the results of one poll or another that report what the public is thinking in an absolute sense – support or opposition for the Keystone pipeline, removing troops from Afghanistan, the health care reform passed in 2010, raising taxes on the “rich,” and – in the case of New York City – the expansion of bike lanes.
The question here is whether scholars can provide advice to public pollsters when they are measuring public attitudes that will be reflected in the marginals. Such reports have a tremendous influence on the political environment, so it would seem important to suggest ways that provide the most realistic results.
The reason I focus on Schuman’s views is that in my commentary in the special issue of Survey Practice, I classified his views into what I tentatively called the “nihilism” school of thought, because – based on his comments, as well as the previous articles where he has expressed such views (noted above) – his position seemed to deny the usefulness of marginals at all. Schuman (and in retrospect, I agree with him) takes issue with the terminology, as well as what I described as a school of “realism.” As I suggested when I wrote the article, the classifications were arbitrary, and I wasn’t tethered to them. So, to avoid confounding the real issue, I would describe the spectrum not from nihilism to literalism (with realism in-between), but as complete acceptance at one end (I still like the word “literalism” here) and complete rejection or disbelief or skepticism at the other end. (There may, of course, be even better terms to use. But for the time being, these terms get at the spectrum that seems to exist.) As for where Schuman’s views fall along this spectrum, I leave that to the reader to decide based on Schuman’s own words.
But that gets back to the real issue: Should marginals ever be treated as though they provide insight into what the public is thinking? And can survey research scholars provide useful advice to pollsters who routinely report such marginals?
Schuman implies in his commentary that as far as marginals are concerned, there is little difference in the results of two polls that were described in the August 2011 Survey Practice issue. One poll used tendentious question wording (“Which comes closer to your point of view” – the expansion of bicycle lanes “is a good thing because it’s greener and healthier for people to ride their bicycle or this is a bad thing because it leaves less room for cars which increases traffic”), while the second one used a more neutral wording (“Do you favor or oppose the expansion of bicycle lanes, or don’t you have an opinion either way?”) The first survey also did not attempt to measure either non-opinion, or intensity of opinion, while the second included measures of both non-opinion and intensity.
The results of the first survey reported a 59% to 35% margin of support for the expansion of bike lanes, with just 7% unengaged. The second reported a virtual split in public opinion, 23% in favor and 21% opposed, with a majority, 56%, unengaged and having no meaningful opinion.
Given such disparate results, is there really no judgment that can be made as to which is a more realistic picture of the public?
The important point to remember is that, from a practical point of view, public policy pollsters cannot avoid dealing with marginals. In the last sentence of his reply to Singer, Schuman (1988:581) writes,
“But I think that we will do better by both the public and the survey profession if we deemphasize the reporting and interpretation of isolated marginals.”
That may be prudent advice for academics who analyze public opinion over time, but in the political world inhabited by public policy pollsters, politicians, pundits, the press and the general public, the advice has little relevance.
Marginals are here to stay, for they are the lifeblood of the American political culture. And researchers would do well to address the techniques of survey research that can provide the most realistic measures of public opinion, rather than giving up hope altogether that a well-designed set of questions can be used to provide insight into public preferences.