In certain litigation surveys, it is necessary to create two groups of respondents by random assignment, and to show the groups different products to test for trademark infringement claims. Cost and timing considerations make it infeasible to use probability samples, and surveys are often taken among respondents recruited in shopping malls or on web survey panels. This paper compares the characteristics of the recruits likely to be obtained by these two approaches and discusses their potential for creating useful data.
Brand image is a valuable resource for many companies. Products such as Mercedes cars, Louis Vuitton handbags and Vitamin Water energy drinks have distinctive looks that consumers use to identify them. These distinctive appearances can be registered and the holders of such trademarks will fight hard to retain their exclusive rights to them. If they feel that a competitor has created a product whose appearance is uncomfortably close to the trademark, they will sue the competitor to withdraw the infringing look.
To win the lawsuit, the trademark holder must demonstrate that the competitor is confusing a substantial share of consumers into thinking that the two companies and their products are linked. For example, if a hypothetical company put out a product called “Cooly Cola” in a red and silver can, Coca Cola might sue to force this competitor to change the name and look of its product. To show confusion, Coca Cola might commission a survey of cola drinkers. The respondents would be randomly assigned to test and control groups. The test group respondents would be shown the Cooly Cola can and asked who put out the product, or if the makers of Cooly Cola were sponsored by or associated with some other manufacturer. The control group respondents would be asked the same questions, but shown a different can. The control group can would look like the Cooly Cola can, except for the allegedly infringing aspects. For example, the control group can might be a different color or the lettering could be changed.
The statistic of interest is the difference between two percentages of respondents, test and control, who identify Coca Cola as the manufacturer or sponsor of Cooly Cola. For example, if 25 percent of the treatment group, but 9 percent of the control group mistakenly identified Coca Cola as the maker of Cooly Cola, we would say that (25 – 9 =) 16 percent of the respondents were confused by the brand name and look of Cooly Cola. By subtracting the two percentages we control for respondents who might name Coca Cola because it was a well known brand or because they could not think of any other product to name.
Because respondents need to look at the products, interviewing for confusion surveys is done in person or on the web. Household surveys based on probability samples are too expensive, so in-person interviewing is usually done in shopping malls. In either case, respondents are recruited and there is no pretense that the surveys are based on probability samples. The question is whether such surveys can provide useful data.
One of the benefits of a probability sample is that researchers have reasonable assurance, given adequate sample size, that the sample will reflect the demographic distribution of the population. To see whether this is likely to occur for web surveys, I consulted a 2007 study of internet use in the United States. The results were posted by the Census Bureau on its Statistical Abstract website. To check this for mall intercept surveys, I purchased two questions concerning the frequency of shopping mall visits on an October 2006 Ipsos-Reid omnibus telephone survey.
As I show in Table 1, 67.3 percent of persons aged 18 and over had the internet at home and 67.4 percent had used it at some location during the past 30 days. There were substantial variations in internet use by age, education, employment status and income. Persons who were younger, who had completed more years of education, who were employed or who had higher incomes were more likely to have access to and use the internet. For example, 92.1 percent of college graduates, but only 45.9 percent of those who did not attend college had used the internet in the past 30 days. None of the other variables considered, such as region, household size or gender created particularly large variations.
Turning to the shopping malls, I learned that 94 percent had gone to a mall at least once in the past year, that 60.3 percent had made seven or more trips during that time period and that the average person made 19.5 yearly trips, going slightly more often than once every three weeks. Variations by demographic category were insubstantial. For all groups, the percentages visiting shopping malls at least seven times per year were between 50 and 70 percent.
The comparisons shown in Table 1 indicate that mall surveys do a better job than web surveys of providing representative cross-sections of American consumers. Mall surveys are more likely than web surveys to provide accurate averages for demographic variables such as age, years of education, employment status and income. Within categories defined by these demographic variables, however, the mall shoppers could differ from other consumers in terms of variables such as interest in shopping, interest in new products and the desire to consume. The random assignment of respondents to treatment and control groups can reduce the impact of such variables on the outcome variable, but the lack of a probability sample should always be kept in mind when interpreting the final results. Even so, shopping mall surveys are more likely than web surveys to represent both high and low income consumers of all ages.
Web surveys have other advantages, including the fact that per-interview costs are less. While the shopping mall interviewers may obtain better data than web survey respondents provide, they can also create serious problems. The interviewers may not be well-trained and supervision can be lax. The independent validation of interviews is needed to prevent the inclusion of unqualified respondents. Review of open-ended answers is needed to prevent the inclusion of summarized answers. These problems are made more likely by the decentralized nature of training and supervision. Careful monitoring is needed to minimize their impact.
Both web and shopping mall surveys are sullied by respondents who take repeated surveys and whose survey-taking experiences may make them unique and unrepresentative, regardless of their demographic characteristics. In certain shopping malls, local employees make repeated trips to the interviewing facility, presumably to be interviewed and pick up the day’s lunch money. The problem is worse for web surveys, as their recruiting procedures attract “professional respondents.” These are people who join the panels of several web survey firms, volunteer for many surveys, and average as many as seven surveys per week. Such respondents may be especially sensitive to details in product appearances that the average consumer would not notice, or they could be so jaded from their repeated survey experiences that their responses would not be thoughtful. Either way, their presence in the data could create biases that we do not yet understand.
In my experience, the problems of shopping mall surveys can be lessened by due diligence, including (1) asking for the replacement of incomplete or otherwise poor data provided by inattentive interviewers, (2) asking that interviewers screen out any potential respondent who has taken part in a market research survey in the past few months and (3) using validation interviews to spot violations of this rule. Instances of outright cheating are rare, but investments of effort and expense are needed to obtain data of acceptable quality.
We have less control of the data quality on our web surveys. Without the probing of an interviewer, answers to open-ended questions are sometimes incomplete and confused respondents cannot get the help they need. The web surveys have lower per-interview costs, but there is no ready way to minimize incomplete answers and respondent confusion.
Most litigation surveys are taken using convenience samples such as those described in this article. They are admitted into evidence nonetheless because such surveys are routinely relied upon by American businesses in conducting their affairs. This is more likely to occur if the sample includes sufficient numbers of the relevant population groups, and if random assignment is used to allocate respondents to test and control groups. The relevant population groups are often defined by demographic variables such as age, income, gender and race and their relevance is determined by the market for the product. The sample members should have similar demographic characteristics to the consumers most likely to use the products at issue.
Shopping mall surveys are more likely than web surveys to represent a broad range of such demographic subgroups. Further research is needed to determine whether this advantage relative to web surveys extends to other criteria such as the likelihood of interviewer or respondent mistakes.