Using Interviewer-respondent Interaction Coding as a Manipulation Check on Interviewer Behavior in Persuading CATI Respondents

Yfke Ongena University of Groningen

Marieke Haan Utrecht University

Abstract

This paper shows how interaction coding of interviewer-respondent interactions was used to perform manipulation checks of CATI interviewer behavior in experimental studies. An experiment in which interviewers were instructed to persuade potential respondents by means of a personal style or a formal style showed no significant effects of the persuasion style on survey participation. By means of interviewer-respondent interaction analysis, we studied the interviews in more depth focusing on the compliance of interviewers with the instructions. First, we found that many respondents immediately complied, but when respondents were reluctant, using any form of persuasion was better than none. Second, interviewers also had success in gaining cooperation when they referred to an argument that they had not been instructed to use. In conclusion, we assume that interviewers using arguments in which they were trained develop too much of an unauthentic routine in expressing these arguments, whereas using arguments outside instructions are likely to be expressed in a more natural, spontaneous way and are therefore more convincing. In addition, this study shows that it is useful to include behavior coding as a manipulation check in experiments involving interviewer behavior.

Introduction

Although it is likely that persuasion attempts by interviewers are an important tool to reduce refusal rates in CATI surveys (Dijkstra and Smit 2002), little is known about the dynamics in interviewer-respondent interaction and critical elements in persuading reluctant respondents.

For most people, survey participation is not an activity of utmost importance. Therefore, Groves et al. (1992) argue that in deciding whether or not to participate, most potential respondents take a heuristic approach, i.e., devoting only small amounts of time or cognitive energy. Groves et al. (1992) apply the six principles of compliance (Cialdini 1988) to the survey participation request. Accordingly, Dijkstra and Smit (2002) assume that most rejections reflect the heuristic approach and that interviewers who are more successful in persuading respondents make use of these compliance principles. Observing interviewer-respondent introductory CATI interactions in a nonexperimental study, they found persuading respondents by means of a more personal style (which emphasized the principle of liking) was more successful than persuading respondents by means of a more formal style (which emphasized the principle of authority). In addition, applying the principle of social validation was not successful. In the current experiment, we aimed to examine these findings and use behavior coding as a manipulation check. In any experiment involving interviewer behavior, a manipulation check is essential to distinguish ineffectiveness of the treatment from inadequate implementation of the treatment. For example, Dijkstra (1987) used behavior coding as a manipulation check in an experiment concerning manipulations of a personal and a formal style of interviewing. Other experiments involving interviewer behavior (e.g., Houtkoop-Steenstra and van den Bergh 2000; O’Neil et al. 1980) did not implement such a manipulation check, while this could have been useful in explaining their findings.

Methods and Data

An experiment was designed to study the effects of two persuasion styles of interviewers on respondents’ participation behavior. The study was implemented in a CATI survey on health-related behaviors and attitudes. It was conducted in February and March 2004. The twelve participating interviewers received general instructions on how to behave during the introductory interaction (see Table 1). Specific instructions, as derived from Dijkstra and Smit (2002), were given to six interviewers trained in the personal style and six trained in the formal style.

Table 1 Overview of instructions in the interviewer groups.

General instruction in all groups
Maintain interaction: never immediately accept a refusal
If the househoulder is busy, try to schedule an appointment
Answer questions of householders
Use your own wording
Emphasize confidential nature of the survey

Principle of compliance Specific instructions
Personal style
Specific instructions
Formal style

Liking Emphasize personal interest Do not mention personal interest
Authority Never refer to the university Emphasize importance for university or science
Social validation Never say most people enjoy the survey Mention that most people enjoy the survey

Data Collection

A sample of 1,500 telephone numbers was drawn from a website with telephone listings of Dutch households. Candidate-interviewers were recruited through various university communication channels. The selection criteria were: some experience and affinity with interviewing, social skills, and availability for all sessions. The interviewers were all female students aged between 19 and 28, who were financially compensated.

The training consisted of three six-hour sessions, during which interviewing techniques were discussed and practiced. The interviewers interviewed for four evenings within a period of two weeks. During the fieldwork, the interviewers were monitored by means of digital recordings. If necessary, interviewers were instructed to improve their behavior.

Coding of the Data

In total, 1,131 contacts with unique phone numbers were analyzed (only the last call to each contact was stored). The sound files were transcribed and coded in Sequence Viewer by two graduate students. The coding scheme that was used was based on a scheme used in earlier studies (Dijkstra and Smit 2002). The coded transcripts were systematically analyzed for rare codes and common errors. The first researcher independently double-coded 109 interactions (i.e., entire calls). The overall Kappa value of coding for all variables (0.74) indicates substantial agreement. More details on the coding scheme and reliability analysis are available upon request from the first author.

Results

Main Effect of Persuasion Style

Although interviewers who were trained in the formal style have a slightly higher cooperation rate (58 percent) than interviewers trained in the personal style (56 percent), the two experimental groups do not differ significantly in participation rates (χ2 [1, n=1,131] 0.498, p=0.48, ω=0.02). This result indicates that by looking exclusively at the manipulation both persuasion styles had similar effects on cooperation. However, since we do not know whether interviewers indeed behaved as instructed, we decided to analyze the interactions in more detail.

Immediate Compliance

Using interviewer-respondent interaction coding, actual interviewer behaviors were analyzed and compared to the instructions the two interviewer groups received. It appears that in most cases that end up in an interview (78 percent), respondents agree to be interviewed immediately after the interviewer’s request. In addition, in six cases, respondents hung up the phone during the request (i.e., blocking declinations, Schaeffer et al. 2013). The proportion of immediate acceptances appears to be slightly higher among interviewers in the formal style (81 percent) than among interviewers in the personal style (75 percent), (χ2 (1, n=1131) 2.58, p=0.11, ω=0.05). However, among the cases where no immediate acceptance occurred (n=594), there is no difference in the proportion of cases ending up in an interview among formal interviewers (22 percent) and personal interviewers (24 percent), (χ2 (1, n=594) 0.55, p=0.45, ω=0.03). So, different training protocols did not differentially increase participation.

Actual Use of Arguments

The use of arguments by interviewers was verified for those cases where there was no immediate acceptance and no blocking declination (n=588). Table 2 provides a manipulation check for implementation of personal and formal persuasion styles. There is no difference in the percentage of cases where interviewers provide any type of argument, although this percentage appears to be slightly higher in the personal style of interviewing. It is striking that the personal interviewers, against instructions, did mention study importance. However, this argument indeed was used more often by the formal interviewers than in the conversational group. Similarly, the social validation argument was used more often in the formal group, whereas personal interviewers more often used personal importance than formal interviewers.

Table 2 Percentage of calls in which interviewers give each type of argument by style of interviewing in which interviewer was trained.

Style of interviewing – treatment
Personal
Formal
Style of argument Percent Percent Chi-square (df=1)
Any argument 65.7% 60.7% 1.66
Study importance mentioned
(principle of authority)
Formal 4.2% 16.5% 24.50***
Mentioning most people enjoy survey (principle of social validation) Formal 0.0% 9.3% 30.13***
Personal importance mentioned (principle of liking) Personal 9.7% 1.1% 20.63***
n= 309 279

***p<0.001.

Participation Per Argument

Since the manipulation check showed interviewers did use different arguments (but sometimes against instructions), we further explored which arguments are more successful in obtaining cooperation. In Table 3, the response rates are given depending on the type of arguments used by interviewers. Four panels are displayed which correspond to the persuasion styles and the compliance principles. In panel A, calls during which the interviewer uses any argument anywhere in the interaction, are compared with calls during which the interviewer gives no argument at all. Panel A shows that it is useful to give any argument after a respondent’s refusal move; the response rate increases to 28 percent when interviewers provide an argument. We also verified whether this effect is the same for formal and personal interviewers, which indeed was the case (personal interviewers: χ2 (1, n=309) 13.1, p<0.01, ω=0.21, formal interviewers: χ2 (1, n=279) 20.9, p<0.01, ω=0.27). Panels B, C, and D show specific arguments within the three principles of compliance. It is possible that interviewers used multiple different arguments within a call. Also, if interviewers used the same argument multiple times, this call still appears only once in the table. Panel B shows that it is useful to mention the importance of the study for the university. Moreover, higher participation rates were found for interviewers who were instructed to use this argument (i.e., in the formal group, χ2 (1, n=279) 0.25, p<0.01, ω=0.03)) but especially for interviewers who were not instructed to use this argument (i.e., in the personal group, χ2 (1, n=309) 8.80, p<0.01, ω=0.07).

Table 3 Percentage of interviews after interviewer’s reaction.

Panel A Mentioning any argument or no argument at all
Any argument No argument Total Chi-square (df=1)
Interview 28% 8% 21% χ2=33.38***
n 372 216 588 ω=0.23

Panel B Principle of authority

Study importance mentioned Study importance not mentioned Total Chi-square (df=1)

Interview 31% 20% 21% χ2=3.50***
n 59 529 588 ω=0.07

Panel C Principle of liking
Personal importance mentioned Personal importance not mentioned Total Chi-square (df=1)

Interview 42% 20% 21% χ2=11.3***
n 33 555 588 ω=0.14

Panel D Principle of social validation

Most people enjoy mentioned Most people enjoy not mentioned Total Chi-square (df=1)

Interview 31% 21% 21% χ2=3.50***
n 26 562 588 ω=0.07

***p<0.001.

A logistic regression model with participation as the dependent variable, and interviewer group (reference=personal), use of the authority argument and interaction was marginally significant χ2 (3, n=588) 7.34, p=0.06, Nagelkerke R2=0.019, with significant between interviewer group and argument (b=1.347, standard error (SE) [b], p<0.01, Exp [b] (95 percent confidence interval)). The odds ratio of 3.95 shows that personal interviewers who use this argument have an odds ratio four times higher than formal interviewers in getting respondents to agree to be interviewed.

Panel C shows that mentioning personal relevance increased participation. Formal interviewers, who were not instructed to use this argument, did use this argument in three cases (which appeared to be two different interviewers), all of which were successful (although due to the low numbers this finding cannot be verified statistically). Finally, panel D shows that mentioning that most people enjoy the survey is only moderately successful. Personal interviewers never used this argument (as instructed), and for formal interviewers, this argument did not significantly increase participation.

Conclusion and Further Research

From our findings, we conclude that a failure of interviewers to apply their training could not account for the absence of a difference in participation between the two groups of interviewers. Although in the majority (78 percent) of the cases that eventually ended up in an interview (n=610), householders complied without any need for persuasion, interviewers managed to successfully persuade 21 percent of householders who initially refused (n=588). Participation among initially declining householders increases close to 30 percent when interviewers use an argument of any kind in persuasion. This effect was not different between the two differently trained groups of interviewers. Therefore, our first conclusion is that any argumentation is better than none. However, the causal direction of this effect is, of course, not clear. It is possible that interviewers’ expectations of success drive the results; interviewers might only make the effort to provide an argument when from the householder’s reaction they expect to be successful in persuading that particular person.

Looking at arguments actually used in the interaction, interesting findings arise. For the importance of the study as an argument, we found a significant interaction effect showing that interviewers trained in the personal style (not instructed to use this argument) had more success in gaining cooperation than interviewers trained in the formal style (instructed to use this argument). Likewise, two interviewers trained in the formal style of interviewing, who stressed (against the rules in their condition) their personal relevance, were equally successful in using that argument as interviewers in the personal style (who were specifically instructed to do so.) Therefore, we posit that interviewers using arguments as instructed develop too much of a unauthentic routine in expressing these arguments, whereas using arguments outside instructions are likely to be expressed in a more natural, spontaneous way and therefore more convincing. In addition, providing interviewers with too specific instructions on which argument they should use also moves them away from tailoring arguments to specific householders. Both tailoring (Groves et al. 1992) and spontaneous behavior (i.e., nonverbatim reading, see Houtkoop-Steenstra and van den Bergh 2000) have been demonstrated to be most effective in persuading reluctant householders.

In addition, this study shows that it is useful to include manipulation checks in experiments involving interviewer behavior. Researchers testing the effects of a manipulation of interviewer behavior should take into account more detailed checks on how these behaviors were actually implemented.

References

Cialdini 1988
Cialdini, R.B. 1988. Influence: science and practice. Scott Foresman, Boston.
Dijkstra 1987
Dijkstra, W. 1987. Interviewing style and respondent behavior: an experimental study of the survey-interview. Sociological Methods and Research, 16(2): 309–334.
Dijkstra and Smit 2002
Dijkstra, W. and J. Smit. 2002. Persuading reluctant recipients in telephone surveys. In: (R. Groves, D. Dillman, J. Eltinge and R. Little, eds.) Survey nonresponse. John Wiley & Sons, Inc., New York.
Groves et al. 1992
Groves, R.M., R.B. Cialdini and M.P. Couper. 1992. Understanding the decision to participate in a survey. Public Opinion Quarterly 56(4): 475–495.
Houtkoop-Steenstra and van den Bergh 2000
Houtkoop-Steenstra, J.P. and H. van den Bergh. 2000. Effects of introduction in large-scale telephone survey interviews. Sociological Methods and Research 28(3): 281–300.
O’Neil et al. 1980
O’Neil, M., R.M. Groves and C. Cannell. 1980. Telephone interview introductions and refusal rates: experiments in increasing respondent cooperation. Proceedings of the American Statistical Association, Section on Survey Research Methods. American Statistical Association, Washington, DC, 1979: 252–255.
Schaeffer et al. 2013
Schaeffer N.C., D. Garbarski, J. Freese and D.W. Maynard. 2013. An interactional model of the call for survey participation. Public Opinion Quarterly 77(1): 323–351.


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.