Loading [Contrib]/a11y/accessibility-menu.js
Skip to main content
Survey Practice
  • Menu
  • Articles
    • Articles
    • Editor Notes
    • In-Brief Notes
    • Interview the Expert
    • Recent Books, Papers, and Presentations
    • All
  • For Authors
  • Editorial Board
  • About
  • Issues
  • Blog
  • Subscribe
  • search

RSS Feed

Enter the URL below into your favorite RSS reader.

http://localhost:26844/feed
Articles
Vol. 9, Issue 2, 2016May 31, 2016 EDT

Using Interviewer-respondent Interaction Coding as a Manipulation Check on Interviewer Behavior in Persuading CATI Respondents

Yfke Ongena, Marieke Haan,
survey practice
https://doi.org/10.29115/SP-2016-0010
Survey Practice
Ongena, Yfke, and Marieke Haan. 2016. “Using Interviewer-Respondent Interaction Coding as a Manipulation Check on Interviewer Behavior in Persuading CATI Respondents.” Survey Practice 9 (2). https:/​/​doi.org/​10.29115/​SP-2016-0010.
Save article as...▾
Download all (1)
  • Original published version of the article
    Download

Sorry, something went wrong. Please try again.

If this problem reoccurs, please contact Scholastica Support

Error message:

undefined

View more stats

Abstract

This paper shows how interaction coding of interviewer-respondent interactions was used to perform manipulation checks of CATI interviewer behavior in experimental studies. An experiment in which interviewers were instructed to persuade potential respondents by means of a personal style or a formal style showed no significant effects of the persuasion style on survey participation. By means of interviewer-respondent interaction analysis, we studied the interviews in more depth focusing on the compliance of interviewers with the instructions. First, we found that many respondents immediately complied, but when respondents were reluctant, using any form of persuasion was better than none. Second, interviewers also had success in gaining cooperation when they referred to an argument that they had not been instructed to use. In conclusion, we assume that interviewers using arguments in which they were trained develop too much of an unauthentic routine in expressing these arguments, whereas using arguments outside instructions are likely to be expressed in a more natural, spontaneous way and are therefore more convincing. In addition, this study shows that it is useful to include behavior coding as a manipulation check in experiments involving interviewer behavior.

Introduction

Although it is likely that persuasion attempts by interviewers are an important tool to reduce refusal rates in CATI surveys (Dijkstra and Smit 2002), little is known about the dynamics in interviewer-respondent interaction and critical elements in persuading reluctant respondents.

For most people, survey participation is not an activity of utmost importance. Therefore, Groves, Cialdini, and Couper (1992) argue that in deciding whether or not to participate, most potential respondents take a heuristic approach, i.e., devoting only small amounts of time or cognitive energy. Groves, Cialdini, and Couper (1992) apply the six principles of compliance (Cialdini 1988) to the survey participation request. Accordingly, Dijkstra and Smit (2002) assume that most rejections reflect the heuristic approach and that interviewers who are more successful in persuading respondents make use of these compliance principles. Observing interviewer-respondent introductory CATI interactions in a nonexperimental study, they found persuading respondents by means of a more personal style (which emphasized the principle of liking) was more successful than persuading respondents by means of a more formal style (which emphasized the principle of authority). In addition, applying the principle of social validation was not successful. In the current experiment, we aimed to examine these findings and use behavior coding as a manipulation check. In any experiment involving interviewer behavior, a manipulation check is essential to distinguish ineffectiveness of the treatment from inadequate implementation of the treatment. For example, Dijkstra (1987) used behavior coding as a manipulation check in an experiment concerning manipulations of a personal and a formal style of interviewing. Other experiments involving interviewer behavior (e.g., Houtkoop-Steenstra and Bergh 2000; O’Neil, Groves, and Cannell 1980) did not implement such a manipulation check, while this could have been useful in explaining their findings.

Methods and Data

An experiment was designed to study the effects of two persuasion styles of interviewers on respondents’ participation behavior. The study was implemented in a CATI survey on health-related behaviors and attitudes. It was conducted in February and March 2004. The twelve participating interviewers received general instructions on how to behave during the introductory interaction (see Table 1). Specific instructions, as derived from Dijkstra and Smit (2002), were given to six interviewers trained in the personal style and six trained in the formal style.

Table 1  Overview of instructions in the interviewer groups.
General instruction in all groups
– Maintain interaction: never immediately accept a refusal
– If the househoulder is busy, try to schedule an appointment
– Answer questions of householders
– Use your own wording
– Emphasize confidential nature of the survey

Principle of compliance Specific instructions
Personal style
Specific instructions
Formal style

Liking – Emphasize personal interest – Do not mention personal interest
Authority – Never refer to the university – Emphasize importance for university or science
Social validation – Never say most people enjoy the survey – Mention that most people enjoy the survey

Data Collection

A sample of 1,500 telephone numbers was drawn from a website with telephone listings of Dutch households. Candidate-interviewers were recruited through various university communication channels. The selection criteria were: some experience and affinity with interviewing, social skills, and availability for all sessions. The interviewers were all female students aged between 19 and 28, who were financially compensated.

The training consisted of three six-hour sessions, during which interviewing techniques were discussed and practiced. The interviewers interviewed for four evenings within a period of two weeks. During the fieldwork, the interviewers were monitored by means of digital recordings. If necessary, interviewers were instructed to improve their behavior.

Coding of the Data

In total, 1,131 contacts with unique phone numbers were analyzed (only the last call to each contact was stored). The sound files were transcribed and coded in Sequence Viewer by two graduate students. The coding scheme that was used was based on a scheme used in earlier studies (Dijkstra and Smit 2002). The coded transcripts were systematically analyzed for rare codes and common errors. The first researcher independently double-coded 109 interactions (i.e., entire calls). The overall Kappa value of coding for all variables (0.74) indicates substantial agreement. More details on the coding scheme and reliability analysis are available upon request from the first author.

Results

Main Effect of Persuasion Style

Although interviewers who were trained in the formal style have a slightly higher cooperation rate (58 percent) than interviewers trained in the personal style (56 percent), the two experimental groups do not differ significantly in participation rates (χ2 [1, n=1,131] 0.498, _p=_0.48, ω=0.02). This result indicates that by looking exclusively at the manipulation both persuasion styles had similar effects on cooperation. However, since we do not know whether interviewers indeed behaved as instructed, we decided to analyze the interactions in more detail.

Immediate Compliance

Using interviewer-respondent interaction coding, actual interviewer behaviors were analyzed and compared to the instructions the two interviewer groups received. It appears that in most cases that end up in an interview (78 percent), respondents agree to be interviewed immediately after the interviewer’s request. In addition, in six cases, respondents hung up the phone during the request (i.e., blocking declinations, Schaeffer et al. 2013). The proportion of immediate acceptances appears to be slightly higher among interviewers in the formal style (81 percent) than among interviewers in the personal style (75 percent), (χ2 (1, n=1131) 2.58, p=0.11, ω=0.05). However, among the cases where no immediate acceptance occurred (n=594), there is no difference in the proportion of cases ending up in an interview among formal interviewers (22 percent) and personal interviewers (24 percent), (χ2 (1, n=594) 0.55, p=0.45, ω=0.03). So, different training protocols did not differentially increase participation.

Actual Use of Arguments

The use of arguments by interviewers was verified for those cases where there was no immediate acceptance and no blocking declination (n=588). Table 2 provides a manipulation check for implementation of personal and formal persuasion styles. There is no difference in the percentage of cases where interviewers provide any type of argument, although this percentage appears to be slightly higher in the personal style of interviewing. It is striking that the personal interviewers, against instructions, did mention study importance. However, this argument indeed was used more often by the formal interviewers than in the conversational group. Similarly, the social validation argument was used more often in the formal group, whereas personal interviewers more often used personal importance than formal interviewers.

Table 2 Percentage of calls in which interviewers give each type of argument by style of interviewing in which interviewer was trained.
Style of interviewing – treatment
Personal
Formal
Style of argument Percent Percent Chi-square (df=1)
Any argument 65.7% 60.7% 1.66
Study importance mentioned
(principle of authority)
Formal 4.2% 16.5% 24.50***
Mentioning most people enjoy survey (principle of social validation) Formal 0.0% 9.3% 30.13***
Personal importance mentioned (principle of liking) Personal 9.7% 1.1% 20.63***
n= 309 279

***p<0.001.

Participation Per Argument

Since the manipulation check showed interviewers did use different arguments (but sometimes against instructions), we further explored which arguments are more successful in obtaining cooperation. In Table 3, the response rates are given depending on the type of arguments used by interviewers. Four panels are displayed which correspond to the persuasion styles and the compliance principles. In panel A, calls during which the interviewer uses any argument anywhere in the interaction, are compared with calls during which the interviewer gives no argument at all. Panel A shows that it is useful to give any argument after a respondent’s refusal move; the response rate increases to 28 percent when interviewers provide an argument. We also verified whether this effect is the same for formal and personal interviewers, which indeed was the case (personal interviewers: χ2 (1, n=309) 13.1, p<0.01, ω=0.21, formal interviewers: χ2 (1, n=279) 20.9, p<0.01, ω=0.27). Panels B, C, and D show specific arguments within the three principles of compliance. It is possible that interviewers used multiple different arguments within a call. Also, if interviewers used the same argument multiple times, this call still appears only once in the table. Panel B shows that it is useful to mention the importance of the study for the university. Moreover, although higher participation rates were not found for interviewers who were instructed to use this argument (i.e., in the formal group, χ2 (1, n=279) 0.25, p= .615, ω=0.03)) higher participation rates were found for interviewers who were not instructed to use this argument (i.e., in the personal group, χ2 (1, n=309) 8.80, p<0.01, ω=0.07).

Table 3 Percentage of interviews after interviewer’s reaction.
Panel A Mentioning any argument or no argument at all
Any argument No argument Total Chi-square (df=1)
Interview 28% 8% 21% χ2=33.38***
n 372 216 588 ω=0.23

Panel B Principle of authority

Study importance mentioned Study importance not mentioned Total Chi-square (df=1)

Interview 31% 20% 21% χ2=3.50
n 59 529 588 ω=0.07

Panel C Principle of liking
Personal importance mentioned Personal importance not mentioned Total Chi-square (df=1)

Interview 42% 20% 21% χ2=11.3***
n 33 555 588 ω=0.14

Panel D Principle of social validation

Most people enjoy mentioned Most people enjoy not mentioned Total Chi-square (df=1)

Interview 31% 21% 21% χ2=1.53
n 26 562 588 ω=0.07

***p<0.001.

A logistic regression model with participation as the dependent variable, and interviewer group (reference=personal), use of the authority argument and interaction was marginally significant χ2 (3, n=588) 7.34, p=0.06, Nagelkerke R2=0.019, with significant between interviewer group and argument (b=1.347, standard error (SE) [b], p<0.01, Exp [b] (95 percent confidence interval)). The odds ratio of 3.95 shows that personal interviewers who use this argument have an odds ratio four times higher than formal interviewers in getting respondents to agree to be interviewed.

Panel C shows that mentioning personal relevance increased participation. Formal interviewers, who were not instructed to use this argument, did use this argument in three cases (which appeared to be two different interviewers), all of which were successful (although due to the low numbers this finding cannot be verified statistically). Finally, panel D shows that mentioning that most people enjoy the survey is only moderately successful. Personal interviewers never used this argument (as instructed), and for formal interviewers, this argument did not significantly increase participation.

Conclusion and Further Research

From our findings, we conclude that a failure of interviewers to apply their training could not account for the absence of a difference in participation between the two groups of interviewers. Although in the majority (78 percent) of the cases that eventually ended up in an interview (n=610), householders complied without any need for persuasion, interviewers managed to successfully persuade 21 percent of householders who initially refused (n=588). Participation among initially declining householders increases close to 30 percent when interviewers use an argument of any kind in persuasion. This effect was not different between the two differently trained groups of interviewers. Therefore, our first conclusion is that any argumentation is better than none. However, the causal direction of this effect is, of course, not clear. It is possible that interviewers’ expectations of success drive the results; interviewers might only make the effort to provide an argument when from the householder’s reaction they expect to be successful in persuading that particular person.

Looking at arguments actually used in the interaction, interesting findings arise. For the importance of the study as an argument, we found a significant interaction effect showing that interviewers trained in the personal style (not instructed to use this argument) had more success in gaining cooperation than interviewers trained in the formal style (instructed to use this argument). Likewise, two interviewers trained in the formal style of interviewing, who stressed (against the rules in their condition) their personal relevance, were equally successful in using that argument as interviewers in the personal style (who were specifically instructed to do so.) Therefore, we posit that interviewers using arguments as instructed develop too much of a unauthentic routine in expressing these arguments, whereas using arguments outside instructions are likely to be expressed in a more natural, spontaneous way and therefore more convincing. In addition, providing interviewers with too specific instructions on which argument they should use also moves them away from tailoring arguments to specific householders. Both tailoring (Groves, Cialdini, and Couper 1992) and spontaneous behavior (i.e., nonverbatim reading, see Houtkoop-Steenstra and Bergh 2000) have been demonstrated to be most effective in persuading reluctant householders.

In addition, this study shows that it is useful to include manipulation checks in experiments involving interviewer behavior. Researchers testing the effects of a manipulation of interviewer behavior should take into account more detailed checks on how these behaviors were actually implemented.

Erratum: This article was updated on March 28, 2018 to correct errors in Table 3 and accompanying text. These changes had no effect on the conclusion. The original version of the article may be found in the supplemental materials.

References

Cialdini, R.B. 1988. Influence: Science and Practice. Boston: Scott Foresman.
Google Scholar
Dijkstra, W. 1987. “Interviewing Style and Respondent Behavior: An Experimental Study of the Survey-Interview.” Sociological Methods and Research 16 (2): 309–34.
Google Scholar
Dijkstra, W., and J. Smit. 2002. “Persuading Reluctant Recipients in Telephone Surveys.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little. New York: John Wiley & Sons, Inc.
Google Scholar
Groves, R.M., R.B. Cialdini, and M.P. Couper. 1992. “Understanding the Decision to Participate in a Survey.” Public Opinion Quarterly 56 (4): 475–95.
Google Scholar
Houtkoop-Steenstra, J.P., and H. Bergh. 2000. “Effects of Introduction in Large-Scale Telephone Survey Interviews.” Sociological Methods and Research 28 (3): 281–300.
Google Scholar
O’Neil, M., R.M. Groves, and C. Cannell. 1980. “Telephone Interview Introductions and Refusal Rates: Experiments in Increasing Respondent Cooperation.” In Proceedings of the American Statistical Association, Section on Survey Research Methods, 252–55. Washington, DC: 1979: American Statistical Association.
Google Scholar
Schaeffer, N.C., D. Garbarski, J. Freese, and D.W. Maynard. 2013. “An Interactional Model of the Call for Survey Participation.” Public Opinion Quarterly 77 (1): 323–51.
Google Scholar

This website uses cookies

We use cookies to enhance your experience and support COUNTER Metrics for transparent reporting of readership statistics. Cookie data is not sold to third parties or used for marketing purposes.

Powered by Scholastica, the modern academic journal management system