As a qualitative research tool, researchers have identified the cognitive interviewing methodology to pretest survey translations (Forsyth et al. 2007; Pan et al. 2007; Sha, Pan, and Lazirko 2012). The U.S. Census Bureau has also conducted large-scale cognitive testing to evaluate translation quality (Carter et al. 2011; P. Goerman et al. 2007; Pan et al. 2009; Sha, Pan, and Lazirko 2012). All of these studies recruited non-English speakers since those are the characteristics of the intended audience for the translated materials. The researchers sometimes documented the challenges of participant recruitment but did not systematically analyze the efficiency of the recruitment methods. Looking at research conducted about non-English speaking research participants, the recruitment methods were not discussed in detail. Instead, the selection of the participants and how their characteristics influenced the results of the cognitive interviews seemed to be the primary concern of the researchers (P. Goerman 2010; P. L. Goerman and Caspar 2010; Park and Wake 2010). Nevertheless, anecdotal observations from recruiters suggested that the word-of-mouth method was efficient in recruiting non-English speakers (Forsyth et al. 2007; Saleska et al. 2009; Sha et al. 2010).
The limited research on the efficiency of the recruitment methods may not be surprising because cognitive testing usually involves a small number of interviews, and such analysis require a comprehensive set of data on recruitment activities and a large number of recruits. Using recruitment records from 845 Chinese and Korean speakers on a multilingual cognitive testing study, this paper affords the opportunity to investigate the efficiency of four common recruitment methods and suggest strategies for recruiting qualitative research participants who have limited English proficiency.
The recruitment data were collected for a large-scale cognitive testing conducted by the U.S. Census Bureau to pretest the Chinese and Korean translations of the American Community Survey (ACS) language assistance guide. The goal of the recruitment was to recruit adult Chinese and Korean speakers who speak little or no English (“monolingual”), while meeting a set of demographic recruitment targets. From December 2009 to July 2011, eight bilingual recruiters contacted 1,084 Chinese and Korean speakers in the DC/Maryland/Virginia are, Illinois, and North Carolina. By the end of the recruitment period, a dataset of 845 cases was available for analysis. More details about the recruitment program can be found in Sha et al. (Sha, Park, and Pan 2012).
Four recruitment methods were used by the recruiters: newspaper advertisement, physical flyers, electronic dissemination, and word of mouth. Each method involved specific activities. For example, the recruiters coordinated the efforts to place print advertisements in popular Chinese and Korean newspapers, posted flyers at churches and businesses frequented by Chinese and Korean speakers, and recruited with the help from community key informants, personal network, or group events. They also spread the message online using subscriber-only e-mail distribution lists and online forums that were created for Chinese and Korean immigrants to exchange information. We did not actively keep track of the bounce-back rate. The recruiters recorded details about the following: the number of minutes spent on each recruitment attempt for an activity (“recruit time”), the number of minutes used to contact and screen potential participants (“contact & screen time”), the frequency of recruitment attempts (# of recruit attempts), the number of calls [inquiries] received from recruits after they saw a recruitment message (# of calls), and the number of complete and incomplete screeners.
We used three measures to examine the efficiency of the recruitment methods. Efficiency is defined as recruiting potential participants faster by spending less time (“time efficiency”), receiving more calls from potential participants after they saw a recruitment message (“outreach capacity”), and recruiting a higher percentage of monolingual participants (“eligibility rate”). For time efficiency, we looked at time spent for recruitment attempts separately from time spent after the recruitment attempts for contacting and screening potential participants (complete or incomplete screeners). The calculation formulas are shown in Table 1.
Three research questions are addressed:
- Which method is the most time efficient?
- Which method presents the best outreach capacity?
- Which method demonstrates the highest eligibility rate?
In the sections below, we report the efficiency of the recruitment methods.
As shown in Figure 1, advertisements in popular in-language newspapers were overall the most time efficient method. On average, recruiters received a response to the advertisement after investing 0.8 minutes in recruitment attempts. Physical flyers, on the other hand, was the least time efficient method to spread the recruitment message — every 13.9 minutes of recruitment attempts only contributed to one call from potential participants. Electronic dissemination ranked the second most time efficient at recruitment attempts (5.4 minutes), followed by word of mouth (11.3 minutes). In addition, newspaper advertisement was also efficient in terms of contacting and screening recruits. It took an average of 7 minutes to contact and complete a screener with a potential respondent. Physical flyer method took slightly less time (6.5 minutes) than newspaper advertisement, and electronic dissemination was a distant third (10.4 minutes). When the screener was incomplete, the physical flyer performed moderately (5.7 minutes), and the time spent was similar to the electronic dissemination method (5.2 minutes). However, both were not as efficient as newspaper advertisement (1.4 minutes). In addition, in all measures of time efficiency, the word-of-mouth approach was not efficient. It took 11.3 minutes of recruitment attempts to reach participants and it was also the least efficient method contacting and screening the recruits (13 and 13.4 minutes).
Figure 1 shows that every time a recruiter released the recruitment messages through a newspaper advertisement, it generated 39.5 calls. This was in strong contrast with the number of calls resulted from electronic dissemination (3.0), physical flyers (2.0), and word of mouth (1.8). In other words, each one recruitment attempt a recruiter took to recruit online, post flyers or use word of mouth only resulted in about two or three calls from potential participants.
Although the word-of-mouth approach was the least efficient in terms of time efficiency and outreach capacity, Figure 2 showed that it had the highest eligibility rate at 86%. In comparison, newspaper ads had the dominant advantage for time efficiency and outreach capacity, but its eligibility rate was low. Only about half (54%) of the screened individuals reached by newspaper advertisement were monolingual. In other words, about half of the screened individuals were ineligible using this method. However, it still performed better than electronic dissemination (49%), which had the lowest eligibility rate among all methods. Physical flyers had a similar eligibility rate to the newspaper ads at 59%.
Recommendations for Participant Recruitment
Based on the results from this study, in general, we recommend a two-step process to recruit non-English speaking participants for qualitative studies. The first step is to build up a pool with a sufficient number of screened individuals available from which to select. To accomplish this goal, we suggest using methods with good time efficiency or strong outreach capacity, which can reach potential participants in a timely and extensive manner. In the current study, we found that newspaper advertisement is best suited to serve this purpose. For non-English speakers, newspapers in their native language may be a major source of information and can be accessed easily and inexpensively. Once most recruitment criteria are fulfilled, the next step is to purposively identify individuals with hard-to-recruit characteristics. Implementing a method with a high eligibility rate is likely to be successful. The word-of-mouth method is a clear choice.
Adopting one recruitment method over another is a decision to be made based on the specific needs of the research project. For example, if a project has a tight schedule but must recruit a wide range of participants in a short period of time, approaches that render high time efficiency and outreach capacity would perform best. Recruiters can reach a lot of potential participants using newspaper advertisements while spending the least amount of time. Electronic dissemination is also a good option to spread the recruitment message because it requires less time at each recruitment attempt (e.g., sending an e-mail is less time consuming than an in-person visit). The caveat about using newspaper advertisement and electronic dissemination to reach a lot of potential participants quickly is that the eligibility criteria cannot be too specific, because they both have low eligibility rate. Nevertheless, our research (Park et al. 2011) reported that the electronic dissemination method tended to reach younger, higher educated people and therefore would be useful if the recruitment criteria required those participant characteristics. Physical flyers were only time efficient when screeners were successfully completed, but did not perform well in all other measures. This method may work better in conjunction with other methods.
If a project is designed to interview participants with a narrow set of selection criteria that are hard to recruit, it may be worthwhile to identify participants with the desired characteristics using methods with a high eligibility rate. In this study, the word-of-mouth approach proved to be the best way of recruiting qualified monolingual participants. With word of mouth, recruiters had to spend more time talking to potential participants or key informants, and a single in-person visit was not always fruitful. Successful recruiters carefully planned communications and were able to build trust because of their interpersonal communication skills. Using recruiters who demonstrate these characteristics and the willingness to network is key to applying this method.
Disclaimer: This paper is released to inform interested parties of research and to encourage discussion of work in progress. Any views expressed on (statistical, methodological, technical, or operational) issues are those of the authors and not necessarily those of the U.S. Census Bureau.
The authors thank Barbara Lazirko, Yuling Pan, Todd Hughes, Dameka Reese, and Debbie Klein of the U.S. Census Bureau. In addition, the authors acknowledge Hyunjoo Park, Lu Liu, Mandy Sha, and Michelle Yuan of RTI International for designing the recruitment data collection tools and the RTI recruiters who collected the recruitment data: Lu Liu, Jiyoung Son, Qinghui Guo, Grace Chan, Suzie Kim, Younhee Harm, Michelle Yuan, and Guangya Liu.