On-the-Job: What to Expect and How to Succeed

Introduction

The work environments of survey researchers and survey methodologists are varied, and their roles are diverse. Across commercial, government, and academic sectors, survey researchers are usually involved in conducting surveys for practical use by stakeholders who need the data for decisions unrelated to the scientific study of survey methodology. Survey methodologists, those who examine and develop survey methods, are often in the academic sector, or in technical or leadership positions in large survey organizations. Yet, most people working in surveys are in a researcher role, rather than a methodologist role. This has implications for training, both in the university and on the job. Although there may be clear distinctions between survey research and survey methodology roles in some contexts, they are both part of the same enterprise. Thus, we use the term “survey researcher” to refer to people with a broad survey methods skill set, and who are involved in various aspects of survey planning, operations, and analysis.

The Job Environment

Survey researchers are well-represented in organizations across employment sectors. In the private sector, these include very large companies and organizations that do surveys on various topics, such as RTI, Westat, NORC, RAND, American Institutes for Research, and Abt SRBI; polling companies like Pew and Gallup; media research companies like Nielsen; and market research companies like Proctor and Gamble, to name only a few major survey research sectors. There are numerous mid-sized and smaller companies, many of which (at the individual or corporate level) are represented by either the American Association for Public Opinion Research (AAPOR), the Council of American Survey Research Organizations (CASRO), the Market Research Association (MRA), or the Survey Research Methods Section (SRMS) of the American Statistical Association (ASA). Within the government sector, the U.S. Census Bureau is the agency most renowned for its dedicated mission to conduct surveys and censuses, but other Federal statistical agencies employ survey researchers, too (see http://fedstats.sites.usa.gov/agencies/ for a list of those agencies).

Some state and city government agencies, such as local health departments or large school systems, have research units that use survey data and conduct surveys. State and local government agencies may work with academic survey research centers and research companies for their public health and educational survey needs, or for polls of their communities. Academic survey research centers design and conduct surveys for a wide range of clients, sometimes including large national surveys (see http://www.aasro.org/ for a list of academic survey research organizations).^[1] In addition, survey researchers are found in academic departments unaffiliated with a survey center or in academic research management roles. Survey researchers and methodologists can be found almost anywhere research is being done.

Required Skills Beyond the Classroom

Survey researchers can find fulfilling employment in any sector, but what can they expect in job duties and on-the-job OJT? Overall, survey research jobs are probably more similar than different across sectors and settings, since they require a common skill set and knowledge base. They are at least similar in that they all deal with the design and analysis of surveys, as opposed to other types of quantitative research. Yet role expectations and work cultures can vary widely. Some skills for success can be learned from academic training, but others are very difficult to teach in academic courses (even practica) and are only learned through extensive “real world” experience. Many of these skills, such as how to run meetings and manage multifaceted projects with complex timelines, are not directly related to survey methods per se, and so receive little if any focus in formal training. Call them “street smarts” or “acquired competencies,” they are the things we learn from doing and experiencing, not from reading and listening to lectures. Reading even the best books on how to conduct a survey, does not teach one how to “run a survey.” We discuss some of these real-world and OJT and learning examples, and ways to promote them in survey contexts.

Collaboration and Teamwork

Survey researchers can expect to interact with colleagues in a range of roles and from various professional and educational backgrounds. Substantive and survey design expertise are both needed to produce high-quality survey data and estimates, so it is common for survey teams to include academic substantive experts, survey production staff, statistical and sampling staff, questionnaire design staff, and management staff. Relationships often last from concept to final product and can continue across projects, years, and decades. Project and personal success are often dependent on being able to work well with others, despite personality conflicts or the difficulty of the task. Survey researchers must have a good understanding of the survey life cycle and sources of survey error in order to contribute fully to these teams.

Trade-offs are inherent in collaborative efforts, and it can be difficult to know when and how to make trade-offs between design purity and operational functionality without some real-world experience. Statisticians and methodologists may propose an ideal survey design only to find out that it exceeds the survey’s budget or that the idea is beyond the technical capacity of the organization. How does a survey researcher adjust his or her design proposal taking practicality into account? Similarly, principal investigators or managers without survey methodology experience may propose techniques that survey methodologists know to be categorically of poor quality. This will require tact on the part of the survey researcher to propose and gain acceptance of a better design. These are examples where survey researchers have a very important responsibility to educate clients and collaborators and help them make good decisions. Survey researchers trained in the survey life cycle should be able to explain the essential aspects of cost/error trade-offs for a design and a survey design team help reach a practical, yet scientifically-sound decision. In our experience, some of the most effective survey researchers who work with diverse teams are excellent listeners. They take in information and synthesize it before making pronouncements about which way the survey should go. Listening is a good foundation for leadership.

Organizational Culture and Methodological Roles

Survey researchers are most effective, and probably happiest, when they understand their role in the larger organizational or project culture. Does the organization expect them to be removed from day-to-day operations, conducting academic-level research and consulting only when invited? Or are they expected to attend regular planning meetings and participate in solving day-to-day data collection problems? Of these two archetypal methodologist roles, the former likely applies in larger organizations, and the latter in smaller ones. In the smallest survey research departments and centers, the survey expert also plays the role of manager, accountant, information technology technician, and other non-researcher roles. They may also find themselves to be the only person with the sufficient quantitative expertise to perform many statistical tasks from data analysis to manuscript preparation. Regardless of role expectations or requirements, survey researchers should be both followers and leaders depending on the context. No university course or corporate training seminar can accurately reflect these cultural expectations or when a survey researcher should listen versus lead. Understanding how decisions are made, who makes them, and (most importantly) why they are made comes from experience and mentoring. Yet it can be a revelatory experience to new researchers to learn that the job they thought was a research job is really more of an operations management, policy analysis, or marketing job. In larger organizations, survey researchers often are hired to fill a specific niche role, with deep expertise in a critical component of surveys or that organization’s survey process. An employee hired to work on sampling and weighting may be seen as overstepping their bounds by trying to contribute to the questionnaire design. In smaller survey organizations, where the survey researcher is a Jack (or Jill)-of-all-trades, the same behavior may be seen as proactive and positive. An employee often only learns what is accepted and expected behavior once he or she oversteps a boundary. Over time, the most successful survey researchers, like people in any field, learn how to align their behaviors with their boss’ (or bosses’) expectations. Even when not directly assigned to a project, successful survey researchers often find ways to help the organization understand and thoughtfully consider survey methods to improve their processes and products. This is another aspect of the survey leadership role.

Concrete Skills to Bring to the Job

Employers who hire survey researchers look for persons with good training in survey methods and practical application knowledge. A solid background in survey methodology is important, particularly for jobs with more independence, specialization, or leadership. Survey researchers are sometimes called on to blend survey design and statistical or social science theory, and to bridge gaps between theory and practice. They are least expected to be current in survey research best practices and methodological findings. In any field, a broad understanding of the myriad technical steps and techniques that are the norm is essential for an employee and priceless to an employer. In survey research, that means a good understanding of the survey life cycle and a working knowledge of each component from coverage and sampling to measurement and data processing, despite the survey researcher’s main focus. The more senior the role, the more important it is to have a broad understanding of survey methods and facility with dominant conceptual paradigms. In survey methodology, that would be the various sources of Total Survey Error, and being able to identify where they arise in the survey process.

Both social science and statistical science survey methodologists should have good quantitative skills as they are increasingly essential for even the most basic survey research roles (see “Kolenikov and Jans, Meyers, and Fricker, this issue). Being quantitative need not mean being purely mathematical. Yet being able to conduct basic quantitative analyses in standard statistical packages such as SAS, R, Stata or even Excel is important in most survey research jobs. Although every survey researcher need not be a programmer, automating analyses and other repetitive tasks becomes essential as one grows in his or her career, and such skills can make that person a very desirable employee. These productivity tips are rarely taught in academic coursework leaving them to OJT. Database processing and manipulation skills are valuable as working with data in various forms is a common part of a survey researcher’s job. Ability to design questionnaires and forms in Word, Web survey software, or other software can be desirable skills as well. Statistical staff need to have a basic understanding of and appreciation for the interpersonal and measurement aspects of survey data collection, so they can design better sampling procedures, nonresponse adjustments, and other statistical aspects of the design. Statistical methodologists will always rely on the expertise of social science methodologists for the arcana that they understand less-fully, and vice versa, but a basic common ground smooths the design process. This is one of the goals of programs like the Joint Program in Survey Methodology.

The interdisciplinary nature of survey methods and its applications often require explaining ideas to and gaining acceptance from technical and nontechnical project members. Thus, collaboration and teamwork require good communication skills. The ability to express ideas clearly and concisely, both verbally and in written form, is important across all types of jobs, even the most technical. This involves both being able to explain technically complex or academic survey methodology topics, and knowing the audience to which the communication is directed.

On-the-job Training (OJT) Scenarios

Where can staff of survey research organizations obtain these skills if they have not obtained them elsewhere? This section addresses some of the ways that we have seen staff nurtured and their careers developed through OJT. Obviously, we cannot address every possible OJT opportunity, so readers should consider applications in their own work settings and the special circumstance they face. For example, a) how do you conduct OJT with freshly-trained employees compared to those with extensive prior experience, b) how can you design experiences and opportunities even if you do not have management and budget authority, and c) where in your own career or your managing/mentoring of other staff could you have used these?

Scenario 1 – Gaining Technical Skills: New employees, particularly recent graduates, may have strong statistics training but much less programming experience. Managers can promote OJT by providing sufficient time and guidance to learn how to program while doing assigned analysis tasks. When staff cannot find solutions to technical problems themselves, it can be helpful to provide support in the form of introductory resources (e.g., books, websites), and people who can help them. Creating connections between staff and other experts in your organization may provide benefits to your staff and operations over time. Training can come in the form of support and mentoring, and need not always involve costly formal instruction.

Scenario 2 – Adapting to the Workplace: Survey researchers come from a wide range of educational backgrounds and professional experiences, which means that they may be very experienced in a particular area of survey research (e.g., questionnaire design), but have less experience with other aspects of survey research (e.g., paradata monitoring). They may also have survey research experience from other organization but do not yet know the detailed operations of their current organization. When employees are expected to perform beyond their previous experiences, some form of OJT may be helpful. Employers can offer staff opportunities to experience other areas of survey research within the organization by assigning them to work closely with experienced staff in other divisions or units, or who are in charge of tasks that are unfamiliar to the new employee. For example, a statistician without previous questionnaire design experience might work on a qualitative questionnaire pre-test. Similarly, a qualitative researcher could help write a quantitative report or learn how to run basic statistical analyses. Researchers who usually work on measurement error could be assigned to new projects to gain some experience working on nonresponse and coverage error, and vice versa. For all staff, shadowing interviewers in the field or in a call center, and conducting interviews themselves is an essential part of understanding the survey life cycle. At a minimum, working closely with staff in operations and research through regular team meetings or collaboration is encouraged to gain understanding of the challenges, constraints, and methods of all phases of the survey process.

Scenario 3 – Getting the Job Done: Part of OJT is developing environments that support the skills that staff need to be productive. Being productive usually involves coordinating activities across staff in different roles. Coordination responsibilities fall to the individual line worker, as well as the manager, to assure a clear understanding of how the survey research activities work together to produce results. Managers can teach by example by being organized themselves, running effective meetings, and assuring that individual employees clearly understand what is expected of them. Individuals are also responsibile for understanding how their work contributes to the entire survey effort. While effective communication and coordination skills often are learned through practice or observing effective leaders, OJT can serve a very helpful role. Offer staff project management and time management training if they want it, and assign it to staff who need it to improve their organizational skills.

Scenario 4 – Creating Exemplary Staff: The scenarios so far have addressed OJT designed to bring staff up-to-speed when they are lacking in some area. However, OJT need not end when all staff meet a minimum or average level of performance. It can also be visionary and promote new skills among well-performing staff. Some employees view proactive career or leadership advancement as part of their career plan, and others are committed to advancing their field of practice, if not their own career per se. Organizations that design advancement and training opportunities that bolster both individual employees and the field as a whole are making positive steps to strengthen their workforce and organizational capacity. Depending on their career goals, staff could pursue formal title changes, new supervision responsibilities, or other status/responsibility changes. For example, every new staff member (depending on experience) could be required to serve as support staff for some number of projects and then be promoted to a project lead role. This can be a good way to encourage mentorship because junior staff will be working closely with senior staff while they are in the “support” role. Alternatively, employees can be assigned to projects that expand their experience with different survey modes, topics, or technologies. Finally, job relevant certification [e.g., project management professional (PMP) or SAS certification] may be appropriate and will bring new skills to the organization.

Positive OJT can be challenging to encourage. Particularly when under pressure to perform, employees may view the time required to develop their own professional skills as a luxury, even if it would make their work more productive and their overall use of time more efficient. Managers may have the same view. However, as the final arbiters of performance and promotion, managers can do much to support and encourage OJT. This can be as simple and unobtrusive as making space in regular staff meetings for employees to share skills and tips, or ask for help. A larger commitment might look like Google’s successful “Google Time” program in which every engineer was given 20 percent of their work time (essentially one day a week) to spend on a work-relevant creative endeavor. Work-relevance was broadly defined, and the only requirement was that they had to share their progress and findings at internal seminars (Johnson 2010). Although Google stopped using this system in 2013,^[2] several of their successful products have been attributed to Google Time projects.

Final Thoughts About Learning on the Job

There is a clear need for continuous learning throughout a survey methodologist’s career, and many opportunities to obtain it. The field of survey methodology evolves as we continue to face new challenges to the science of accurately measuring and making inferences about populations. A number of emerging issues impact the traditional survey design, data collection, and estimation environment. These include alternative platforms for contacting and collecting data from respondents, nonprobability samples, the so-called “big data” of social media, and other organic data. All of these require bringing the science of survey methods to bear in developing appropriate design, data collection, and estimation methods.

There are existing opportunities to continually learn in the work environment, particularly for those early in their career. Formal coursework is always an option, and the proliferation of online training has made it much easier for employees to balance training with work (see “Kolenikov, Jans, O’Hare, and Fricker” in this issue).

Extensive statistical training (written and video) is available online, too. Rather than send an employee to an in-person SAS training for hundreds of dollars, why not have them watch videos at http://www.ats.ucla.edu/stat/sas/ as an example? Listservs like AAPORnet and SRMSnet still remain good resources for helping survey researchers solve problems and represent some of the best aspects of our field twist on OJT and mentoring. Some of the skills we have discussed in this article do not appear in any formal training program. Internships and mentorships can be a great introduction to the work environment. For example, RTI International has an internal mentorship program in which senior staff are matched with junior staff and serve as professional mentors beyond what can usually be provided in day-to-day supervision and management. In addition, the American Statistical Association (ASA) and the American Association of Public Opinion Research (AAPOR) have developed internship and mentorships programs that connect junior professionals with senior professionals. Internships typically provide opportunities to learn the procedures of conducting the survey – the practical application of survey methods training. A mentor provides guidance on how to handle decisions and relationships in the workplace, based on their senior experience. Mentoring and advising need not be in-person and by people in the same organization as the mentee. While remote mentoring has probably always been done to some degree, social media have made this possibility very accessible. Junior staff looking for advice about their careers or technical problems can use numerous forums, blogs, and other networking sites to fill this gap (e.g., www.statckexchange.com; www.researchgate.com; www.facebook.com; www.linkedin.com). Survey methodologist bloggers provide a different twist on “passive mentoring,” by writing instructive or thought-provoking posts that focus on contemporary issues and shared challenges in survey research (e.g., http://jameswagnersurv.blogspot.com/). We think staff should be encouraged to use social media, blogs, websites, and other similar methods to further their training and mentoring to the degree they are allowed by corporate policy and not abused, of course.

The good news for survey researchers is that there are many job opportunities in a variety of work environments. Further, as survey methods become more complex and diversify in our information-hungry society, there is a growing need for experienced survey researchers to address these needs. Survey researchers are well-positioned for successful careers.

Disclaimer

Any views expressed are those of the authors and do not reflect those of the U.S. Census Bureau or Abt SRBI.

Stas Kolenikov wishes to acknowledge and thank Abt SRBI for supporting and encouraging his participation in and contributions to this special issue.