Training Needs in Survey Research Methods: An Overview

Barbara O'Hare; Matt Jans; Stanislav (Stas) Kolenikov

doi:10.29115/SP-2015-0010

The Case for Assessing Current Survey Research Training

The twentieth century witnessed a dramatic rise in the need for, interest in, and use of public opinion findings and related social and behavioral data derived from surveys. Results from survey research now drive policies, decisions, and scientific knowledge in almost every sector of our social, educational, and political spheres. This has led to increasing demand for workers trained in survey research and survey methodology.

The rapid change in the field of survey research calls for appropriately-focused and specialized training. The last quarter century has seen an explosion in research on survey methods in both the statistical and nonstatistical aspects of the field, an explosion that has been accompanied by a corresponding growth in the survey methodology literature. This period has also seen important developments in training opportunities for survey methodologists in universities, government agencies, and in professional organizations. Survey methodology now has a well-established standing as a discipline of specialization, rather than just a field of practice. Despite its established status, survey methodology is in a state of transition for many reasons, including falling response rates and an increasing number of alternative modes of data collection. Much research and many new developments should be expected in the next decade, requiring survey that are methodologists trained to address these challenges.

What skills do employers look for in survey specialists? Hiring managers (i.e., senior researchers leading teams of methodologists) may have specific ideas of the required skills, but their wish lists of required job skills may be so ambitious that they are intimidating to applicants. Overly-ambitious job descriptions may be impossible to fill with a single person because survey methods have become so complex. However, students of survey methodology, statistics, and social science programs who are interested in survey research careers, must understand the strengths in their training and the gaps that need to be filled to get closer to an employer’s wish list and be an easily-hirable survey specialist.

The news is good for students of survey methodology. Survey organizations in all sectors are continuously hiring statisticians and survey methodologists. This special issue of Survey Practice was inspired by a panel on “Training for Survey Research” held in June 2014 by the Baltimore-DC Chapter of American Association for Public Opinion Research (AAPOR), and organized by Scott Fricker (Bureau of Labor Statistics). Stanley Presser (University of Maryland) moderated the session, and panelists included Graham Kalton (Westat), Barbara O’Hare (U.S. Census Bureau), Scott Keeter (Pew), Mikelyn Meyers (U.S. Census Bureau), Jill Dever (RTI International), and Danni Mayclin (Energy Information Agency). They addressed four core questions:

What skills does a survey researcher need to contribute effectively to the field?
What are survey organizations looking for when they hire?
What are the pros and cons of current educational programs?
How might programs better prepare students for survey careers?

The panelists recognized common themes as they shared personal experiences and perspectives on survey research and methodology training. All recognized the growing technical and substantive complexity of survey research and the need for existing training programs to respond, the need for new and continued learning to be successful on the job, the essential role of collaboration and communication in conducting and evaluating surveys, and that these components draw upon diverse academic, technical, and experiential skillsets. This issue summarizes and develops these ideas to help move the field forward.

The Backdrop of Survey Research and Survey Methodology

Survey methodology is relatively young, having evolved from early-to-mid-twentieth century roots in statistical sampling, psychological measurement and quantitative social science generally (see, for example, Groves’ 2011 article “Three Eras of Survey Research” in the Public Opinion Quarterly special issue on the journal’s 75^th anniversary.^[1] Other articles in the issue contain insights from many of the pioneers in survey methodology). The study of survey methodology and the practice of survey methods are inextricably intertwined. As the utility of systematic surveys to gain perspective on issues from commerce to public health to television viewing habits grew, so did the businesses, government agencies, and academic survey research centers that conducted this research. Survey researchers in those organizations put forward best practices for our field and turned a niche methodology into a mainstay of the information age. Survey methodologists were not far behind, sometimes serving dual roles with survey researchers, questioning and evaluating the methods of sampling, question asking, and survey data analysis used to move this field ahead. A core set of principles and perspectives dubbed “survey methodology” began to emerge. Survey practitioners whose academic origins were in sociology, psychology, statistics, biostatistics, and other fields began to see surveys as more than just tools to evaluate social phenomena. Like any other research methodology or epistemological framework, survey methods could be questioned, evaluated, and innovated upon. The Joint Program in Survey Methodology (JPSM) first formalized this independent yet interdisciplinary field into a program of study under the name “Survey Methodology” in 1993 as a collaboration between the University of Maryland the University of Michigan, and Westat, an employee-owned contract research organization.^[2] The goal of JPSM was to harness the unique expertises of each party and provide state-of-the-art survey methodology training for the U.S. Federal Statistical System, but its student base has reached multiple survey research sectors.

The core distinction between survey research and survey methodology is that the former focuses on conducting surveys for researchers or clients with their own substantive (i.e., non-methodological) interests (e.g., to answer sociological, public health, or economic questions), while the latter views the process of survey data collection as its substantive topic of study. Many fields use survey research methods in the pursuit of academic, business, or public wellness goals, but survey methodology concerns itself with the assessment and improvement of survey methods, process of data collection and whether (and how) analysis of data collected from sample surveys is affected by aspects of survey design and data collection protocol, independent of the topic of the survey itself.

Key Panel Messages

Survey Research and Methodology should Celebrate their Interdisciplinary Histories

The parent fields of survey methodology are social science and statistical science, broadly speaking. Social science survey methodology claims its lineage from fields like psychology, sociology, education, linguistics (mostly sociolinguistics and psycholinguistics), and political science. The survey research techniques generally proscribed to social science survey methodology include questionnaire design, question pretesting, cognitive and social psychological aspects of asking and answering survey questions (and contacting sample units), interactions between interviewers and survey respondents, and related topics that have to do with the psychology and sociology of response and nonresponse at the unit and item level. The statistical aspects of survey methodology include topics like sampling frame development, sample design and sampling techniques, weighting sample data, unit and item nonresponse adjustment (e.g., weighting and imputation), statistical disclosure control, imputation of missing data, statistical analysis of complex sample survey data, and other quantitative topics. Consequently, survey methodology training programs are essentially interdisciplinary.

The Joint Program in Survey Methodology was established with the goal of providing students with a general understanding of all aspects of survey research and preparing students with the skills needed to produce high quality surveys. The program includes two distinct, but overlapping tracks: statistical science and social science. The program was established particularly to train workers in the United States Federal statistical agencies on census and survey methods relevant to the production of official statistics (Clark and Groves 2002). The program quickly drew students from commercial, academic/scientific organizations, as well as government agencies. In addition to the mission of training the workforce in current survey methods, the program had a goal of establishing a program for survey methods research and training that would advance survey methodology (Tourangeau and Lepkowski 2008).

There are currently three doctoral programs in the United States (University of Michigan,^[3] University of Maryland,^[4] and University of Nebraska),^[5] and one in the United Kingdom (University of Essex^[6]), specifically claiming the survey methodology identity. Several other graduate programs train researchers in the more statistical aspects of survey methodology, including the statistics program at Iowa State University^[7] in the United States and the Ph.D. program in Social Statistics and Demography at the University of Southampton, UK.^[8] Other programs in the United States and around the world offer survey methodology training at the MS/MA level, or offer similar training under different names (e.g., University of Connecticut Graduate Program in Survey Research,^[9] Utrecht University in the Netherlands,^[10] University of Wollongong in Australia^[11]), or graduate level certificates (University of Illinois at Chicago Survey Research Methods Certificate Program,^[12] Certificate Program in Survey Methodology at University of North Carolina, Chapel Hill).^[13] The Survey Research Methods Section of the American Statistical Association maintains a list of such programs (see http://www.amstat.org/sections/srms/college.html).

Other pathways into the profession include graduate programs in mathematical statistics or biostatistics, or in social sciences under advisership of survey researchers that have their academic home in social science departments. Graduates of such programs may benefit from complementary training, as discussed in later articles of this Survey Practice issue.

Survey Research and Methodology must Think Toward the Future

Whether or not survey methodology needs to have a clearly-defined academic identity is debatable. Is survey methodology better served as a hybrid field drawing from established disciplines and offering interdisciplinary training, or is it better served by organizing under a common framework like Total Survey Error? Both perspectives have their strengths and weaknesses, but it is clear that the services and perspectives survey methodologists can provide are unique, yet not unchallenged in academic and professional circles. It is easy to argue that the combination of social and statistical sciences that comprise survey methodology training is rare. Yet other hybrid fields like data science have found ways to rebrand and reshape topics that used to hide from each other in disconnected departments like statistics, mathematics, information science, and computer science.

The survey methods field has grown substantially and has become more technically challenging, increasing the demand for advanced training and for continued learning throughout the careers of survey methodologists. There have been many advances in survey methodology since the establishment of JPSM in 1993 including growth in the literature, number of peer-reviewed journals, international conferences and workshops, and academic courses and programs. Survey specialists need a broad skillset including a solid background in theory and methods, good communication skills, and knowledge of a range of special survey topics, whether in statistics or social science areas. These issues all point to the need for expanding knowledge in the current survey environment that requires new thinking on how to train methodologists to cover all areas.

The study and practice of survey methods needs highly-qualified experts to meet today’s challenges. Survey research is much more than designing questionnaires, collecting data in standard environments, and estimating results from a high-responding probability sample. Survey methods will continue their dependence on interdisciplinary study because related fields, such as data science or communications, are critical to improving survey methods. We need to attract qualified undergraduates to our exciting, growing field through mechanisms such as undergraduate minors in survey statistics or survey methods, or direct recruiting of undergraduates into graduate programs.

The need for training in survey methods is as important as ever. Our economy, health systems, educational institutions, and even personal lives are increasingly information-driven, and the information we need often comes from surveys or questionnaires about behaviors, attitudes, and other personal facts. We are certain that the training structures first established decades ago are no longer sufficient to train future survey researchers. We hope this issue of Survey Practice promotes the necessary dialogue.

Articles in This Issue

This special issue of Survey Practice seeks to propel the discussion of survey methodology, its orientation, training, and professional outlets by expanding on the topics addressed at the DC-AAPOR panel. Beyond this brief introductory article, it includes the following:

“Social Science Survey Methodology Training: Understanding the Past and Assessing the Present to Shape our Future” by Matt Jans, Mikelyn Meyers, and Scott Fricker focuses on social science aspects of survey methodology training, reviewing curricula of existing survey methodology programs and outlining an ideal program of study.
“Training for the Modern Survey Statistician” by Stas Kolenikov lists the key proficiencies in the quantitative aspects of survey statistics and their relations to typical graduate programs in statistics and/or survey methodology. Kolenikov presents the perspectives of current students, statisticians or biostatisticians who want to rebrand themselves and find new careers in survey methodology, and employers.
“On-the-Job: What to Expect and How to Succeed” by Barbara O’Hare, Stas Kolenikov, and Matt Jans discusses the mix of academic and professional skills of a successful survey methodologist, such as communication skills, ability to collaborate with researchers outside of one’s home discipline, and understanding of the survey lifecycle and sources of survey error.
“Future Training of Survey Methodologists” by Stas Kolenikov, Matt Jans, Barbara O’Hare, and Scott Fricker looks at the prospects of survey training through the traditional academic channels, at the formats of survey training outside of the traditional university classroom, and at the additional directions of interdisciplinary survey methodology research that years to come will likely demand.

We hope you enjoy this special issue of Survey Practice, and carry forward this discussion in your own work and training environments.

Disclaimer

Any views expressed are those of the authors and do not reflect those of the U.S. Census Bureau or Abt SRBI.

The authors enjoyed immensely the cross-fertilization of ideas that were generated on the DC-AAPOR panel, on the calls and in the email exchanges during the months of collaborative work on these papers. While all papers have nominal authors, it is fair to say that all members of the core team (Barbara O’Hare, Stas Kolenikov, Matt Jans, Scott Fricker, and Mikelyn Meyers) have contributed to every paper. Stas Kolenikov wishes to acknowledge and thank Abt SRBI for supporting and encouraging his participation in and contributions to this special issue.