Future Training of Survey Methodologists

Introduction

The preceding four papers in this issue outlined the evolution of survey methodology and survey research training programs, summarized the current state of training, and discussed the continually evolving challenges to adapt survey methods in various contexts. “Training Needs in Survey Research Methods: An Overview” by O’Hare, Jans, and Kolenikov provided our motivation for the issue in terms of the need within the discipline to assess where we stand, and outlined survey training in wide brush strokes. “Social Science Survey Methodology Training: Understanding the Past and Assessing the Present to Shape Our Future” by Jans, Meyers, and Fricker focused specifically on the skills and training required of specialists in social science aspects of survey methodology, whose responsibilities typically include cognitive and social aspects of measurements, such as questionnaire design and testing, interviewer-respondent interactions, and translation and cultural adaptions. “Training for the Modern Survey Statistician” by Kolenikov discussed the skillset required of a specialist in survey statistics, whose responsibilities typically cover frame construction and design of the sample on the front end of a study, and weighting and imputation on the back end. “On-the-Job: What to Expect and How to Succeed” by O’Hare, Kolenikov, and Jans described the job environment of a survey methodologist as that of a constant immersion in the survey cycle and in the interaction with other specialists, including substantive area experts, project administrators, clients, and stakeholders. Where do we see our future survey methodologists coming from, and how are they going to accumulate the required skills? We see several potential routes.

Challenges We Face

Current and future survey methodology training needs are motivated in part by technological advances and social changes of an increasingly-electronic and wireless society and how those changes affect our approaches to sampling and measurement. Greater cell phone use, household Internet penetration, the development of mobile broadband, and increased ownership of smartphones and other handheld computing devices^[1] have each changed how people access the Internet and interact with one another. They also offer a tremendous range of new data collection modes and tools that have never been available so easily before. The survey industry can now provide modes that are extremely convenient for respondents, and easily collect data that previously required specialized expensive equipment such as photographs, sound, geolocation, and activity. Coverage, both in terms of geographic coverage and the proportion of the population that owns devices capable of gathering these auxiliary data, is still far from complete, but will only increase over time. In addition to planned data collection using these devices, the flood of organic data, or “big data,” that come from them is staggering. Social networks and individual mobile devices create new opportunities for measuring people in situ with little intervention from researchers or conscious action from respondents. To incorporate these data sources into traditional survey measurement and sampling requires a sophisticated understanding of the technological aspects of data collection, such as the data streams related to social networking software and the sources and structure of automatically-collected administrative data. Given falling response rates and shrinking survey budgets, we are in an environment that almost requires incorporating new sources of data with traditional survey methods. Our ability to adapt to this environment may make the difference between a future with, versus one without, surveys as we know them. Thus, today’s survey researchers must understand the sources of total survey error associated with various sample frames and data collection modes (both traditional and nontraditional) and must know how to optimize data collection plans by choosing freely between available methods. Survey organizations increasingly are interested in specialists that can deliver survey design and data products that respond to these challenges. Both the newly minted and seasoned survey researcher should expect to need to acquire these skills. What does this mean for the future of survey methodology training?

Changes to Existing “Traditional” Survey Methodology Programs

Survey methods training undoubtedly will continue to rely heavily on the foundational elements and core competencies delivered by traditional academic survey methodology programs. The three doctoral programs in the United States – the Joint Program in Survey Methodology (JPSM) at University of Maryland, the Program in Survey Methodology (PSM) at University of Michigan, and the Survey Research and Methodology Program (SRAM) at University of Nebraska, Lincoln – have awarded about 50 survey methodology doctorate degrees by the end of 2014, and scores more master’s degrees. The impact of the first graduates of these programs has spread widely throughout academia, government agencies, and the private sector. The current annual supply of survey methodologists from JPSM, MPSM, and SRAM seems to fluctuate around approximately five PhDs and 20 master’s degrees a year across the three programs. Other programs that train students in survey methodology and survey research increase this output. For example, the Graduate Program in Survey Research at the University of Connecticut (UConn) awards about seven survey research MA degrees a year (all trained online), and other certificate programs like the University of Illinois at Chicago (UIC) Survey Research Methods Certificate Program provide additional experts to the field. Yet the field seems able and eager to absorb far greater numbers.

To address the needs of working professionals, a number of survey methodology programs offer (or soon will offer) online training. UConn was an early adopter of this approach, offering both online certificate and master’s programs. JPSM’s 6-course (18-credit) graduate certificate programs in survey statistics^[2] and in survey methodology^[3] will be online in the near future. The JPSM survey statistics certificate is aimed at individuals who have graduate-level coursework in statistics but desire specific knowledge and training in survey statistics, and is well-suited for early-to-mid-career professionals who want to enter the survey statistics job market. Its survey methodology certificate program is geared toward working professionals with at least one year of experience in a survey data collection setting but whose education did not include courses in survey statistics and methodology. JPSM also offers a leaner educational credential called the Citation in Survey Methodology, which consists of one online course, “Fundamentals of Survey Methodology” and eight short courses. (See section on short courses below.)

Continuing Professional Education

In addition to traditional degree programs that involve regular student-instructor interactions and various forms of student assessments (home assignments, quizzes, exams within each course; thesis for the program as a whole), there are a number of continuing professional education offerings in the discipline, of which short courses and webinars are the most common. Short courses offer specialized and intensive training in specific topic areas in a limited amount of time (e.g., between two hours and five days). Short courses typically involve lectures and individual or group exercises and are aimed at upgrading or reinforcing participants’ knowledge and skills. Survey methodology short courses are offered through academic departments, by professional organizations, and at professional conferences. For example, attendees of the American Association for Public Opinion Research (AAPOR) and the Joint Statistical Meetings (JSM) annual conferences can register for in-person half- and full-day short courses (aka continuing education courses) on such topics as cognitive interviewing, multilevel modeling of complex survey data, the use of paradata to inform responsive design decisions, Bayesian analysis, total error in project survey management, etc.^[4] JPSM regularly offers 1- and 2-day short courses on topics ranging from data confidentiality to writing questions for surveys, to introduction to the federal statistical system, to construction of price indices (see https://projects.isr.umich.edu/jpsm/index.cfm). Similarly, the Southampton Statistical Science Research Institute in the United Kingdom offers a series of Courses in Applied Social Surveys (CASS) ranging in length from one day to one week, covering topics from data linkage to structural equation modeling to meta-analysis of rare events (see http://www.southampton.ac.uk/s3ri/cpd/courses.page). A major strength of these conference- and academic-based short courses is that they are offered by leading survey methods researchers and practitioners, and many offer certificates of completion that may be used to meet work-related professional development requirements.

One limitation of the current survey methodology short course offerings is that most can only be taken in-person. Webinars offer a more flexible format for survey professionals to refresh their methodological training and gain exposure to new concepts and techniques. These generally are shorter than short courses, lasting up to an hour of online presentation and lecture on a defined topic, and they may be live or recorded. In 2014, AAPOR offered 14 webinars ( http://www.aapor.org/source/education/webinar_recordings.cfm#.VFfH5slC7II), on topics such as the use of incentives in survey research, keeping online audiences engaged, or web survey usability testing. Other professional organizations, such as the Survey Research Methods Section of the American Statistical Association, or regional chapters of AAPOR, also offer webinars to their members. Some short courses and webinars are free, but some require a fee. Some issue a certificate of completion, but others do not. Webinars rarely, if ever, provide such credentials. The content and format of these two training vehicles shed some light on the hot topics in survey methodology, and give the reader a sense of where these educational opportunities can be sought.

Emerging Training Formats

New educational formats will emerge as web-based technologies evolve and our expectations about training change. One of the greatest training innovations of the past few years is the massive open online course (MOOC). MOOCs are typically offered for free, or for very low fee, by accredited universities through websites that aggregate courses across universities (e.g., www.coursera.org or www.edx.org). They typically involve online lectures, student discussion sections via online boards, and traditional homework assignments. Enrollment can run into the tens of thousands, although the number of students who regularly submit assignments and receive a certificate of completion usually is much lower (i.e., in single to low double digits).

In 2014, Frederick Conrad (University of Michigan) and Frauke Kreuter (University of Maryland) co-taught the first known survey methodology MOOC titled “Questionnaire Design for Social Surveys” through www.coursera.org.^[5] The course was offered twice, and there were about 18,000 students registered in each session, representing more than 100 countries (with approximately 33% of students from North America, 25% from Europe, and 40% from developing countries). The class consisted of weekly online lecture videos (about 10 minutes each), which were usually viewed by about a third of students. The videos contained one or two integrated quiz questions, and students were assigned peer-reviewed homework projects. (The Coursera platform sends one student’s homework to another randomly chosen student in the class.) The very diverse international audience produced interesting cross-cultural dynamics. For example, an assignments on sensitive questions produced lively discussions revealing vast differences in perceptions about what is considered sensitive in the various cultures represented.

This experience demonstrates some of the new opportunities and challenges of transitioning existing training protocols into digital environments. We see enormous promise in this approach and expect offerings in survey methodology to be expanded to cover other topics. The world of online education evolves very quickly, and the reader of this article is advised to check with the content providers such as JPSM or MOOC platforms such as Coursera to assess currently-available offerings.

New Programs Within Old Programs

New training directions are being explored within existing survey methodology programs. An initiative on Survey Informatics at the SRAM program at the University of Nebraska, Lincoln is being spearheaded by Allan McCutcheon. It seeks to establish an interdisciplinary specialization across four science, technology, engineering and mathematics (STEM) disciplines: 1) survey research and methodology, 2) computer science and engineering, 3) statistics, and 4) mathematics. The specialization will involve faculty from these departments, as well as industry partners. The program will produce survey specialists with unique expertise in technological foundations of survey data collection and analysis of survey data collected online. While the proposal is still under review at the time of this writing, we are extremely excited about this development, as it reflects a move to integrate survey methodology with the burgeoning field of data science.

Although not being actively pursued by any survey methodology program yet, there is a new type of graduate degree that we may see in survey methodology in the future. The Professional Science Master’s (PSM) degree ( http://www.sciencemasters.com/) differs from traditional master’s programs by splitting its requirements between content classes and business administration/leadership classes. For instance, the Master of Business and Science (MBS) program in statistics and biostatistics at Rutgers University,^[6] the largest provider of the PSM degrees, requires 19 credit hours in business and 24 credit hours in the science component of the program (statistics). The business classes include courses like finance and accounting, marketing for decision making, marketing for science and technology, principles of communication and professional development, and ethics for science and technology management. The courses in statistics include required courses in basic probability, methods of inference, design of experiments and regression analysis, and additional electives in statistics and biostatistics, including a course in survey sampling and acceptance sampling theory aimed at industrial quality control. Additionally, the program requires a minimum of 130 hours of internship in a commercial setting under the guidance of a professional mentor. Although there are no survey methodology PSM/MBS programs at the time of this writing, the (PSM) coursework balance offers exactly what other articles in this issue stress: a combination of training in technical and methodological skills with training in operational aspects of our field. This combination of academics and experience creates graduates that employers find irresistible.

Conclusions

Conducting high-quality sampling and data collection requires that training programs remain contemporary as well as cover fundaments. All surveys face challenges ranging from technical and operational (e.g., quickly-changing phone number frames; variety of platforms for web surveys, including mobile devices; distributed big data systems or human resources databases to pull establishment data from) to social and cognitive (e.g., falling response rates, multilingual, and multicultural populations), to mathematical and methodological (e.g., nonprobability samples, small area estimation problems, and statistical disclosure control). A successful survey methodology researcher needs to have familiarity with multiple facets of the discipline. As society and technology change, our methods of data collection change in response. The content and format of future survey methodology training needs to keep up with these changes. While traditional survey methodology training provides the best base for the skillset, such training needs to be complemented by practical expertise and additional professional training in the current topics provided via short courses, MOOCs and other complementary educational formats.

Even though traditional academia remains the major source of newly-trained survey methodologists, that number is not sufficient to fully cover the field’s existing needs. The discipline of survey methodology, including its industry and government branches, should continue devising new ways to promote awareness of the need for survey methodologists and additional ways to train them. The traditional route has been top down, where survey organizations or industry partners reach out to university administrators and funding agencies to set up survey methodology programs, such as the Joint Program in Survey Methodology.^[7] In addition, we can generate demand for survey methodology and statistics courses in colleges if we, as ambassadors of the survey research discipline, can find credible ways to reach out to undergraduate and master students across the nation to make them aware of the exciting career possibilities in survey methodology.

Professional education formats (e.g., short courses and webinars) and online education (e.g., MOOCs) offer viable training options for working survey professionals who need survey methodology training but who do not have the time for graduate training. As we stressed throughout these articles, employers are interested in tangible survey skills and obtaining these from complementary training formats is a good starting point. We pose two questions for academic survey research and methodology programs to ponder. First, can survey methodology training draw any tips from applied fields like medicine, teaching, or clinical psychology that have large practice components as a formal part of their training (e.g., internships and rotations)? Second, can short courses, MOOCs and other flexible training modes be integrated with established academic venues to serve as testing grounds for new methodology courses and can those courses be adopted by academic programs later when shown to be popular and fill a training gap?

We began this special issue of Survey Practice by saying: “The news is good for students of survey methodology.” We close it by saying that the future looks brighter, as working methodologists have a variety of formats by which to expand their training. Survey organizations should reasonably expect to see these formats on applicants’ resumes and promote them as a form of professional training for their existing workforce.

Acknowledgment

Stas Kolenikov wishes to acknowledge and thank Abt SRBI for supporting and encouraging his participation in and contributions to this special issue.

Disclaimer

Any views expressed are those of the authors and do not reflect those of Abt SRBI, the U.S. Census Bureau or the U.S. Bureau of Labor Statistics.