Training for the Modern Survey Statistician

Introduction

This paper describes what, in our view, a modern survey statistician must know to be effective in research, study design and survey analyses positions, be that in academia, industry or government. Note that the boundaries between these fields are quite permeable, as many survey researchers migrate between these sectors throughout their careers. As discussed in the introductory article of this issue of Survey Practice,^[1] the field of survey methodology and survey research in general draws heavily on statistical science. In a typical training setting, a survey statistician is receiving his or her training in a “regular” statistics department or in one of several survey methodology programs. We believe that a person with the skillset that we are outlining below is easily employable in survey research organizations, including government agencies, survey research companies and academic survey research centers. A search on “survey statistician” at www.usajobs.gov consistently produces more than a dozen openings at any given time, and given that most positions are open for a month or two, there’s a demand for at least 50 survey statisticians across the Federal Statistical System alone; private industry creates at least as much demand. We provide some pointers for other tracks into careers in survey statistics, including researchers coming from other, more social science oriented, branches of survey methodology, as well as the more mainstream statisticians looking for a career in survey statistics. Survey statisticians in training may find it worthwhile to check how their toolbox compares to our “ideal” survey statistician, and seek opportunities for complementary training.

As compared to the mainstream statistics training, a survey statistician needs to learn yet another inferential paradigm. The traditional frequentist approach takes off the launching pad by saying, “Let’s assume an independent and identically distributed (i.i.d.) normal distribution with mean μ and variance σ²”, and the Bayesian approach chimes in with “Let’s add a conjugate prior on top of that normal distribution.” Complementary to these, design-based survey statistics operates primarily with the finite population sampling paradigm. Unlike both the frequentist paradigm and the Bayesian paradigm that mostly tend to deal with continuous and/or exponential family distributions in Euclidean spaces, design-based inference relies on discrete probability and sample inclusion indicators, i.e., the space of the possible samples that can come out of a given sampling design. Understanding inference with respect to the sampling distribution that is driven by the sampling design, and taking the values of response variables as fixed quantities, requires a certain switch in the mindset that may not be easily achievable with a single course in sampling.

Besides the basic understanding of the finite population inference in sampling statistics, survey statisticians often need to work on sampling designs and inform the project directors and principal investigators about the trade-offs that the different design options would achieve. Such trade-offs often include cost versus precision trade-off, choice of modes, and sometimes a multi-way trade-offs between cost, variance and coverage of the population of interest (i.e., bias). While some statistics departments offer courses in decision theory as a part of statistical theory sequences, and the issue of study design comes up in biostatistics in the form of power analysis and sample size calculations, the issues of optimizing the multiple features of a complex design to balance the cost with precision and fitness for purpose seem to be more typical for survey statistics.

Within a survey organization, a survey statistician needs to have a stronger grasp of statistics and quantitative methods than a survey methodologist who works with the cognitive and behavioral issues of survey response, or with substantive areas such as polling or public health. In smaller organizations, survey statisticians may find themselves to be the quantitative resource, and they may be expected to consult internally on other areas of statistics such as regression or multivariate analysis. Knowledge of these statistical methods is usually achieved in either the mainstream statistics departments, or on the quantitative tracks of social science departments.

Besides this methodological knowledge, survey statisticians need to have applied survey statistics skills to be productive in their survey organizations. To translate methodological knowledge of sampling statistics into data products, survey statisticians need to know how to work with frames from which their samples are (to be) drawn (which includes finding existing sampling frames or developing new ones, cleaning frames, and effectively utilizing frame information to optimize the study budgets achieving the maximum precision and accuracy), and how to weight the final data.

Below, we present the core competencies in survey statistics as progressing from the minimal qualifications that any survey researcher must possess to the top quality, doctorate-level skills. For an additional perspective on training survey statisticians, see the Morris Hansen lecture by Sharon Lohr.^[2]

Core Statistical Science Survey Competencies for All Survey Methodologists

The core statistical competencies are the skills necessary for survey methodologists of any flavor to successfully design, implement, and evaluate basic sampling schemes and basic built-in sample experiments. Additionally, survey researchers may need to contribute to scientific reports should they be sole methodological experts in a given project. Finally, this set of core competencies will be necessary to read and comprehend modern literature on survey methodology.

• Foundations of statistical methods
  • Basics of (frequentist) estimation and hypothesis testing
  • Linear regression model: estimation, testing, diagnostics
  • Logistic regression model: estimation, testing, diagnostics
• Foundations of sampling
  • Notions of simple random sampling (SRS), stratified and cluster sampling
• Foundations of complex survey data analysis
  • Modifications of the standard procedures for complex survey data
  • Concept of design effect and variance estimation
• Frames typically available for the general population samples
  • Nature of nonresponse on different frames
  • Coverage properties of different frames
  • Natural modes for the existing frames

We expect that the first set of items will be covered by general statistics education. On the other hand, we found that the last set of items constitutes a training gap, and the discipline of survey methodology as a whole will benefit from better documentation and training in the issues of sampling frames. For the middle two points, we can provide a bookshelf of critical resources. These books are often used by instructors of courses in sampling, analysis of survey data, survey methodology and total survey error, or for self-study by statisticians catching up with survey methodology. Finally, having these books at arm’s length in the office will prove handy when references to specific formulae, statistical software code, or methods reference will be necessary as a part of a paper, proposal or methods report. The list is in no way exhaustive, but having books of this kind in personal professional library, and knowing what one can find in these books, will be helpful for the early career survey methodologists.

• Bookshelf (examples of critical resources)
  • Lohr, S.L. 2009. Sampling: design and analysis (2nd ed.). Duxbury Press, Pacific Grove, CA.
  • Heeringa, S., B.T. West and P.A. Berglund. 2010. Applied survey data analysis, Chapman & Hall/CRC, Boca Raton, FL.
  • Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer and R. Tourangeau. 2009. Survey methodology (2nd ed.). Wiley.
  • Statistics Canada. 2003. Survey methods and practices. Catalogue no. 12-587-XPE. Ottawa, Canada.
  • Kish, L. 1965. Survey sampling. Wiley Classics. Wiley, New York.

Statistical Science Survey Competencies: Master’s Level

For a survey methodologist to qualify as a quantitative specialist, capable of producing cost-and-precision-effective sampling designs and weights that minimize noncoverage and nonresponse biases while maintaining precision of the target statistics of their studies, their list of skills needs to be substantially wider than the generic ones outlined above. Thinking of the daily jobs of (entry, master-level) survey statisticians, the following skills are universally necessary across the survey organizations in a typical position that is often titled “survey statistician” or “sampling statistician”:

• Foundations of statistical inference
• Sampling techniques
  • (Stratified) SRS sampling as a building block
  • Cluster sampling: area samples; household samples
  • Probability proportional to size (PPS) sampling: basic understanding of the complications of unequal probability sampling
  • Systematic sampling: uses and limitations
  • Sample size calculations and power analysis
• Analysis of complex survey data
  • Familiarity with at least one major package with support of complex survey data features (probability weights, stratification, clustering, finite population corrections, replicate weights, calibrated weights): R, Stata, SAS, SUDAAN. Note that SPSS does not fully qualify as it does not support replicate weights, while support for calibrated weights is limited in SAS and Stata.
  • Variance estimation by linearization and by replicate, or resampling, methods: basic application
  • Construction and role of weights
  • Design effects
  • Analysis of subpopulations
  • Implications of complex survey designs for typical analyses (tabulation, regression, generalized linear models): Basic notions and application
• Frames, their properties and uses
  • Relations between frames and population
  • Frame undercoverage and overcoverage
  • Area sampling frames: U.S. Census geography; unit listing; HOI, CHUM, and other coverage enhancement procedures
  • Random digit dialing (RDD) sampling frames:
    • Landline frame: 100-blocks, 1+blocks, Mitofsky-Waksberg procedures
    • Cellphone frame: 1000-blocks, rate centers, billing zip codes, activity flags
    • Dual-frame sampling methodologies which incorporate landline and cellphone frames
  • Address-based sampling (ABS) based on U.S. Postal Service (USPS) information
  • Establishment frames: e.g., Dunn & Bradstreet
  • Interplay between frames and modes; transfers from a natural mode to a different mode (e.g., mail-to-web push) and breakoffs associated with these transfers
• Weighting
  • Steps in weighting
  • Basics of frame integration
  • Raking/weight calibration
    • Basic concepts
    • Implementation in the statistical package of choice
    • Sources of the calibration targets: ACS, NHIS, Census Summary Files
• Interplay of statistics and social sciences in applied survey research
  • Nonresponse analysis and nonresponse adjustments
  • Response propensity modeling
• Statistical programming and computer proficiencies
  • Ability to write scripts in at least one major package (R, Stata, SAS, SUDAAN) to automate complex operations
  • Code documentation
  • Cleaning and restructuring data
  • Version control
• Communication and presentation skills
  • Data visualization
  • Explaining statistical concepts and findings to non-statistical audiences
  • Proposal and grant writing skills
• Bookshelf (examples of critical resources)
  • All of the earlier suggestions
  • Korn, E.L. and B.I. Graubard. 1999. Analysis of health surveys. John Wiley and Sons, New York.
  • At least one of:
    • Kish, L. 1965. Survey sampling. John Wiley and Sons, New York.
    • Cochran, W.G. 1977. Sampling techniques. John Wiley & Sons, New York.
    • Levy, P.S. and S. Lemeshow, 2003. Sampling of populations: methods and applications (3rd ed.). John Wiley & Sons, New York.
  • Long, J.S. 2008. The workflow of data analysis using Stata. Stata Press, College Station, TX.

All frames listed above are specific for the United States. Other countries may have richer or poorer frames depending primarily on how the national statistical agencies operate. For instance, unlike the United States, some European countries have population registers that make it possible to take balanced samples with respect to demographic variables. Also unlike the United State, the legislative environment of other countries can make it possible to conduct cell phone surveys differently, e.g., by using predictive dialers or by unsolicited text/short message service (SMS) messages.

Sampling frames, their properties and coverage is what will probably change the fastest over time, among the proficiencies listed. Fifteen years ago, we did not have to dial cell phones, and there was little loss of coverage, due to low single percentage of cell only population, in not doing so. Ten years ago, we had yet to hear of address-based sampling from the USPS Delivery Sequence File for general population surveys. Five years ago, the RDD frames did not have the cell phone flags from the cell sample providers with which to subsample or stratify. Maybe we should expect yet another new frame to appear and become operational by 2020 or 2025.

Statistical Science Survey Competencies: Doctorate Level

Doctorate training in survey statistics is a rare luxury. Most PhD graduates are being produced by the major survey methodology programs: the Joint Program in Survey Methodology at the University of Maryland ( http://www.jpsm.umd.edu/), the Michigan Program in Survey Methodology at the University of Michigan ( http://psm.isr.umich.edu), and the Research and Methodology Program at the University of Nebraska Lincoln ( http://sram.unl.edu). Several statistics programs produce doctorate level survey statisticians, including Iowa State University (five survey statistics faculty), Colorado State University (two survey statistics faculty), University of Waterloo (three survey statistics faculty), and a scattering of isolated survey statisticians who work in departments of statistics, biostatistics or mathematics. Generalizations from this sample size of less than a dozen are somewhat difficult. At the risk of erring on the side of including too many ideas, we think that the strongest doctorate level researchers will possess all of the masters-level competencies, plus a selection of:

• Advanced statistical inference
  • Asymptotic theory
  • Bayesian methods
  • Sensitivity analysis and diagnostics for main methods (linear and logistic regression)
• Thorough understanding of survey statistics
  • Design-based inference, Horvitz-Thompson, and Yates-Grundy-Sen estimators and their formal justification
  • Model-based inference
  • Variance estimation by linearization and by replicate methods: properties, limits of applicability
  • Calibrated estimators, variance estimation with calibrated data
  • Subtle points of variance estimation: degrees of freedom, domains/subpopulations, single PSU/stratum
  • Implications of complex survey designs for typical analyses (tabulation, regression): methodological foundations
  • Implications of complex survey designs for advanced analyses: multilevel models, survival models, structural equation models, etc.; interplay of the study designs, estimation methods and inferential goals
  • Statistical disclosure control
• Sampling
  • Power and sample size calculations
  • Optimization by analytical and numeric tools
  • Rare populations
  • Longitudinal studies
  • Nonstandard schemes (prediction theory, restrictive, balanced and cube sampling, network samples, indirect sampling, program evaluation research design, and samples, etc.)
• Advanced quantitative techniques to support research in survey methodology
  • Missing data: main concepts and imputation techniques
  • Mixed and multilevel models (applications: interviewer effects, small area estimation)
  • Latent variable models and/or psychometric measurement theory (applications: questionnaire design and instrument testing)
  • Small area estimation
  • Geographic Information Systems (GIS; applications: ABS designs, transportation studies)
  • Data science and machine learning (applications: record matching, propensity modeling)
• Frames
  • Frame integration techniques
  • Quantifying undercoverage and overcoverage
• Statistical programming
  • Ability to develop moderate to complex programs in at least one major package (R, Stata, SAS, SUDAAN)
  • Recyclable code: SAS macros, R functions, and Stata programs
  • Use of programming industry techniques (version control, project management tools, automated and semi-automated reproducible report generation with tools like RWeave)
  • Monte Carlo simulations
• Communication and presentation skills
  • Evidence of successful presentations at major conferences
  • Evidence of successful statistical consulting to internal and external clients through a survey research center in the home university
  • Soliciting and addressing clients’ needs
  • Proposal grant writing skills
• Bookshelf (examples of critical resources)
  • Valliant, R., J.A. Dever and F. Kreuter. 2013. Practical tools for designing and weighting survey samples. Springer, New York.
  • Lumley, T.S. 2010. Complex surveys: a guide to analysis using R. Wiley, Hoboken, NJ.
  • One or more of survey statistics theory books
    • Särndal, C.-E., B. Swensson and J. Wretman. 2003. Model assisted survey sampling. Springer, New York.
    • Thompson, M.E. 1997. Theory of sample surveys (Vol. 7 ). Monographs on statistics and applied probability. Chapman & Hall/CRC, New York.
    • Fuller, W.A. 2009. Sampling statistics. Wiley, Hoboken, NJ.
  • Additional sampling books:
    • Hansen, M.H., W.N. Hurwitz and W.G. Madow. 1953. Sample survey methods and theory. Wiley Classics. Wiley, New York.
    • Thompson, S.K. 1992. Sampling. Wiley, New York.
    • Lehtonen, R. and E. Pahkinen. 2004. Practical methods for design and analysis of complex surveys (2nd ed.). Wiley, Hoboken, NJ.
  • Edited volumes:
    • Skinner, C.J., D. Holt and T.M. Smith (eds.). 1989. Analysis of complex surveys. Wiley, New York.
    • Chambers, R.L. and C.J. Skinner (eds.). 2003. Analysis of survey data. Wiley, New York.
    • Pfeffermann, D. and C.R. Rao (eds.). Handbook of statistics. Sample surveys. (Vol. 29A) Design, methods and applications; (Vol. 29B) Inference and analysis. North Holland, Amsterdam, The Netherlands.
  • Specialized books, per needs in one’s research or application area:
    • Rao, J.N.K. 2003. Small area estimation. John Wiley and Sons, New York.
    • Wolter, K.M. 2007. Introduction to variance estimation (2nd ed.). Springer, New York.
    • Särndal, C.E. and S. Lundström. 2005. Estimation in surveys with nonresponse. Wiley, Hoboken, NJ.
    • Cox, B.G., D.A. Binder, B.N. Chinnappa, A. Christianson, M. J. Colledge and P.S. Kott (eds.). 1995. Business survey methods. Wiley, New York.
    • Valliant, R., Dorfman, A.H. and R.M. Royall. 2000. Finite population sampling and inference: a prediction approach. Wiley, New York.
    • Hundepool, A., J. Domingo-Ferrer, L. Franconi, S. Giessing, E.S. Nordholt, K. Spicer, and P.P. de Wolf. 2012. Statistical disclosure control. Wiley, Hoboken, NJ.
  • Resources on the advanced quantitative techniques of one’s specialization (e.g., multilevel modeling, machine learning, longitudinal data, etc.)

Core Social Science Survey Competencies for Statistical Science Survey Methodologists

To be effective in survey organizations, survey statisticians need to have the basic understanding of the context of survey operations.

• Data collection methods
• Modes of data collection – pros, cons, and the (changing) interaction effects with type of data collected and type of sample units (e.g., sensitive questions are sometimes best collected through a noninterviewer mode such as ACASI; long lists of response categories cannot be administered on the phone; web surveys implicitly assume access to the Internet which is not yet universal in the general population; lower literacy population groups require presence of an interviewer; including all the questions necessary for weighting, such as demographic and phone use, in the instruments; etc.)
• Psychology of survey response
• Foundations of questionnaire design
• Response rate calculations, American Association for Public Opinion Research (AAPOR) dispositions and standard definitions
• Total survey error

While these topics can be learned on the job, systematic exposure to them in courses and practica in survey methodology will be very helpful and will likely help students develop these skills faster.

Training Gaps

We see several gaps between training and practice that survey statisticians need to overcome to become proficient and employable. The areas where the practice of survey data collection has been moving faster than academic research and textbooks are inevitably at risk to become such training gaps. These are especially important for the self-studying, early-to-mid-career statistician considering a career shift into survey statistics.

Training Gap 1: Survey statistics in the general statistics curricula

The “typical” graduate programs in statistics does not provide sufficient training for their graduates to be effective in survey settings. In most programs, survey methodology related training is confined to a single course on sampling, which is often taught as an upper undergraduate or applied graduate course from a book like Lohr (2009) or Levy and Lemeshow (2003). As our prior discussion suggests, a survey statistician needs to know much, much more. Extensive additional specialized training will be required for nearly all statisticians coming from these traditional (bio)statistical programs.

Training Gap 2: Sampling frames

Even when a statistics student does receive more advanced coursework in survey statistics, one area where we expect the training to be lagging behind the practice is that of sampling frames: their construction, usage, manipulation and integration. The textbook setting is typically that of a list sample with known population total. In reality, such unicorns do not exist. The existing lists inevitably need to be cleaned and supplemented, and if they do not exist, lists must be constructed, as in area sampling, via field enumeration that itself is subject to error. Error properties of sampling frames are often studied by a survey statistician in collaboration with the field managers, cognitive scientists, and technical experts in the area or discipline on which the frame is based (in the case of areal samples, geographers and Geographic Information System (GIS) specialists; in the case of school surveys, education researchers and administrators; etc.). In RDD phone sampling, population totals are really murky, as frames contain large proportions of ineligible elements. Survey statisticians need to work closely with project managers to be aware of the adjustments in the samples, calling protocols and other aspects of data collection to track subtle changes in probability of selection that may need to be reflected in weighting. One of the articles in this special issue,^[3] discusses collaborative work environments of survey methodologists in more detail.

Finally, improvements in survey coverage are often associated with using multiple frames (such as the landline and cell phone frames in phone surveys). Hence, modern survey statisticians need to be familiar with the procedures of frame integration and have an understanding of the advantages and disadvantages of the various frame integration methods and approaches.

Training Gap 3: Weighting

Another area where training gaps exist is that of weighting survey data. Most textbooks discuss the basic elements of the weights calculated as the inverse probabilities of selection and used in the Horvitz-Thompson estimator. These weights would work fine for a frame of perfect coverage and a sample with complete response. However, in the current practice of weighting, the inverse selection probabilities are but the first step in weighting, sometimes the easiest of all the steps. For the survey to be representative of the target population and the weighted estimates to be sufficiently accurate, much more work is often needed. The full description of the weighting protocols in the most complex studies may take many pages to capture several steps in the weight construction to correct for data collection issues like eligibility, frame overlap, and nonresponse. The final steps of weight construction usually involve weight calibration, and survey statisticians need to be familiar with both the methodology and the applied aspects that range from trimming to looking for the right source of the population totals.

Training Gap 4: Design trade-offs and multicriteria optimization

For most practical studies, there is usually a variety of survey designs that may be applicable. Nearly every general population survey can conceptually be conducted by a real face-to-face multistage survey, a phone survey, or a mail survey, with either of these having an option to complete the instrument in the original contact mode, or in a less expensive mode such as the web. These modes have vastly different costs, as well as notably different response propensity properties that may imply different degree of weighting required, and thus the ultimate accuracy and precision properties of the design. Besides these big picture choices, within a given class of designs, there may be continuous options, such as sample sizes at each level in multistage samples, the number of and allocations between strata (a common theme in studies of rare populations when preliminary information on the target population prevalence in the general population is available), and allocations between frames in multiframe designs (such as allocations to the landline and cell-phone samples in the dual frame RDD designs). A solid background in both discrete and continuous optimization methods will be an important asset for a survey statistician. In academic settings, courses in optimization theory and applications are typically located outside the home departments of statistics or survey methodology programs in departments such as applied mathematics, computer science, and less frequently, in pure mathematics. Such courses, in turn, require calculus, and sometimes discrete mathematics and linear algebra, as prerequisites.

How to Overcome These Gaps

While the list of (mostly mathematical) sampling books may easily run over a couple dozen titles, the books where steps in frame construction and issues of weighting are touched upon are much scarcer. Valliant, Dever and Kreuter (2013) devote a good fraction of their hefty volume to weighting, and Statistics Canada (2003) manual, as well as Johnson’s (2014) edited handbook^[4] have chapters with some coverage of frames and weighting as well.

Other sources that interested trainees can turn to would include resources provided through the professional societies. Condensed knowledge on practical topics, including frames and weighting, is often presented in short courses and webinars, which are discussed in more detail in the final article of this issue.^[5] Many valuable observations, however, do not make it beyond AAPOR or Joint Statistical Meeting presentations, making it necessary for survey statisticians to attend these conferences in order to keep themselves up to date with the existing techniques and methods.

Discussion

Arguably, the lists of skills presented above are quite ambitious. To some extent, they reflect the trade-offs that most programs have in picking the topics and courses: does the academic training need to generate strong generalists, or specialized hyperfocused researchers, or something in-between? If each major bullet point corresponds roughly to one full-semester graduate course, then covering all masters-level competencies will easily populate a typical 2-year program requiring 30–40 credit hours. In reality, few if any programs offer classes in all of these areas. The implication for the statisticians-in-training is that they should seek complementary training formats and internship opportunities in survey organizations to get a glimpse of these aspects of survey profession. The implications for employers in survey organizations are that, unfortunately, our “ideal” survey statisticians do not exist, and hence, survey organizations should reasonably be prepared to lower the bar somewhat in terms of the entry-level positions requirements and to spend time and resources providing supplementary training in these positions.

While doctorate training is nominally longer and is expected to run for approximately five years, the course sequencing might be even tighter for doctorate-level training. Most doctorate students are expected to take most if not all of their courses in the first two years of the program, concentrating on research after that. Yet our “desirable” doctorate graduate in survey statistics is expected to pick up at least another five courses to become proficient in a variety of additional areas. Some of these courses may be relatively esoteric and not easily found within the existing offerings. For example, programming techniques may be considered too basic to be offered by a computer science department at a graduate level, and classes on highly specialized quantitative techniques like multilevel or structural equation models may not be offered at all because of the lack of expertise among the faculty; or, where offered, may not be considered rigorous enough to satisfy requirements of a statistics program. While we believe that the additional expertise that could be accumulated through these courses will eventually be beneficial to the researcher, we cannot recommend these classes over the required dissertation research.

We reasonably expect that the competencies we have outlined have a certain window of time before they expire, and new sets of skills may be necessary. The survey world is a rapidly changing one, and new technologies bring new frames and new modes every few years, with older technologies becoming obsolete and barely usable. For instance, no survey organization can rely solely on landline phone samples to provide an acceptable degree of coverage since early 2010s, and only dual frame RDD samples can be expected to reach the majority of population. Likewise, new statistical techniques are constantly being developed, and new computing platforms and paradigms change how we collect and access the data. We discuss some of these issues in the closing article of this issue.^[6]

Disclaimer

The author would like to acknowledge suggestions, contributions, and discussions with Jill Dever, Graham Kalton, Barbara O’Hare, and Matt Jans. He also wishes to acknowledge and thank Abt SRBI for supporting and encouraging his participation in and contributions to this special issue. Any views expressed are those of the author and do not reflect those of Abt SRBI.