Introduction

When undertaking an extensive questionnaire (re)design, it is typical to plan multiple rounds of large-scale qualitative testing. For example, a current project at the US Census Bureau to redevelop the Annual Social and Economic Supplement (ASEC) of the Current Population Survey for web administration includes four rounds of large qualitative tests over four years: exploratory interviews, two rounds of cognitive testing, and usability testing. This effort is executed through a contract (which has inherent restrictions of timeline, cost, and scope) and is further constrained by target dates for the field test and live implementation. This rigidity can be problematic when test results seem inconclusive. If more cognitive testing is desired, adding time and money for an extra round might be impossible, particularly in the late phases of a project. This makes each round critical, increasing pressure to solve every problem within the limited number of rounds. MicroTesting was developed to introduce some flexibility into this type of large-scale testing, allowing for additional research without adjusting the overall plan. By conducting MicroTesting as needed, researchers can feel greater confidence in, and raise the quality of, the larger rounds of testing.

What is MicroTesting?

MicroTesting is a short, focused qualitative test (e.g., exploratory interviews, cognitive testing, usability, etc.). Each round includes six to ten people, and interviews take approximately twenty minutes. Respondents interact with a short instrument, often a segment of a larger questionnaire, that targets a few specific research questions. Recruitment can be simplified to include only the people most relevant to those research questions.

Some of the benefits of MicroTesting include:

  • Offers a nimble approach to address problems quickly, without having to wait for the next round of large-scale testing;

  • Reduces the pressure of having to force a decision prior to the next round of testing, or cramming too much into a round;

  • Allows A/B testing so that the larger round can refine a single option; and

  • Gives the methodologist a tool to test a recommendation when applicable literature is not available.

If using a contract for the large-scale testing, MicroTesting will most likely be conducted outside of that contract (i.e., in-house). This way, it can be implemented precisely when it is needed and concluded when the desired feedback has been obtained. This agility provides efficiency, both in terms of time and cost.

MicroTesting Applications

  1. The ASEC team successfully implemented MicroTesting on a subset of the questionnaire asking about health insurance. Of particular concern was an early filter question that determined what types of insurance were in the household. Refining the wording of the response options was proving difficult. MicroTesting was employed, focusing on a few subgroups aligned with certain response options. For example, people 65 and older were interviewed to determine what keywords must be shown to help them choose Medicare as their insurance, even if they purchase supplemental plans. Also, teachers were recruited to help further refine the response option mentioning insurance through a “school,” which was meant to capture university students, not teachers. For this test, MicroTesting was concluded after eight interviews, over ten days.

  2. Prior to the invention of MicroTesting, exploratory interviews were conducted concerning “health insurance” and related terms which would be used repeatedly in the ASEC questionnaire. This early testing found that respondents did not uniformly understand: “health insurance,” “health plan,” and “health coverage.” To some people, two or more of these phrases were synonymous; to others, some or all phrases were different. Misunderstanding of the terms was evident throughout the test results. While the report provided enough information to choose a single term to take forward, MicroTesting could have been conducted to provide reassurance that this key term was the best option for all respondents.

  3. An innovative application of MicroTesting could involve usability testing question formats in the early phases of a project. For example, the question designers could have several ideas as to how to format roster questions asking for relationships between household members. The options can include a series of questions, a sequence of grids, or a mix of the two. To narrow down what should be presented in cognitive testing, different formats could be MicroTested with respondents of various household compositions. This would allow the cognitive test to focus more time on the questionnaire, rather than the roster.

Variations of MicroTesting Protocols

The methodology is adaptable to the problem(s) being addressed. Researchers can adjust the timing and number of rounds of MicroTesting to suit their goals. For example, testing can be conducted:

  • As a single round of testing on a small section of the survey.

  • Iteratively, conducting multiple rounds on the same section of the survey, editing the questions between each round.

  • Concurrently, executing two or more rounds of testing at the same time but on separate portions of the instrument with unique recruitment quotas for each test.

MicroTesting does not always have to be implemented between rounds of the larger tests. There may be situations where testing occurs alongside a larger test, such as the need to test a few survey questions on a specific hard-to-reach population that is not part of the recruitment quotas.

Practicalities

As explained, MicroTesting is meant to complement a large-scale qualitative test, not replace it. In addition, it does require resources (particularly people) that may have not been designated at the start of the project, or in the contract. To prevent disappointment that MicroTesting cannot be implemented due to resource strain, researchers can plan ahead, designating points in the timeline where MicroTesting could be implemented, if needed.

Although the MicroTests are small, all of the processes to conduct a qualitative test of high quality must be in place. A researcher may need to program a web instrument, make arrangements for a small incentive, and/or brief potential observers on neutral probes when questioning respondents. They certainly must design a protocol, execute recruitment, document respondents’ consent, conduct interviews, and produce a memo on the findings. Since the scale was small and involved a senior methodologist, these steps were executed quickly at the Census Bureau. In addition, recruitment was simplified by inviting a subset of Census Bureau Research Panel members with the desired characteristics to participate.

For researchers conducting federal studies, the Office of Management and Budget (OMB) is also part of this process. As the testing is likely to include more than nine people (and planning for more cases would provide essential flexibility), MicroTesting and the estimated time burden must be accounted for in the OMB clearance.

Conclusion

MicroTesting as a research tool proved to be an asset during ASEC’s qualitative testing. With this technique, much progress was made on some of the most difficult questions. The internal clients involved in MicroTesting were particularly pleased with the detail they could gain from focusing on such a small section of the instrument. In addition, they were brought closer to the instrument, respondents, and overall testing process. Respondents were highly engaged, and given the short time commitment, remained motivated for the duration of the test. The US Census Bureau will continue to incorporate this technique into testing and encourages others to explore its value for their research.

Corresponding author contact information

Jennifer Sinibaldi
US Census Bureau
4600 Silver Hill Road
Washington, DC 20233
jennifer.sinibaldi@census.gov