Using a Seeded Sample to Measure Response among Homeschooling Households

Estimates of the size of the homeschooling population derived from government-sponsored surveys have been questioned because of concerns that homeschooling families may be less willing to participate in government studies. To estimate whether the response propensity of homeschooling households differs from that of non-homeschooling households, a sample of likely homeschooling households was seeded in a federal education survey. Cases in the seeded sample were treated to the same field procedures as the survey’s sample to allow comparisons of the two samples’ response patterns. While the seeded sample is an excellent method, in theory, for examining response propensity, the study revealed significant practical problems in implementing the study.

Introduction

The National Household Education Surveys Program (NHES), sponsored by the National Center for Education Statistics (NCES), provides one of the few sources of information on the homeschooling population in the United States. As such, the NHES data is important for tracking this segment of the student population. However, there are concerns about the willingness of some homeschooling households to participate in government-sponsored studies (Henke et al. 2000; Mayberry et al. 1995). Alternatively, some homeschooling households may respond at high rates to studies about family involvement in education, such as the NHES, because the survey topic is particularly salient. Topic saliency has been shown to increase survey participation (Groves, Presser, and Dipko 2004).

If homeschooling households are opting out, they are likely doing so before it can be established whether the household is homeschooling.^[1] To study this nonresponse, we compiled a list of likely homeschooling households and seeded the resulting sample in the 2007 NHES telephone sample. The NHES is representative of U.S. households, the vast majority of which are not homeschooling. By treating the cases in the seed sample with the same call and field procedures as the main random digit dialed (RDD) sample (blind administration), we sought to determine if there was a difference in the response propensity of homeschooling households and in which direction.

Data and Methods

The 2007 NHES was comprised of two parts – an eligibility screener and a main interview on family involvement in education. Households were eligible if there was a school-age child between the ages of 3–20 residing in the household. Homeschooling status was also established in the screener. The respondent for the main interview was the householder who knew the most about the sampled child’s education. Data were collected in spring 2007. The homeschooling seed sample yielded approximately 100 homeschooled students and the RDD approximately 300.

Constructing the homeschooling seed sample. The goals for the seed sample (referred to from here on as the “List sample”) were to find a list that would: 1) consist of households with school-aged children who were being homeschooled; 2) include phone numbers and mailing addresses; 3) be nationally representative and capture the diversity of homeschoolers; 4) be able to be delivered to NCES in October 2006; and 5) be administered blind to List sample households under the same survey conditions as the main study.

We explored a variety of sources that could contain known or likely homeschooling households, including State Education Agencies, homeschooling supply vendors, membership organizations, magazine subscription lists, and mailing list vendors. Records available through SEAs met most study criteria, but were rejected because of the high likelihood of sample bias: homeschooling households reluctant to participate in surveys sponsored by the government may also be less likely to comply with state homeschooling reporting requirements. Also, drawing a list from SEAs could have been time consuming, since it would be necessary to work with a number of states rather than one centralized point of contact.

While records available from homeschooling membership organizations were promising for the study, it became clear early on that it would be difficult, if not impossible, to gain access to the organizations’ databases. Several organizations explicitly noted on their websites that they do not provide their member lists to outside groups or required the member organization to contact their membership prior to permitting the use of their list in the NHES. Another potential problem with membership organizations is that members may differ from those who are nonmembers, again introducing a potential source of bias.

An online search for homeschooling magazines resulted in several hundred hits. However, we were stymied somewhat by the digital age: many of the sites identified were electronic newsletters, e-magazines, blogs, and listservs, each of which are not likely to have access to a large enough database of homeschoolers or contain the needed contact information. Subsequently, we were able to identify a list of printed magazines with national readership. Although this option showed promise, we did not pursue it further because homeschooling magazine subscriptions, along with lists from other sources, could be obtained through mailing list vendors.

We chose homeschooling mailing lists because they met most of the study goals. First, the mailing lists contained information that allowed us to make inferences about a household’s homeschooling status. Second, the mailing lists provided us access to a range of sources of homeschooling lists which could provide access to a diverse sample of homeschooling households. Third, the mailing lists were marketed as having updated and complete mailing addresses for a high percentage of records. Fourth, mailing lists did not require pre-study contact with households, thus allowing for blind administration to the List sample.

We also encountered problems with the mailing lists. Some owners of homeschooling lists refused to rent the list for use in the NHES because they did not believe their members would want to participate in the survey. As such, we were unable to rent some promising lists. To address this problem, we sought a homeschooling-affiliated endorsement to help gain list approval, but were refused. Time-constraints prevented us from pursuing additional endorsements. Also phone numbers were not always included in the record files or updated. This issue was addressed by performing a reverse directory search to locate the phone numbers of households in the List sample, which identified phone numbers in 60 percent of the records.

Another key goal in constructing the sample of likely homeschoolers was to represent a range of homeschooling households. To increase its diversity, the List sample was drawn from three separate mailing lists, each of which was marketed as being comprised of homeschooling households. The first list, referred to as the “Actives list”, was comprised of 170,000 households that had purchased homeschooling materials in the two years prior to the last update in June 2006. Past research suggests that approximately 70 percent of households that are currently homeschooling were homeschooling in the previous year^[2] By limiting the records to households that had made a purchase in the last two years, we presumably increased our likelihood of identifying active homeschooling households. The second list, referred to as the “Christian list”, included roughly 146,000 likely Christian homeschooling households. The households were identified through direct mail and point of sale records (e.g., purchase of Christian-based homeschooling materials). This was an older list. Newer records were last added in May 2000 and addresses were last compared to the National Change of Address (NCOA) registry in July 2006. Accordingly, an unknown percentage of households in this list were expected to no longer be homeschooling. The third list, referred to as the “General list”, included 118,000 household records compiled through direct mail and Internet inquires. This list was last updated in March 2004 and compared to the NCOA registry in December 2005.

A random sample of each of the lists was drawn. To increase the likelihood that the seeded List sample was comprised of households that were actively homeschooling, the sample was drawn primarily from the Actives list (70 percent). A total of about 2,400 cases were randomly selected from the sample frame and were seeded in the 2007 NHES.

Results

How well did the List sample meet the study objectives? Table 1 shows the extent to which the List sample provided access to homeschooling households with age eligible children. Roughly 2 in every 5 cases in the List sample included an age-eligible child. Among households with an age-eligible child, 29.7 percent were confirmed to be currently homeschooling a child. The overall hit rate – the percentage of all records in the List sample confirmed to have both an age eligible child and to be currently homeschooling – was 12.6 percent (42.3 percent X 29.7 percent).

Cooperation Rates. The cooperation rate of the List sample was roughly 14 points higher than that of the RDD sample at 75.9 percent vs. 61.9 percent. Because the overall hit rate for the List sample was low, we cannot infer that the higher cooperation rate observed in the List sample indicates that homeschooling households cooperate at higher rates than non-homeschooling households. However, by simulating the cooperation rates using varying assumptions from the percentage of refusal cases in the List sample that were homeschooling, we can provide some evidence that suggests the cooperation rates of homeschooling households were not likely lower than that of non-homeschooling households.^[3]

Table 2 presents the results of this simulation exercise. The formula used to calculate the cooperation rates is as follows:

$$\text{COOP}_\text{H}= \frac{\text{I}_\text{H}}{\text{I}_\text{H} + _\text{eh}\text{R}}$$

where:

COOP=Cooperation rate

I=Complete interview

R=Refusal

H=Homeschooled

h=Estimated proportion of cases homeschooled

e=Estimated proportion of cases that are age-eligible

We estimate the age-eligibility rate to be the same as the rate for completed cases, which is 43 percent. As seen in Table 2, the cooperation rate of homeschooling households was 76 percent if we assume that 30 percent of all refusals were homeschooling households, 61 percent if 40 percent of all refusals were homeschooling households, and so on. Even if 100 percent of all refusals are from homeschooling households, the estimated cooperation rate is still near 50 percent.

Call effort. In Table 3, we look at the level of call effort required to complete a screener interview. Cases in the List sample completed screener interviews, on average, within a fewer number of calls (3.6) compared with the RDD sample (4.3). However, this difference appears to be driven by non-homeschooling households in the List sample.

Another way to look at the call effort is to examine the percentage of cases completed within one, two to three, or more than three calls. The results of these analyses are presented in the bottom half of Table 3. Again, we found evidence of a lower call effort in the List sample when compared to the RDD sample. And again, these differences appear to be driven by non-homeschooling cases. However, we found no measurable differences in the percentage of screener interviews completed after the first, second, third or higher call when we compared confirmed homeschooling cases in the List sample to those in the RDD sample, as well as to non-homeschooling cases in the RDD sample.

Table 4 presents the results of our analysis of refusal patterns. Compared with the RDD sample, a smaller percentage of cases in the List sample ever refused to participate in the screener interview (58 percent versus 48 percent). Roughly a third of confirmed homeschooling cases in both the List and RDD samples ever refused a screener interview compared with over half (59 percent) of non-homeschoolers in the RDD

sample and half (49.7 percent) of non-homeschoolers in the List sample. Looking at the distribution in the number of times a case was refused, again we found a smaller percentage of homeschooling households in the RDD (10.1 percent) and the List (16.1 percent) sample refused two or more times compared to non-homeschoolers.

Implications for Survey Practice

This study illustrates the potential benefits of using a seeded sample to study nonresponse for key groups, while also showing its significant limitations. We faced several challenges in implementing this study, including time constraints, barriers to access to potential sample sources and quality of sample records. Most membership lists would not allow for blind administration and single lists were not sufficiently diverse. Most problematic in the mailing list we selected was the list quality: the bulk of households in the List sample were found to be ineligible.

Like many surveys concerned about nonresponse bias, it is difficult to obtain information about survey nonresponders. Therefore, a single nonresponse study is not likely to capture enough information to fully analyze and explain response propensity, even for a specific subgroup. Despite the shortcomings of the current study, the limited results do not suggest that all or most homeschooling households have low response propensity. It is more likely that some have low response propensity, but some have high response propensity. However, differential response propensity among homeschoolers can bias estimates that are directly related to the characteristics of nonresponding homeschoolers, such as household characteristics, reasons for homeschooling, and interaction with local public schools. These potential sources of bias should be the subject of further research.

Acknowledgment

The authors gratefully acknowledge the funding support provided by National Center for Education Statistics (NCES), U.S. Department of Education. The views and opinions expressed in this paper are those of the authors and do not necessarily represent those of NCES or the U.S. Department of Education. The authors also are grateful for the computational assistance and research support provided by Akemi Kinukawa and Camille Whitney.