Enhancing Address Based Sampling with Contingent Incentives and Sample Indicators

In February 2010, a test was conducted among Nielsen’s TV Diary Surveys to identify improvements in the cost model associated with the Address Based Sampling (ABS) methodology. The goals of this test were as follows:

1. Evaluate the effectiveness of using a Black race indicator resulting from the use of geo-coding to differentially incent addresses in the pre-recruitment stage.
2. Evaluate the effectiveness of using sample indicators at the diary mailing stage.
3. Evaluate the use of contingent (“pay for performance”) incentives in combination with non-contingent (“pay up front”) incentives.

Nielsen’s TV Diary Survey measures the TV viewing and tuning habits in 185 markets. Since November 2008, this survey has been conducted using ABS. Specifically, addresses are randomly selected from the U.S. Postal Service and matched against an external database to append such information as phone number, name of head of household, age of head of household, etc. The TV Diary Survey is split into two different samples: the Regular Sample and the Oversample. The Regular Sample is randomly sampled to be representative of the U.S. population. However, because of lower response rates among certain hard-to-reach demographics, namely age of head of household (AOH) less than 35 years, Black, and Hispanic, we use the Oversample to supplement the Regular Sample.

Test Design

Sample indicators are used at the pre-recruitment stage for Regular Sample addresses. These indicators include the vendor supplied AOH indicator and the Hispanic surname indicator (used only in Spanish treatment markets.^[1] In addition to these indicators, which have been used since the implementation of ABS, a new Black indicator was also used for this test. This Black indicator is applied to addresses in Census block groups with 70% or greater penetration of Black households (in Black treatment markets) as determined by the Nielsen Company. Pre-recruitment incentives, ranging from $0 to $5, are calculated based on these sample indicators.

Sample indicators were also used at the TV Diary mailing stage to determine incentive levels for those test households without a response to the AOH, Black race, or Hispanic identity questions (for the control group, only known responses are used to calculate the diary incentive amounts). In the case of the Oversample, the sample indicators could also result in test households staying in sample that may not have before because of missing responses to these questions (for the control group, only known responses are used to determine eligibility in the Oversample).

In production,^[2] all diary incentives are non-contingent and are mailed with the diary package. Those incentives range from $1 to $30. This test used both non-contingent and contingent incentives. In an effort to effectively reallocate the pay out of incentives, the non-contingent incentive was lowered to $1 or $2. This allowed for higher contingent incentives to the hard-to-reach groups (AOH<35, Black, and Spanish speaking Hispanic). Table 1 shows the non-contingent and contingent incentive structure for this test.

All contingent incentives are assumed to be cash and were sent in a separate mailing after diary processing. For the test, incentive levels were determined by responses to the AOH, Black race, Hispanic identity, and language questions, as well as the sample indicators.

Cooperation Rates^[3]

Overall, the Regular Sample test homes cooperated at a lower rate than the control households. Specifically, of the households that were mailed a TV Diary, 24.2% returned a useable diary in the control group, while only 23.7% returned a useable diary in the test group. While this difference was not statistically significant at the 0.05 significance level (p-value = 0.099), we felt the decrease merited additional investigation.

To determine the probable cause for the decrease in cooperation rates among the Regular Sample, we looked at the effect of each change introduced in the test. First, we investigated the impact of the higher pre-recruitment incentives. Among households with the Black indicator in Black treatment markets, the cooperation rate for the test homes (18.1%) was 3 percentage points (ppt.) higher than the cooperation rate for the control cell (15.1%), which was significant (p-value = 0.027)

Next, we investigated the impact of the contingent incentives and using sample indicators at the diary mailing stage. After splitting the Regular Sample into 3 analogous groups for both the test and control, we observed the following cooperation rates:

1. control households that received a $0–3 non-contingent incentive had a slightly higher cooperation rate than test households that received a $0 contingent incentive (45.2% vs. 42.7%; p-value = 0.16)
2. control households that received a $5–10 non-contingent incentive had a significantly lower cooperation rate than test households that received a $20 contingent incentive (13.1% vs. 21.6%; p-value < 0.000)
3. control households that received a $30 non-contingent incentive had a significantly higher cooperation rate than test households that received a $50 contingent incentive (43.8% vs. 29.4%; p-value < 0.000)

The above results are influenced by both contingent incentives and sample indicators.^[4] To isolate the effect of the contingent incentives, we simulated test results assuming that sample indicators were not used. Based on the following, we see significant drops in cooperation rates for all groups:

1. among the low-incentive groups, control households had a 2.6 ppt. higher cooperation rate than the test households (45.2% vs. 42.6%; p-value < 0.000)
2. among the medium-incentive groups, control households had a 1.5 ppt. higher cooperation rate than the test households (13.1% vs. 11.6%; p-value < 0.000)
3. among the high-incentive groups, control households had a 3.0 ppt. higher cooperation rate than the test households (43.8% vs. 40.8%; p-value = 0.004)

For the Oversample, the sample indicators were used to include addresses where we did not make contact, thus making a previously unmailable address mailable. As a result, we saw higher Oversample mail rates. However, although we mailed to more addresses, there were significant drops in cooperation rates. Specifically, the cooperation rate among control households was 39.9%, compared to only 22.2% among test households (p-value < 0.000).

When we combined the Regular Sample and the Oversample, we saw a 2 ppt. drop in cooperation rates. As such, although we see positive results among households with a Black indicator, the overall drop in the Diary Survey cooperation rates shows that contingent incentives, as tested, result in a significant drop in intabs.^[5]

Sample Composition

In addition to cooperation rates, we also looked at the AOH, Black race, and Hispanic identity distributions for the Regular Sample intabs. We did not observe any statistically significant changes in any of these distributions for the total Regular Sample.

When we look at the sample composition for the Regular Sample homes with the Black indicator, we see that the test had a significantly lower proportion of AOH 50+ households when compared to control (see Table 2). While the test also had a higher Black penetration than the control, the difference was not significant (this could be due to relatively small sample size).

The indicators were relatively accurate in identifying household demographics. Specifically, among intabs, the AOH<35 indicator had a 78.8% accuracy, the Black indicator had a 85.2% accuracy, and the Hispanic indicator had a 87.8% accuracy.

Conclusion

Even though the sample indicators accurately identified household characteristics, the use of the contingent incentives resulted in significantly lower cooperation rates for the TV Diary sample. This was true even for those households offered the $50 contingent incentive. In addition, we saw no improvement in the Regular Sample demographics. Based on these results, we did not implement contingent incentives nor use sample indicators at the mailing stage.

However, among households with the Black indicator, we saw higher cooperation rates, as well as a higher Black penetration. Consequently, we implemented the use of the Black Indicator at the pre-recruitment stage in our production sample.