Incentive Impact on Data Quality, Sample Composition, and Respondents’ Topic Interest

Michael Soszynski

doi:10.29115/SP-2024-0013

Soszynski, Michael. 2024. “Incentive Impact on Data Quality, Sample Composition, and Respondents’ Topic Interest.” Survey Practice 17 (September). https://doi.org/10.29115/SP-2024-0013.

View more stats

Abstract

This article investigates the impact of offering a small incentive for a web survey of utility customers in the Mid-Atlantic United States. We explore background demographics and characteristics, response patterns, and data quality. Offering a small incentive was found to increase response rates. We did not find convincing evidence that offering an incentive affects data quality. However, we did find some evidence that the lack (or presence) of an incentive may impact sample composition and failing to provide an incentive may lead to more responses from people who are more interested in the survey topic.

Introduction

Survey researchers may seek to reduce costs by choosing not to offer an incentive. However, there is evidence that surveys with guaranteed incentives tend to improve response rates when compared to surveys with no incentive or a lottery/prize drawing (see for example, Halpern et al. 2011 and Dykema et al. 2024). Further, studies have shown that the provision of an incentive may increase response rates from hard-to-reach groups with little or no impact on data quality.

For example, a mail-administered consumer preference survey focused on ethanol fuel found that providing an incentive tended to increase the portion of respondents who were less educated and who were less familiar with the survey topic (Petrolia and Bhattacharjee 2009). Another study found that using an incentive was an effective and cost-efficient strategy to increase response rates for hard-to-reach caregivers in a phone survey (Fomby, Sastry, and McGonagle 2017).

Continued research is needed on the impact of incentives on data quality and sample composition (Singer and Ye 2013). Specifically, researchers have noted that the available evidence on the impacts of monetary incentives in web surveys is more limited compared to other administration modes (Abdelazeem et al. 2023).

In this study we consider whether an incentive impacts data quality or sample composition through the investigation of non-observation and observation effects. To measure non-observation effects, we compared the reported demographic makeup of an incentive and no incentive group. For observation effects, we investigated whether the presence of an incentive led to satisficing or affected data quality or response patterns. To operationalize satisficing, we used multiple indicator variables. For each group, we reviewed the amount of time spent on the survey, the length of open-ended responses, and the tendency to straightline responses to grid or matrix-style questions. Response patterns were examined by comparing the groups’ rates of item non-substantive response and satisfaction and how they answered attitudinal and behavioral questions.

Methods

In 2022 and 2023, utility customers in a Mid-Atlantic state in the United States were invited to a web survey via email. The purpose of the survey was to help inform an evaluation of a behavior modification or Home Energy Reports (HER) program.

The survey invitations offered either no incentive or a $5 incentive. Initially, the survey did not offer an incentive; a $5 incentive was added to improve the response rate and meet survey quota requirements. Incentive group respondents were sent $5 electronic gift cards through an online fulfillment company 5-10 days after survey completion. Respondents were informed that the gift card would be sent to them via email 5-10 days following survey submission, and they could redeem it to be used at one of over 80 different retailers.

We selected groups, or “lots,” of invitees for the survey from the HER program tracking file. Tracking data contained customer names, email addresses, and account information as well as whether customers had received reports. The selection of lots was not random, but no underlying characteristics were considered in selection – that is, the lots were selected in a non-systematic fashion. For example, customers in rows 2 through 500 were selected to receive the survey invite with no offer of an incentive. After initial lots did not result in a sufficient number of completes, later lots were offered incentives (for example, those in row 9,530-12,500 were selected to be offered an incentive). Tracking data did not list customers in any discernable order (e.g., date of report enrollment, geographic area, alphabetically) and lots were pulled from tracking data in an ad hoc fashion. Incentive assignment was unrelated to HERs treatment group or date of HERs enrollment.

Program data did not include respondent characteristics, so it was not possible to determine or use demographic or background characteristics for the assignment of the incentive and no incentive groups. Therefore, incentive assignment should not have led to any potential impact on the outcomes of this study. About the same portion of control and treatment group customers were offered an incentive.

All survey questions required a response to allow respondents to move forward. However, for each of the demographic and background questions, either “Prefer not to answer” or “Don’t know” were provided as options.

For nominal-level variables, the test statistics presented are chi-square tests for categorical variables and two-sample z-tests for proportions for “Don’t know” and “Prefer not to answer.” In cases in which categorical variables had expected values less than 5 and tests were significant, Fisher’s Exact Test was used (statistics are denoted with two hyphens for these tests). For ordinal-level variables, the test statistic is the z score for Mann-Whitney U. We acknowledge and recognize both Type I and Type II error concerns as legitimate, thus we present individual p-values as well as 95% confidence intervals for differences between groups.

Results

Table 1 shows the response rate for the survey. The presence of an incentive was associated with a 2.1 percentage point higher response rate (95% CI [1.77, 2.39]).^[1]

Table 1.Summary of Survey Response Rate

Metric	Incentive	No Incentive	Total	Statistic	p-value
Initial Invitation	13,805	23,446	37,251	-10.37 (two-sample z-test)	<0.001
Bounce/Email Failed	927	1,573	2,500
Complete	387	203	590
Response Rate (Complete/ (Initial Invite-Bounces)	3.0%	0.9%	1.7%

Non-observation effects

No underlying customer characteristics were considered (or available to be considered) during incentive group assignment, thus the groups should not have differed on any demographic or household characteristics variables.

The incentive group had a larger portion of respondents identify as Asian (9% compared to 2% of the no incentive group). Also, the ages of the groups differed, with the no incentive group skewing older, with a larger portion identifying as being over 55 years old compared to the incentive group. The incentive group tended to have larger household sizes. Forty-two percent of the incentive group indicated their household had three or more members; this compared to 28% of the no incentive group. To sum up, the incentive group tended to be younger, have larger households, and be more likely to identify as Asian ethnicity.

Because underlying or benchmark demographic and background information was not available, we cannot state whether or not survey respondents were demographically representative of the population. Though lots were not selected using demographic or background characteristics as this information was unavailable to the survey administration team, we must caveat these findings and note that observed differences between the groups may be a function of population differences that we have no way of knowing about. Complete tables with survey respondents’ reported demographic and household information are provided in the Appendix.

Observation Effects

We investigated whether the presence of an incentive affected data quality or response patterns.

Satisficing

To gauge whether the presence of an incentive was related to satisficing, we reviewed the amount of time spent on the survey for the groups, length of open-ended responses, and the tendency of each group to straightline their responses to grid or matrix-style questions.

We did not find evidence that providing an incentive led to satisficing or less nuanced or thoughtful responses. The time to complete the survey, length of open-ended responses, and straightlining rates were similar for the two groups. After removing outliers, the average survey time response time was 9:56 for the incentive group, and 10:20 for the no incentive group, respectively. This difference was not significant at the p<0.05 level.

Customers who received Home Energy Reports and indicated they were dissatisfied or felt they were inaccurate were asked open-ended questions to get feedback on how to improve the reports. Though the groups had differing response lengths, these differences were not statistically significant. Additionally, not only were the differences not statistically significant, but the trends for the two questions went in opposite directions. The incentive sample gave shorter responses to the first question but longer ones to the second question. We note that the sample sizes for these open-ended questions were smaller than other analyses in this paper.

Table 2.Length of Open-Ended Responses

	Average Number of Words		Statistic (z test for two proportions)	p-value	Difference (95% CI)
Question	Incentive	No Incentive	Statistic (z test for two proportions)	p-value	Difference (95% CI)
What did you find inaccurate in your Home Energy Report?	9.83 (n=41)	11.16 (n=25)	-0.39	0.70	-1.3 [-8.13, 5.5]
How could we improve the Home Energy Report?	27.87 (n=15)	21.94 (n=16)	0.51	0.61	5.9 [-16.9, 28.8]

The survey contained five grid-style Likert questions. We used these questions to investigate straightlining. Each respondent was asked up to four grid-style questions that requested they provide a scale rating for several sub-components. We assigned each respondent a straightlining score from 0% to 100% (count of straightlined grid questions/total grid questions asked). Each grid-style question was considered independently. For example, respondents were asked to rate their level of agreement on a scale from 0 (strongly disagree) to 10 (strongly agree) for a grid with nine statements. If a respondent selected the same response (e.g., 9 for all statements, 0 for all statements) as their level of agreement for all nine statements, they were considered to have provided a straightline response for that grid.

The average straightline score for the incentive group was 15.1%, and the no incentive group had an average score of 17.9% (difference of -2.8 with 95% CI [-9.2, 3.6]). More intuitively, of all matrix or grid questions asked to the incentive group, 18.9% had straightlined responses, compared to 22.3% of the no incentive group (difference of -3.4 with 95% CI [-10.3, 3.5]). The difference between groups was not significant, and the result went against the hypothesis that the incentive group would be more likely to straightline grid-style questions.

Item Non-Substantive Response

The incentive and no incentive groups had similar rates of item nonresponse for demographic questions (choosing “Prefer not to answer” or “Don’t know”). Respondents who received an incentive provided non-substantive responses for an average of 4.1% of demographic questions, compared to 3.6% of the no incentive group (difference of 0.5% 95% CI [-2.7, 3.7]).

Satisfaction

Survey respondents who received reports were asked to rate their satisfaction with the reports, as well as their accuracy and the perceived value of the components. Eleven questions were asked in the first year and seven in the second year; four of the value ratings were removed in the second year, as their subject overlapped with the satisfaction ratings. Though there were differences between the groups, none of them were significant and generally the groups provided similar satisfaction ratings. For example, 70.1% of the no incentive group said the reports were accurate compared to 65.8% of the incentive group. About 78% of both groups indicated they were satisfied with the reports overall.

Leverage-salience theory

The leverage-salience theory suggests that people respond at higher rates to surveys when the topic is of more interest (or more salient) to them (Groves, Singer, and Corning 2000). Without any other lever, the main motivation for taking a survey is typically the salience of a topic to a potential respondent.

Though we did not find evidence of satisficing or any significant differences in data quality, there was some evidence that the two groups had differing levels of interest in the survey topic. A larger portion of the no incentive group indicated that they read all the reports (see Table 3).

Table 3.How often did you read the Home Energy Reports in the last 12 months?

Frequency	Incentive (n=196)	No Incentive (n=107)	Statistic	p-value	Difference (95% CI)
I read all the reports	57.1%	71.0%	-2.16 (Mann Whitney U test)	0.02	13.9 [2.9, 24.9]
I read most of the reports	27.0%	16.8%			-10.2 [-19.7, -0.8]
I read about half of the reports	4.6%	1.9%			-2.7 [-6.6, 1.2]
I read a few of the reports	11.2%	10.3%			-0.9 (-8.2, 6.3]

Each year, the respondents were asked attitudinal questions. Respondents were asked to rate their level of agreement on up to nine questions. The portion of each group that agreed with each statement was similar. However, a higher portion of the no incentive group indicated they agreed that scarce energy supplies will be a major problem in the future and that it is possible to save energy without sacrificing comfort by being energy efficient.

Table 4.Energy efficiency beliefs and attitudes^[2]

Question	Incentive	No Incentive	Statistic	p-value (z test for two proportions)	Difference (95% CI)
Energy efficiency saves money. (n= 203, 387)	84%	87%	0.956	0.339	3.0% [-2.9, 8.9]
I am aware of how much energy my household consumes each month. (n= 203, 387)	65%	69%	1.064	0.287	4.4% [-3.9,11.9]
I am not very concerned about the amount of energy used in my home. (n= 65, 222)	16%	17%	0.135	0.892	0.7 [-9.6, 11.0]
I am too busy to worry about making energy-related improvements in my home. (n= 203, 387)	17%	11%	-1.695	0.090	-5.2% [-10.9, 0.5]
Scarce energy supplies will be a major problem in the future. (n= 203, 387)	55%	69%	3.513	<0.001	14.9 [6.89, 22.9]
There is very little I can do to reduce the amount of energy I am now using. (n= 203, 387)	25%	31%	1.550	0.121	6.0% [-1.7, 13.6]
It is possible to save energy without sacrificing comfort by being energy efficient. (n= 203, 387)	65%	75%	2.550	0.011	10.3 [2.7, 17.8]
I know of steps I could take to reduce my household energy use. (n= 203, 387)	58%	57%	-0.408	0.683	-1.7 [-10.1, 6.7]
I am interested in reducing my household energy use. (n= 138, 165)	75%	71%	-0.689	0.491	-3.5 [-13.6, 6.5]
I intend to reduce my household energy use in the next 12 months. (n= 203, 387)	52%	44%	-1.869	0.062	-8.1 [-16.6, 0.3]

The portion of incentive and no incentive groups that reported purchasing or installing energy-efficient equipment in the past 12 months or taking energy-efficient actions was similar (see Table 5). This finding is consistent with the fact that, in the above table, the two groups differed on the two statements about attitudes or beliefs, while they were similar on the items that had more to do with their actual behaviors or intentions:

There is very little I can do to reduce …
I know of steps I could take…
I am interested in reducing…
I intend to reduce…

Table 5.Reported Energy-Saving Actions & Purchases

Question	Incentive (n=387)	No Incentive (n=203)	Statistic	p-value (z test for two proportions)	Difference (95% CI)
Purchased Item	63.3%	60.2%	0.76	0.45	3.2 [-5.2, 11.3]
Taken action	87.6%	89.6%	-0.75	0.45	-2.1 [-7.3, 3.3]

These findings, in addition to the incentive group being more prone to marking “Don’t know” for home size and age questions (see Table A1) seem to give some evidence that the no incentive group had more interest in and awareness of the survey topic. They tended to know more about their home and be more interested in their energy efficiency, compared to the incentive group (though the groups reported similar behaviors or intentions).

Controlling for multiple comparisons

It is important to recognize that making multiple comparisons elevates the risk of finding statistically significant differences between groups and erroneously rejecting the null hypothesis. Increasing the number of comparisons or statistical tests between groups raises the chances of false positives. For example, if we conducted 20 statistical tests and the null hypothesis was true for all of them, we would expect to reject the null hypothesis at the p<0.05 level for one of the tests, just due to chance. To account for the increased risk of a false positive (Type I Error), some researchers call for corrections, or to aim to control the False Discovery Rate using a higher significance threshold (McDonald 2014). Others caution that these adjustments could cause researchers to miss possibly important findings by increasing the number of false negatives (Rothman 1990).

Once we controlled for multiple comparisons, evidence that the two groups differed diminished. We used the Benjamini-Hochberg procedure to control for false discovery. Two differences remained significant with an adjusted threshold for significance of p<0.002: the no incentive and incentive groups’ reported age and level of agreement to the question on scarce resources.

Discussion

Offering a small incentive increased the response rate. We did not find evidence that response quality differed between the groups. But there was some evidence that the offer of an incentive may impact the types of customers that choose to take a survey and their response patterns. The groups provided similar satisfaction ratings, but the incentive group was younger, more non-white, more likely to rent, and more prone to marking “Don’t know” for home size and age questions (see Table A1). Further, responses to level of agreement questions provided evidence that the no incentive group tended to be more engaged or interested in the topic (though groups reported similar intentions or actual behavior).

The incentive and no incentive groups were distinct and there were not any known differences between the groups. They received similar recruitment messages and completed the same survey instrument. The groups were not known to have systematic differences or to have been chosen in a manner that could have impacted results. The only potential impact may have been that the no incentive group was invited to take the survey before the incentive group. However, there is no compelling reason that this would impact sample composition, response patterns, or data quality, as all invitees had significant time to complete the survey once they had been invited and the two groups were distinct.

Recent research has questioned the validity of standard weighting models’ assumption that whether and how people respond is not correlated and suggest survey researchers in various fields should investigate and account for this correlation more actively (Bailey 2023). This paper provides evidence that relying on individuals willing to provide responses without an incentive may create non-ignorable nonresponse. We found several differences between the no incentive and incentive groups. Modeling selection and adjusting results such that the incentive group is weighted heavier may create truer estimates.

Additional research is needed to validate this paper’s findings. Other potential areas to explore include different geographies or target populations, survey topics, and incentive levels (or forms of incentives). Another area to strengthen this study could be to seek larger sample sizes. The sample sizes of 387 and 203 provided decent power to detect moderate differences, but less power to detect very small differences. Leaving Type I error at the default 5% rate, these sample sizes deliver at least 76% power to detect a 10-percentage point difference and at least 31% power to detect a 5-percentage point difference.

Submitted: March 04, 2024 EDT

Accepted: August 09, 2024 EDT

References

Abdelazeem, Basel, Aboalmagd Hamdallah, Marwa A. Rizk, Kirellos S. Abbas, Nahla A. El-Shahat, Nouraldeen Manasrah, Mostafa R. Mostafa, and Mostafa Eltobgy. 2023. “Does Usage of Monetary Incentive Impact the Involvement in Surveys? A Systematic Review and Meta-Analysis of 46 Randomized Controlled Trials.” PloS One 18 (1): e0279128. https://doi.org/10.1371/journal.pone.0279128.

Google Scholar

Bailey, Michael. 2023. “A New Paradigm for Polling.” Issue 5.3, Summer 2023 5 (3): 5–3. https://doi.org/10.1162/99608f92.9898eede.

Google Scholar

Dykema, J., J. Stevenson, C. P. Jones, and B. Day. 2024. “Guaranteed Incentives and Prize Drawings: Effects on Participation, Data Quality, and Costs in a Web Survey of College Students on Sensitive Topics.” Social Science Computer Review 42 (3): 777–93. https://doi.org/10.1177/08944393231189853.

Google Scholar

Fomby, P., N. Sastry, and K. A. McGonagle. 2017. “Effectiveness of a Time-Limited Incentive on Participation by Hard-to-Reach Respondents in a Panel Study.” Field Methods 29 (3): 238–51. https://doi.org/10.1177/1525822X16670625.

Google Scholar

Groves, Robert M., Eleanor Singer, and Amy Corning. 2000. “Leverage-Saliency Theory of Survey Participation.” Public Opinion Quarterly 64 (3): 299–308. https://doi.org/10.1086/317990.

Google Scholar

Halpern, S. D., R. Kohn, A. Dornbrand-Lo, T. Metkus, D. A. Asch, and K. G. Volpp. 2011. “Lottery-Based versus Fixed Incentives to Increase Clinicians’ Response to Surveys.” Health Services Research 46 (5): 1663–74. https://doi.org/10.1111/j.1475-6773.2011.01264.x.

Google Scholar

McDonald, John H. 2014. Handbook of Biological Statistics. 3rd ed. Baltimore: Sparky House Publishing.

Google Scholar

Petrolia, Daniel R., and Sanjoy Bhattacharjee. 2009. “Revisiting Incentive Effects: Evidence from a Random-Sample Mail Survey on Consumer Preferences for Fuel Ethanol.” The Public Opinion Quarterly 73 (3): 537–50. http://www.jstor.org/stable/40467617.

Google Scholar

Rothman, Kenneth J. 1990. “No Adjustments Are Needed for Multiple Comparisons.” Epidemiology (Cambridge, Mass.) 1 (1): 43–46. https://doi.org/10.1097/00001648-199001000-00010.

Google Scholar

Singer, Elanor, and Cong Ye. 2013. “The Use and Effects of Incentives in Surveys.” Annals of the American Academy of Political and Social Science 645 (1): 112–41. https://doi.org/10.1177/0002716212458082.

Google Scholar

Appendix

Table A1.Home characteristics by incentive group

Question	Response	Incentive (n = 387)	No Incentive (n = 203)	Statistic	p-value	Difference (95% CI)
Do you own or rent your home?	Rent	9.3%	5.9%	-- (Fisher’s Exact Test)	0.20	3.4% [-1.1, 7.7]
	Own	90.4%	94.1%	-- (Fisher’s Exact Test)	0.20	-3.6% [-8.0, 0.7]
	Don't know	0.3%	0.0%	-0.724 (two-sample z-test)	0.47	0.3% [-0.2, 0.8]
Which of the following best describe your home?	Single-family detached	72.9%	78.3%	-- (Fisher’s Exact Test)	0.71	-5.5% [-12.7,1.7]
	Duplex	4.1%	2.0%			2.2% [-0.6, 4.9]
	Triple decker	1.0%	0.5%			0.5 [-0.9, 1.9]
	Apartment/condo in a 2-4 unit building	3.6%	4.9%			-1.3 [-4.8, 2.2]
	Apartment/condo in a 5+ unit building	5.7%	3.9%			1.7 [-1.8, 5.3]
	Townhouse or row house	10.9%	9.4%			1.5 [-3.6, 6.6]
	Mobile home or trailer	1.3%	1.0%			0.3 [-1.5, 2.1]
	Other	0.3%	0.0%			0.3 [-0.2, 0.8]
	Don't know	0.3%	0.0%	-0.72 (two-sample z-test)	0.47	0.3 [-0.2, 0.8]
When was your home built?	Before 1960	29.7%	29.1%	-0.15 (chi-square test)	0.44	0.7 [-7.1, 8.4]
	1960 to 1979	22.5%	24.6%			-2.1 [-9.4, 5.1]
	1980 to 1999	25.1%	29.1%			-4.0 [-11.6, 3.6]
	2000 to 2009	10.1%	13.8%			-3.7 [9.3, 1.9]
	2010 or later	7.5%	2.0%			5.5 [2.3, 8.8]
	Don't know	5.2%	1.5%	-2.19 (two-sample z-test)	0.028	3.7 [0.9, 6.5]
How many people, including you, currently live in your household?	1	17.6%	22.7%	-2.87 (Mann Whitney U Test)	0.004	-5.1 [-12.0, 1.8]
	2	38.8%	47.3%			-8.5 [-16.9, -0.1]
	3	18.9%	11.8%			7.0 [1.1, 12.9]
	4	17.1%	10.8%			6.2 [0.5, 11.9]
	5	5.2%	1.5%			3.7 [0.9, 6.5]
	6 or more	1.0%	3.9%			-2.9 [-5.8, 0.0]
	Prefer not to say	1.6%	2.0%	0.38 (two-sample z-test)	0.71	-0.4 [-2.7, 1.9]
About how many square feet is your home?	Less than 1,000 square feet	5.9%	5.4%	-0.74 (chi-square test)	0.46	0.5 [-3.4, 4.4]
	1,000-1,999 square feet	43.4%	43.8%			-0.4 [-8.9, 8.0]
	2,000-2,999 square feet	26.1%	33.5%			-7.4 [-15.2, 0.4]
	3,000-3,999 square feet	7.8%	6.4%			1.3 [-2.9, 5.6]
	4,000 or more square feet	3.9%	4.9%			-1.1 [-4.6, 2.5]
	Don’t know	12.9%	5.9%	-2.64 (two-sample z-test)	0.01	7.0 [2.4, 11.7]
What is the main fuel used for heating your home?	Electricity	8.0%	3.9%	-- (Fisher’s Exact Test)	0.238	4.1 [0.3, 7.9]
	Natural Gas	88.4%	93.6%			-5.2 [-9.9, -0.6]
	Propane	0.5%	0.0%			0.5 [-2.2, 1.8]
	Oil	1.3%	1.5%			-0.2 [-2.2, 1.8]
	Wood	0.3%	0.5%			-0.2 [1.3, 0.9]
	Don't know	9.8%	8.4%	-1.13 (two-sample z-test)	0.26	1.1 [-0.5, 2.6]
What fuel does your main water heater use?	Electricity	86.0%	89.2%	-- (Fisher’s Exact Test)	0.85	1.4 [-3.4, 6.3]
	Natural gas	0.3%	0.0%			-3.1 [-8.6, 2.4]
	Oil	3.9%	2.5%			0.3 [-0.2, 0.8]
	Don’t know	9.8%	8.4%	-0.90 (two-sample z-test)	0.37	1.4 [-1.5, 4.3]

Table A2.Demographic characteristics by incentive group

Question	Response	Incentive (n = 387)	No Incentive (n = 203)	Statistic	p-value	Difference (95% CI)
What is your age?	Under 35 years old	6.2%	1.5%	-5.03 (Mann Whitney U Test)	<0.01	4.7 [1.8, 7.6]
	35-55 years old	35.7%	20.2%			15.5 [8.2, 22.8]
	Over 55 years old	54.5%	74.4%			-19.9 [-27.7, -12.1]
	Prefer not to answer	3.6%	3.9%	0.19 (two-sample z-test)	0.84	-0.3 [-3.6, 2.9]
What is the primary language spoken in your home?	English	94.3%	96.1%	-- (Fisher’s Exact Test)	0.99	-1.7 [-5.3, 1.8]
	Spanish	2.1%	2.0%			-0.1 [-2.3, 2.5]
	Chinese	0.8%	1.0%			-0.2 [-1.8, 1.4]
	Hindi	0.5%	0.0%			0.5 [-0.2, 1.2]
	Gujarathi	0.5%	0.0%			0.5 [-0.2, 1.2]
	Portuguese	0.5%	0.5%			0.0 [-1.2. 1.2]
	Polish	0.3%	0.0%			0.3 [-0.2. 0.8]
	Other	0.3%	0.0%			0.3 [-1.0, 1.6]
	Prefer not to answer	0.8%	0.5%	-0.39 (two-sample z-test)	0.69	0.3 [-1.0, 1.6]
Which of the following best describes the race or ethnic background you identify with?^a	Black or African American	6.2%	3.9%	-- (Fisher’s Exact Test)	<0.01	2.3 [-1.3, 5.9]
	Hispanic or Latino/Latina	7.5%	5.4%			2.1 [ -2.0, 6.1]
	American Indian and Alaska Native	0.3%	0.0%			0.3 [ -0.2, 0.8]
	Asian	8.8%	2.0%			6.8 [3.4, 10.2]
	Native Hawaiian and Other Pacific Islander	0.0%	0.5%			-0.5 [-1.5, 0.5]
	White	71.1%	76.8%			-5.8 [13.1, 1.6]
	Prefer not to answer	9.3%	13.3%	1.49 (two-sample z-test)	0.14	-4.0 [-9.5, 1.5]
Household Income	Under 250% FPL	21.2%	18.2%	-0.81 (Mann Whitney U Test)	0.67	2.7 [-4.0, 9.4]
	Between 250-400% FPL	12.4%	11.8%			0.6 [-4.9, 6.1]
	Over 400% FPL	39.0%	41.4%			-2.6 [-11.0, 5.7]
	Don't know	3.1%	3.9%	0.54 (two-sample z-test)	0.59	-0.8 [-0.4, 2.3]
	Prefer not to answer	24.3%	24.6%	0.09 (two-sample z-test)	0.93	-0.3 [-7.6, 7.0]

^aTotal is greater than 100% because customers could select more than race or ethnic background.

Invite #1 (Report Recipient with incentive)

Subject: Earn $5 and Help [UTILITY] Improve Its Home Energy Reports

[UTILITY] is interested in your feedback to help them improve their home energy reports. We would greatly appreciate your taking a few minutes to provide your feedback. Your responses will be kept anonymous and confidential. If you have a few minutes to complete this survey, we’ll send you a $5 gift certificate that you can use at one of dozens of online stores.
[SURVEY LINK]

We would greatly appreciate your taking a few minutes to provide your feedback. If you have questions or require technical assistance, please respond to this email at [EMAIL] or contact the [UTILITY] evaluation contractor at [PHONE NUMBER] / [EMAIL ADDRESS].
If you wish to no longer receive emails about this survey, please click on the “Unsubscribe” link below. Thank you in advance for your time!
Kind Regards,
Contractor to [UTILITY]
[OPT OUT LINK]
The company administering the survey conducts research to support research and evaluation in the energy sector. Survey data are not shared with third parties for marketing purposes. Our full privacy statement is linked here:
xxx.com/privacy

Invite #2 (Report Recipient without incentive)

Subject: [UTILITY] Needs Your Feedback in its Home Energy Reports

[UTILITY] is interested in your feedback about the Home Energy Reports it sends you through email or the mail. Your reports contain information about your home energy use and tips for saving energy. Your responses will be kept anonymous and confidential.
[SURVEY LINK]
We would greatly appreciate your taking a few minutes to provide your feedback. If you have questions or require technical assistance, please respond to this email at [EMAIL] contractor at [PHONE NUMBER] / [EMAIL ADDRESS].
If you wish to no longer receive emails about this survey, please click on the “Unsubscribe” link below. Thank you in advance for your time!
Kind Regards,
Contractor to [UTILITY]
[OPT OUT LINK]
The company administering the survey conducts research to support research and evaluation in the energy sector. Survey data are not shared with third parties for marketing purposes. Our full privacy statement is linked here:
xxx.com/privacy

Invite #3 (Non-Report Recipient without incentive)

Subject: [UTILITY] Needs Your Feedback in its Energy Efficiency Program

[UTILITY] is interested in your feedback to help them improve their services. We would greatly appreciate your taking a few minutes to provide your feedback. Your responses will be kept anonymous and confidential.
[SURVEY LINK]
If you have questions or require technical assistance, please respond to this email or contact us at [EMAIL ADDRESS].
If you wish to no longer receive emails about this survey, please click on the “Unsubscribe” link below. Thank you in advance for your time!
Kind Regards,
Contractor to [UTILITY]
[OPT OUT LINK]
The company administering the survey conducts research to support research and evaluation in the energy sector. Survey data are not shared with third parties for marketing purposes. Our full privacy statement is linked here:
xxx.com/privacy

Invite #4 (Non-Report Recipient with incentive)

Subject: Earn $5 and Help [UTILITY] Improve Its Energy Efficiency Programs

[UTILITY] is interested in your feedback to help them improve their services. We would greatly appreciate your taking a few minutes to provide your feedback. Your responses will be kept anonymous and confidential. If you have a few minutes to complete this survey, we’ll send you a $5 gift certificate that you can use at one of dozens of online stores.
[SURVEY LINK]
We would greatly appreciate your taking a few minutes to provide your feedback. If you have questions or require technical assistance, please respond to this email at [EMIAL] or contact the [UTILITY] evaluation contractor at [PHONE NUMBER] / [EMAIL ADDRESS].
If you wish to no longer receive emails about this survey, please click on the “Unsubscribe” link below. Thank you in advance for your time!
Thank you in advance for your time!
Kind Regards,
Contractor to [UTILITY]
[OPT OUT LINK]
The company administering the survey conducts research to support research and evaluation in the energy sector. Survey data are not shared with third parties for marketing purposes. Our full privacy statement is linked here:
xxx.com/privacy

CI, Confidence interval
Table displays agreement as defined with a rating of 7 or higher on scale from 0 (strongly disagree) to 10 (strongly agree). The questions differed slightly in 2022 and 2023; the number of responses to each question is displayed in parenthesis.