So You Want to Survey a State Legislator? Call Me, Maybe

Introduction^[1]

In 2021, New Jersey enacted into law a bipartisan-endorsed bill establishing a first-in-the-nation statewide account of the gender, race, and ethnicity of elected and appointed officials. The Center for American Women and Politics and the Eagleton Center for Public Interest Polling —both at the Eagleton Institute of Politics at Rutgers, The State University of New Jersey— were tasked with collecting the information following enactment. Former New Jersey State Senator Loretta Weinberg, one of the bill’s sponsors, argued the legislation provided greater transparency of those in charge: “We often talk in politics and policy about making sure ‘everyone has a seat at the table.’ Right now, that is not the case in New Jersey. This legislation will begin to change that gross oversight once and for all.”^[2] Despite the near-unanimous support from lawmakers^[3] and the legislation’s laudable goals, we faced two problems: 1) how and where to reach this elite population and 2) how to compel them to complete the five-minute survey.

The survey design employed a mixed mode recruitment strategy, using a customized multi-contact sample frame assembled from publicly available information.^[4] The survey included 16 questions (see Appendix 1). Importantly, only demographic questions were asked, not anything political or explicitly ideological.

The target population for the project authorized the establishment of two censuses: one for elected municipal, county, and state officials, excluding local school boards, and another for appointed officials serving terms in 2022. The analysis presented here focuses on the 120 members of the New Jersey State Legislature (80 state assembly members, 40 state senators) for two reasons. First, we expected a high cooperation rate from these members, given near unanimous support for the legislation. Second, we had contact information for all state legislators; thus, we can attribute a lack of compliance with the law to nonresponse rather than coverage error.

Our initial recruitment wave utilized publicly available legislative emails for all members. Two recruitment attempts yielded a cumulative 24% response rate (Table 1), despite fielding while both chambers were in session (see Appendix 2). Given the survey’s short length and wide legislative support for the project, the response rate observed was abysmal. This lackluster response raised suspicion that emails, like postal mail, were filtered by full-time staffers responsible for correspondence in the district offices on behalf of part-time legislators.

To gain a figurative foot in the door, we altered our strategy among the nonrespondents, implementing phone outreach to those members’ Chiefs of Staff and legislative aides. The calls followed a loose script (see Appendix 3), treating the calls as a conversation. To confer the legitimacy of the survey with the staffer answering the phone, we relied upon the name recognition of “Rutgers” and “Eagleton.” We then emphasized the survey was authorized by the legislature, and when applicable, we referenced the member’s vote in support of the bill. After the first few calls were greeted with some reluctance, we altered our script to explicitly and emphatically state the survey was purely demographic in nature, with no political or policy questions included, and could be completed in under five minutes. The combined brand recognition and assurances about the content and timing, anecdotally, appeared to generate more receptivity from the person answering the phone.

These conversations served a two-fold purpose: we confirmed that legislators themselves infrequently accessed their legislative emails and that staffers were gatekeepers, who, in turn, served as valuable facilitators for our recruitment efforts. Given the indirect use of official legislative email accounts by members, we inquired if we could re-forward the survey directly to the staffer with whom we were speaking with, to assist us in obtaining the legislator’s completion. By circumventing the normal email routing process, which likely obscured our previous email recruitment efforts given the volume of emails received, we estimate this personalized outreach strategy more than doubled our response rate.

It is likely that endogenous factors influenced the overall response rate among some legislators. During fielding, 35% of members were women and more than 55% of members in both chambers were Democrats. Seventy-one percent of women completed the survey compared to 47% of men. No significant difference was observed by race or ethnicity among respondents.

Table 2 below highlights three important points. First, calling was important. Men and women in the legislature were more likely to respond following the direct calls; a majority of those who completed the survey did so following the calling wave. Second, follow-up emails had mixed efficacy. Men were less responsive to the follow-up, compared to women. Women responded to the follow-up email at a slightly higher rate compared with the original. Finally, and unsurprisingly, repeated contact, in whatever form, was crucial.

While email has become ubiquitous and is touted as a way constituents can contact legislators, it may not be the best mode of recruitment. Calling the office directly can bypass roadblocks associated with the volume of emails received. We observed that calling offices greatly improved the response rate compared with the less personalized emails. Future research involving elected officials should be attentive to the tradeoffs here, balancing increased response rates with additional costs associated with direct and personal recruitment.

Appendices

Appendix 1: Survey Instrument

Introduction:

On January 18th, 2022, Governor Phil Murphy signed a law that was unanimously passed by the legislature (P.L.2021, c.414. S4004), which establishes a database with key demographic information for all elected and appointed officials in New Jersey.

Under the statute, the Center for American Women and Politics and the Eagleton Center for Public Interest Polling, at the Eagleton Institute of Politics at Rutgers, the State University of New Jersey, have been directed to collect this information. The data will be made available in a public database and will be used in a public report summarizing the demographics of elected officials in the state of New Jersey.

Your participation will help ensure compliance with the legislation’s directives as well as accuracy of information reported. This survey will take less than 5 minutes.

Please click “NEXT” to continue.

C1:
First, please confirm some information about yourself.
Your name is {Pipe in name}. Is this correct?

1. Yes
2. No (please correct your name in the space provided)
   [Open ended text box]

C2:
The office you currently hold is {Pipe in office} in {Pipe in municipality}. Is this correct?

1. Yes
2. No (please correct your office information in the space provided)
   [Open ended text box]

C3:

Your office mailing address is:
{Pipe in office address}
Is this correct?

1. Yes
2. No (please correct your office mailing address in the space provided)
   [Open ended text box]

C4:
Your official government email address is {Pipe in email address}. Is this correct?

1. Yes
2. No (please correct your email address in the space provided)
   [Open ended text box]

C5:
Your office phone number is {Pipe in phone number}. Is this correct?

1. Yes
2. No (please correct your phone number in the space provided)
   [Open ended text box]

PLEASE CHECK OVER CAREFULLY BEFORE SUBMITTING. YOU WILL NOT BE ABLE TO GO BACK.

T1:
When did you first start serving in your current office?
              Term start: [Month] [Year]

T2:
When does your current term expire?
              Term end: [Month] [Year]

Hispanic:
Are you Hispanic, Latino, or Spanish Origin?

1. No, not Hispanic, Latino, or Spanish origin
2. Yes, Mexican American or Chicano
3. Yes, Puerto Rican
4. Yes, Cuban
5. Yes, another Hispanic, Latino, or Spanish origin (please specify)
   [Open ended text box]

Race:
Listed below are U.S. Census racial categories. What is your race? Please select all that apply.

1. White

2. Black or African American

3. Middle Eastern/Northern African

4. Chinese

5. Filipino

6. Asian Indian

7. Vietnamese

8. Korean

9. Japanese

10. Other Asian (please specify [Open ended text box]

11. Native Hawaiian

12. Samoan

13. Chamorro

14. Other Pacific Islander (please specify) [Open ended text box]

15. Other (please specify [Open ended text box]

Gender:
Do you describe yourself as a man, a woman, or in some other way?

1. Man
2. Woman
3. Other (please specify)
   [Open ended text box]

Transgender:
Do you identify as transgender?

1. Yes
2. No

Appendix 2: Recruitment Emails

May 19, 2022 Recruitment Email

{Date}

Dear {Name},

The Eagleton Center for Public Interest Polling is asking for your help with a short questionnaire about elected officials. Your participation is incredibly important. The survey will take less than 5 minutes.

Per P.L.2021, c.414. S4004, passed unanimously by the legislature and signed into law by the governor on January 18, 2022, the Center for American Women and Politics and the Eagleton Center for Public Interest Polling, at the Eagleton Institute of Politics at Rutgers, the State University of New Jersey, has been directed to establish a database with key demographic information of certain appointed positions and elected offices. The information provided here will be available in a public database and will be used in a public report on the demographics of elected officials in the state of New Jersey.
Your participation will ensure compliance with the legislation’s directives as well as accuracy of information reported.

PARTICIPATE HERE:
${l://SurveyLink?d=Take the Survey}

If you have any questions about this research project, you may contact Dr. Ashley Koning, director of the Eagleton Center for Public Interest Polling, by email at poll@eagleton.rutgers.edu or by phone at 848-932-8995. If you have any questions about your rights as a research participant, you can contact the Institutional Review Board at Rutgers University (which is a committee that reviews research studies in order to protect research participants) at: Arts & Sciences Institutional Review Board, Rutgers University, the State University of New Jersey. Liberty Plaza / Suite 3200 335 George Street, 3rd Floor, New Brunswick, NJ 08901, 732-235-2866, humansubjects@orsp.rutgers.edu.

Many thanks,

 

Ashley Koning, Ph.D.
Assistant Research Professor
Director, Eagleton Center for Public Interest Polling
Eagleton Institute of Politics
Rutgers, The State University of New Jersey
Wood Lawn, 191 Ryders Lane
New Brunswick, NJ 08901
Office: 848.932.8940
poll@eagleton.rutgers.edu
eagletonpoll.rutgers.edu
www.facebook.com/RutgersEagletonPoll
Twitter @EagletonPoll
Follow the link to opt out of future emails:
${l://OptOutLink?d=Click here to unsubscribe}

June 14, 2022 Recruitment/Reminder Email

Your participation will ensure compliance with the legislation’s directives as well as accuracy of information reported.

PARTICIPATE HERE:
${l://SurveyLink?d=Take the Survey}

If you have any questions about this research project, you may contact Dr. Ashley Koning, director of the Eagleton Center for Public Interest Polling, by email at poll@eagleton.rutgers.edu or by phone at 848-932-8995. If you have any questions about your rights as a research participant, you can contact the Institutional Review Board at Rutgers University (which is a committee that reviews research studies in order to protect research participants) at: Arts & Sciences Institutional Review Board, Rutgers University, the State University of New Jersey. Liberty Plaza / Suite 3200 335 George Street, 3rd Floor, New Brunswick, NJ 08901, 732-235-2866, humansubjects@orsp.rutgers.edu.
Many thanks,

 

Ashley Koning, Ph.D.
Assistant Research Professor
Director, Eagleton Center for Public Interest Polling
Eagleton Institute of Politics
Rutgers, The State University of New Jersey
Wood Lawn, 191 Ryders Lane
New Brunswick, NJ 08901
Office: 848.932.8940
poll@eagleton.rutgers.edu
eagletonpoll.rutgers.edu
www.facebook.com/RutgersEagletonPoll
Twitter @EagletonPoll
Follow the link to opt out of future emails:
${l://OptOutLink?d=Click here to unsubscribe}

Appendix 3: Calling script template

Hello, I am [NAME] calling from the Eagleton Center for Public Interest Polling at Rutgers University. We are calling to request the Senator/Assemblyman’s help with implementation of S-4004, which establishes a public database with key demographic information of appointed and elected officials. We’re just collecting demographic information, there are no political or policy questions. We sent your office an email invitation to a survey that includes the information about your member that we just need to be reviewed and confirmed.

Would you be able to confirm that you receive the email request?

Is there an alternative email we should use for your member or a staff member in order to review this request.