Open-Ended Survey Questions: Item Nonresponse Nightmare or Qualitative Data Dream?

Angie L. Miller Indiana University Bloomington

Amber D. Dumford Indiana University Bloomington

Abstract

The purpose of this research was to explore whether those survey completers with certain demographic and personal characteristics, including gender, age, cohort, number of children, marital status, citizenship, race, current employment status, income, and institutional satisfaction level, are more or less likely to respond to open-ended questions placed at the beginning, middle, and end of an online alumni survey. Using data from the Strategic National Arts Alumni Project (SNAAP), a series of chi-squared and means comparisons analyses were done to compare whether or not respondents provided an answer to three different open-ended questions throughout the survey. Findings suggest that there are some group differences in likelihood of response, which could be explained by time burden, negativity bias, and self-identification as “other.”

Background

Surveys are widely used in higher education (Kuh and Ikenberry 2009; Porter 2004), and alumni surveys have become an important tool for programmatic and institutional assessment. Unfortunately, alumni surveys often have low response rates because of bad contact information and other reasons such as suspicion of money solicitation or decreased loyalty after graduation (Smith and Bers 1987). Yet even with relatively few respondents, institutions may be able to glean information on important concerns of respondents in the form of qualitative data derived from open-ended survey questions (Geer 1991; Krosnick 1999). Although those collecting this qualitative data receive benefits, a largely recognized disadvantage of open-ended questions is the heavy burden on respondents (Dillman 2007). Existing research suggests that open-ended questions have much higher rates of item nonresponse than other types of survey items (Millar and Dillman 2012). Another concern is that even when one has many open-ended responses at hand, how well do the responses represent the opinions of the entire group? Are some types of respondents more likely to complete open-ended questions? Previous research has shown that some personal characteristics, such as language fluency and positive affect (Wallis 2012), can increase the likelihood of responding to open-ended questions. Survey mode can play a role in nonresponse on open-ended items, and research suggests that for online surveys there may be differences in nonresponse across types of devices (Lambert and Miller 2014; Peytchev and Hill 2010). The purpose of this study is to explore whether those with certain demographic and personal characteristics, including gender, age, cohort, number of children, marital status, citizenship, race, current employment status, income, and institutional satisfaction level, are more or less likely to respond to open-ended questions placed at the beginning, middle, and end of an online alumni survey.

Method

Participants

The data used for this study was from the 2011 administration of the Strategic National Arts Alumni Project (SNAAP). SNAAP is a multi-institution online alumni survey designed to obtain knowledge of arts education. The participants were 33,801 alumni from 57 different arts high schools, undergraduate, and graduate colleges or arts programs within larger universities. Participating institutions provided the researchers with population information, including name, email address, phone number, mailing address, degree level, cohort (year of graduation), and major/arts field. All alumni with email addresses were invited to participate. No more than five contact messages (initial email invitation plus up to four reminder emails) were sent to alumni; this data was collected from September 2011 to November 2011. Of those who responded, 2,606 were high school level alumni (8 percent); 23,607 undergraduate level alumni (70 percent); and 7,588 graduate level alumni (22 percent). Of these alumni, 38 percent were male, 62 percent female, and 0.2 percent transgender. The majority of alumni (87 percent) reported their ethnicity as Caucasian. The overall response rate was 18 percent, which was derived by dividing the total number of respondents by the total number of alumni contacted (minus undeliverable emails). The average institutional response rate was 21 percent, which was derived by calculating the response rate for each institution and averaging those response rates. Because these analyses compared respondents on questions at the beginning, middle, and end of the survey, in order to prevent any bias from partial respondents only those who completed the entire survey (did not drop out before making it to the end of the survey) were included. This lowered the eligible number to 27,212. The characteristics of these respondents remained consistent with the entire sample. The average duration for those who completed the survey was 28 minutes.

Materials

The measures were questions included in a larger survey administered to participants online. Participants were emailed an invitation including a link to the survey. Participants could log in multiple times, so they were not constrained to complete all questions during a single session. Participants were not required to answer any of the items; therefore, they could advance through the survey even if they did not respond to individual items throughout the instrument.

The open-ended questions included in the analyses were three different items, selected due to placement on the survey instrument. (SNAAP contains 11 different open-ended items overall.) One item was selected from near the beginning of the survey (appears as the 17th of 82 total questions), one from the middle (appears as the 44th of 82), and one from near the end (appears as the 80th of 82). The item from the near-beginning asked respondents if there was anything their institution could have done better to prepare them for further education or career; the middle item asked them to describe how their arts training is or is not relevant to their current work; and the near-end item asked them to describe any additional information about their education, life, and/or career that were not adequately covered on the survey. From each of these questions, a binary variable was created based on whether or not the respondent provided an answer. In order to be classified as providing an answer, the respondent had to enter at least one character in the accompanying text box.

To compare the characteristics of those who did provide responses to those who did not, the demographic and personal variables included gender, age group, graduation cohort, number of children, marital status, citizenship, race/ethnicity, current employment status, income, and institutional satisfaction level. Citizenship (i.e., whether or not respondent was a US citizen) was a binary variable. Age, graduation cohort, and number of children were ordinal variables that contained recoded group ranges. Race/ethnicity was a “check all that apply” question and therefore was made up of seven binary race/ethnic variables. Gender, marital status, and current employment status were categorical variables, made up of three, four, and seven response options, respectively. Income was an ordinal measure, using midpoints of ranges; overall institutional satisfaction was also ordinal, using a four-point scale from “Poor” to “Excellent.” For a complete list of items and response options, see the Appendix.

Analyses

A series of fourteen chi-squared analyses was done for each of the three open-ended question binary variables. The chi-squared analyses were run for gender, age group, graduation cohort, number of children, marital status, citizenship, each race/ethnicity option, and current employment status. Three independent samples t-tests were completed for institutional satisfaction and each of the open-ended question binary variables. Three nonparametric Mann-Whitney U tests were completed for each of the comparisons of income, as this variable used midpoints for recoding and the skewed variance violated the parametric assumptions of the independent samples t-test.

Results

Descriptive Statistics

In looking at the percentages of responses for the open-ended questions, there are much higher percentages of responses for the near-beginning and middle questions than for the near-end item, keeping in mind that only those who reached the end of the survey are included in this analysis. For the near-beginning question, 68 percent of respondents provided an answer. For the middle question, 79 percent of respondents provided an answer. For the near-end question, 24 percent of respondents provided an answer.

Chi-Squared Analyses

When looking at comparisons based on gender, the results indicated that females were significantly more likely to answer the near-beginning and middle questions, but for the near-end question there were no significant differences (see Table 1 for χ2 values). For age, those groups over 50 were significantly more likely than their younger counterparts to answer all three questions. For graduation cohort, a similar pattern occurs, with those graduating in or before the year 1990 being significantly more likely to answer all three questions. Furthermore, for marital status, those who are single were significantly less likely to answer all items, which relates to age as well, as many of those who are single are also younger. For number of children, those with no children under 18 dependent on them for support were more likely to answer all three questions. Looking at current employment status, those who were unemployed and looking for work, retired, or selected “other” (and had the opportunity to supply an answer in a corresponding “other” text box) were more likely to answer all three open-ended items. Those who reported they were US citizens when attending their institutions were also more likely to answer all three questions.

Table 1 Chi-squared values and response patterns.

Near-Beginning Item Middle Item Near-End Item

χ2 value Likely to Respond? χ2 value Likely to Respond? χ2 value Likely to Respond?
Gender 16.29*** Females more likely 50.57*** Females more likely 0.02
Age group 145.54*** Over 50 more likely 191.26*** Over 50 more likely 355.54*** Over 50 more likely
Graduation cohort 159.22*** 1990 and before more likely 177.45*** 1990 and before more likely 258.08*** 1990 and before more likely
Number of dependents 48.01*** No dependents more likely 19.74*** No dependents more likely 34.87*** No dependents more likely
Marital status 26.63*** Singles less likely 35.11*** Singles less likely 35.96*** Singles less likely
US citizenship 10.58** Citizens more likely 65.48*** Citizens more likely 20.94*** Citizens more likely
Race (white) 1.82 11.09** Whites more likely 2.71
Race (black) 13.51*** Blacks more likely 2.11 0.08
Race (American Indian) 4.69* American Indians more likely 0.90 6.01* American Indians more likely
Race (Asian) 49.74*** Asians less likely 55.53*** Asians less likely 32.59*** Asians less likely
Race (other) 55.53*** “Others” more likely 10.833** “Others” more likely 59.47*** “Others” more likely
Current employment status 256.42*** Unemployed, retired, and “other” more likely 268.34*** Unemployed, retired, and “other” more likely 375.31*** Unemployed, retired, and “other” more likely

*p<0.05, **p<0.01, ***p<0.001.

Some different patterns occur when looking at the binary race variables. White/Caucasian individuals were more likely to answer the middle item, while Black individuals were more likely to answer the near-beginning item. Furthermore, American Indians were more likely to answer the near-beginning and near-end items, but not the middle item. Asian individuals were consistently less likely to answer all three items, while interestingly those who selected the “other” race response option (some of whom also wrote in the “other” text box) were consistently more likely to answer all three items. No significant differences were found for Hispanic or Native Hawaiian respondents.

Means and Other Ordinal Comparisons

The results of the independent samples t-tests showed that those who answered the near-beginning and near-end questions were significantly less satisfied with their overall institutional experience (see Table 2 for test statistics). In looking at income (recoded into midpoints of ranges), the Mann-Whitney U test indicated that those who answered the open-ended questions had a significantly lower income that those who did not, which was consistent across all three questions (see Table 3 for test statistics).

Table 2 Group means comparisons for institutional satisfaction.

Answered Question? (Group Mean) Answered Question? (Group Mean) t value df Sig. Effect Size (d)

No Yes
Near-beginning item 3.57 3.40 20.33 19914.16 *** 0.26
Middle item 3.44 3.45 –0.907 27082 0.01
Near-end item 3.46 3.42 4.20 9767.14 *** 0.06

***p<0.001.

Table 3 Group mean rank comparisons for income.

Answered Question? (Mean Rank) Answered Question? (Mean Rank) Mann-Whitney U value Sig.

No Yes
Near-beginning item 13860.60 13173.39 74282028.0 ***
Middle item 12056.14 11307.10 54347479.5 ***
Near-end item 12099.86 11409.52 40097544.5 ***

***p<0.001.

Discussion

There are several potential explanations for the various patterns found in the results, many of which support previous research and survey methodology knowledge. Completing open-ended response options requires a greater amount of time and mental effort than most close-ended questions (Dillman 2007); thus, it is not surprising that those with no dependents, are retired or unemployed, and older are more likely to provide open-ended responses. Time burden seems to fall more heavily on certain groups than others. Therefore, survey designers must choose open-ended questions wisely if they wish for as many types of respondents as possible to complete them. They should also consider placing the most important open-ended questions toward the beginning, as these showed more responses than those near the end (although they should continue to avoid open-ended questions as the very first item in the survey, which can result in survey abandonment) (Dillman 2007).

Another explanation for the patterns of results may be that those with negative feelings are more likely to voice their opinions as comments in the open-ended items, using them as a platform for their complaints. This negativity bias has been found in research with workplace environments (Poncheri et al. 2008), and may explain why those who are unemployed and looking for work are more likely to respond to these items. These alumni in particular might be frustrated with their situation, and feel that their institution, who provided them with their degree, should shoulder some of the responsibility. Furthermore, those who provided open-ended responses had significantly lower levels of income and were significantly less satisfied with their institutional experience, compared to those who left the questions blank. It seems that the disgruntled alumni are more willing to spend the time and effort to provide a response to the open-ended questions. While this might initially seem like an unwanted result, it may actually be that these alumni have the best insight on improvements to curriculum and programming. However, given the potential negativity bias, survey researchers need to carefully craft the stems of open-ended questions to illicit both valences of response.

A third and quite interesting pattern was found concerning the use of the “other” response option, as those individuals who prefer to describe themselves as “other” are also more likely to respond to open-ended questions throughout the survey. A cursory review of the open-text boxes that accompany the “other” employment and race options shows a considerable number of responses that actually do fall into one of the categorizations, but the respondents choose to further explicate on themselves. For instance, some respondents reported their “other” race as things like “Caucasian/American Indian,” even though the race/ethnicity question was in a check-all-that-apply format and one could have simply checked both of these response options. It seems that those respondents choosing to identify themselves as an “other,” thinking of themselves as a unique individual, are more likely to provide responses to open-ended questions. Perhaps they are simply more verbose, or they have a disposition that resists the confinement of categorization. It is possible that this “other” preference is a distinct response style, and more qualitative analysis of the “other” text boxes is needed.

Although there are several strengths of this study, some limitations should be noted. Given the data collection procedures and response rates, the sample may not be representative of all arts alumni, or alumni in general, and caution should be made when making generalizations. More sophisticated analyses, such as logistic regression, could shed further light on which combinations of groups can predict the likelihood of responses in larger, more diverse samples. One particular source of bias may be that alumni who are either extremely dissatisfied or satisfied are more likely to respond to the survey itself, although follow-up studies on SNAAP pilot tests indicate that these differences are negligible (Kennedy et al. 2010). There may be differences between partial and full completers, although our sample did not differ substantially on key characteristics.

Overall, the findings suggest that while a great deal of information can be gained from open-ended survey questions, some groups are more likely than others to provide responses, and this should be kept in mind when designing future surveys and interpreting one’s qualitative survey results. More research is also needed to explore the patterns of results concerning the influence of question placement and issues such as type of device on whether particular groups respond to open-ended questions, as well personal and environmental influences that may contribute to this “other” survey response style.

References

Dillman 2007
Dillman, D.A. 2007. Mail and internet surveys: the tailored design method (2nd ed.). John Wiley & Sons, Inc., New York.
Geer 1991
Geer, J.G. 1991. Do open-ended questions measure “salient” issues? Public Opinion Quarterly 55(3): 360–370.
Kennedy et al. 2010
Kennedy, J., S.J. Tepper and A.D. Lambert. 2010. An analysis of mode effects in three survey modes in the Strategic National Arts Alumni Project. In: Proceedings of the 2nd International Workshop on Internet Survey Methods. Daejeon, South Korea.
Krosnick 1999
Krosnick, J. 1999. Survey research. Annual Review of Psychology 50(1): 537–567.
Kuh and Ikenberry 2009
Kuh, G.D. and S.O. Ikenberry. 2009. More than you think, less than we need: Learning outcomes assessment in American higher education. National Institute of Learning Outcomes Assessment, University of Illinois and Indiana University, Urbana, IL.
Lambert and Miller 2014
Lambert, A.D. and A.L. Miller. 2014. Living with smartphones: Does completion device affect survey responses? Research in Higher Education. doi: 10.1007/s11162-014-9354-7.
Millar and Dillman 2012
Millar, M.M. and D.A. Dillman. 2012. Do mail and internet surveys produce different item nonresponse rates? An experiment using random mode assignment. Survey Practice 5(2). http://www.surveypractice.org/index.php/SurveyPractice/article/view/48/html.
Peytchev and Hill 2010
Peytchev, A. and C.A. Hill. 2010. Experiments in mobile web survey design: similarities to other modes and unique considerations. Social Science Computer Review 28(3): 319–335. doi: 10.1177/0894439309353037.
Poncheri et al. 2008
Poncheri, R.M., J.T. Lindberg, L.F. Thompon and E.A. Surface. 2008. A comment on employee surveys: negativity bias in open-ended responses. Organization Research Methods 11(3): 614–630. doi: 10.1177/1094428106295504.
Porter 2004
Porter, S.R. 2004. Raising response rates: What works? New Directions for Institutional Research 2004(121): 5–21.
Smith and Bers 1987
Smith, K. and T. Bers. 1987. Improving alumni survey response rates: an experiment and cost-benefit analysis. Research in Higher Education 27(3): 218–225.
Wallis 2012
Wallis, P. 2012. Profiling college students who skip open-ended items win questionnaires with varied item formats. In: Proceedings of the Annual Meeting of the American Educational Research Association. Vancouver, British Columbia: AERA.

Appendix. List of Variables from SNAAP Survey Instrument Used in Analyses

Item # Variable Question Text Response Values and Labels
9 instexp Now, back to your time at [INSTITUTION]. Overall, how would you rate your experience at [INSTITUTION] while pursuing your [HIGH SCHOOL/UNDERGRADUATE/GRADUATE] degree? 1=Poor
2=Fair
3=Good
4=Excellent
17 edpreptxt Is there anything that [INSTITUTION] could have done better to prepare you for further education or for your career? Text box
Please describe.
42 curemp What is your currenr employment status? 1=Full-time (35 hours or more per week)
2=Part-time only (fewer than 35 hours per week)
3=Unemployed and looking for work.
4=In school full time
5=Caring for family full time
6=Retired
7=Other
curempothtxt Write in value: Other employment status Text box
44 wktraintxt Please describe how your arts training is or is not relevant to your current work. Text box
59 income What was your individual annual income in 2010? (Do not include spousal income or interest on jointly-owned assets.) 1=$10,000 or less
2=$10,001 to $20,000
3=$20,001 to $30,000
4=$30,001 to $40,000
5=$40,001 to $50,000
6=$50,001 to $60,000
7=$60,001 to $70,000
8=$70,001 to $80,000
9=$80,001 to $90.000
10=$90,001 $100,000
11=$100,001 to $150,000
12=More than $150,000
13=I prefer not to answer
* income_R Recoded from question 59 to remove 13 values and use midpoints of range as value: What was your individual annual income in 2010? (Do not include spousal income or interest on jointly-owned assets.) Number box
68 gender What is your gender? 1=Male
2=Female
3=Transgender
69 age What is your age? In years: Number box (no decimals)
* age_R Recoded from question 69 to put in ranges: What is your age? 1=24 or younger
2=25 to 29
3=30 to 39
4=40 to 49
5=50 to 59
6=60 or older
72 marital What is your marital status? 1=Single (never married)
2=Married or domestic partner
3=Divorced/separated
4=Widowed
73 dependent How many children (under 18) live with you or are dependent on your income for support? Enter total number: Number box (no decimals)
* children Recoded fiom question 73 to put in ranges: How many children (under 18) who live with you or are dependent on your income for support? 0=No dependents
1=1
2=2
3=3 or more
74: What is your race/ethnicity? Check all that apply.
74.1 race_white White or Caucasian
74.2 race_blck Black or African American
74.3 race_hisp Hispanic, Latino, or Spanish origin 0=Not Selected
74.4 race_amerind American Indian or Alaska Native 1=Selected
74.5 race_haw Native Hawaiian or other Pacific Islander
74.6 race_asian Asian
74.7 race_oth Other
race_othtxt Write in value: Other race-ethnicity Text box
75 citizen While enrolled at [INSTITUTION] were you a US citizen? 0=No
1=Yes
80 finalcomments If there are additional things you would like to tell us about your education, life, and/or career that were not adequately covered on the survey, please do so here. (Please note that you also have a chance to make comments to SNAAP about the questionnaire and the project below.) Text box
* Cohort Last year attended reported by school in alumni file Number box
* Cohort_R Recode of cohort: Last year attended in ranges 1=1980 and before
2=1981–1990
3=1991–1995
4=1996–2000
5=2001–2005
6=2006–2010


About Survey Practice Our Global Partners Disclaimer
The Survey Practice content may not be distributed, used, adapted, reproduced, translated or copied for any commercial purpose in any form without prior permission of the publisher. Any use of this e-journal in whole or in part, must include the customary bibliographic citation and its URL.