The Veterans Health Administration (VHA) All Employee Survey (AES) is a voluntary, confidential annual survey of employee job satisfaction and employee perceptions of workplace climate. Survey researchers (e.g., Rogelberg and Stanton 2007; Stocke 2006) recommend offering several response modes to a survey. This serves to increase participation of different demographic groups and obtain results that adequately represent them all. The practice of offering Web, Interactive Voice Response (IVR) and paper response options has been adopted for the AES, but the majority of respondents across the survey years consistently choose the Web (over 75% since 2004, over 90% since 2008). The AES thus can be seen as predominantly a Web survey. The data obtained in all of the three modes are combined prior to analyzing and reporting the results. The objective of this paper was to examine if respondents selecting each of the modes differed in demographic characteristics, in how many questions they answered, and in the answers they gave.

The AES serves to systematically track employee perceptions of the organization, communicate them clearly between employees and management, and then to hold leaders accountable for articulating action plans and implementing improvements informed by the results of the system assessments (Young 2000). AES results are broadly disseminated within the organization and included in action plans at all of the VHA facilities, at the regional level, and nationally. Performance standards for VHA managers and executives require broadly sharing AES results with their workforce and planning specific improvements based on the survey information. Such an important organizational role of the AES requires that the AES results be representative of the different demographic groups of employees.

Given the increased popularity of Web-based tools and the decreased use of paper surveys within organizations, we also examined how any differences changed over time. The AES dataset has several features generally rare in organizational surveys. These features include data for the same survey that are available for several years (2004, 2006, annually thereafter) and for several modes (paper, IVR, Web-based), for multiple local organizations (hospitals, clinics, offices) all within the same organizational structure (VHA). In addition, the VA is the 2nd largest federal agency in the United States and its workforce is extremely broad and diverse. For example, the VHA AES workforce includes surgeons, policemen, engineers, canteen workers, pharmacists, photographers, clerks, designers and many other occupations, spread over all of the 50 states. AES ratings of the workplace environment thus reflect a rich variety of geographically, demographically and professionally different perspectives. We believe that these features of the AES dataset make for an informative comparison of survey results across administration modes. These results generalize to many contexts where organizational surveys are used.

Method

Data collection

The annual administration of the AES is preceded by an organization-wide marketing where all VHA employees are encouraged to take part in the AES. Through a single, widely distributed instruction document, employees receive detailed information on how to access each of the 3 modes: IVR, paper, and Web-based. Access to the Web-based questionnaire is through a URL maintained by a contractor. IVR access is through a toll free telephone number. Employees selecting the paper mode have to contact a site coordinator at each location to receive the questionnaire and a postage paid envelope to return it. In all cases, employees are given a 7-digit work group code to be entered on the questionnaire and are instructed to also enter an occupation code from a list. They are given assurance that responses would be confidential. The confidentiality of responses is strictly maintained throughout the data collection, analysis, and reporting processes. The AES is open for 3 weeks. After the survey closes, data are processed and reported to the organization within one month.

Data analysis

We calculated response rates using The American Association for Public Opinion Research (2008) RR2 method. Surveys with at least 80% of the non-demographic questions answered were considered ‘complete’, and surveys with at least 50 percent but less than 80% of the non-demographic questions answered were designated as ‘partial’. Only ‘complete’ and ‘partial’ questionnaires appear in the numerator of the response rates. The denominator of the response rates was the total number of VHA employees at the time the survey was offered. We used Cohen’s D (Cohen 1988) to calculate the effect size between Web-based versus IVR and Web-based versus paper modes for the 58 survey questions for all respondents and also for selected demographic subgroups. An effect size greater than or equal to 0.2 was considered to be of practical significance. We used the t-test (p(t)≤0.05) to determine the statistical significance of mean differences between Web-based versus IVR and Web-based versus paper modes for the 58 survey questions for all respondents and for demographic subgroups.

Results

Overall, from 2004 through 2008, the number of completed AES questionnaires increased from 104,155 to 163,831 (Table 1). Response rates based on the number of VHA employees were 51.5%, 69.7%, and 72.4% for 2004, 2006, and 2008, respectively. During this interval, the proportion of respondents selecting IVR or paper modes decreased, and the proportion selecting Web-based surveys increased.

Table 1  Mode selection by respondents to the VHA All Employee survey in years 2004, 2006, 2008.
2004
IVR
 Paper
 Web
 Total
n Row % n Row % n Row %
All 13,958 13.40 9,750 9.36 80,447 77.24 104,155
Sex Male 4,818 12.6 4,837 12.65 28,591 74.76 38,246
Female 8,274 13.05 4,632 7.3 50,516 79.65 63,422
Age group (years) <20 134 34.72 20 5.18 232 60.1 386
20–29 525 10.06 420 8.05 4,275 81.9 5,220
30–39 1,836 11.4 1,198 7.44 13,078 81.17 16,112
40–49 4,162 12.47 2,904 8.7 26,313 78.83 33,379
50–59 5,136 13.45 3,847 10.07 29,202 76.48 38,185
>=60 1,284 15.53 1,058 12.8 5,925 71.67 8,267
Hispanic Yes 1,137 15.3 1,172 15.77 5,122 68.93 7,431
no 11,910 12.79 8,155 8.75 73,087 78.46 93,152
Race American 134 11.54 121 10.42 906 78.04 1,161
Indian
Asian 512 10 325 6.35 4,283 83.65 5,120
Balck 2,755 13.97 3,138 15.91 13,827 70.12 19,720
Multiracial 1,737 53.78 203 6.28 1,290 39.94 3,230
Native 57 7.85 75 10.33 594 81.82 726
Hawaiin
White 7,091 10.49 2,506 7.7 55,278 81.8 67,575
Occupation gp Administration 4,740 10.39 2,632 5.77 38,249 83.84 45,621
MD 611 9.61 417 6.56 5,332 83.84 6,360
Nurse 3,874 14.71 1,328 5.04 21,134 80.25 26,336
Other clinical 1,544 12.3 769 6.13 10,238 81.57 12,551
Unknown 1,078 49.11 796 36.26 321 14.62 2,195
Wage grade 2,111 19.03 3,808 34.33 5,173 46.64 11,092
Time with VHA < 6 months 567 15.17 435 11.64 2,735 73.19 3,737
6–12 months 910 14.79 583 9.48 4,660 75.74 6,153
1–3years 1,729 11.1 1,436 9.22 12,409 79.68 15,574
4–5 years 1,096 12.07 767 8.45 7,216 79.48 9,079
6–10 years 1,709 12.76 1,232 9.2 10,453 78.04 13,394
11–20 years 3,958 13.04 2,775 9.14 23,630 77.82 30,363
>20 years 2,940 12.54 2,277 9.72 18,219 77.74 23,436
Supervisory level None 8,724 12.05 7,327 10.12 56,347 77.83 72,398
First line 2,448 17.48 1,079 7.7 10,479 74.82 14,006
Manager 990 12.34 611 7.62 6,422 80.04 8,023
Executive 571 9.49 321 5.34 5,122 85.17 6,014
Senior exec. 146 11.96 89 7.29 986 80.75 1,221
2006
IVR
 Paper
 Web
 Total
n Row % n Row % n Row %
All 12,527 8.54 8,251 5.63 125,848 85.83 146,626
Sex Male 4,664 8.67 3,925 7.29 45,216 84.04 53,805
Female 7,220 8.29 3,065 3.52 76,830 88.19 87,115
Age group (years) <20 171 41.5 23 5.58 218 52.91 412
20–29 564 7.5 321 4.27 6,632 88.23 7,517
30–39 1,542 7.06 771 3.53 19,515 89.4 21,828
40–49 3,371 7.94 1,934 4.55 37,177 87.51 42,482
50–59 4,880 8.79 3,065 5.52 47,594 85.69 55,539
>=60 1,352 10.39 839 6.45 10,823 83.16 13,014
Hispanic
Yes 1,112 10.7 824 7.93 8,455 81.37 10,391
No 10,757 8.34 6,007 4.66 112,171 87 128,935
Race American 27 1.93 83 5.92 1,291 92.15 1,401
Indian
Asian 20 0.27 216 2.89 7,248 96.85 7,484
Black 193 0.76 2,294 8.98 23,048 90.26 25,535
Multiracial 11,010 76.31 122 0.85 3,296 22.84 14,428
Native 25 2.2 74 6.5 1,039 91.3 1,138
Hawaiin
White 145 0.17 3,856 4.53 81,085 95.3 85,086
Occupation gp Administration 4,068 6.58 1,998 3.23 55,752 90.19 61,818
MD 630 6.67 355 3.76 8,465 89.58 9,450
Nurse 3,478 8.83 1,431 3.63 34,500 87.54 39,409
Other clinical 1,373 7.66 538 3 16,003 89.33 17,914
Unknown 199 33 404 67 0 0 603
Wage grade 2,779 15.94 3,525 20.22 11,128 63.84 17,432
Time with VHA < 6 months 557 11.42 270 5.53 4,052 83.05 4,879
6–12 months 609 10.15 325 5.42 5,064 84.43 5,998
1–3years 1,682 6.88 1,213 4.96 21,564 88.16 24,459
4–5 years 1,071 7.49 699 4.89 12,530 87.62 14,300
6–10 years 1,693 8.19 1,049 5.08 17,927 86.73 20,669
11–20 years 3,326 8.68 1,850 4.83 33,147 86.49 38,323
>20 years 2,878 8.8 1,590 4.86 28,234 86.34 32,702
Supervisory level None 8,268 8.02 5,523 5.35 89,348 86.63 103,139
First line 2,169 11.37 804 4.21 16,104 84.42 19,077
Manager 763 7.56 392 3.89 8,931 88.55 10,086
Executive 474 6.16 175 2.27 7,051 91.57 7,700
Senior exec. 130 8.63 41 2.72 1,336 88.65 1,507
2008
IVR
 Paper
 Web
 Total
n Row % n Row % n Row %
All 8,334 5.09 3,913 2.39 151,584 92.52 163,831
Sex Male 3,564 6.03 1,764 2.98 53,791 90.99 59,119
Female 4,361 4.42 1,079 1.09 93,319 94.49 98,759
Age group (years) <20 107 26.03 13 3.16 291 70.8 411
20–29 403 3.84 198 1.89 9,893 94.27 10,494
30–39 1,038 4.01 280 1.08 24,553 94.91 25,871
40–49 2,036 4.53 701 1.56 42,206 93.91 44,943
50–59 3,228 5.5 1,201 2.05 54,216 92.45 58,645
>=60 1,107 6.44 436 2.54 15,648 91.02 17,191
Hispanic Yes 847 6.27 250 1.98 11,515 91.3 8,334
No 7,067 4.93 2,534 1.77 133,692 93.3 151,584
Race American 86 5.03 43 2.51 1,581 92.46 1,710
Indian
Asian 276 3.02 100 1.09 8,764 95.89 9,140
Black 2,171 6.83 995 3.13 28,635 90.04 31,801
Multiracial 976 19.32 53 1.05 4,024 79.64 5,053
Native 30 2.11 40 2.82 1,349 95.07 1,419
Hawaiin
White 3,833 3.76 1,486 1.46 96,509 94.78 101,828
Occupation gp Administration 2,466 3.78 890 1.36 61,890 94.86 65,246
MD 368 3.49 161 1.53 10,021 94.99 10,550
Nurse 1,850 4.1 583 1.29 42,682 94.61 45,115
Other clinical 812 3.75 213 0.98 20,642 95.27 21,667
Unknown 137 35.49 249 64.51 0 0 386
Wage grade 2,701 12.94 1,817 8.71 16,349 78.35 20,867
Time with VHA < 6 months 481 5.26 211 2.31 8,447 92.43 9,139
6–12 months 717 5.77 259 2.08 11,461 92.15 12,437
1–3years 1,076 4.1 514 1.96 24,658 93.94 26,248
4–5 years 760 4.33 285 1.63 16,493 94.04 17,538
6–10 years 1,252 4.92 451 1.77 23,731 93.3 25,434
11–20 years 1,742 5.08 558 1.63 32,000 93.29 34,300
>20 years 1,858 5.61 568 1.72 30,690 92.67 33,116
Supervisory level None 5,685 4.85 2,269 1.93 109,377 93.22 117,331
First line 1,366 6.71 312 1.53 18,667 91.75 20,345
Manager 481 4.48 147 1.37 10,105 94.15 10,733
Executive 283 3.39 69 0.83 7,990 95.78 8,342
Senior exec. 66 3.95 25 1.5 1,578 94.55 1,669

Mode preferences for demographic subgroups

For 2004, 2006, and 2008, males were more likely to select paper and less likely to select Web-based questionnaires than females. In 2004, males were less likely to select the IVR mode than females were, but more likely than females to select this mode in 2006 and 2008.

For all years, the less-than-20 age group selected the IVR mode much more frequently than the other age groups. For all years, selection of paper and IVR over Web-based modes tended to increase with age-group.

For all years, Hispanic respondents were more likely to select the IVR and paper modes and less likely to select the Web-based mode than non-Hispanics. These differences narrowed between 2004 and 2006 and again between 2006 and 2008.

Mode selection for the 6 racial classifications was variable among groups and over years. In general Blacks and American Indians were similar in their mode preference, as were Asians and Whites. Those respondents self-classifying as Multi-racial were much more likely to select IVR over the paper and Web-based modes.

Within years, mode selection was similar among groups based on length of time working at the VA and based on supervisory level.

For all years, there was little variation in mode selection among occupation groups except for wage grade employees (hourly paid occupations that typically do not require extensive professional training, such as housekeepers or canteen workers). Wage grade employees were much more likely to select the IVR and paper modes and less likely to select the Web-based modes than other occupation groups. These differences narrowed between 2004 and 2006 and between 2006 and 2008.

Mode preference and number of demographic and non-demographic responses missing

In 2004, respondents selecting each of the 3 modes had similar patterns of the number of missing responses for the 7 demographic questions. In 2006 and 2008, however, respondents selecting the paper mode had a higher proportion of unanswered demographic questions than respondents selecting the IVR and Web-based modes (Figures 1–3).

Figure 1  Relative frequency of demographic questions missing out of 7, 2004 AES.
Figure 2  Relative frequency of demographic questions missing out of 7, 2006 AES.
Figure 3  Relative frequency of demographic questions missing out of 7, 2008 AES.

For all years, respondents selecting the IVR mode had a higher proportion of questionnaires with more than 5 non-demographic questions missing out of 58 than respondents selecting Web-based surveys (Figure 4). Respondents selecting the paper mode had intermediate proportions of surveys with more than 5 non-demographic responses missing.

Figure 4  Percent of surveys with more than 5 non-demographic questions missing out of 58, AES.

Mode preference and effect size for non-demographic questions across subgroups

For all years, all respondents and 58 question means, there were no questions with effect size (Web-based versus paper modes) ≥0.2. There were 4, 7, and 15 questions with effect size ≥0.2 (Web-based versus IVR modes) in 2004, 2006, and 2008, respectively. Only one question with effect size ≥0.2 (Web-based versus IVR modes) in each of 2006 and 2008 had means that were statistically significantly different.

Statistically significant effect sizes ≥0.2 were more common among the demographic subgroups studied (males, females, <20 year olds, >60 year olds, whites, blacks, <6 month tenure, >20 years tenure, supervisors, and non-supervisors) than in the overall VHA population. Among the demographic subgroups, there were more statistically significant effect sizes ≥0.2 for the Web vs. IVR contrast compared to the Web vs. paper contrast.

Discussion

This paper sought to determine if respondents selecting each of the modes differed in demographic characteristics, in how many questions were answered, and in the answers they gave. We also wanted to know how any differences changed over time.

Over the 6 year period reported here, respondents to the VA’s AES were increasingly likely to select the Web-based mode when given the option of using Web-based, IVR and paper modes. Possible reasons for this trend include increasing familiarity with the Internet and computers among the general population, the relative ease of taking a survey electronically, and efforts by the VA to increase access to computers for employees who do not regularly use them in their jobs. Regardless of the reasons for this trend, if it continues, the VA might be able to consider eliminating the paper and/or IVR modes because the loss of information from the reduced number of completed questionnaires would be small in relation to the costs of offering these modes.

The mode preferences among the subgroups within the major demographic categories were similar for each year with few exceptions. The <20 year old age group was much more likely to select the IVR mode than the other age groups. Wage grade employees were more likely to select the IVR and telephone modes than were the other occupation groups. One possible explanation for these preferences could be that both groups might have more limited access to a computer. The higher preference of multiracial respondents for the IVR compared to other racial groups is difficult to explain, especially since it consistently occurs in all years. If only the Web-based mode were used for AES, it is possible that at least some of the information for the <20 year old, wage grade, and multiracial employees would not have been captured. Of course, it is not possible to determine how many employees choosing the IVR and paper modes would have used the Web-based mode if it were the only one available.

The number of missing demographic questions out of a total of 7 was similar among modes in 2004. In 2006 and again in 2008, the proportion of surveys with 6 or more demographic questions missing increased dramatically for respondents selecting the paper mode. Because demographic questions are placed at the end of the questionnaire, a possible explanation for the increase in demographic questions missing for the paper mode is respondent fatigue. However, the proportion of paper surveys with more than 5 non-demographic questions missing increased only slightly from 2004 to 2008, and this proportion was less than that for the IVR mode. It is unlikely that respondent fatigue would affect only the demographic questions and not the non-demographic questions, and only for the IVR mode. Another possible explanation is that as the number of respondents selecting the paper mode decreases, the employees still using it are somehow less likely to answer demographic questions. These employees become a larger relative proportion of the paper responses and thus so does the number of questionnaires with ≥6 missing demographic questions. Again, if this occurred, a similar effect should be seen among the IVR surveys, but it is not.

Based on effect size, respondents selecting the paper mode were more similar to respondents selecting the Web-based mode than those selecting the IVR mode were to those selecting the Web-based mode. This finding held for the total population and also for most of the demographic subgroups. These results suggest that, regardless of differences in question completion rates, paper surveys did not add much additional information on the surveyed population beyond what was captured by the Web-based surveys. On the other hand, the overall survey means would have been different had the information from the IVR surveys not been included.

Limitations

Although the VA tries to make all 3 modes available equally to all employees, we have no way to assess how much convenience influenced choices of survey mode. Differences among modes with regard to missing responses and question means could be due to something inherent in the mode itself, to the person selecting the mode (all things being equal), or to the relative ease with which each mode can be accessed by particular respondent groups.

Since demographic answers are self-disclosed, it is not possible to independently determine the likely accuracy of the responses.

We are not able to assess if respondents would switch to a different mode or would not participate if a particular mode was not offered as an option.

Conclusions

The AES is very important to the VA’s efforts to improve its efficiency and effectiveness while maintaining customer and employee satisfaction. In order to accommodate employee preferences and thereby maximize response rates, the AES is offered in 3 modes. Although employees are increasingly likely to select the Web-based mode, in 2008 approximately 7.5% of employees were still using the IVR and paper modes. There were few differences in mode preference among demographic groups and subgroups. There were also few statistically significant differences in question response means among these groups when Web-based is compared to paper mode. There were more such differences between Web-based and IVR modes. It is unlikely that elimination of the paper mode would lead to significant changes in the demographics of the respondents or to the answers they provide. An increase in missing demographic questions could be expected. If the IVR mode were to be eliminated, there would be little change in the demographic characteristics of the respondent and the proportion of demographic questions missing, but response means could be expected to be different. If the trend toward Web-based mode continues, any overall changes to demographic patterns, missing rates, or response means due to elimination of the IVR or paper modes can be expected to diminish.