The APA Salaries in Psychology Survey is conducted on a bi-ennial basis by the American Psychological Association, a non-profit member association. The purpose of the survey is to collect current salary information at the masters and doctoral levels for all APA members. Meta-data on which this article is based were collected in June of 2009. For the 2009 effort an item on current employment setting was presented to respondents using two different formats (open text vs. forced choice drop down). Members of the sample were randomly assigned to one of two groups (Branch A or Branch B) for the on-line survey. Results showed that participants who received the open text question format were significantly more likely to provide employment setting data compared to participants who received the forced choice drop-down format, and participants who received the open text question format were significantly more likely to provide usable data compared to participants who received the forced choice drop-down format.

Background

The APA Salaries in Psychology data are overwhelmingly doctoral level, and this reflects the prevailing situation among APA members. As is the case with any member association, the APA is interested in using technology to ease the survey response burden on its members in order capture more robust data. For many years APA has struggled with accurately and consistently describing the members’ employment settings and positions. For all past efforts the item requesting data on members current employment setting has been offered as an open text box with a numbered list of popular employment settings for master’s or doctoral-level individuals in psychology. As individuals with a doctorate degree in psychology have expanded their employment setting choices the employment setting list has lengthened, requiring more space on both paper and online instruments. Past attempts to utilize space more efficiently in online instruments through the use of a ‘pop-up box’ containing the employment settings that participants could click on resulted in technical issues and respondents abandoned the survey in frustration.

Since past efforts have only provided the employment setting question in the form of an open text box with a numbered list of employment setting options, a certain percentage of respondents provided incomplete data that could not be analyzed or they skipped the question entirely. Unusable or incomplete data account for as little as 2 percent (2007 APA Salaries in Psychology Survey) to as much as 20 percent (2005 APA Salaries in Psychology Survey) of the employment item analysis for previous efforts. Other survey efforts from APA experience similar broad ranges of missing and unusable data for the employment setting item depending upon survey effort and year.

Methods

To that end an experiment testing different question formats was implemented in the 2009 APA Salaries in Psychology Survey effort for the item on current employment setting. Members of the sample for the online instrument were randomly assigned to one of two groups. Each group was presented with the identical question using different formats, either an open text box, which had been used in prior surveys with varying degrees of success, or a drop-down forced choice design. This would be the first attempt at testing the effectiveness of different question formats for the employment setting item for item response rate and data quality.

Individuals who requested a paper version of the instrument (N=10) were not included in this investigation, rather, only data for those individuals who responded to the online survey (N=12,825) were used.

Data collection began in June of 2009 with an approximate sample of 47,239 members receiving the survey. The true overall response rate for this effort was approximately 27 percent. Data indicate that 2009 respondents and those in the sample were quite similar with respect to key variables of interest. Differences, where they existed, were not substantive.

Groups were created using a branching feature within the online survey tool that randomly assigned 50 percent of respondents to the question formatted as an open text box (Branch A), and 50 percent to the forced choice drop-down box format (Branch B). Appendix A is a visual representation of the two different question formats for Branch A and for Branch B.

Respondents in the two groups did not differ substantively in their demographics (mean year degree was earned, disability status, gender, race/ethnicity, etc.) from each other. Table 1 and Table 2 compare the demographic characteristics of respondents to Branch A and respondents to Branch B.

Table 1  Characteristics of APA members by question item format, 2009.
Branch A Branch B Total
N % N % N %
N= 6,374 49.7 6,441 50.3 12,815 100.0
Gender Female 3,603 56.5 3,584 55.6 7,187 56.0
Male 2,771 43.5 2,856 44.3 5,627 43.9
Not specified 0 0.0 1 0.0 1 0.0
Race/ethnicity African American/Black 211 3.3 197 3.1 408 3.2
American Indian or Alaskan Native 20 0.3 19 0.3 39 0.3
Asian 145 2.3 176 2.7 321 2.5
Caucasian/White 5370 84.2 5,384 83.6 10,754 83.9
Hispanic 197 3.1 212 3.3 409 3.2
Native Hawaiian or Pacific Islander 12 0.2 8 0.1 20 0.2
Other 0 0.0 0 0.0 0 0.0
Multiple racial/ ethnic background 103 1.6 104 1.6 207 1.6
Not specified 316 5.0 341 5.3 657 5.1
Disability No 5,912 92.8 5,976 92.8 11,888 92.8
Yes 186 2.9 182 2.8 368 2.9
Not specified 276 4.3 283 4.4 559 4.4
Highest Degree Earned Doctorate 6,135 96.3 6,210 96.4 12,345 96.3
Master’s 231 3.6 227 3.5 458 3.6
Other Degree 5 0.1 4 0.1 9 0.1
Not Specified 3 0.0 0 0.0 3 0.0
Licensure Status Currently Licensed and/or certified 3,913 61.4 3,910 60.7 7,823 61.0
Not currenlty licensed and/or certified 2,460 38.6 2,531 39.3 4,991 38.9
Psychology Major Field DHS 3,459 54.3 3,492 54.3 6,951 54.2
Research/Other 2,915 45.7 2,949 45.7 5,864 45.8
Not Specified 0 0.0 0 0.0 0 0.0
Early Career Doctorate No 4,804 75.4 4,889 75.9 9,693 75.6
Yes 1,570 24.6 1,552 24.1 3,122 24.4

Source: APA Salary Survey, 2009. Compiled by APA Center for Workforce Studies, June 2010.

Table 2  Statistics of APA members by question item format, 2009.
Year Degree Earned Branch A Branch B
N 5,722 5,788
Mean 1991 1990
Median 1992 1992
Mode 2006 2005
Std. Deviation 11.129 11.186
Not Specified 67 79
Hours per week in primary position
N 5,592 5,652
Mean 41.06 41.15
Median 40 40
Mode 40 40
Std. Deviation 11.554 11.457
Not Specified 197 215
Total years work experience
N 5,530 5,605
Mean 18.22 18.44
Median 18 18
Mode 25 30
Std. Deviation 11.225 11.251
Not Specified 259 262

Source: APA Salary Survey, 2009. Compiled by APA Center for Workforce Studies, June 2010.

This experiment focused on the following:

  1. Comparison of item response rates utilizing an open-ended text item (Branch A) versus a series of drop-down forced choice items (Branch B), and,
  2. Comparison of amount of usable data received from respondents through an open-ended text item (Branch A) versus a series of drop-down forced choice items (Branch B).

These questions were addressed using univariate and bivariate descriptive statistics (e.g., percentages and cross-tabulations with chi-square).

Findings/Results

Of the 12,825 survey participants, 6,374 (49.7 percent) received the open text box question format (Branch A) and 6,441 (50.3 percent) received the series of drop-down forced choice question format (Question B).

Response Rate was higher for Branch A Participants

The response rate for the employment setting question differed dramatically based on the format of the item: open text box, or forced choice drop down menu.

Figure 1 presents response rates for both experimental groups. While approximately 97 percent of persons who received the open text box format (Branch A) provided some sort of response, 86 percent of individuals who received the forced choice drop-down format provided a response. Put another way, individuals who received Branch B were almost 5 times more likely to not respond to the employment setting item than individuals who received Branch A.

Figure 1  Response rates for Question Item Branches A and B.

Participants who were randomly assigned to Branch A (open text format) were significantly more likely to provide data than were those who were assigned to Branch B (series of forced choice drop-down items) (χ2=459.723, df=1, p<0.001).

Incompatible Data More Likely for Branch B Participants

Additionally, approximately 1 percent of those who received Branch B and who responded, provided a response that was not fit for analysis. Data that could not be reconciled or verified through other sources of information were removed from analysis.

Participants who were randomly assigned to Branch A (open text question item format) were significantly more likely to provide usable employment setting data, than were those who were assigned to Branch B (series of forced choice drop-down items) (χ²=58.461, df=1, p<0.001).

Discussion

Our results indicated that in comparison with a forced choice drop-down question format, the open text box with list of employment settings format seemed to promote not only a better response for that item, but collected higher quality data on employment setting as well. Based on the results it was determined that the open text box question format should continue to be the format used in APA surveys that inquire about members current employment settings. Preliminary data from three other survey efforts currently in the field that utilize the open text format for the employment setting item reveal that there continues to be variability in the response rate for this question (from one to four percent item nonresponse dependant upon the particular survey effort) (2009 APA Doctorate Employment Survey 2010 APA Employment Update), although it is much lower than the item nonresponse for the forced choice drop-down format.

In theory the use of multiple forced choice drop down questions might be thought to yield ‘cleaner’ data since respondents are limited to choosing an employment setting from only four or five increasingly narrow options at a time, it seems that not allowing respondents to view the complete list of employment settings (currently 72 settings) may increase frustration by not being able to immediately see one’s employment setting, leading to abandonment of the question item all together.

It should be noted that despite the open text box format being a better choice in terms of yielding the most robust data, there were still disadvantages that must be addressed. These include the high variability in item nonresponse by survey effort, and the ever increasing number of employment setting codes, forcing the allocation of more space to the list in both online and paper surveys.

Based on the results of this survey format experiment some of our recommendations for survey researchers are as follows:

  1. Thorough review of the characteristics of your survey population prior to instrument creation. This may provide insight into which question item format will yield the most accurate survey results.
  2. While not always possible, keeping abreast of the characteristics of your survey population will also assist in the creation of employment setting lists that are as concise and accurate as possible, ensuring it takes up the least amount of space in your survey (whether it be online or paper format) as absolutely necessary, saving you money and the respondent time.
  3. Finally, careful examination of all survey item formats should be conducted with every new iteration, no matter how long a particular method has been in use. With the constant emergence of new research and technologies, question item formats that may have worked well in past survey efforts can be quickly supplanted with something even better. However, before altering formats careful consideration should be made to ensure that new methods of asking items will not adversely affect any longitudinal work being conducted.

Conclusion

We conclude that until a better alternative for collecting usable data on employment setting can be found, it is better to use the open text box with a displayed accompanying setting list than to potentially lose more data through item nonresponse, or collect data that is determined to be unusable and must be removed from analysis, through the use of a forced choice drop-down format. More research is needed to evaluate alternatives for displaying the employment setting question item to respondents, while both maintaining and increasing data quantity and quality.

Appendix A

Employment Settings Branches A & B (participant’s viewed either one question item format or the other)

Branch A

Employment Setting. Please indicate your PRIMARY employment setting as a two digit code. Please scroll down to see a complete list of employment setting codes. (PRIMARY SETvTING IS THE ONE IN WHICH YOU SPEND THE MOST TIME)

University (not medical or independent professional school) Settings

01  Psychology department
02  Education department
03  Business school or department
04  Other academic department or unit
05  Management or administrative office
06  Student counseling or services center
07  Research center or institute
24  Professional school of psychology, University based
09  Other university setting

Four-year College Settings

11  Psychology department
12  Education department
13  Business school or department
14  Other academic department or unit
15  Management or administrative office
16  Student counseling or services center
17  Research center or institute
19  Other four-year college setting

Other Educational Settings

21  Two-year college
22  Medical school, psychiatry department
23  Medical school, other than psychiatry department
28  Independent professional school of psychology
29  Professional schools not listed above (e.g., law, nursing)
25  Elementary or secondary school
26  School system district office
27  Other educational setting (e.g., vocational or special education)

Private Practice Settings

41  Individual private practice
42  Group psychological practice
43  Primary care group practice

Organized Human Service Settings

31  Public general hospital
32  Private general hospital
33  City/county/state psychiatric hospital
34  Not for profit, private psychiatric hospital
36  For profit, private psychiatric hospital
35  VA medical center
37  Military hospital (e.g., Air Force)
71  Rehabilitation facility
47  Counseling or guidance center (not university or college)
44  Outpatient mental health clinic, free-standing
45  Community mental health center or clinic (CMHC)
73  Primary care office/community health center
40  Nursing home
48  Specialized health service (e.g., substance abuse or mental retardation)
39  Preferred Provider Organization (PPO)
38  Independent Practice Association (IPA)
46  Health Maintenance Organization (HMO), excluding IPA
70  Other managed care setting
49  Other human service setting not listed above

Business, Government and Other Employment Settings

51  Self-employed (not private practice)
52  Consulting firm
53  Private research organization or lab
54  Govt. research organization or lab
56  Independent consultant
72  Union
55  Business or industry (excluding consulting firm or research organization)
61  Criminal justice system
62  Military service (not military hospital)
63  Federal government agency (other than above settings)
64  State government agency (other than above settings)
61  Local government agency (other than above settings)
62  Other non-profit organization
69  Other non-educational or non service setting not listed above

Branch B

Employment Setting. Please indicate the ONE employment category and corresponding setting that best describes your PRIMARY employment. (PRIMARY SETTING IS THE ONE IN WHICH YOU SPEND THE MOST TIME)

Please select a general employment category from the list below. You will be able to refine this selection.

—Please Select—
Educational: University (not medical or professional school)
Educational: Four-Year College
Educational: Other Educational Setting
Private Practice Setting
Organized Human Services
Business, Government, and Other Employment Setting

(refining option for university) Please select a University employment setting from the list below.

—Please Select—
01  Psychology department
02  Education department
03  Business school or department
04  Other academic department or unit
05  Management or administrative office
06  Student counseling or services center
07  Research center or institute
24  Professional school of psychology, University based
09  Other university setting
Please specify other University employment setting: ___________

(refining option for four year college) Please select a Four-Year College employment setting from the list below.

—Please Select—
11  Psychology department
12  Education department
13  Business school or department
14  Other academic department or unit
15  Management or administrative office
16  Student counseling or services centeresearch center or institute
19  Other four-year college setting
Please specify other Four-Year College employment setting: ___________

(refining option for other educational) Please select an Other Educational employment setting from the list below.

—Please Select—
21  Two-year college
22  Medical school, psychiatry department
23  Medical school, other than psychiatry department
28  Independent professional school of psychology
29  Professional schools not listed above (e.g., law, nursing)
25  Elementary or secondary school
26  School system district office
27  Other educational setting (e.g., vocational or special education)
Please specify Other Educational employment setting: ___________

(refining option for private practice) Please select a Private Practice employment setting from the list below.

—Please Select—
41  Individual private practice
42  Group psychological practice
43  Primary care group practice

(refining option for organized human service) Please select an Organized Human Service setting from the list below.

—Please Select—
31  Public general hospital
32  Private general hospital
33  City/county/state psychiatric hospital
34  Not for profit, private psychiatric hospital
36  For profit, private psychiatric hospital
35  VA medical center
37  Military hospital (e.g., Air Force)
71  Rehabilitation facility
47  Counseling or guidance center (not university or college)
44  Outpatient mental health clinic, free-standing
45  Community mental health center or clinic (CMHC)
73  Primary care office/community health center
40  Nursing home
48  Specialized health service (e.g., substance abuse or mental retardation)
39  Preferred Provider Organization (PPO)
38  Independent Practice Association (IPA)
46  Health Maintenance Organization (HMO), excluding IPA
70  Other managed care setting
49  Other human service setting not listed above
Please specify other Organized Human Service employment setting: ___________

(refining option for business, government or other) Please select a Business, Government, or Other employment setting from the list below.

—Please Select—
51  Self-employed (not private practice)
52  Consulting firm
53  Private research organization or lab
54  Govt. research organization or lab
56  Independent consultant
72  Union
55  Business or industry (excluding consulting firm or research organization)
61  Criminal justice system
62  Military service (not military hospital)
63  Federal government agency (other than above settings)
64  State government agency (other than above settings)
65  Local government agency (other than above settings)
66  Other non-profit organization
69  Other non-educational or non service setting not listed above
Please specify other Business, Government, or Other employment setting: ___________