Loading [Contrib]/a11y/accessibility-menu.js
Skip to main content
Survey Practice
  • Menu
  • Articles
    • Articles
    • Editor Notes
    • In-Brief Notes
    • Interview the Expert
    • Recent Books, Papers, and Presentations
    • All
  • For Authors
  • Editorial Board
  • About
  • Issues
  • Blog
  • Subscribe
  • search

RSS Feed

Enter the URL below into your favorite RSS reader.

http://localhost:23905/feed
Articles
Vol. 12, Issue 1, 2019July 29, 2019 EDT

Effects of Push-To-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments

Layla Parast, PhD, Megan Mathews, MA, Marc Elliott, PhD, Anagha Tolpadi, MS, Elizabeth Flow-Delwiche, PhD, William G. Lehrman, PhD, Debra Stark, MBA, Kirsten Becker, MS,
experimentnonresponseweb surveyresponse rate
https://doi.org/10.29115/SP-2019-0008
Photo by Ali Yahya on Unsplash
Survey Practice
Parast, PhD, Layla, Megan Mathews, MA, Marc Elliott, PhD, Anagha Tolpadi, MS, Elizabeth Flow-Delwiche, PhD, William G. Lehrman, PhD, Debra Stark, MBA, and Kirsten Becker, MS. 2019. “Effects of Push-To-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments.” Survey Practice 12 (1). https:/​/​doi.org/​10.29115/​SP-2019-0008.
Save article as...▾

View more stats

Abstract

Recent work involving survey administration among emergency department (ED) patients has demonstrated very low response rates using single-mode (e.g., mail-only or web-only) approaches. In this study, the authors tested several mixed-mode protocols that focus on a push-to-web approach for survey administration in this challenging but important setting. A random sample of 26,991 ED discharged-to-community (DTC) patients discharged in January 2018–March 2018 from 16 hospital-based EDs nationwide were randomized to nine survey arms, eight of which involved a web survey with one or more of the following methods of invitation to the web survey: an emailed link to the web survey, a texted link to the web survey, and/or a mailed paper invitation with information on how to access the web survey. The reference arm was standard mixed mode (mailed survey with telephone follow-up). Invitation(s) to the web survey was followed by (1) a mailed survey, (2) telephone follow-up, or (3) both. The overall response rate across all arms was 18.6% (American Association for Public Opinion Research [AAPOR] RR1). None of the tested web-survey arms had significantly higher response rates than standard mixed mode (25.5%). Protocol comparisons demonstrated that both text outreach and including a telephone mode increased response rates. When examining response by completion mode, the arms involving text invitation had a higher percentage of completed surveys that were completed by web than arms not involving text invitation. Characteristics (e.g., age) of telephone and web respondents were more representative of the sampled population than mail respondents. Older patients and male patients were significantly less likely to be reachable by email or text. These results demonstrate that sequential multi-mode approaches involving web, mail, and telephone with multiple methods of web survey invitation show the greatest promise for obtaining higher response rates and more representative respondents from the ED population.

Introduction

Measuring and reporting patient experience with health care through surveys provides useful information to guide consumer decision-making when selecting a provider and contributes to the evaluation of provider performance. For example, results from the Hospital Consumer Assessment of Healthcare Providers and Systems (CMS 2018a) Survey of patients’ experiences of hospital care are publicly reported as measures and star ratings and used in hospital payment (CMS 2018d; Elliott et al. 2016; Giordano et al. 2010). Patient experience surveys also inform publicly reported information about the quality of hospice care and the care received by Medicare beneficiaries who belong to Accountable Care Organizations (CMS 2018a; Hospice 2018).

Given that these survey results inform evaluations of provider performance, it is essential that they accurately reflect the experiences of the patient population by having a reasonably high survey response rate, obtaining a representative sample of respondents, and utilizing statistical methods to ensure representative and unbiased results. In this paper, we examine a patient experience survey of hospital-based emergency department (ED) patients discharged to home after their ED visit (rather than admitted to the hospital), referred to as discharged-to-community (Parast et al. 2019) patients. As previous studies have found very low survey response rates among such patients, less than 20% (Mathews et al. 2019; Weinick et al. 2014), we sought to understand how response rates could be improved in this population.

Previous work in the ED DTC population has shown that single-mode protocols, such as mail-only or web-only, and distribution within the ED do not perform well (1–14% response rate), but sequential mixed-mode approaches may have promise (Mathews et al. 2019; Parast et al. 2019). In this paper, we describe results from a randomized test of nine experimental survey administration protocols, eight of which began with an invitation to a web survey, then followed by a mailed survey, a telephone survey or both. The primary goals of this experiment were to (1) determine whether any of the experimental protocol arms with a web survey component could obtain a higher response rate than a two-stage non-web sequential mixed mode protocol (mail followed by telephone) and (2) identify protocol components (methods of survey invitation) that were most beneficial in terms of improving the likelihood of response.

Methods

Sample and Survey Administration

The study was designed as a randomized experiment in which randomization of patients to survey arms and the sample size targets were designed to achieve adequate power to detect small differences in response rates when comparing mode protocols. A total of 16 large, geographically dispersed hospitals participated in the experiment. To be eligible, a hospital had to have adequate volume (14,000 or more annual ED visits), routinely collect patient email addresses, and have a process to obtain and document consent to contact patients via text message. Hospitals participated voluntarily and did not receive any compensation. Adult (18 years and older) patients were eligible for the study except those who (1) were admitted to the hospital as a result of the ED visit; (2) left the ED without being seen; (3) requested not to be contacted (“no publicity” patients); (4) were excluded due to state regulations; (5) had a primary diagnosis of mental health or substance use (International Statistical Classification of Diseases and Related Health Problems [ICD]-10 codes F01–F99); (6) were transferred to another hospital; (7) were discharged anywhere other than home (e.g., to a skilled nursing facility); (8) had a foreign home address; (9) were court/law enforcement patients (prisoners); (10) spoke a language other than English; or (11) had died in the ED or prior to sampling. Exclusions (1)–(4) were implemented by each participating hospital; exclusions (5)–(11) were implemented by the study team and are listed in order of decreasing frequency.

A total of 26,991 ED DTC adult patients discharged between January 2018 and March 2018 from the 16 hospitals were sampled by the study team and randomized to nine protocol arms within each hospital. Table 1 provides a detailed description of protocol components and timing information for the nine protocol arms. Documented consent to contact patients by text was required for this study. Some hospitals included this consent in their general consent to treat documentation, others employed a separate consent form. Absent consent, patients were not contacted by text. Data collection began between 11 and 64 days (average of 31 days) after discharge, and all survey outreach efforts ended 42 days after the first contact attempt.

Table 1.Protocols for each arm.
Contact day Arm 1: Mail + telephone (standard mixed mode) Arm 2: Paper invite + email (x2) + mail (x2) Arm 3: Paper invite + email (x3) + mail Arm 4: Email (x4) + telephone Arm 5: Email (x4) + mail + telephone Arm 6: Email (x4) + mail (x2) Arm 7: Text + email + text + email + mail (x2) Arm 8: Paper invite + text + email (x2) + mail Arm 9: Paper invite (URL only) + email (x2) + mail (x2)
1 Mailed survey Mailed invitation (containing QR code & URL) Mailed invitation (containing QR code & URL) Emailed link Emailed link Emailed link Texted link Mailed invitation (containing QR code & URL) Mailed invitation (containing URL only)
2 Emailed link Emailed link Emailed link Emailed link
4 Emailed link Emailed link Emailed link Texted link
6 Emailed link Emailed link Emailed link Emailed link Emailed link Emailed link Texted link Emailed link
8 Emailed link & mailed survey Emailed link Telephone attempt 1 (5 max) Mailed survey Mailed survey Mailed survey Emailed link Emailed link & mailed survey
10 Emailed link Emailed link
14 Mailed survey Mailed survey
22 Telephone attempt 1 (5 max) Mailed survey Telephone attempt 1 (5 max) Mailed survey Mailed survey Mailed survey

Notes: QR = quick response

The study survey instrument contained 38 questions and is available online (CMS 2018a). A completed survey was defined as one in which at least 50% of the items applicable to all patients[1] (21 items) were answered. This study was approved by the RAND Institutional Review Board.

Statistical Analysis

We examined response rates using the American Association for Public Opinion Research Response Rate 1 definition (Kaplowitz et al. 2012). To compare protocol pairings, we used logistic regression in which the outcome was individual-level response (0/1) with an indicator for one survey arm within the pair. The two arms being compared were typically different in a single way, such as the number or type of reminders, or a difference in timing. We also examined the response rate by each completion mode and compared characteristics of respondents by completion mode. Throughout, all regression models were adjusted for patient age, gender, Emergency Severity Index (ESI)[2], primary diagnosis[3], sampling batch[4], and indicators for hospital.

To examine the potential for nonresponse bias, we conducted a nonresponse analysis by fitting a logistic regression predicting response (0/1) from available characteristics, tested for differences in nonresponse results by arm by adding an interaction between patient characteristic and arm indicators, and compared respondent characteristics by arm.

Finally, we examined whether patient characteristics were associated with the ability to contact a patient by email or text, using age, gender, ESI, primary diagnosis, sampling batch, and hospital as predictors in a logistic regression. We defined “being able to contact patient by email” as the patient having a non-missing email address provided in the contact information, and consenting to email contact (if required by the hospital). Similarly, for text, we defined “being able to contact patient by text” as the patient having a mobile telephone number provided in the contact information, and consenting to text contact (required by all hospitals).

Results

Table 2 presents response rate results; the overall response rate was 18.6%. The response rate in Arm 5 (Email (x4) + Mail + Telephone) was highest (27.3%) but was not significantly higher than Arm 1, standard mixed mode (25.5%). The response rates for all other arms were significantly lower than standard mixed mode.

Table 2.Response rates, overall, and by survey arm.
Arm 1: Mail + telephone Arm 2: Paper invite + email (x2) + mail (x2) Arm 3: Paper invite + email (x3) + mail Arm 4: Email (x4) + telephone Arm 5: Email (x4) + mail + telephone Arm 6: Email (x4) + mail (x2) Arm 7: Text + email + text + email + mail (x2) Arm 8: Paper invite + text + email (x2) + mail Arm 9: Paper invite (URL only) + email (x2) + mail (x2) Overall
N Sampled 2,844 2,848 2,846 3,193 3,192 3,195 3,191 2,841 2,841 26,991
N Ineligible 137 (4.8%) 49 (1.7%) 26 (0.9%) 152 (4.8%) 172 (5.4%) 45 (1.4%) 38 (1.2%) 36 (1.3%) 49 (1.7%) 704 (2.6%)
N Respondents 690 433 378 697 824 482 555 405 429 4,893
Response rate among eligible 25.5% (ref) 15.5%*** 13.4%*** 22.9%* 27.30% 15.3%*** 17.6%*** 14.4%*** 15.4%*** 18.60%

Notes: *p<0.05, **p<0.01, ***p<0.001. Stars reflect results of a logistic regression model in which the outcome was individual level response (0/1) with indicators for survey arm; where arm 1 was the reference arm; and adjusting for sampled patients age, gender, Emergency Severity Index, primary diagnosis, sampling batch, and indicators for hospital

Table 3 presents logistic regression results for each of the eight selected comparisons. For each of these protocol comparisons, in addition to differences in the protocol components (e.g., email only vs. email and text), there were also differences in the timing of outreach efforts and differences in the patient population that could be reached using the different protocol components (e.g., mailed survey vs. email). These results show that the following factors increased the likelihood/odds of response: (1) using telephone in place of a second mailing (odds ratio [OR] =2.41), (2) using a text invitation and a second mailed survey instead of a paper invitation (OR=1.34), (3) adding a mailed survey to a web-phone protocol (OR=1.30), (4) adding a second mailing instead of having a third reminder email (OR=1.25), and (5) using a combination of text invitation/reminder and email reminders instead of only email invitation/reminders (OR=1.25).

Table 3.Protocol comparisons.
Protocol component ARM of interest Reference arm Adjusted Odds Ratio (95% CI)
Email (x4) addition ARM 5: Email (x4) + mail + telephone ARM 1: Mail + telephone 1.12 ( 0.99 , 1.27 )
QR code added to paper invitation ARM 2: Paper invitation + email (x2) + mail (x2) ARM 9: Paper invitation (URL only) + email (x2) + mail (x2) 1.02 ( 0.87 , 1.19 )
Second mailing instead of 3rd email reminder ARM 2: Paper invitation + email (x2) + mail (x2) ARM 3: Paper invitation + email (x3) + mail 1.25 ( 1.06 , 1.47 ) **
Paper invitation instead of 2 additional email reminders ARM 2: Paper invitation + email (x2) + mail (x2) ARM 6: Email (x4) + mail (x2) 1.01 ( 0.86 , 1.18 )
Two texts instead of two email reminders ARM 7: Text + email + text + email + mail (x2) ARM 6: Email (x4) + mail (x2) 1.25 ( 1.08 , 1.45 ) **
Text invitation and second mailed survey instead of paper invitation ARM 7: Text + email + text + email + mail (x2) ARM 8: Paper invitation + text + email (x2) + mail 1.34 ( 1.16 , 1.56 ) ***
Mailed survey ARM 5: Email (x4) + mail + telephone ARM 4: Email (x4) + telephone 1.30 ( 1.15 , 1.46 ) ***
Telephone addition (instead of 2nd mailing) ARM 5: Email (x4) + mail + telephone ARM 6: Email (x4) + mail (x2) 2.41 ( 2.10, 2.76 ) ***
Text instead of email for 1 reminder ARM 8: Paper invitation + text + email (x2) + mail ARM 3: Paper invitation + email (x3) + mail 1.15 ( 0.98 , 1.36 )

Notes: Models adjust for sampled patients age, gender, Emergency Severity Index, primary diagnosis, sampling batch, and indicators for hospital.
Stars reflect significance level: *p<0.05, **p<0.01, ***p<0.001; CI = confidence interval

Table 4 presents the percentage of patients responding by each mode by arm. The reference mode here was Arm 5 (Email [x4] + Mail + Telephone) because it was the only arm with all three survey completion modes. Among arms that included telephone, 14.1%–18.1% of patients responded by telephone; for arms with mail, 7.0%–10.5% of patients responded by mail; and for arms with web, 4.8%–7.5% responded by web. In all arms that included telephone (which was the terminal mode), the majority of responses were by telephone. Overall, the most productive mode in terms of obtaining completed surveys was telephone, then mail, then web. The web mode consistently resulted in the lowest percentage of responses, though Arms 7 and 8 (which also included text messages) obtained a significantly higher proportion of responses by web (>7%) than did Arm 5. These results also reveal that Arm 5 (Email [x4] + Mail + Telephone) had significantly fewer mail respondents and telephone respondents compared to other arms yet had the highest response rate, illustrating that use of an initial web mode has the potential to reduce costs associated with mail and telephone contact.

Table 4.Responses by completion mode, among all sampled patients.
Arm 1: Mail + telephone Arm 2: Paper invite + email (x2) + mail (x2) Arm 3: Paper invite + email (x3) + mail Arm 4: Email (x4) + telephone Arm 5: Email (x4) + mail + telephone Arm 6: Email (x4) + mail (x2) Arm 7: Text + Email + text + email + mail (x2) Arm 8: Paper invite + text + email mail Arm 9: Paper invite (URL only) + email (x2) + mail (x2) Overall
% Completed by web -- 5.00% 6.40% 4.80% 5.4% (ref) 5.50% 7.5%*** 7.5%** 5.60% 5.30%
% Completed by mail 10.3%*** 10.5%*** 7.00% -- 7.8% (ref) 9.8%* 10.1%*** 7.00% 9.8%* 8.00%
% Completed by telephone 15.10% -- -- 18.1%*** 14.1% (ref) -- -- -- -- 5.30%
% Non-respondents 74.50% 84.5%*** 86.6%*** 77.1%*** 72.7% (ref) 84.7%*** 82.4%*** 85.6%*** 84.6%*** 81.40%

Note: % is among all eligible sampled patients assigned to each arm; *p<0.05, **p<0.01, ***p<0.001; stars reflect results of a logistic regression model in which the outcome (0/1) using that completion mode with indicators for survey arm, where arm 5 was the reference arm, and adjusting for sampled patients age, gender, Emergency Severity Index, primary diagnosis, sampling batch, and indicators for hospital. Arms without an option to complete by a respective completion mode are excluded from the respective model.

Table 5 presents respondent characteristics within each completion mode. All characteristics had significant differences in distributions across completion modes. The age distribution of telephone respondents was most similar to the sampled population. Compared to mail respondents, web respondents tended to be younger and more closely reflect the sampled eligible population. For example, while 38.5% of the eligible sampled population were 18–34 years old, only 11.0% of mail respondents were in this age range, as were 18.7% of web respondents and 30.8% of telephone respondents. Thus, among 18–34-year-old patients, the telephone mode most closely reflects the sampled population (although still lower than the true percentage), while the mail mode does not reflect the true percentage, and web mode is in between. In addition, the telephone mode obtained more patients with poorer health and lower education, as well as more racial-ethnic minorities, compared to mail and web mode.

Table 5.Summary of respondent characteristics, by completion mode.
Characteristic Sampled Web Mail Phone Chi-squared
P-value
N (%) N (%) N (%) N (%)
Age
18-24 4114 (15.4%) 98 (6.6%) 81 (4.4%) 169 (12.1%) <0.001
25-34 6175 (23.1%) 171 (12.1%) 139 (6.6%) 264 (18.7%)
35-44 4517 (17.6%) 243 (17.6%) 160 (7.7%) 204 (14.3%)
45-54 3865 (14.3%) 227 (16.5%) 226 (11.0%) 216 (15.4%)
55-64 3332 (13.2%) 260 (18.7%) 415 (19.8%) 231 (16.5%)
65-74 2162 (7.7%) 237 (16.5%) 476 (23.1%) 167 (12.1%)
75-84 1416 (5.5%) 127 (8.8%) 388 (18.7%) 96 (6.6%)
85 + 706 (2.2%) 41 (3.3%) 218 (9.9%) 39 (3.3%)
Gender
Female 16013 (60.5%) 932 (66.0%) 1315 (62.7%) 837 (60.5%) 0.004
Male 10274 (39.6%) 472 (34.1%) 788 (37.4%) 549 (39.6%)
Emergency Severity Index
Unknown 3327 (13.2%) 246 (17.6%) 320 (15.4%) 135 (9.9%) 0.002
High urgency 2956 (11.0%) 164 (12.1%) 323 (15.4%) 205 (14.3%)
Medium urgency 12209 (46.2%) 682 (48.4%) 978 (46.2%) 654 (47.3%)
Low urgency 7795 (29.7%) 312 (22.0%) 482 (23.1%) 392 (28.6%)
Primary diagnosis
Unknown 3970 (15.4%) 241 (17.6%) 348 (16.5%) 221 (15.4%) <0.001
Infectious and parasitic diseases 410 (1.1%) 12 (1.1%) 18 (1.1%) 15 (1.1%)
Neoplasms 51 (<0.1%) 3 (<0.1%) 4 (<0.1%) 0 (0.0%)
Endocrine; nutritional and metabolic diseases and immunity disorders 414 (1.1%) 21 (1.1%) 42 (2.2%) 31 (2.2%)
Diseases of the blood and blood-forming organs 73 (<0.1%) 5 (<0.1%) 12 (1.1%) 4 (<0.1%)
Mental Illness 127 (<0.1%) 6 (<0.1%) 1 (<0.1%) 2 (<0.1%)
Diseases of the nervous system and sense organs 1928 (7.7%) 109 (7.7%) 153 (7.7%) 103 (7.7%)
Diseases of the circulatory system 2052 (7.7%) 176 (12.1%) 228 (11.0%) 128 (8.8%)
Diseases of the respiratory system 3060 (12.1%) 123 (8.8%) 261 (12.1%) 152 (11.0%)
Diseases of the digestive system 1575 (5.5%) 88 (6.6%) 107 (5.5%) 83 (5.5%)
Diseases of the genitourinary system 1618 (6.6%) 101 (7.7%) 122 (5.5%) 86 (6.6%)
Complications of pregnancy; childbirth and the puerperium 1307 (5.5%) 46 (3.3%) 35 (2.2%) 59 (4.4%)
Diseases of the skin and subcutaneous tissue 701 (2.2%) 24 (2.2%) 50 (2.2%) 36 (2.2%)
Diseases of the musculoskeletal system and connective tissue 1996 (7.7%) 91 (6.6%) 155 (7.7%) 117 (8.8%)
Congenital anomalies 5 (<0.1%) 1 (<0.1%) 0 (0.0%) 0 (0.0%)
Injury and poisoning 4061 (15.4%) 224 (16.5%) 345 (16.5%) 208 (15.4%)
Symptoms signs and ill-defined conditions and factors influencing health 2608 (9.9%) 120 (8.8%) 205 (9.9%) 128 (8.8%)
Residual codes; unclassified 331 (1.1%) 13 (1.1%) 17 (1.1%) 13 (1.1%)
Self-reported general health
Unknown -- 35 (2.2%) 33 (1.1%) 39 (3.3%) <0.001
Poor -- 68 (4.4%) 116 (5.5%) 110 (7.7%)
Fair -- 235 (16.5%) 478 (23.1%) 350 (25.3%)
Good -- 447 (31.9%) 693 (33.0%) 394 (28.6%)
Very Good -- 448 (31.9%) 541 (25.3%) 315 (23.1%)
Excellent -- 171 (12.1%) 242 (11.0%) 178 (13.2%)
Self-reported mental health
Unknown -- 38 (2.2%) 33 (1.1%) 42 (3.3%) <0.001
Poor -- 43 (3.3%) 64 (3.3%) 62 (4.4%)
Fair -- 125 (8.8%) 236 (11.0%) 210 (15.4%)
Good -- 322 (23.1%) 492 (23.1%) 348 (25.3%)
Very Good -- 441 (31.9%) 704 (33.0%) 352 (25.3%)
Excellent -- 435 (30.8%) 574 (27.5%) 372 (26.4%)
Self-reported education
Unknown -- 46 (3.3%) 41 (2.2%) 45 (3.3%) <0.001
8th grade or less -- 23 (1.1%) 81 (4.4%) 38 (2.2%)
Some high school -- 38 (2.2%) 169 (7.7%) 143 (9.9%)
High School or GED -- 259 (18.7%) 610 (28.6%) 368 (26.4%)
Some college or 2-year degree -- 505 (36.3%) 645 (30.8%) 502 (36.3%)
4-year college graduate -- 253 (17.6%) 279 (13.2%) 142 (9.9%)
More than 4-year college degree -- 280 (19.8%) 278 (13.2%) 148 (11.0%)
Self-reported race/ethnicity
Unknown -- 59 (4.4%) 62 (3.3%) 79 (5.5%) <0.001
Hispanic -- 125 (8.8%) 144 (6.6%) 188 (13.2%)
Native American -- 9 (1.1%) 18 (1.1%) 15 (1.1%)
Black -- 104 (7.7%) 213 (9.9%) 231 (16.5%)
Asian or Pacific Islander -- 32 (2.2%) 62 (3.3%) 25 (2.2%)
White -- 1044 (74.8%) 1572 (74.8%) 726 (52.8%)
Multiracial -- 31 (2.2%) 32 (1.1%) 122 (8.8%)

Notes: Chi-squared values excluded missing or unknown response categories, and excluded categories with zero cells. A p-value of <0.05 is considered statistically significant and is bolded here. The mental illness primary diagnosis category contains diagnoses beyond the ICD-10 F codes that were included in the ineligibility criterion, such as suicidal ideation. All self-reported variables were summarized among responders only as this information was obtained from the survey.

Our nonresponse analyses showed that the likelihood of responding was significantly higher for older patients and for patients with certain primary diagnoses (such as diseases of the nervous system and sense organs, diseases of the circulatory system, and injury and poisoning, relative to patients with a primary diagnosis of infectious and parasitic diseases) and significantly lower for male patients and for low-urgency patients (results not shown). The association between response likelihood and age varied significantly by arm such that older patients were even more likely to respond in arms without a telephone component (results not shown). This result is also reflected in Appendix Table 1, which compares respondent characteristics by arm; arms with telephone have a higher percentage of younger patients responding and more closely reflect the sampled population.

Overall, across all eligible patients in this study, 30.1% could be contacted by email, and 59.3% could be contacted by text. Appendix Table 2 presents logistic regression results examining the likelihood of being able to be contacted by email and by text by demographic characteristics. For both email and text, the ability to contact patients varied greatly by hospital (range of 0.4%–59.7% and median of 28.4% for email; range of 1.3%–85.6% and median of 77.2% for text). Patients aged 25–74 were significantly more likely to be able to be contacted by e-mail compared to the youngest patients (18–24 years old; OR range 1.20–1.43, p<0.01), while patients 85 and older were significantly less likely to be able to be contacted by email (OR=0.54, p<0.001). For text, compared to patients 18–24 years old, patients 45 and older were significantly less likely to be able to be contacted by text (OR range 0.11–0.77, p<0.001). For both email and text, males were significantly less likely to be able to be contacted (OR=0.57, p<0.001 for email; OR=0.83, p<0.001 for text).

Discussion

Results from this study, in which ED DTC patients were randomly assigned to one of nine survey protocols, further demonstrate the difficulty in obtaining even moderate response rates in an ED DTC population. The overall response rate in this study was 18.6%, compared to national average response rates of 27% and 29% for the HCAHPS and Home Health CAHPS surveys, respectively (CMS 2018c, 2018e). However, this low response rate is not surprising given that the ED population is younger, more mobile, and tends to provide incomplete or poor-quality contact information (Elliott et al. 2009; Klein et al. 2011; Weinick et al. 2014). While the testing of eight web-based protocols is instructive, more work is needed to identify ways to improve response rates in this population.

Contact via text message is a promising method of invitation to a web survey, given the finding that more patients could be contacted by text than by email and that text improved response rates over email-only invitation. However, contacting patients via text incurs administrative and legal complications because the Telephone Consumer Protection Act of 1991 requires prior consent to contact via text. The extent to which the hospital-initiated patient consent protocol employed in this study would be feasible among hospitals nationally is not known.

On the other hand, results clearly demonstrated the benefits of including a telephone mode as part of the survey protocol. Telephone mode captures respondents who are not well represented in web and mail modes, including patients who are younger, in poorer health, with less education, and belong to racial/ethnic minorities. Telephone mode increases representativeness and response rates when added to other modes, as found in other health survey settings (Klein et al. 2011). Our results suggest that future protocols for this survey should include a telephone component. Our nonresponse analysis reinforces this conclusion given that differences between the respondent population and sampled population were more extreme using protocols without telephone. While hospital scores calculated using survey results are likely to employ case-mix adjustment to account for some respondent characteristics, there is still interest in having response rates as high and representative of the population as possible (Klein et al. 2011). Mode protocols that include telephone are more likely to achieve this goal than mode protocols without a telephone component.

This study demonstrated that patients who can be reached by email or text differ with respect to age and gender from those who cannot, which is consistent with previous work on this topic (Fowler Jr, et al. 2019). And while a web-only survey using an email and/or text invitation would poorly represent the emergency department patient population, the results presented here reveal that these modes of invitation do enhance contact with a hard-to-reach younger population—but should be followed by mail and/or telephone survey attempts.

Our study was limited to sixteen hospitals that met specific criteria, including willingness to voluntarily participate and to collect and document consent for contact via text. However, the hospitals were geographically diverse, and most importantly, the randomization of patients to modes within hospitals ensures internal validity of comparisons between the experimental arms (Klein et al. 2011). Particularly with respect to results on web survey response and contact via email and text, results are likely to vary depending on the percentage and characteristics of patients within an ED who provide consent to contact and provide email and mobile telephone contact information.

Overall, this study demonstrates that sequential multi-mode approaches involving web, mail, and telephone with multiple methods of web survey invitation will likely obtain the highest response rates in the ED population. Further work is needed to improve response rates. Future research might explore embedding the first survey item in the emailed/texted web survey invitation (Liu and Inchausti 2017), streamlining the introductory screen language, and varying the email/text subject line (Kaplowitz et al. 2012; Trouteaud 2004).

Appendices

Appendix Table 1.Summary of respondent characteristics, by arm.
Characteristic Sampled Arm 1: Mail + Phone Arm 2: Paper Invite + Email (x2) + Mail (x2) Arm 3: Paper Invite + Email (x3) + Mail Arm 4: Email (x4) + Phone Arm 5: Email (x4) + Mail + Phone Arm 6: Email (x4) + Mail (x2) Arm 7: Text + Email + Text + Email + Mail (x2) Arm 8: Paper Invite + Text + Email (x2)+ Mail Arm 9: Paper Invite (URL only) + Email (x2)+ Mail (x2) Chi-squared P-value
N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%) N (%)
Age
18-24 4114 (15.4%) 64 (8.8%) 20 (4.4%) 8 (2.2%) 72 (9.9%) 83 (9.9%) 15 (3.3%) 33 (5.5%) 31 (7.7%) 22 (5.5%) <0.001
25-34 6175 (23.1%) 104 (15.4%) 39 (8.8%) 37 (9.9%) 108 (15.4%) 123 (15.4%) 43 (8.8%) 54 (9.9%) 39 (9.9%) 27 (6.6%)
35-44 4517 (17.6%) 81 (12.1%) 54 (12.1%) 45 (12.1%) 109 (15.4%) 112 (13.2%) 47 (9.9%) 70 (12.1%) 42 (9.9%) 47 (11.0%)
45-54 3865 (14.3%) 90 (13.2%) 51 (12.1%) 52 (14.3%) 105 (15.4%) 114 (14.3%) 53 (11.0%) 81 (14.3%) 57 (14.3%) 66 (15.4%)
55-64 3332 (13.2%) 142 (20.9%) 87 (19.8%) 72 (18.7%) 111 (15.4%) 140 (16.5%) 93 (19.8%) 105 (18.7%) 77 (18.7%) 79 (18.7%)
65-74 2162 (7.7%) 102 (14.3%) 89 (20.9%) 73 (19.8%) 102 (14.3%) 133 (16.5%) 106 (22.0%) 101 (18.7%) 74 (18.7%) 100 (23.1%)
75-84 1416 (5.5%) 75 (11.0%) 63 (14.3%) 61 (16.5%) 65 (8.8%) 77 (8.8%) 79 (16.5%) 70 (12.1%) 56 (14.3%) 65 (15.4%)
85 + 706 (2.2%) 32 (4.4%) 30 (6.6%) 30 (7.7%) 25 (3.3%) 42 (5.5%) 46 (9.9%) 41 (7.7%) 29 (7.7%) 23 (5.5%)
Gender
Female 16013 (60.5%) 405 (58.3%) 278 (63.8%) 240 (63.8%) 430 (61.6%) 538 (64.9%) 317 (66.0%) 346 (62.7%) 254 (62.7%) 276 (63.8%) 0.252
Male 10274 (39.6%) 285 (41.8%) 155 (36.3%) 138 (36.3%) 267 (38.5%) 286 (35.2%) 165 (34.1%) 209 (37.4%) 151 (37.4%) 153 (35.2%)
Emergency Severity Index
Unknown 3327 (13.2%) 85 (12.1%) 71 (16.5%) 64 (16.5%) 88 (12.1%) 104 (12.1%) 81 (16.5%) 70 (12.1%) 59 (14.3%) 79 (18.7%) 0.514
High urgency 2956 (11.0%) 91 (13.2%) 71 (16.5%) 50 (13.2%) 103 (14.3%) 117 (14.3%) 62 (13.2%) 89 (16.5%) 56 (14.3%) 53 (12.1%)
Medium urgency 12209 (46.2%) 326 (47.3%) 179 (41.8%) 179 (47.3%) 327 (47.3%) 395 (48.4%) 229 (47.3%) 275 (49.5%) 199 (49.5%) 205 (47.3%)
Low urgency 7795 (29.7%) 188 (27.5%) 112 (26.4%) 85 (22.0%) 179 (25.3%) 208 (25.3%) 110 (23.1%) 121 (22.0%) 91 (22.0%) 92 (20.9%)
Primary diagnosis
Unknown 3970 (15.4%) 106 (15.4%) 62 (14.3%) 66 (17.6%) 104 (15.4%) 149 (17.6%) 87 (17.6%) 103 (18.7%) 67 (16.5%) 66 (15.4%) 0.330
Infectious and parasitic diseases 410 (1.1%) 1 (<0.1%) 2 (<0.1%) 1 (<0.1%) 6 (1.1%) 11 (1.1%) 6 (1.1%) 4 (1.1%) 8 (2.2%) 6 (1.1%)
Neoplasms 51 (<0.1%) 0 (0.0%) 2 (<0.1%) 0 (0.0%) 0 (0.0%) 1 (<0.1%) 1 (<0.1%) 0 (0.0%) 2 (<0.1%) 1 (<0.1%)
Endocrine; nutritional and metabolic diseases and immunity disorders 414 (1.1%) 16 (2.2%) 12 (3.3%) 5 (1.1%) 11 (1.1%) 16 (2.2%) 12 (2.2%) 9 (1.1%) 6 (1.1%) 7 (1.1%)
Diseases of the blood and blood-forming organs 73 (<0.1%) 2 (<0.1%) 1 (<0.1%) 3 (1.1%) 1 (<0.1%) 5 (1.1%) 2 (<0.1%) 5 (1.1%) 1 (<0.1%) 1 (<0.1%)
Mental Illness 127 (<0.1%) 2 (<0.1%) 1 (<0.1%) 1 (<0.1%) 0 (0.0%) 3 (<0.1%) 1 (<0.1%) 1 (<0.1%)
Diseases of the nervous system and sense organs 1928 (7.7%) 50 (7.7%) 30 (6.6%) 27 (6.6%) 49 (6.6%) 56 (6.6%) 31 (6.6%) 48 (8.8%) 37 (8.8%) 37 (8.8%)
Diseases of the circulatory system 2052 (7.7%) 69 (9.9%) 47 (11.0%) 40 (11.0%) 74 (11.0%) 82 (9.9%) 62 (13.2%) 54 (9.9%) 48 (12.1%) 56 (13.2%)
Diseases of the respiratory system 3060 (12.1%) 79 (11.0%) 66 (15.4%) 35 (8.8%) 76 (11.0%) 88 (11.0%) 53 (11.0%) 51 (8.8%) 50 (12.1%) 38 (8.8%)
Diseases of the digestive system 1575 (5.5%) 40 (5.5%) 21 (4.4%) 19 (5.5%) 45 (6.6%) 45 (5.5%) 30 (6.6%) 35 (6.6%) 23 (5.5%) 20 (4.4%)
Diseases of the genitourinary system 1618 (6.6%) 52 (7.7%) 24 (5.5%) 26 (6.6%) 47 (6.6%) 53 (6.6%) 22 (4.4%) 39 (6.6%) 28 (6.6%) 18 (4.4%)
Complications of pregnancy; childbirth and the puerperium 1307 (5.5%) 19 (3.3%) 11 (2.2%) 5 (1.1%) 27 (4.4%) 32 (4.4%) 10 (2.2%) 13 (2.2%) 12 (3.3%) 11 (2.2%)
Diseases of the skin and subcutaneous tissue 701 (2.2%) 19 (3.3%) 8 (2.2%) 10 (2.2%) 21 (3.3%) 14 (2.2%) 7 (1.1%) 12 (2.2%) 13 (3.3%) 6 (1.1%)
Diseases of the musculoskeletal system and connective tissue 1996 (7.7%) 47 (6.6%) 28 (6.6%) 31 (7.7%) 63 (8.8%) 62 (7.7%) 27 (5.5%) 46 (8.8%) 22 (5.5%) 37 (8.8%)
Congenital anomalies 5 (<0.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (<0.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%)
Injury and poisoning 4061 (15.4%) 111 (16.5%) 69 (15.4%) 71 (18.7%) 113 (16.5%) 118 (14.3%) 79 (16.5%) 77 (14.3%) 60 (14.3%) 79 (18.7%)
Symptoms signs and ill-defined conditions and factors influencing health 2608 (9.9%) 68 (9.9%) 44 (9.9%) 32 (8.8%) 55 (7.7%) 83 (9.9%) 47 (9.9%) 55 (9.9%) 26 (6.6%) 43 (9.9%)
Residual codes; unclassified 331 (1.1%) 9 (1.1%) 5 (1.1%) 6 (1.1%) 5 (1.1%) 6 (1.1%) 4 (1.1%) 3 (<0.1%) 2 (<0.1%) 3 (1.1%)
Self-reported general health
Unknown -- 21 (3.3%) 10 (2.2%) 2 (<0.1%) 16 (2.2%) 18 (2.2%) 9 (2.2%) 10 (2.2%) 13 (3.3%) 8 (2.2%) 0.021
Poor -- 45 (6.6%) 24 (5.5%) 16 (4.4%) 51 (7.7%) 59 (7.7%) 26 (5.5%) 19 (3.3%) 27 (6.6%) 27 (6.6%)
Fair -- 152 (22.0%) 81 (18.7%) 73 (19.8%) 160 (23.1%) 197 (24.2%) 114 (24.2%) 108 (19.8%) 77 (18.7%) 101 (23.1%)
Good -- 210 (30.8%) 141 (33.0%) 124 (33.0%) 223 (31.9%) 234 (28.6%) 133 (27.5%) 198 (35.2%) 122 (29.7%) 149 (35.2%)
Very Good -- 178 (25.3%) 126 (28.6%) 116 (30.8%) 157 (22.0%) 214 (26.4%) 149 (30.8%) 148 (26.4%) 120 (29.7%) 96 (22.0%)
Excellent -- 84 (12.1%) 51 (12.1%) 47 (12.1%) 90 (13.2%) 102 (12.1%) 51 (11.0%) 72 (13.2%) 46 (11.0%) 48 (11.0%)
Self-reported mental health
Unknown -- 21 (3.3%) 11 (2.2%) 1 (<0.1%) 18 (2.2%) 19 (2.2%) 10 (2.2%) 8 (1.1%) 15 (3.3%) 10 (2.2%) 0.008
Poor -- 22 (3.3%) 20 (4.4%) 11 (3.3%) 35 (5.5%) 25 (3.3%) 18 (3.3%) 12 (2.2%) 17 (4.4%) 9 (2.2%)
Fair -- 91 (13.2%) 45 (9.9%) 32 (8.8%) 93 (13.2%) 108 (13.2%) 51 (11.0%) 63 (11.0%) 43 (11.0%) 45 (11.0%)
Good -- 175 (25.3%) 105 (24.2%) 75 (19.8%) 166 (24.2%) 213 (25.3%) 97 (19.8%) 143 (25.3%) 76 (18.7%) 112 (26.4%)
Very Good -- 198 (28.6%) 136 (31.9%) 143 (37.4%) 187 (26.4%) 237 (28.6%) 169 (35.2%) 169 (30.8%) 129 (31.9%) 129 (29.7%)
Excellent -- 183 (26.4%) 116 (26.4%) 116 (30.8%) 198 (28.6%) 222 (26.4%) 137 (28.6%) 160 (28.6%) 125 (30.8%) 124 (28.6%)
Self-reported Education
Unknown -- 21 (3.3%) 13 (3.3%) 3 (1.1%) 22 (3.3%) 26 (3.3%) 13 (2.2%) 12 (2.2%) 13 (3.3%) 9 (2.2%) 0.116
8th grade or less -- 17 (2.2%) 13 (3.3%) 8 (2.2%) 20 (3.3%) 18 (2.2%) 19 (4.4%) 18 (3.3%) 13 (3.3%) 16 (3.3%)
Some high school -- 65 (9.9%) 40 (8.8%) 17 (4.4%) 59 (8.8%) 55 (6.6%) 24 (5.5%) 36 (6.6%) 24 (5.5%) 30 (6.6%)
High School or GED -- 186 (27.5%) 99 (23.1%) 91 (24.2%) 177 (25.3%) 225 (27.5%) 125 (26.4%) 126 (23.1%) 98 (24.2%) 110 (25.3%)
Some college or 2-year degree -- 217 (31.9%) 138 (31.9%) 143 (37.4%) 243 (35.2%) 292 (35.2%) 160 (33.0%) 188 (34.1%) 130 (31.9%) 141 (33.0%)
4-year college graduate -- 93 (13.2%) 65 (15.4%) 62 (16.5%) 86 (12.1%) 89 (11.0%) 61 (13.2%) 91 (16.5%) 66 (16.5%) 61 (14.3%)
More than 4-year college degree -- 91 (13.2%) 65 (15.4%) 54 (14.3%) 90 (13.2%) 119 (14.3%) 80 (16.5%) 84 (15.4%) 61 (15.4%) 62 (14.3%)
Self-reported race/ethnicity
Unknown -- 38 (5.5%) 14 (3.3%) 4 (1.1%) 36 (5.5%) 33 (4.4%) 18 (3.3%) 18 (3.3%) 23 (5.5%) 16 (3.3%) <0.001
Hispanic -- 68 (9.9%) 32 (7.7%) 26 (6.6%) 83 (12.1%) 91 (11.0%) 42 (8.8%) 51 (8.8%) 32 (7.7%) 32 (7.7%)
Native American -- 7 (1.1%) 4 (1.1%) 1 (<0.1%) 7 (1.1%) 9 (1.1%) 4 (1.1%) 4 (1.1%) 2 (<0.1%) 4 (1.1%)
Black -- 91 (13.2%) 48 (11.0%) 30 (7.7%) 92 (13.2%) 124 (15.4%) 35 (7.7%) 54 (9.9%) 31 (7.7%) 43 (9.9%)
Asian or Pacific Islander -- 15 (2.2%) 9 (2.2%) 9 (2.2%) 7 (1.1%) 23 (3.3%) 14 (3.3%) 17 (3.3%) 15 (3.3%) 10 (2.2%)
White -- 431 (62.7%) 320 (73.7%) 301 (79.2%) 419 (60.5%) 499 (60.5%) 365 (75.9%) 398 (71.5%) 291 (71.5%) 318 (73.7%)
Multiracial -- 40 (5.5%) 6 (1.1%) 7 (2.2%) 53 (7.7%) 45 (5.5%) 4 (1.1%) 13 (2.2%) 11 (2.2%) 6 (1.1%)

Notes: Chi-squared values excluded missing or unknown response categories, and excluded categories with zero cells. A p-value of <0.05 is considered statistically significant and is bolded here. The mental illness primary diagnosis category contains diagnoses beyond the ICD-10 F codes that were included in the ineligibility criterion, such as suicidal ideation. All self-reported variables were summarized among responders only, as this information was obtained from the survey.

Appendix Table 2.Characteristics of patients who can be contacted by email and/or text.
Email Text
Characteristic Provided non-missing email and consent to be emailed
N (%)
Adjusted odds ratio (95% CI) Provided mobile telephone and consent to be texted
N (%)
Adjusted Odds Ratio (95% CI)
Age
18–24 1,131 (27.5%) -- 2,729 (66.3%) --
25–34 1,982 (32.1%) 1.30 ( 1.18 , 1.42 ) *** 4,276 (69.2%) 1.03 ( 0.92 , 1.15 )
35–44 1504 (33.3%) 1.43 ( 1.30 , 1.58 ) *** 4,276 (69.2%) 1.03 ( 0.92 , 1.15 )
45–54 1,172 (30.3%) 1.33 ( 1.20 , 1.48 ) *** 2,888 (63.9%) 0.77 ( 0.69 , 0.87 ) ***
55–64 938 (28.2%) 1.20 ( 1.07 , 1.34 ) ** 2,316 (59.9%) 0.61 ( 0.53 , 0.69 ) ***
65–74 678 (31.4%) 1.29 ( 1.14 , 1.46 ) *** 1,837 (55.1%) 0.43 ( 0.37 , 0.49 ) ***
75–84 382 (27.0%) 1.03 ( 0.88 , 1.20 ) 955 (44.2%) 0.24 ( 0.21 , 0.28 ) ***
85 + 133 (18.8%) 0.54 ( 0.44 , 0.67 ) *** 443 (31.3%) 0.11 ( 0.10 , 0.14 ) ***
Gender
Female [reference] 5,515 (34.4%) -- 9,801 (61.2%) --
Male 2,405 (23.4%) 0.57 ( 0.53 , 0.60 ) *** 5,792 (56.4%) 0.83 ( 0.78 , 0.89 ) ***
Emergency Severity Index
High urgency [reference] 877 (29.7%) -- 1,766 (59.7%) --
Medium urgency 3,553 (29.1%) 1.05 ( 0.95 , 1.16 ) 7,357 (60.3%) 1.13 ( 1.00 , 1.26 ) *
Low urgency 2,068 (26.5%) 0.97 ( 0.87 , 1.08 ) 4,755 (61.0%) 1.08 ( 0.95 , 1.22 )
Primary diagnosis
Infectious and parasitic diseases [reference] 108 (26.3%) -- 304 (74.1%) --
Neoplasms 17 (33.3%) 0.97 ( 0.50 , 1.89 ) 28 (54.9%) 0.65 ( 0.32 , 1.34 )
Endocrine; nutritional and metabolic diseases and immunity disorders 124 (30.0%) 1.03 ( 0.74 , 1.43 ) 254 (61.4%) 0.94 ( 0.64 , 1.37 )
Diseases of the blood and blood-forming organs 32 (43.8%) 1.68 ( 0.95 , 2.96 ) 53 (72.6%) 1.86 ( 0.82 , 4.25 )
Mental Illness 26 (20.5%) 0.62 ( 0.37 , 1.03 ) 71 (55.9%) 0.38 ( 0.23 , 0.62 ) ***
Diseases of the nervous system and sense organs 658 (34.1%) 1.14 ( 0.88 , 1.46 ) 1,195 (62.0%) 0.85 ( 0.64 , 1.14 )
Diseases of the circulatory system 701 (34.2%) 1.19 ( 0.92 , 1.54 ) 1049 (51.1%) 0.68 ( 0.51 , 0.90 ) **
Diseases of the respiratory system 872 (28.5%) 0.94 ( 0.74 , 1.21 ) 1,942 (63.5%) 0.96 ( 0.72 , 1.27 )
Diseases of the digestive system 496 (31.5%) 1.00 ( 0.77 , 1.30 ) 983 (62.4%) 0.91 ( 0.68 , 1.22 )
Diseases of the genitourinary system 558 (34.5%) 1.16 ( 0.90 , 1.50 ) 1,038 (64.2%) 0.98 ( 0.73 , 1.31 )
Complications of pregnancy; childbirth and the puerperium 515 (39.4%) 1.23 ( 0.95 , 1.60 ) 997 (76.3%) 0.83 ( 0.61 , 1.13 )
Diseases of the skin and subcutaneous tissue 187 (26.7%) 1.00 ( 0.74 , 1.34 ) 459 (65.5%) 1.02 ( 0.73 , 1.42 )
Diseases of the musculoskeletal system and connective tissue 548 (27.5%) 0.97 ( 0.75 , 1.25 ) 1,301 (65.2%) 1.04 ( 0.78 , 1.39 )
Congenital anomalies 1 (20.0%) 0.52 ( 0.06 , 4.60 ) 4 (80.0%) 0.62 ( 0.06 , 6.35 )
Injury and poisoning 1,120 (27.6%) 0.94 ( 0.74 , 1.20 ) 2,384 (58.7%) 0.79 ( 0.60 , 1.04 )
Symptoms signs and ill-defined conditions and factors influencing health 823 (31.6%) 1.09 ( 0.85 , 1.40 ) 1,530 (58.7%) 0.88 ( 0.66 , 1.16 )
Residual codes; unclassified 83 (25.1%) 0.88 ( 0.61 , 1.26 ) 194 (58.6%) 0.63 ( 0.43 , 0.94 ) *
Hospital
A [reference] 688 (41.2%) -- 1,413 (84.7%) --
B 262 (16.7%) 0.29 ( 0.24 , 0.34 ) *** 1,290 (82.2%) 0.72 ( 0.59 , 0.88 ) **
C 525 (33.4%) 0.76 ( 0.66 , 0.89 ) *** 307 (19.5%) 0.03 ( 0.02 , 0.03 ) ***
D 313 (19.1%) 0.34 ( 0.29 , 0.40 ) *** 1,407 (85.6%) 0.76 ( 0.62 , 0.94 ) *
E 939 (55.1%) 1.75 ( 1.52 , 2.01 ) *** 213 (12.5%) 0.01 ( 0.01 , 0.02 ) ***
F 104 (6.5%) 0.10 ( 0.08 , 0.12 ) *** 1,255 (79.0%) 0.43 ( 0.35 , 0.52 ) ***
G 554 (33.4%) 0.70 ( 0.60 , 0.81 ) *** 1,081 (65.2%) 0.19 ( 0.16 , 0.23 ) ***
H 957 (59.7%) 2.20 ( 1.91 , 2.54 ) *** 782 (48.8%) 0.10 ( 0.08 , 0.12 ) ***
I 732 (44.0%) 1.11 ( 0.96 , 1.28 ) 1,268 (76.3%) 0.32 ( 0.26 , 0.39 ) ***
J 7 (0.4%) 0.01 ( 0.00 , 0.01 ) *** 22 (1.3%) <0.01 (<0.01, <0.01) ***
K 425 (26.0%) 0.46 ( 0.39 , 0.54 ) *** 23 (1.4%) <0.01 (<0.01, <0.01) ***
L 430 (26.5%) 0.52 ( 0.45 , 0.61 ) *** 1,292 (79.6%) 0.64 ( 0.52 , 0.77 ) ***
M 469 (28.5%) 0.53 ( 0.45 , 0.61 ) *** 1,283 (78.1%) 0.35 ( 0.29 , 0.43 ) ***
N 620 (36.8%) 0.78 ( 0.68 , 0.90 ) *** 1,359 (80.6%) 0.40 ( 0.33 , 0.49 ) ***
O 422 (25.4%) 0.47 ( 0.40 , 0.54 ) *** 1,260 (75.9%) 0.34 ( 0.28 , 0.41 ) ***
P 473 (28.3%) 0.52 ( 0.45 , 0.61 ) *** 1,338 (80.0%) 0.39 ( 0.32 , 0.48 ) ***

Notes: *p<0.05, **p<0.01, ***p<0.001; OR = odds ratio; CI = confidence interval. Stars reflect results from a logistic regression model with the outcome of whether a patient could be contacted by email [or text] (0/1) with the following predictors: patient age, gender, Emergency Severity Index, primary diagnosis, and hospital. Sampling batch was also included as a predictor but is not included here; surveys were fielded in 6 two-week batches. The mental illness primary diagnosis category contains diagnoses beyond the ICD-10 F codes that were included in the ineligibility criterion, such as suicidal ideation. Percentages shown are row percentages.


  1. Applicable-to-all items exclude items within the About You and Your Healthcare sections as well as items that require a screener question.

  2. ESI was categorized as follows: Immediate (ESI=1) and Emergent (ESI=2) were categorized as high urgency; Urgent (ESI = 3) was categorized as medium urgency; Semi-urgent (ESI = 4) and Non-urgent (ESI=5) were categorized as low urgency.

  3. Primary diagnosis, obtained from ICD-10 codes, was categorized into 17 categories using the Healthcare Cost and Utilization Project (HCUP) Clinical Classification Software.

  4. Surveys were fielded in 6 two-week batches.

References

CMS. 2018a. “CAHPS® Survey for Accountable Care Organizations Participating in Medicare Initiatives.” http:/​/​acocahps.cms.gov.
———. 2018b. “Emergency Department Patient Experience of Care (EDPEC) Survey.” Centers for Medicare and Medicaid Services. https:/​/​www.cms.gov/​Research-Statistics-Data-and-Systems/​Research/​CAHPS/​ed.html.
———. 2018c. “Home Health CAHPS Publicly Reported State and National Averages: July 2017 - June 2018.” https:/​/​homehealthcahps.org/​Portals/​0/​PublicReporting/​PubRptNtlAndStateAvg2017-07thru2018-06.pdf.
———. 2018d. “Hospital Compare: Patient Survey (HCAHPS) - National.” Centers for Medicare and Medicaid Services. https:/​/​data.medicare.gov/​Hospital-Compare/​Patient-survey-HCAHPS-National/​99ue-w85f.
———. 2018e. “Summary of HCAHPS Survey Results: January 2017 to December 2017 Discharges. Hospital Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey.” https:/​/​www.hcahpsonline.org/​globalassets/​hcahps/​summary-analyses/​summary-results/​2018_10_summary-_analysis_states_results.pdf.
Elliott, Marc N., Megan K. Beckett, William G. Lehrman, Paul Cleary, Christopher W. Cohea, Laura A. Giordano, Elizabeth H. Goldstein, and Cheryl L. Damberg. 2016. “Understanding the Role Played by Medicare’s Patient Experience Points System in Hospital Reimbursement.” Health Affairs 35 (9): 1673–80.
Google Scholar
Elliott, Marc N., Alan M. Zaslavsky, Elizabeth Goldstein, William Lehrman, Katrin Hambarsoomians, Megan K. Beckett, and Laura Giordano. 2009. “Effects of Survey Mode, Patient Mix, and Nonresponse on CAHPS® Hospital Survey Scores.” Health Services Research 44 (2): 501–18.
Google Scholar
Fowler Jr, Floyd J., Carol Consenza, Lauren A Cripps, Susan Edgman‐Levitan, and Paul D Cleary. 2019. “The Effect of Administration Mode on CAHPS Survey Response Rates and Results: A Comparison of Mail and Web‐based Approaches.” Health Services Research 54 (3): 714–21.
Google Scholar
Giordano, Laura A., Marc N. Elliott, Elizabeth Goldstein, William G. Lehrman, and Patrice A. Spencer. 2010. “Development, Implementation, and Public Reporting of the HCAHPS Survey.” Medical Care Research and Review 67 (1): 27–37.
Google Scholar
Hospice, C.A.H.P.S. 2018. “CAHPS Hospice Survey Quality Assurance Guidelines.” https:/​/​www.hospicecahpssurvey.org/​en/​quality-assurance-guidelines/​.
Kaplowitz, Michael D., Frank Lupi, Mick P. Couper, and Laurie Thorp. 2012. “The Effect of Invitation Design on Web Survey Response Rates.” Social Science Computer Review 30 (3): 339–49.
Google Scholar
Klein, David J., Marc N. Elliott, Amelia M. Haviland, Debra Saliba, Q. Burkhart, Carol Edwards, and Alan M. Zaslavsky. 2011. “Understanding Nonresponse to the 2007 Medicare CAHPS Survey.” The Gerontologist 51 (6): 843–55.
Google Scholar
Liu, Mingnan, and Nick Inchausti. 2017. “Improving Survey Response Rates: The Effect of Embedded Questions in Web Survey Email Invitations.” Survey Practice 10 (1): 2791.
Google Scholar
Mathews, Megan, Layla Parast, Anagha Tolpadi, Marc Elliott, Elizabeth Flow-Delwiche, and Kirsten Becker. 2019. “Methods for Improving Response Rates in an Emergency Department Setting - A Randomized Feasibility Study.” In press.
Parast, Layla, Megan Mathews, Anagha Tolpadi, Marc Elliott, Elizabeth Flow-Delwiche, and Kirsten Becker. 2019. “National Testing of the Emergency Department Patient Experience of Care (EDPEC) Discharged to Community (DTC) Survey and Implications for Adjustment in Scoring.” Medical Care 57 (1): 42–48.
Google Scholar
Trouteaud, Alex R. 2004. “How You Ask Counts: A Test of Internet-Related Components of Response Rates to a Web-Based Survey.” Social Science Computer Review 22 (3): 385–92.
Google Scholar
Weinick, Robin M., Kirsten Becker, Layla Parast, Brian D. Stucky, Marc N. Elliott, Megan Mathews, Chris Chan, and Virginia Kotzias. 2014. “Emergency Department Patient Experience of Care Survey: Development and Field Test.” Santa Monica, CA: RAND Corporation. https:/​/​www.rand.org/​pubs/​research_reports/​RR761.html.

This website uses cookies

We use cookies to enhance your experience and support COUNTER Metrics for transparent reporting of readership statistics. Cookie data is not sold to third parties or used for marketing purposes.

Powered by Scholastica, the modern academic journal management system