Introduction

This paper details the importance of a flexible evaluation methodology that allowed West Virginia University Health Affairs Institute (HAI) to make recommendations on the ideal length of training for Direct Care Professionals (DCPs) during the first eight months of a training surge, known as the pilot phase. West Virginia (WV) is facing challenges around a shortage of DCPs: one study projected a shortage of 2,800 DCPs in WV per year through 2030 (Christiadi and Deskins 2021). Thus, in 2022, the WV Direct Care Taskforce, led by the AARP WV, was charged with developing legislative and regulatory recommendations to increase WV’s direct care workforce. The direct care workforce, as it pertains to this project, provides hands-on support to clients in their private homes. This workforce is a critical component of the healthcare system, responsible for providing essential care and support to individuals with disabilities, chronic illnesses, and other healthcare needs. Not only is the current workforce insufficient to meet needs, but the state’s older population is projected to increase by 2030 (Christiadi and Deskins 2021; PHI National 2022), increasing demand for such services. Without interventions, future needs will likely go unmet (West Virginia Direct Care Task Force 2022; Institute of Medicine (US) Committee on the Future Health Care Workforce for Older Americans 2008). The projected DCP shortage is problematic for healthcare providers, family caregivers, and clients alike, as it can result in inadequate care, increased healthcare costs, and poorer health outcomes for clients in need (Institute of Medicine (US) Committee on the Future Health Care Workforce for Older Americans 2008; Seavey 2004).

Improving training and, in turn, employee readiness to meet the demands of the field is crucial. However, gathering key feedback from a WV workforce poses challenges for evaluators. WV is among the three most rural states in the nation, and it is the only state entirely within the Appalachian region. Geographical challenges and a complex historical context contributing to mistrust in research means collecting information from these populations can be difficult (Kennedy-Rea et al. 2021), making a flexible evaluation approach critical.

To address the DCP shortage, the Direct Care Taskforce produced a list of recommendations around compensation, education, and training (West Virginia Direct Care Task Force 2022). Among the recommendations selected by the WV Department of Human Services (DoHS), Bureau for Medical Services, was the implementation of a Direct Care Workforce Training Surge to recruit and train up to 800 DCPs over two years to address WV’s immediate shortage. Following this recommendation, HAI partnered with subject matter experts to develop the WV Pathways to Professional In-Home Care Training (WV Pathways Training). DoHS provided funding for trainee stipends to help offset costs like childcare and transportation, wage replacement during training, and incentives to participate in follow-up interviews. Trainees accessed training content via Activated Insights’ online learning platform, referred to as a Learning Management System (LMS).

HAI conducted a state- and country-wide scan of DCP training and found variation in length and content. In WV, many agencies offered minimal (two days) initial training. The Direct Care Taskforce recommended more in-depth initial training (upwards of four weeks), hypothesizing that additional training would lead to greater job preparedness and ultimately increased retention. The pilot phase was used to assess training length to 1) maximize trainee sense of preparedness for the job; 2) be palatable for agencies in terms of logistics and cost; and 3) improve job retention.

HAI consulted with Activated Insights, WV-based provider organizations, and national subject matter experts to select content for training of three different lengths:

  1. A two-day course, focusing on state requirements

  2. A five-day course, allowing for exposure to additional training content

  3. A seven-day course, allowing for more in-depth exposure to additional training content

While the overarching training surge took place in 2024 and 2025, the first eight months were considered a pilot phase, during which focus was on determining an ideal training length based on evaluation data. During the pilot phase, 356 individuals were trained.

The Mixed Methods Approach

During the pilot phase, four training providers across multiple locations offered rotating two-day, five-day, and seven-day trainings. Participants were assigned a training length based on geographic proximity to a location and length of training at that location at the time of sign up; participants could not choose their training length. Overall, 44% (n=158) of individuals trained during the pilot phase attended a two-day training, 37% (n=132) attended a five-day training, and 19% (n=66) attended a seven-day training. To determine an ideal training length, HAI aimed to compare the three groups based on the following variables: employment after training, test scores from the LMS, DCP self-reported competency, knowledge gain, satisfaction, and preparedness for the job. The Quantitative Data Collection and Initial Analysis section below outlines HAI’s initial setup for data collection, followed by Pivots and Resulting Changes in Qualitative Data Collection that outlines necessary pivots and the importance of a flexible approach, allowing HAI to compare the three groups based on the outlined variables.

The evaluation was guided by the Centers for Disease Control and Prevention’s (CDC) Program Evaluation Framework (Centers for Disease Control and Prevention 2024), which positions all stakeholders as experts and emphasizes frequent, low-burden feedback. Complementing the Framework, the evaluation used an explanatory sequential mixed methods design, where quantitative data collection and analysis informed the design of qualitative data collection approaches and instruments (Creswell and Plano Clark 2011).

Figure 1 depicts how utilizing the Framework in combination with an explanatory sequential mixed methods design ensured an iterative approach to data collection and analysis, while allowing a range of stakeholder input during all stages of the training surge pilot phase.

Figure 1
Figure 1.Explanatory Sequential Project Design

Quantitative Data Collection and Initial Analysis

The first step in an explanatory sequential mixed methods design is quantitative data collection. To gain initial insight into which training lengths led to the most DCPs hired after training, as well as DCP self-reported competency, knowledge gain, satisfaction, and preparedness for the job, five surveys were developed:

  • Registration

  • Touchpoint

  • Post-Training

  • 2-Week

  • 6-Week

While surveys were initially intended to be administered through the LMS, exporting data proved challenging. Therefore, one month into training, the evaluators pivoted to administering all surveys online using Qualtrics. Survey questions were developed based on a review of the literature, conversations with training organizations, and feedback from stakeholders, including national subject matter experts and state partners.

On the first day of training, trainees filled out a Registration Survey that captured standard demographic data. In addition to the Registration Survey, trainees completed Touchpoint Surveys throughout the training at the end of days two, five, and seven (Table 1). The Touchpoint Survey collected information on:

  • How training influenced trainees’ understanding of DCP jobs

  • DCP confidence in their ability to provide in-home care

  • Overall satisfaction with training

  • Whether the training met expectations

  • Trainee intention to accept a DCP job, if offered

In addition to Registration and Touchpoint Surveys, trainees completed Post-Training Surveys, which collected information on training content and materials, satisfaction with trainers, and perceived knowledge gains related to providing direct care before and after training.

Survey response rates were calculated based on 1) all trainees that completed training during the pilot phase (n=356, May-December 2024), and 2) trainees that attended training after switching survey administration from the Activated Insights platform to Qualtrics (n=339, June-December 2024). Response rates were higher once surveys were available on Qualtrics: Registration Survey (86% vs. 90%), Touchpoint Surveys (79% vs. 83%), and Post-Training Survey (80% vs. 84%). Overall, response rates were extremely high as trainees were on-site with dedicated time to complete these surveys, as well as access to technical support, devices, and an internet connection. Timing of surveys differed based on training length (Table 1).

Table 1.Trainee Survey Cadence by Training Length
Training Length Registration Survey Touchpoint Survey Post-Training Survey
2-Day Training End of Day 1 End of Day 2 End of Day 2
5-Day Training End of Day 1 End of Days 2 and 5 End of Day 5
7-Day Training End of Day 1 End of Days 2, 5, and 7 End of Day 7

After training concluded, trainees were invited to take two follow-up surveys, one 2-Week Survey and one 6-Week Survey, using links sent via email. Both surveys collected information on whether trainees were currently employed as DCPs. If trainees indicated they were not currently employed as DCPs, the survey prompted them to select from four other statuses to understand what happened after they completed training. The 2-Week Survey was also designed as a recruitment tool, asking trainees to participate in a 30-to-60-minute interview about their experience.

Approximately three months into data collection, anecdotal feedback provided by training agency staff indicated that sending 2-Week and 6-Week Survey invitations via text message instead of email might elicit more responses, leading to a pivot in how surveys were distributed. Text messages were sent from a WV phone number approximately two and six weeks after training completion, with a reminder sent one week after the initial message. Text messages were short, stating “Please follow this link to let us know what you thought of the WV Pathways to Professional In-Home Care Training.” for the 2-Week Survey and “One last time we’d like to hear about your experience in the Professional In-Home Care Training!” for the 6-Week Survey.

HAI planned to use Activated Insights data on LMS test attempts, failures, and scores to determine any difference in knowledge gain between the different training lengths. While it was expected that this data would be available, initial exports from the LMS included most recent test attempts only. HAI pivoted and incorporated retrospective post-test questions on self-reported trainee knowledge gain into the Post-Training Survey:

  1. How much did you know about direct care work before this training?

  2. How much do you know about direct care work after taking the training?

Retrospective post-tests allow respondents to express, after receiving an intervention, how much they knew before the intervention, and how much they knew after, in two questions posed at the same time, after the training. Studies find that retrospective post-assessments show comparable results to traditional pre-post assessments and avoid response shift bias (Bhanji et al. 2012; Cors and Bell 2022).

Pivots and Resulting Changes in Qualitative Data Collection

In an explanatory sequential mixed methods design, quantitative findings inform the design of qualitative data collection. To gain deeper insight into trainees’ experience, competency level, satisfaction, and preparedness to provide direct care, HAI sought to interview trainees directly. Semi-structured interviews are ideal during the second step of an explanatory sequential mixed methods design, as they allow researchers to design a script with specific questions while also allowing participants to share thoughts on topics they find important. This provides the interviewer with the opportunity to further explore themes that emerge during the interview (Adams 2015).

After an initial two months of data collection and analysis during the pilot phase, based on survey data and feedback provided by trainers during regular touchpoints, interview guides were developed for five groups HAI hoped to establish from 2-Week and 6-Week Survey data: 1) did not complete the training (not employed), 2) not offered a job after training (not employed), 3) did not accept the job (not employed), 4) were offered a job and accepted but have since quit (not employed), 5) individuals employed as a DCP. Interview topics included input on training content and length, experience with technology and resources, marketing materials, experience with recruitment and scheduling, as well as questions about employment.

The 2-Week Survey was designed as a recruitment tool for interviewees; however, pivoting from an email to a texting approach did not improve response rates, making meaningful analyses and further data collection via interviews not feasible. During a five-month period, only 17 individuals indicated they were interested in being contacted about a follow-up interview; of those 17, seven were reached, but only two completed phone interviews.

Due to the low survey response rate and difficulty following up with those respondents willing to be interviewed, HAI pivoted to a cold calling approach. This required a more condensed interview guide allowing evaluators to gather the most important information from any trainee, regardless of employment status, in a shorter time span (15 to 20 minutes, instead of 30 to 60 minutes). Interview topics still included feedback on training content and length, employment related questions, and reasons for participating in the training. However, some questions such as ‘We have heard that some days are more intense or challenging than others. Any thoughts on this?’ and marketing related questions were removed, since answers to these questions could be found elsewhere (trainer feedback, zip codes attached to survey data, etc.). Technology and resource-related questions, and descriptions of the recruitment and scheduling process were only discussed if time permitted.

In the cold calling approach, two phone call attempts were made for trainees who had completed training at least six weeks prior. During regular touchpoints, trainers mentioned trainees’ enthusiasm for a training completion certificate. Therefore, HAI offered $25 Visa gift card incentives to complete an interview during calls, and an opportunity to verify their email address, ensuring they would receive their completion certificate. Once HAI cold called 73 trainees, interviewing 19, no new information emerged and thus interviews concluded. Interviews included trainees that attended two-day, five-day, and seven-day trainings, were trained across locations by all four training providers, and were employed and not employed as DCPs at the time of the interview. Interestingly, in many cases, trainee excitement about the certificate outweighed the excitement expressed for Visa gift cards, emphasizing trainees’ intrinsic motivation to complete the training, and pride in the work. This again highlights the importance of a flexible evaluation approach: understanding trainees’ intrinsic motivation allowed evaluators to tailor the incentive to the target population. HAI ensured that all trainees were made aware that receiving a training certificate was not contingent upon participation in the follow-up interview.

Concluding Discussion

Using a flexible explanatory sequential mixed methods approach allowed for a better understanding of DCP employment after training and self-reported experiences with the three training lengths than a solely qualitative or quantitative study would have. The first quantitative portion of the evaluation took place in an environment where trainees had access to an internet connection, devices, and technical support; response rates were high. Post-training, response rates were significantly lower; it was no longer feasible to collect adequate data from trainees via online surveys. Thus, one benefit of the mixed methods approach described in this paper is the flexibility to change direction and work with partners to understand the best way to reach the population, focusing on information gaps from the quantitative stage. Instead of relying on additional survey data, phone interviews allowed a deeper understanding of survey results, and a clearer picture of trainee experiences.

During trainee interviews, it became apparent that many trainees were intrinsically motivated to complete the WV Pathways Training, citing a drive to help others and pride in the work. Trainees expressed appreciation for receiving the completion certificate as it physically documented their achievement in something they were proud of. HAI leveraged this intrinsic motivation to improve interview response rates by asking to verify trainee email addresses (to send the completion certificate) before requesting an interview when calling trainees. This highlights not only the need for flexible methodology, but also the importance of understanding and responding accordingly when working within a population that historically has been considered “hard-to-reach.”

The flexible mixed methods approach used to evaluate the WV Pathways Training gave HAI the ability to pivot along the way and apply results quickly, within the first eight months of the training surge, rather than once the entire training surge was over. As data were analyzed and recommendations became apparent, HAI implemented changes while training was ongoing to increase DCP preparedness and employment in real time. The flexible approach further allowed HAI to recommend an ideal training length of five days, which was implemented at all training locations after the pilot phase concluded.

Corresponding author contact information

Lena Stevens, PhD
Health Affairs Institute, West Virginia University
64 Medical Center Dr., PO Box 9191, Morgantown, WV 26506