Development of a tool for assessing the performance of long-term care systems in relation to care transition: Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC)

Abstract

Background

Improving the quality and safety of care transitions is a priority in many countries. Carrying out performance measurements play a significant role in improving quality of decisions undertaken by different actors involved in reforms. Therefore, the main objective of this paper is to present the development of an evaluation tool for assessing the performance of long-term care systems in relation to care transition, namely the Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC). This study is performed as part of a larger European TRANS-SENIOR project.

Methods

The development of the TCAT-LTC involved three steps. First, we developed a conceptual model based on Donabedian’s quality framework and literature review. Second, we carried out a thorough process of item pool generation using deductive (systematic literature review) and deductive-inductive methods (in-depth interviews) with experts in the field of long-term care. Third, we conducted preliminary validation of the tool by asking experts in research and practice to provide an opinion on a tool and to assess content validity. Future fourth step will involve a tool’s pilot with country experts from Germany, the Netherlands and Poland.

Results

By applying methodological triangulation, we developed the TCAT-LTC, which consists of 2 themes, 12 categories and 63 items. Themes include organizational and financial aspects. Organizational aspects include categories such as communication, transfer of information, availability and coordination of resources, training and education of staff, education/support of the patient/informal caregiver, involvement of the patient/informal caregiver, telemedicine and e-Health, and social care. Financial aspects include categories such as primary care, hospital, and long-term care. We also present the instructions on the application of the TCAT-LTC.

Conclusions

In this paper, we presented the development of the TCAT-LTC evaluation tool for assessing the performance of long-term care systems in relation to care transition. The TCAT-LTC is the first tool to assess the performance of long-term care systems in relation to care transition. Assessments can be carried out at the national and international level and enable to monitor, evaluate, and compare performance of the long-term care systems in relation to care transition within and across countries.

Background

Care transitions are vulnerable exchange points for older adults with complex care needs [1, 2]. Older persons often require care services from different practitioners in multiple settings, but practitioners tend to work in silos and are unaware of services delivered in previous settings [3]. Lack of coordination, communication, and transfer of information between the settings may lead to poorly executed transitions [4, 5]. Nonetheless, not only organizational aspects may affect the care transition of older adults. A recent study by Wieczorek and colleagues [6] pinpointed the importance of financial aspects (provider payment mechanism, reward, and penalty) and their impact on care transition in long-term care systems. A growing body of evidence suggests that a high proportion of care transitions among older adults is far from optimal. Fragmented care transitions are often associated with preventable adverse events, rehospitalizations and compromised patient outcomes [7,8,9]. Moreover, suboptimal care transitions may lead to unnecessarily high rates of health service use and health care spending in both, health and social care systems [10]. The recommendation of the World Health Organization is to avoid, if possible, or to optimize transitions between the settings as they are high-risk scenarios for patient safety [11]. Given the importance of this issue, improving the quality and safety of care transitions is an international priority, and efforts are being made by governments worldwide to optimize care transitions [12, 13].

Nonetheless, to improve quality of decisions undertaken by different actors such as practitioners, managers, governments, policymakers, and payers/insurers, health system performance measurements are needed [14]. Performance measurement instruments have two important goals, first, to promote accountability, and second, to improve the performance of the system. According to Donabedian [15], there are three approaches to assessment. The first approach focuses on the “structure”; the second one focuses on the “process” and the last one on “outcomes”. Assessments examining the “structure” study the settings and instrumentalities with which care is delivered. It might refer to the adequacy of facilities and equipment but also to the training and qualifications of the staff. At the same time, examining “process of care” allows us to answer the question: of whether health care (in this case, transitional care) is properly practised. Process measures may be indicators of future success or failure [15]. Process indicators are easy to measure, to interpret, provide clear pathways for action, and capture aspects of care that are valued by patients [16]. The last approach focuses on “outcomes” and has been widely used as an indicator of the quality of medical care. Outcome indicators reflect the impact of the health care service on the patient. Examples of outcome measures include mortality, survival, disease prevalence etc. Nevertheless, the use of outcome as the criterion for quality is questioned because many other factors other than medical care could affect the outcome [16].

Currently, to the best of our knowledge, there is no assessment tool dedicated to measuring the performance of long-term care systems in relation to care transition. Existing tools, such as Care Transition Measure (CTM) and Partners at Care Transitions Measure (PACT-M) do not assess care transition as part of the long-term care system. There are plenty of measures that assess only selected aspects related to care transition (e.g., discharge planning, patients’ experience) or focus on care transition between specific settings such as the hospital, home etc. [17,18,19]. For instance, the Care Transition Measure (CTM) is a tool used to assess the quality of the transition between hospital and home [20]. Similarly, PACT-M also focuses on care transition from hospital to home [21]. Existing tools, even though valued, have a narrow focus. According to the Institute of Medicine [22] and the report “To Err is Human” efforts to improve patient safety should be centered around the system rather than providers. Likewise, OECD report titled “Caring for Quality in Health” also emphasizes the importance of systemic changes and their impact on quality and efficiency of care [23]. For the purpose of this study, we define long-term care system as all organizations, providers, individuals, and actions with the primary aim to promote, maintain and/or improve the wellbeing, health and functional ability of individuals with limitations in intrinsic capacity [24].

The main objective of this paper is to present the development of an evaluation tool for assessing the performance of long-term care systems in relation to care transition. We provide details of the methods used to develop this tool, which was named Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC), as well as the tool itself and the guide on how to apply it. The results of the application of the tool will be reported elsewhere. This study is performed as part of a larger European TRANS-SENIOR project focused on avoiding unnecessary care transitions and improving care for transitions that are needed.

Methods

The development of the TCAT-LTC involved three steps (Fig. 1). We followed guidelines on scale development by DeVellis [25]. First (1), we developed a conceptual model based on Donabedian’s quality framework and literature review carried out by Wieczorek and colleagues [4, 6, 15]. Second (2), we carried out a thorough process of item pool generation using deductive and inductive methods as recommended by DeVellis and Morgado [25, 26]. In this step, we performed a systematic literature review (deductive method) and semi-structured, in-depth interviews (deductive-inductive method) with experts in the field of long-term care. Third (3), we conducted preliminary validation of the tool by asking experts in research and practice to provide an opinion on the tool and to assess content validity. Future fourth step will involve a tool’s pilot with country experts from Germany, the Netherlands and Poland.

Fig. 1

Flowchart of the tool development process

Step 1. Development of a conceptual model

For the purpose of this study, we defined transitional care as “a set of actions designed to ensure the coordination and continuity of health care as patients transfer between different locations or different levels of care within the same location. Representative locations include (but are not limited to) hospitals, sub-acute and post-acute nursing facilities, the patient’s home, primary and specialty care offices, and long-term care facilities” [27] p556. Thus, in this study, we focus on care transitions occurring in both, health care and social care sector, and between those sectors. We adopt this approach given the focus of our study on long-term care systems. World Health Organization [24] suggests that a long-term care system encompasses all organizations, providers, individuals, and actions that’s objective is to promote, maintain or improve the wellbeing, health, and functional ability of persons with limitations in intrinsic capacity. Moreover, given that the presented study is conducted along the European TRANS-SENIOR project that focuses on the optimization of care transitions of older adults, the primary focus of this study is on older adults. This patient group is particularly often in need of long-term care services and therefore, at higher risk of care transitions. Even though, the focus of our study is on older adults, the results of this study could be used for other patient groups as well. However, it is crucial to consider the specific needs of studied groups that might differ from those of older patients. We built the assessment tool involving two approaches out of three proposed by Donabedian, namely structure and process [15]. By focusing on these two approaches, we want to provide the evaluators with a better understanding of the relative magnitude of associations between structure and process and their impact on quality of care [15]. Through a literature review, we defined important core organizational and financial aspects that are relevant to care transition and decided that TCAT-LTC will focus on the following areas:

• How well is long-term care system performing when it comes to organizational aspects of care transition?

• How well is long-term care system performing when it comes to financial aspects of care transition?

Step 2. Item pool generation

Item pool generation had two phases. First, we used a combination of deductive and inductive methods to build on the item pool, namely, we conducted a systematic literature review and semi-structured in-depth interviews with experts in long-term care. Second, we carried out multiple meetings with the research team to discuss the relevance and clarity of items and to refine the item list.

Literature review

We used MEDLINE, Embase and CINAHL to search for relevant studies between 2005 and 2020 using three components to build the search terms: (1) old or geriatric or senior; (2) care transition or coordinated care or care continuity; (3) financing or organization. The search strategy was consulted with an academic health sciences librarian. The detail on the review methodology can be found in the published articles [4, 6] and on the International Prospective Register of Systematic Reviews (PROSPERO) platform under identification number CRD42020162566. The review results were used to build on the item pool by identifying key core organizational and financial aspects that are relevant for care transition.

Semi-structured, in-depth interviews with experts

Design

We used a qualitative research design to understand what kind of organizational and financial aspects affect care transition in long-term care systems. Detailed information on the interviews is provided in Appendix 1 using the COnsolidated criteria for REporting Qualitative research (COREQ) checklist [28]. Below, some key methodology aspects are presented.

Participants

We used a purposive sampling method to identify country experts in long-term care and care transition in Germany, the Netherlands, and Poland. To be included in the study, participants had to (1) represent either providers from primary care, hospital, long-term care or payers/insurers. Also, they had to (2) have some experience with care transitions of older adults and (3) be familiar with one of the long-term care systems in Germany or the Netherlands or Poland. They also had to (4) speak English, German or Polish. We contacted by e-mail 23 potential participants and only one of the approached participants did not respond to the invitation to the study. We provided the respondents with detailed information about the study prior to the interview. All participants suggested the time and the mode/place for the interview. All the interviewees provided informed consent and voluntarily participated in the study. In total, 22 semi-structured interviews were conducted with country experts (8 experts from Germany, 8 experts from the Netherlands (one dyadic interview) and 7 experts from Poland).

Data collection

Interviews were conducted by the main researcher Estera Wieczorek (EW) with the help of a second researcher Christoph Sowada (CS). More information about the members of the research team and interviews can be found in the Appendix 1. At first, the interview guide was built based on the results from the literature review. The interview guide was discussed, modified, and accepted by the research team. The relevant topic list can be found in the Appendix 2. The first three interviews confirmed that the guide was clear to participants and thus, no adjustments were needed. The interviews were scheduled in the place/mode and at the time suggested by the participant. Majority of the interviews (18 out of 22) were carried out online due to the COVID-19 pandemic. Three interviews were face-to-face and carried out in the workplace of the participants, and one respondent provided the answers through e-mail. All interviewees were carried out once (without repeated interviews) with only the participant and an interviewer/s being present. Each interview lasted, on average 52 minutes (range: 27-107 minutes) and was recorded. Field notes were also taken during the interview. We then transcribed the recordings using Verbatim method (word by word) and sent the transcripts for a member check. Only 2 respondents provided some minor changes to the transcripts. Ethical considerations regarding this study are explained in the Appendix 1.

Data analysis

All the data was downloaded, coded, and analyzed using the method of qualitative content analysis. The analysis was facilitated with the use of ATLAS.ti Version 22. All interviews were coded using a deductive-inductive approach, i.e. the initial set of codes (themes/categories) was informed by the priori literature review, while additional codes (sub-themes/sub-categories) emerged from the interviews. Interviews in English and Polish were coded by the main researcher EW, who is a native Polish speaker, and a fluent English speaker. Interviews in German were coded by a second researcher CS, who is a native German speaker, fluent Polish, and English speaker; the main researcher EW was also involved to ensure uniformity of coded data. The results were used to challenge the categories coming from the literature review, refine categories, and develop items.

Step 3. Preliminary validation of the tool

The preliminary validation of the tool was performed in two stages. First, the tool was discussed at four separate research team meetings to check for the clarity of the items and to agree on the first draft of the final item pool. Second, we sent an invitation by e-mail to 6 experts in research and practice to preliminary validate the TCAT-LTC tool. Expert panel consisted of 5 experts - two professors and an associate professor in aging and long-term care, an associate professor and assistant professor in health system organization and financing. Experts received an online document and were requested to fill out the form regarding TCAT-LTC tool. The form included a definition of transitional care and short information about the study, the questions regarding the relevance and clarity of each indicators/items. Relevance of an item was rated using a rating scale with 3 response categories: “very relevant”, “somehow relevant”, “not relevant”. Moreover, next to each indicator, experts were invited to provide comments and suggestions for improvement. At last, the form included optional fields where experts could provide general comments and suggestions regarding each category of indicators (e.g., communication), and propose items that should be added to each category. Respondents had 4 working days to provide responses and to send the filled form back by e-mail. All experts could contact the main researcher EW in case of questions. After receiving responses from the experts, the research team met again to analyze the responses. The results were used to review and refine items and categories, and to further improve the tool.

Results

Step 1. Development of conceptual model

Based on Donabedian’s three-components approach, structure measures may have an effect on process measures, and ultimately affect the outcome measures [15]. Based on Donabedian’s quality framework, organizational and financial aspects could be recognized as structure and process indicators. Systematic literature review that we performed served as a theoretical foundation and was conducted to identify general organizational and financial aspects that may affect care transition (Fig. 2). In line with Donabedian’s model, these aspects may affect the outcome (e.g. quality of care transition).

Fig. 2

Organizational and financial aspects that affect care transition

Step 2. Item pool generation

The literature identified in the search pointed out to multiple organizational and financial aspects that may affect care transition in long-term care systems. Organizational aspects included: communication among involved professional groups, transfer of information and care responsibility of the patient, coordination of resources, education and involvement of the patient and family, training and education of staff, e-Health and social care. Financial aspects included: provider payment mechanism, rewards and penalties. More detailed information on the review findings can be found in recently published articles [4, 6]. Findings from the systematic review provided us with a guiding framework for developing the qualitative study.

After developing the guiding framework for our qualitative study, we conducted 22 interviews with country experts from Germany, the Netherlands and Poland (8 experts from Germany, 7 experts from the Netherlands and 7 experts from Poland). Of those, 18 participants represented providers (7 individuals represented long-term care, 6 primary care and 5 hospital), and four respondents represented payers/insurers. The analysis of the in-depth interviews revealed important organizational and financial aspects affecting care transition in their countries. The exact results and codes for each category, for each country can be found in Appendix 3. We used the responses from the experts to challenge the categories coming from the literature review, refine categories, and develop items. During interviews, experts were asked to discuss in detail all organizational and financial aspects that may affect care transition. There were also requested to indicate potential problems and solutions. Their responses enabled us to build a detailed and comprehensive item pool by developing items for each category. For instance, when discussing the category related to availability and coordination of resources, experts suggested a different type of resources relevant for care transition, among others – human resources. Moreover, country experts elaborated on communication in more detail and provided us with items that make communication effective (e.g., timely and direct communication between providers). At the same time, we also used responses from the interview to create new categories of items. For example, some respondents emphasized the importance of including patient and carer in decision-making process and considering their preferences. As a result, involvement of the patient/family/informal caregivers’ category was added.

Step 3. Preliminary validation of the tool

Research team members met 4 times to analyze and refine each category and item included in the tool. After each session, adjustments to the tool have been made by unanimous decision of the team members. During the fourth meeting, the research team agreed on the final version of the tool, which was sent to six experts for validation. All six experts in research and practice responded to our invitation to provide us with their opinion and feedback on the tool. Nonetheless, one of the experts could not provide the response due to time constraints. Five of the experts sent their responses via e-mail and provided us with the items’ relevance rating, comments, and suggestions for improvement. Almost all experts recognized the relevance of the items included. Nevertheless, for a couple of items, the relevance and clarity were questioned. Experts also proposed to clarify and merge some items. After receiving filled forms from the experts, the research team met again to analyze each response. As a result, we adjusted the names of categories, combined, or removed items following the sumscore decision rule (defined as the total score for an item across all judges) (threshold accepted - more than 50% of respondents had to consider the item as “not relevant”) [26], and we changed the names of some items. The results on the relevance of each item can be found in Appendix 4. Additionally, we added some more explanations to some items. During an online meeting research team unanimously agreed on the new version of the tool.

Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC)

By applying methodological triangulation based on the three steps presented above, we finalized the TCAT-LTC presented in Table 1. The tool focuses on care transitions occurring in both, health care and social care sector, and between those sectors. TCAT-LTC is designed as an assessment tool that can be used internally or externally by different stakeholders at different levels of the LTC system. TCAT-LTC consists of 2 themes, namely, organizational and financial aspects. Organizational aspects are divided into 8 categories, and there are 3 categories regarding financial aspects. Organizational aspects include categories: communication, transfer of information, availability and coordination of resources, training and education of staff, education/support of the patient/informal caregiver, involvement of the patient/informal caregiver, telemedicine and e-Health, social care. Financial aspects include following categories: primary care, hospital, long-term care. Each category entails dedicated items. In total, TCAT-LTC consists of 63 items. TCAT-LTC could be completed by hand or electronically. Optimally, the assessment should be carried out by at least 2 experts in the field of transitional care of older adults. Moreover, the experts should be aware of the functioning and financing of health and long-term care systems in the assessed country. Experts might make use of data previously collected for other reports and assessments, for instance, health system performance assessment framework of a given country. Nonetheless, some of the information will need to be generated anew. With all necessary information available, the completion of the assessment takes around 2-3 hours, depending on the level of expertise of evaluators. We recommend performing an assessment of the performance of long-term care systems in relation to care transition using TCAT-LTC at least once a year.

Table 1 Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC)

Each question/item can be graded on a three-grade scale. Depending on the answer, countries can score 3, 2 or 1 points, where 3 points are the highest score, and 1 point is the lowest score. If the answer for an item was “not applicable” then the item is excluded from the assessment. Similarly, in case of missing data, there should be an annotation “missing data”, and such an item is excluded from the assessment. Nonetheless, respondents may use “not applicable” and “missing data” options only in justified cases. The exact instructions for the scoring of each question in the TCAT-LTC can be found in Table 2.

Table 2 Transitional Care Assessment Tool in Long-Term Care (TCAT-LTC) Instruction

At the end of the questionnaire, the total score can be calculated. Evaluators should first sum up the scores from all items for which responses were provided, and then divide the total sum by the maximum number of points that could be scored for all items (excluding items with answer “not applicable”, “missing data”). At last, the divided score should be multiplied by 100% to obtain score as a percentage.

For instance, a country scored 142 points in 61 items (2 items were excluded because there were not applicable), therefore, (142 / 183 * 100% = 77,6%). The score can be used as a rough indication on the performance of a country’s long-term care system in relation to care transition. The higher the percentage, the more items considered important for care transition have been addressed by the long-term care system.

Discussion

The objective of this paper was to present the development of an evaluation tool for assessing the performance of long-term care systems in relation to care transition. We elaborated in detail on the methods used to develop the tool. The TCAT-LTC is, to our knowledge, the first tool that looks at the performance of long-term care systems in terms of organizational and financial aspects, and their relation to care transition.

The proposed TCAT-LTC assess long-term care performance in relation to care transition using a structure and process approach. The TCAT-LTC consists of 63 questions/items, grouped into 2 themes (organizational and financial) and 12 categories. Many of the items in the TCAT-LTC are related and may influence one another. For instance, the number of staff in LTC, number of beds in LTC facilities and appropriateness of reimbursement level may have an impact on waiting time for LTC. The TCAT-LTC shows the interrelation between organizational and financial aspects, and structure and process.

As confirmed by the experts’ validation, the TCAT-LTC is a helpful tool that separates the long-term care system into manageable parts by identifying organizational and financial aspects that are relevant to care transition. Assessments using the tool can be carried out at the national and international level to help to monitor, evaluate, and compare performance of the long-term care systems in relation to care transition within and across countries. Moreover, the TCAT-LTC aims to inform decision-makers and thus, improve the quality of the decisions undertaken by different stakeholders regarding care transition. Applying the TCAT-LTC enables us to shed light on high-performing countries when it comes to care transition in the long-term care systems. As a result, countries may use this knowledge to learn from pioneers by adapting strategies and solutions that proved to be effective.

Evaluation of long-term care (LTC) systems is very important but understudied subject. Monitoring the performance of long-term care systems is necessary for the identification of current issues and for informing evidence-based policy-making. Reforms cannot take place without a sound understanding of how long-term care system is performing. There are a few existing frameworks for LTC system performance assessment that originated in different parts of the world [29,30,31,32]. Their common goal is to better understand the LTC system. One of the tools measures Long-Term Services and Supports across five dimensions, including effective transitions. Nonetheless, this tool uses an outcome approach to performance instead of structure and process [30]. Such approach has certain limitations and should be used with discrimination as suggested by Donabedian [15].

We acknowledge that the completion of this tool might have the unintentional effect of diverting resources. Nevertheless, the completion of the tool by staff that is familiar with transitional care and LTC of older adults should not take longer than 2-3 hours. Performing assessment with the TCAT-LTC is an essential step in promoting accountability and improving the performance of the LTC system.

Limitations

Although we performed an exhaustive process of tool development, this study has some limitations. First, we are aware that the literature review that we performed may not have identified all relevant literature due to heterogeneity of terminology for care transitions. Moreover, qualitative interviews were carried out by two interviewers and in three different languages. Therefore, there may have been some discrepancies between the interviewers and between the languages in which the interviews were carried out. Furthermore, for the theoretical analysis, we did not use target population opinion to theoretically refine the items and to analyze the tools’ content validity. Instead, we only used expert judges. Future studies are recommended to involve target population groups as it enables to identify and eliminate potential problems in the scale (to test the language and level of comprehension). Another limitation of our study is the absence of direct input from patients and their informal caregivers. We acknowledge that involving their opinion and perspectives is important in future research and policymaking. We are also aware that some of the items in the tool might not be specific enough, and this may cause an ambiguous understanding of the items. Few non-specific items in our tool are due to the variability and complexity of long-term care systems that could be assessed with this tool. Given, there is still a need for thorough validation of the tool. Future validation might further refine items that enable us to provide more detailed and clear explanations of the scoring system. Tools’ pilot test is our next step. We plan to test the TCAT-LTC in Germany, the Netherlands and Poland.

Strengths

Our study had some strengths as well. Item generation process is one of the most important steps in the scale development process. For this purpose, we used a combination of both deductive and inductive approaches for item generation to strengthen the validity of the tool. Twenty-five different experts in the field of long-term care and transitional care from three different countries – Germany, the Netherlands and Poland were involved at different stages in this study. This comprehensive approach helped us to ensure that key items are included in the tool.

Conclusions

In this paper, we presented the development of the TCAT-LTC evaluation tool for assessing the performance of long-term care systems in relation to care transition. We also presented the instructions on the application of the TCAT-LTC. The TCAT-LTC is the first tool to assess the performance of long-term care systems in relation to care transition. Assessments using the TCAT-LTC can be carried out at the national and international level, which can help to monitor, evaluate, and compare the performance of the long-term care systems (in relation to care transition) within and across different countries. Performing assessment with the TCAT-LTC can be an important first step toward optimizing care transitions for older adults and their informal caregivers. This is particularly important due to ageing population and thus, increased proportion of individuals with complex health and social care needs. Feedback on the application of the tool is welcomed as it will help us to further refine the TCAT-LTC.