Skip to main content

Fostering the implementation of transitional care innovations for older persons: prioritizing the influencing key factors using a modified Delphi technique

Abstract

Background

Transitions in care for older persons requiring long-term care are common and often problematic. Therefore, the implementation of transitional care innovations (TCIs) aims to improve necessary or avert avoidable care transitions. Various factors were recognized as influencers to the implementation of TCIs. This study aims to gain consensus on the relative importance level and the feasibility of addressing these factors with implementation strategies from the perspectives of experts. This work is within TRANS-SENIOR, an innovative research network focusing on care transitions.

Methods

A modified Delphi study was conducted with international scientific and practice-based experts, recruited using purposive and snowballing methods, from multiple disciplinary backgrounds, including implementation science, transitional care, long-term care, and healthcare innovations. This study was built on the findings of a previously conducted scoping review, whereby 25 factors (barriers, facilitators) influencing the implementation of TCIs were selected for the first Delphi round. Two sequential rounds of anonymous online surveys using an a priori consensus level of > 70% and a final expert consultation session were performed to determine the implementation factors’: i) direction of influence, ii) importance, and iii) feasibility to address with implementation strategies. The survey design was guided by the Consolidated Framework for Implementation Research (CFIR). Data were collected using Qualtrics software and analyzed with descriptive statistics and thematic analysis.

Results

Twenty-nine experts from 10 countries participated in the study. Eleven factors were ranked as of the highest importance among those that reached consensus. Notably, organizational and process-related factors, including engagement of leadership and key stakeholders, availability of resources, sense of urgency, and relative priority, showed to be imperative for the implementation of TCIs. Nineteen factors reached consensus for feasibility of addressing them with implementation strategies; however, the majority were rated as difficult to address. Experts indicated that it was hard to rate the direction of influence for all factors.

Conclusions

Priority factors influencing the implementation of TCIs were mostly at the organizational and process levels. The feasibility to address these factors remains difficult. Alternative strategies considering the interaction between the organizational context and the outer setting holds a potential for enhancing the implementation of TCIs.

Peer Review reports

Background

Transitions in care are common among older persons and entail the movement between different settings and healthcare providers [1, 2]. Research shows that older persons have at least one transition towards their end of life, and one in five experience an adverse event in common transition from hospital to home [1, 3]. Transitional care innovations (TCIs) are emerging evidence-based interventions (EBIs) designed to enhance the continuity and coordination of care for older persons when transferring between multiple care settings [4,5,6]. Numerous TCIs demonstrated promising evidence for their effectiveness, such as in relation to reducing hospital readmissions, shortening hospital stay, preventing unnecessary admission to a nursing facility, averting hospitalization during an emergency department visit, or improving quality of life [2, 6,7,8,9,10].

While the positive outcomes of TCIs are encouraging, the successful translation of these innovations from trials into “real-world settings” is a main challenge [4, 11]. The implementation of TCIs in practice remains slow and ambiguous with a lack of rigorous evidence on how to best achieve translation [11]. The key components of most TCIs cross the care continuum and involve multiple care settings, which render them intricate and multifaceted [6, 12]. Therefore, integrating TCIs into an existing healthcare system with specific regulations, reimbursement, and funding mechanisms is a starting point of an onerous roadmap to their implementation [11, 13]. Moreover, TCIs normally involve two or more care settings or organizations that can be at different levels of readiness for implementing new interventions. Hence, misalignment of the different organizations’ readiness for change, which encompasses factors such as staff commitment, receptive context for innovation, priority setting, change agents, or dedicated resources, is a basic difficulty in implementation of TCIs [4, 13]. Correspondingly, while the older persons remain the core and common element across various TCIs, the heterogeneity of their care needs prevails. For instance, transitions in care for older persons with dementia [14] differ from those who suffer from the consequences of a stroke [15], which in turn, adds to the complexity of implementing TCIs.

Understanding these challenges and the various interacting constituents of TCIs illuminates the realm of implementing such complex healthcare interventions [16]. Consequently, several research efforts identified factors (barriers, facilitators) influencing the implementation of TCIs in order to better inform implementers and enhance the process [6, 13, 17]. Failure to target the right older population, discontinuous information exchange among care providers, and a lack of organizational resources with low priority given to innovation were among the prominent factors reported to hinder the implementation of TCIs [6]. In contrast, predominant facilitators included a demonstrated advantage of the TCIs for the stakeholders, the presence of frontline staff with a care transition role, as well as a continuous evaluation and monitoring process [6]. However, other factors such as leadership engagement and external policies and incentives were highly reported in the literature, but with a mixed (occasionally enabling/occasionally hindering) influence [6].

Although an awareness of these common factors helps to overcome the challenges of implementing TCIs, this compilation results in multiple and diverse factors, which are highly variable across multiple contexts. Moreover, there is a lack of prioritization based on the importance of the influencing factors, and there is a dearth in evidence on the feasibility of addressing the barriers and leveraging the facilitators with implementation strategies.

This study builds from a scoping review by the research team that identified 25 prominent factors influencing the implementation of TCIs [6]. The study aims to achieve expert consensus on the i) direction of influence (hindering, facilitating) of the known factors relevant to the implementation of TCIs that were predetermined from the literature, ii) the relative degree of importance for each factor in the implementation of TCIs, and iii) the feasibility of addressing each factor with implementation strategies for TCIs.

The overall objective is to derive a prioritized list by degree of importance and feasibility of the factors influencing the implementation of TCIs.

Methods

All methods used to carry out this study are in accordance with relevant published guidelines and regulations for the Delphi technique, and this report of the study followed the guidelines for reporting Delphi studies (see Additional file 1) [18,19,20,21].

Ethics approval and consent to participate

Ethical approval was granted by the Maastricht University Faculty of Health, Medicine, & Life Sciences Ethics Committee (approval no. FHML-REC/2020/088). Informed consent was obtained from all participants prior to the study and by including a consent statement in each survey round as the initial question, whereby participants needed to agree in order to progress.

Modified Delphi study approach

A three-step modified Delphi study was conducted with a panel of international experts in the fields of implementation of innovations, transitional care, and long-term care (LTC) between July 2020 and March 2021. The method consisted of two sequential rounds of an online survey and a final group discussion session performed through an online video-conferencing platform.

In this study, the Delphi technique was used as a practical and iterative method to obtain broad perspectives of an experienced mix of experts (in this case, from different countries and, therefore, long-term healthcare system backgrounds) and to achieve consensus in an area where there is not enough evidence [19, 22, 23]. Specifically, a modified Delphi method was chosen. This approach differs from the classical one, because first the content for round one was based on pre-determined items derived from data collected from other resources prior to the Delphi study (in this case, a scoping review) and hence utilized close-ended questions [22,23,24,25]. Second, the final round was held as a face-to-face group discussion session with the experts [22, 26,27,28,29]. A rating approach was used, whereby a panel of experts anonymously took part in surveys in different rounds. The findings and feedback of round one led the development of round two, and the final expert consultation session was based on input from the previous two rounds [22].

Participants

The expert profile was defined as individuals with extensive research and/or real-life experience in development, implementation, and evaluation of transitional care innovations (programs, models, and interventions) in LTC settings; healthcare innovations; LTC; or implementation science. Purposive and snowball sampling methods were used to recruit experts internationally.

An initial list of potential experts was developed based on professional contacts of the research team, authors of 21 published studies on the implementation of TCIs (from a previously published scoping review; Fakha et al. 2021) [6], and established contacts from the 3rd UK Implementation Science Conference – July 2020. We aimed for a minimum of 20 participants, as generally recommended [30]. Initially, 62 experts were purposively contacted, and an additional three potential experts were contacted as a result of the snowball technique. All experts were invited to participate in the study by sending them individual recruitment emails along with an invitation letter describing the overall study background, aims, and methodology.

Data collection

Survey design and development

Qualtrics software, an online survey tool, was used to develop and conduct the survey. This entailed sending the different questionnaires of each round to the participants. A total of 25 factors identified from the results of a previously published scoping review study conducted by the research team [6] were used to develop the survey content. Three of these factors were split into two subparts for description clarity, and a final list of 28 factors was thus used in round one (see Table 1). The factors were grouped into the five domains of the Consolidated Framework for Implementation Research (CFIR): intervention characteristics, outer setting, inner setting, characteristics of individuals, and process [31]. The survey consisted of three sections and explored the following for each of the 28 factors: i) direction of influence as hindering or facilitating to the implementation of TCIs; ii) importance of influence to the implementation of TCIs; and iii) feasibility (easiness/difficulty) of addressing the factor with implementation strategies for TCIs (see Additional file 2: survey round 1). The survey was piloted among the research team and amended accordingly.

Table 1 Predetermined factors (n = 28) for Delphi round one grouped into CFIR domains

Round one

Personal links to the survey were sent in individual emails to the participants. A consent statement was the initial question, and participants needed to agree in order to progress. Participants were asked to rate each factor on a five-point Likert scale, either in ascending order or from negative to positive [18]. Ratings used per section were as follows: For direction of influence: 1) strongly hindering, 2) hindering, 3) neither hindering nor facilitating, 4) facilitating, 5) strongly facilitating. For importance of influence: 1) not important, 2) slightly important, 3) moderately important, 4) very important, 5) extremely important. For feasibility: 1) very difficult, 2) difficult, 3) neither difficult nor easy, 4) easy, 5) very easy. Moreover, participants were requested to provide additional comments in free-text boxes provided per each section, as well as overall suggestions for additional factors relevant to the implementation of TCIs. The survey required approximately 20 min to complete. Instructions were sent to participants on how to complete the survey, and they were given two weeks to respond. A reminder was sent to participants who did not complete the survey within the two-week period.

Round two

Survey round two was conducted online and in a manner similar to that for round one. Only participants who completed round one of the survey were invited to round two. This survey included factors that did not reach consensus from round one and additional factors that were suggested by participants, an established approach using the Delphi technique [22, 23]. The definitions of a few factors were revised based on comments of participants in round one (see Additional file 3: survey round 2). Participants were asked to again rate the factors using the same method as round one, but with knowledge of their individual ratings and the group ratings for each factor from the first round. In addition, a summary report on the results of round one was provided to all participants prior to the second round.

Final round: expert consultation session

The final round was comprised of two online video call meetings lasting two hours each, and performed in the same manner and using the same content, through a data-protected, web-based conferencing platform. All participants of round two were invited to join and were assigned to either of the two meetings according to their availability. In order to limit bias, AF and BdB facilitated both sessions, and TvA was an observant who also intervened when necessary to ensure participation from all experts. The sessions’ discussions were recorded and later transcribed. The goal of these sessions was to allow interaction between the experts and provide further clarifications on the overall results from previous rounds. The specific aim, determined by the earlier results, was to i) narrow down the important factors to the “must have” factors, ii) obtain further insights on the feasibility of addressing the important factors, and iii) receive suggestions for potentially relevant implementation strategies. Initially, the results of rounds one and two were presented, and then participants were asked for their overall reflections. Afterwards, two open and predetermined questions were used to guide the discussion in order to allow for further deliberations, as follows:

  1. a)

    What are your views on the top factors? And if you were to choose five “must-have” factors, what would they be?

  2. b)

    Please can you explain why the majority of factors that reached consensus on feasibility (including top important ones) were rated as difficult to address when developing implementation strategies? Any advice on how to approach this? Which strategies would you suggest to tackle each factor?

Data analysis

Responses were analyzed after the completion of each survey round, and rating scores were calculated as percentages using SPSS Statistics 25 software. Consensus was determined as reached if over 70% of the respondents rated the i) direction of influence of each factor as either ‘strongly hindering’/‘hindering’ (combined score), ‘strongly facilitating’/‘facilitating’ (combined score), or neither. Similarly, consensus on the ii) importance of factors was reached if either combined scores for ‘extremely important’/‘very important’ or combined scores for ‘moderately important’/‘slightly important’ or ‘not important’ were over 70%, and for iii) feasibility ‘very easy’/‘easy’, ‘very difficult’/‘difficult’, or ‘neither’. This level of agreement was used and considered appropriate in previous Delphi studies [23, 24, 32,33,34]. Free-text comments from rounds one and two were analyzed thematically and discussed among the research team to identify any additional factors or to rephrase and clarify the definitions.

Thematic analysis: final expert consultation session

The transcripts of the two final meetings were compiled together and analyzed thematically by three researchers (AF, BdB, TvA) following the six-step method described by Braun and Clarke (2006) [35, 36]. An inductive form of thematic analysis was performed, and the codes created were data-driven. Following data coding, themes were developed and then reviewed iteratively. A thematic map was developed, and a clear delineation of the final themes was discussed and agreed upon by the research team.

Results

The overall study design, number of participants, and results per round are summarized in Fig. 1.

Fig. 1
figure 1

Flow of rounds, participants, and factors through the modified Delphi study*. *Final number of factors that reached consensus is a cumulative build up between the consecutive rounds

Round one

Initially, 32 experts agreed to participate in the study (round one), out of which 29 responded to the first survey (45% response rate, based on initial number of invited participants). Table 2 presents the participant demographics and professional backgrounds. More than half of the participants had at least 10 years of experience. The majority had a current role in research, mainly in implementation science and transitional care.

Table 2 Participant demographics (n = 29)

Eighteen factors out of 28 reached consensus on the direction of influence; however, only one factor (complexity) was generally seen as a barrier, while 17 were seen as facilitators (see Additional file 4). Fourteen factors reached consensus on importance (see Table 3), with all but one factor, rated as very/extremely important in influencing the implementation of TCIs. Engaging key stakeholders ranked as the most important influencing factor. Nine factors reached consensus on feasibility, with only one factor (planning) rated as easy/very easy to address with implementation strategies (see Table 4). The organization’s culture surpassed the other factors as most difficult.

Table 3 Factors that reached consensus on importance of influence on the implementation of TCIs, in order of ranking
Table 4 Factors that reached consensus on feasibility (easy/difficult to address with implementation strategies), in order of ranking

Thematic analysis of the free-text comments in the first round indicated that the direction of influence for the factors was very difficult to assess. The participants mentioned that factors can behave differently according to the specific context where TCIs are implemented. Therefore, it was hard to judge the ultimate influence of each factor. “Factors that hinder can paradoxically also be factors that facilitate and vice versa” (Expert 8, round 1). “Whether these factors are hindering or facilitating depends highly on the specific nature of this factor in the organization, so culture can be facilitating if there is an innovative culture, but hindering if there is a conservative culture without openness to innovation” (Expert 7, round 1).

Moreover, the experts identified an additional seven factors to explore for consensus in the consecutive round. These factors were recognized across the five CFIR domains and included power of decision-makers, sense of urgency, adoption of change in work processes, financing of TCIs’ implementation, inter-organizational collaborations, previous experiences with implementation of innovations, and co-design of the TCIs (see Additional file 3: survey round 2). According to the experts’ comments, the definitions of some factors were revised.

Round two

Twenty-eight of the 29 participants, who completed round one, completed this round (97% response rate). In this round, rating the factors’ direction of influence was omitted. The consensus results from round one were skewed mostly to one direction (facilitating) and hence were ruled as of low relevance and non-conclusive by the research team.

A further nine factors reached consensus as very/extremely important and one as slightly/moderately. Additionally, six out of the seven newly added factors reached consensus as very/extremely important to the implementation of TCIs, with financing of TCIs’ implementation ranked as highest (see Table 3). Ten additional factors achieved consensus on feasibility in this round, with leadership engagement as the most difficult and transition roles as a neutral factor (see Table 4). A further two of the seven new factors (financing of TCIs’ implementation, adoption of change in work processes) reached consensus as difficult/very difficult.

Factors of highest importance

Of the total 30 factors that reached consensus on the importance of influence following rounds one and two, the majority were linked to the inner setting. Within this domain, leadership engagement, availability of resources including HIT systems, and information continuity between care providers had the highest consensus levels on importance as compared to other factors such as the organizational culture. Whereas the engagement of stakeholders and organizations/external context was of highest importance within the implementation process and exceeded other factors, such as planning, reflecting and evaluating, and transition roles. In comparison, factors (skills and competencies, role, knowledge and beliefs) related to the characteristics of individuals had a lower level of consensus on their importance. Moreover, the personal attributes of older persons such as their motivation, values, or intellectual ability were rated as slightly/moderately important, while factors relating to healthcare professionals were seen as very/extremely important. As for the intervention characteristics, the relative advantage and benefits of the innovation as perceived by stakeholders (older persons and healthcare providers) as well as the degree of involvement of these stakeholders in its design prior to implementation were the most important factors. Alternatively, the demonstrated evidence strength and quality of the TCIs appear to be of least importance to influence the implementation. In contrast, external incentives and policy supporting the TCIs’ implementation and national financing structures, such as a healthcare services reimbursement system, were the important factors within the outer setting.

A final list of the 11 factors that ranked as most important with consensus of 85% and above is presented in Table 5. Once more, these key factors were predominantly related to the inner setting, and only three were linked to the implementation process. The engagement of leaders and key stakeholders was confirmed by the experts as essential in influencing the implementation of TCIs. “If key stakeholders are in favor of an innovation, good communication can really help, but when they are not in favor, it can really hinder an implementation process” (Expert 7, round 1). Nevertheless, the continuity of information and communication across multiple organizations came in third place, which could be explained by the nature of transitional care involving several care settings. “It is difficult to coordinate care that goes beyond the boundaries of a specific organization” (Expert 7, round 1). Moreover, the availability of organizational resources as well as the existing financing structures to support the implementation of TCIs were important influencers. “In transitional care also the reimbursement system in healthcare can play an important role. If an intervention does not fit the current system, this can be a real challenge for the implementation process” (Expert 7, round 1). “Lack of money and lack of management support can stop efforts very quickly” (Expert 5, round 1).

Table 5 Final list of most important factors and indication of feasibility†

Feasibility

Around only half (54%) of the total number of factors reached consensus on feasibility, with the majority rated as difficult to address with implementation strategies and repeatedly linked to the organizational context. An attempt to address the organizational culture was regarded by experts as topmost difficult and as the least feasible approach to take. “And since we’re talking about implementing a new model or some sort of a change, it’s always a culture change conversation, and there are a lot of things involved in changing culture...” (Expert 12, consultation session). Moreover, experts indicated that it was challenging to assess the feasibility for each factor, since it depends on the context where the TCIs’ implementation is happening. “The difficulty to address these items in practice can vary a lot from project to project” (Expert 5, round 1). “Ideally, each site should identify the areas that are strengths and challenges in relation to the intervention, and from there identify which strengths they can leverage” (Expert 18, round 1). In addition, consensus on the feasibility of specific factors revealed that, while these factors are very important in influencing the implementation of TCIs, it is most likely difficult to consider, control, or change them with strategies (see Table 5).

Final round: expert consultation session

Fourteen experts participated in this session and two overarching themes emerged, which are described as follows:

Theme one: 'The Catalysts’

This theme describes a combination of temporal and interconnected factors that were seen as essential prerequisites for starting the implementation of TCIs. From the 11 key important factors from the previous rounds, the experts identified five factors that are the catalysts to launch any effort to implement TCIs. These factors were the sense of urgency, relative priority, financing and resources, leadership engagement, and engagement of key stakeholders across the continuum of transitional care. Sense of urgency was identified as a primary factor to induce any change within organizations and even to create priorities and allocate resources needed for the implementation of TCIs. Whereas relative priority was regarded as a “stop/go” flag for the implementation of TCIs, it also depends on from whom or where this priority is coming.

“First to get that sense of urgency and then it realigns priorities.” (Expert 12, consultation session)

“And since we're talking about implementing a new model or some sort of a change. It is always a culture change conversation, and there are a lot of things involved in changing culture. Having people feel like this is important; maybe you realize the priorities in their head, but I think the urgency comes first, in my mind.” (Expert 12, consultation session)

“So this is the sort of stop/go regardless of leadership engagement, regardless of HIT systems, regardless of result. You are not going to have resources, you are not going to have engagement unless something is the priority, so for me this is almost like a step before.” (Expert 4, consultation session)

The experts reconfirmed that the engagement of leaders, key stakeholders such as representatives of multiple organizations, older persons, and family caregivers is the backbone for implementing TCIs. However, it is crucial to first identify the role and responsibility of each stakeholder in the implementation process and then to create the right engaging activity and sustain it.

“Engaging key stakeholders is a means to an end is kind of an initial, you know piece of it is a catalyst for all of the other things that happen up and down those levels.” (Expert 9, consultation session)

“…that you have a successful implementation, and that is stakeholder engagement and the leadership engagement.” (Expert 6, consultation session)

Furthermore, it was indicated that resources including HIT systems and funds would only be made available if leadership and key stakeholders are involved early on. As for the factor of financing the implementation of TCIs, it was discussed that reimbursement or financing structures could be varied in transitional care, especially when several organizations are involved. Therefore, the key element is to demonstrate the TCIs’ value for care and the return on investment in order to feedback into the loop of leadership engagement and prioritizing its implementation. The proposed interrelationships among these catalyst factors are depicted in Fig. 2.

Fig. 2
figure 2

Depiction of the interrelationships among the catalyst factors influencing the implementation of TCIs

Theme two: Alignment – 'The Driver'

This theme highlights the importance of aligning the motivation for change across various organizations and levels in the healthcare system. Experts implied that regardless of ‘catalysts’ being present, the alignment of forces to drive the implementation of TCIs across the continuum of transitional care is a key issue, yet often the “blind spot”. Aligning the different priorities, interests, motivations, innovation readiness, financial incentives, and agendas of the organizations involved can help drive the implementation of TCIs. Moreover, experts agreed on the significance of considering at which level the implementation should occur and to check if the stakeholders are aligned in their need and motivation for implementing TCIs. Lastly, the experts believed that it is a crucial driver to ensure that the TCIs are in alignment with the older person’s care needs.

Feasibility – The experts concurred that developing implementation strategies to address the important factors is largely dependent on the context, individuals involved, and the care continuum. Specifically, trying to overcome hindering factors linked to the organizational context was seen as a known challenge and hard to successfully address with implementation strategies. “Organizational inertia, culture, its also infrastructure, its processes are inert, is because what they’ve done has worked so far. If they’re surviving it’s because they’ve done something that, for whatever reason, has worked” (Expert 9, consultation session). However, experts highlighted that the focus could be shifted to creating strategies to induce change at the individual level, which may ultimately improve the organization’s willingness to innovate.

Discussion

Experts in this study prioritized 11 factors as the most important in the implementation of TCIs. Amongst these factors, the majority were organizational factors, primarily the leadership engagement, availability of resources, information continuity, sense of urgency, and relative priority. Moreover, engagement of stakeholders linked to the implementation process was seen as another priority factor. However, the study results demonstrated a prevalent agreement among experts on the difficulty to address these priority factors with implementation strategies. Nevertheless, ensuring the alignment of the organizations’ interests, agendas, incentives, and priorities was established as a crucial “stepping stone” in implementing TCIs across the transitional care continuum. The current findings are congruent with earlier research indicating that organizational factors, chiefly leadership, resources, and communication, significantly influence the implementation of EBIs in healthcare settings [37,38,39,40].

In this study, experts concurred strongly that a high commitment of organizational leadership is the dominant factor in initiating the implementation of TCIs and supporting the overall change process. Furthermore, leadership has the ability to respond to a sense of urgency to innovate within an organization and keep it as a priority. Therefore, this suggested a versatile nature of leadership influence on the implementation of TCIs by being not only a precursor but also a probable moderator or mediator. This resonates with recent studies that recognized a contingent relationship between leadership influence and other implementation factors [37, 38, 41]. The other two priority factors — resource allocation and engagement of key stakeholders (i.e., healthcare professionals and staff required for implementation of TCIs) — were acceded by our Delphi panel as dependent on the existence of a supportive leadership influence. Correspondingly, Gifford et al. describe the potential effective contribution of leadership to promote a successful implementation of evidence in healthcare practice [42].

Our panel agreed that engagement of key stakeholders was significant in the implementation of TCIs at all levels of the transitional care continuum. Similarly, engaging multidisciplinary healthcare teams and key staff in various care settings has been widely reported as an integral element and a necessary process activity for implementing innovations in transitional care and LTC in general [6, 38, 39, 43,44,45,46]. Nevertheless, despite the well-known importance of stakeholders’ engagement in implementation, there is still vagueness and limited evidence on its definition, who qualifies as a stakeholder, and which best practices to employ [47, 48].

In relation to this, and to our surprise, the importance of engaging the older persons and their family or informal caregivers in the implementation of TCIs was not something our experts reached consensus on. We would have expected that factors related to the older persons including their knowledge, perceptions, attitudes toward and value placed on the TCIs’ services, personal care needs, and an overall consideration of “what matters to them?” would be prioritized as very important by the experts. Acknowledging the characteristics and interests of the older persons was revealed as instrumental in other studies describing the process of uptake and implementation of interventions in transitional care [38, 49]. Although person-centered transitional care, whereby TCIs are tailored to older persons’ needs and preferences, is generally seen as important [50]; our results indicate that involving the target group in the implementation of TCIs is less evident. Likewise, Olsen et al. highlight that engaging the older persons and listening to their needs and wishes are fundamental factors in delivering transitional care interventions, yet there appears to be other significant and overlooked constraints at the organizational and system levels [51].

The feasibility of addressing the agreed upon priority factors with implementation strategies was concurred by the experts as mostly challenging, particularly for the organizational factors. Contrary to our expectations, these results do not inform the development of strategies for implementation of TCIs. Notably, a number of taxonomies and compilations of strategies were developed to aide in implementing EBIs in healthcare [52,53,54,55]. Moreover, some of these strategies were matched to the relative influencing factors in general healthcare settings [54, 56]. In addition, although organizational leadership was rated as difficult to address in this Delphi study, there is evidence on an emerging strategy: the leadership and organizational change for implementation (LOCI) [57]. Among its aspects, LOCI focuses on leveraging the leaders’ readiness for implementing EBIs, training them on how to overcome implementation barriers, and promoting them to be proactive and create a shared vision within the organization [57]. On the other hand, the body of research on implementation strategies is not specific to transitional care, although it is starting to be utilized in implementing certain TCIs [43]. Therefore, our findings add to the evolving literature by indicating that practitioners and researchers in the field of implementing TCIs perceive that strategies should be tailored to the specific settings involved.

Implications for implementing innovations in transitional care

Practice

In light of our findings, we ask healthcare practitioners (leaders, managers, frontline staff, and other professionals) wanting to implement a TCI, to start by conducting a local needs assessment. It is crucial to know the inner settings of LTC organizations, as well as the inter-organizational differences and dynamics, as this is also a key message from previous studies [39, 58]. To better assess the organizational readiness for putting a TCI into practice, we recommend utilizing our list of ‘priority factors’ as a starting point. Exploring these factors locally will provide an early essential awareness and knowledge of what will most likely help or hinder the implementation of a TCI. Based on the existing literature, we hereby provide hands-on suggestions for addressing the priority factors when implementing a TCI [53, 54, 56]. For example, having both intra/inter-organizational discussions among care providers and key stakeholders can help identify existing problems in care transitions of older persons between different settings, and hence create a sense of urgency or prioritize the need for a TCI as a solution [53, 56]. This, in turn can be used to build a case and to present it to the leaders of LTC organizations, as to obtain support for implementation. Furthermore, engaged leaders can help secure required resources through practical ways by looking for funding options for initial implementation, such as responding for governmental calls to fund implementation of innovations in practice or restructuring organizational incentives. In addition, the creation of inter-organizational working groups of key stakeholders can assist in following through the implementation process of a TCI and making necessary decisions and adjustments [53].

Correspondingly and given the big role of organizational factors, we also highlight the potential value of using insights from four prominent organizational theories (transaction cost economics, institutional theory, contingency theory, and resource dependency theory) in implementing TCIs [59]. For example, with keeping in mind the disparities across different healthcare systems, a healthcare manager can assess the transaction cost of implementing an intervention and consider the possibility of outsourcing a TCI’s components or services to another institution. Otherwise as denoted by resource dependency theory, healthcare managers and leaders can establish inter-organizational partnerships and alliances to acquire resources needed. Moreover, as per both institutional and contingency theories, healthcare managers and leaders can promote the adoption and implementation of a TCI within their organization by copying successful innovative behaviours of other organizations in the environment, and boost the organization’s agility to respond to external factors.

Future research

Further investigation of the prioritized factors in the actual implementation of TCIs in practice can provide a better understanding of how they exist and interact. In addition, the development and testing of a set of tailored, effective, and feasible strategies to target these priority factors influencing the implementation of TCIs is required.

Strengths and limitations

This study gathered consensus by drawing on an international panel of experts from the fields of innovation, implementation, and transitional care, which allowed to obtain focused perspectives. Moreover, the use of an online survey permitted a high response, and the final session with the experts was instrumental to understanding the consensus results and enriched the study.

Alternatively, there were some limitations. First, selection bias could play a role. A majority of the panel were scientists rather than professionals from practice, which may have led to an under-representation of insights from real-life context. This could also explain the panel members’ difficulties in assessing implementation factors’ direction of influence and their focus on organizational factors, as their direct care experience might have been limited and/or mainly in the past. Furthermore, the majority of experts came from European healthcare systems. Insights from other alternative models of healthcare systems (e.g. the USA) were under-represented in our panel of experts, thus limiting the applicability of the findings. Second, repeating the survey with other panel experts might have led to other results. However, to improve the reliability, we aimed for a large sample of experts from various backgrounds and countries. Also, we performed the two survey rounds in a consistent manner. Third, we provided the group ratings from round one in the consecutive round, which could be viewed as a pressure to obtain consensus. However, one can also argue that panel members are entitled to receiving core results, besides we followed the Delphi methodology carefully, used findings of a previous literature study to inform the survey, and based our work on an established implementation framework. Lastly, the final round being a qualitative group discussion, holds the limitation pertaining to typical group dynamics and power/confidence of expression. However, the session also enriched our study and facilitators for the session made sure that all participants were heard, and encouraged an open discussion.

Conclusions

Though many factors are relevant in the implementation of TCIs, experts conceded that the priority factors in the implementation of TCIs are leadership engagement, sense of urgency to innovate, relative priority given to a TCI, availability of organizational resources, and engagement of key stakeholders. Results from our study enable the selection of relevant strategies for implementation of TCIs, yet special attention should be given to inter-organizational factors, which are seen as difficult to address, as well as the interrelationships revealed between these factors. This study provides novel guidance for healthcare researchers and practitioners, opting to improve transitional care for older persons, to better navigate the implementation process of innovations, and deter efforts based on intuition rather than evidence.

Availability of data and materials

Data generated and analyzed during this study are avaliable from the corresponding author on reasonable request.

Abbreviations

TCI(s):

Transitional care innovation(s)

EBIs:

Evidence-based interventions

LTC:

Long-term care

CFIR:

Consolidated Framework for Implementation Research

HIT:

Health information technology

LOCI:

Leadership and organizational change for implementation

References

  1. Baxter R, Shannon R, Murray J, O’Hara JK, Sheard L, Cracknell A, et al. Delivering exceptionally safe transitions of care to older people: a qualitative study of multidisciplinary staff perspectives. BMC Health Serv Res. 2020;20(1):780.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Hirschman KB, Hodgson NA. Evidence-based interventions for transitions in care for individuals living with dementia. Gerontologist. 2018;58(suppl_1):S129–s40.

    Article  PubMed  Google Scholar 

  3. Abraham S, Menec V. Transitions between care settings at the end of life among older homecare recipients: a population-based study. Gerontol Geriatr Med. 2016;2:2333721416684400.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Wee SL, Vrijhoef HJ. A conceptual framework for evaluating the conceptualization, implementation and performance of transitional care programmes. J Eval Clin Pract. 2015;21(2):221–8.

    Article  PubMed  Google Scholar 

  5. Prusaczyk B, Fabbre V, Morrow-Howell N, Proctor E. Understanding transitional care provided to older adults with and without dementia: a mixed methods study. Int J Care Coord. 2020;23(1):14–23.

    Article  Google Scholar 

  6. Fakha A, Groenvynck L, de Boer B, van Achterberg T, Hamers J, Verbeek H. A myriad of factors influencing the implementation of transitional care innovations: a scoping review. Implement Sci. 2021;16(1):21.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Dresden SM, Hwang U, Garrido MM, Sze J, Kang R, Vargas-Torres C, et al. Geriatric emergency department innovations: the impact of transitional care nurses on 30-day readmissions for older adults. Acad Emerg Med. 2020;27(1):43–53.

    Article  PubMed  Google Scholar 

  8. Finlayson K, Chang AM, Courtney MD, Edwards HE, Parker AW, Hamilton K, et al. Transitional care interventions reduce unplanned hospital readmissions in high-risk older adults. BMC Health Serv Res. 2018;18(1):956.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Hwang U, Dresden SM, Rosenberg MS, Garrido MM, Loo G, Sze J, et al. Geriatric emergency department innovations: transitional care nurses and hospital use. J Am Geriatr Soc. 2018;66(3):459–66.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Sezgin D, Hendry A, Liew A, O'Donovan M, Salem M, Carriazo AM, et al. Transitional palliative care interventions for older adults with advanced non-malignant diseases and frailty: a systematic review. J Integr Care. 2020;28(4):387–403.

    Article  Google Scholar 

  11. Naylor MD, Bowles KH, McCauley KM, Maccoy MC, Maislin G, Pauly MV, et al. High-value transitional care: translation of research into practice. J Eval Clin Pract. 2013;19(5):727–33.

    PubMed  Google Scholar 

  12. Naylor MD, Shaid EC, Carpenter D, Gass B, Levine C, Li J, et al. Components of comprehensive and effective transitional care. J Am Geriatr Soc. 2017;65(6):1119–25.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Lutz BJ, Reimold AE, Coleman SW, Guzik AK, Russell LP, Radman MD, et al. Implementation of a transitional care model for stroke: perspectives from frontline clinicians, administrators, and COMPASS-TC implementation staff. Gerontologist. 2020;60(6):1071–84.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Prusaczyk B, Olsen MA, Carpenter CR, Proctor E. Differences in transitional care provided to patients with and without dementia. J Gerontol Nurs. 2019;45(8):15–22.

    Article  PubMed  Google Scholar 

  15. Markle-Reid M, Valaitis R, Bartholomew A, Fisher K, Fleck R, Ploeg J, et al. An integrated hospital-to-home transitional care intervention for older adults with stroke and multimorbidity: a feasibility study. J Comorb. 2020;10:2235042X19900451.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Khalil H, Kynoch K. Implementation of sustainable complex interventions in health care services: the triple C model. BMC Health Serv Res. 2021;21(1):143.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Weeks LE, Barber B, MacDougall ES, Macdonald M, Martin-Misener R, Warner G. An exploration of Canadian transitional care programs for older adults. Healthc Manage Forum. 2021;34(3):163–8.

    Article  PubMed  Google Scholar 

  18. Toma C, Picioreanu I. The Delphi technique: methodological considerations and the need for reporting guidelines in medical journals. Int J Public Health Res. 2016;4(6):47–59.

    Google Scholar 

  19. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    CAS  PubMed  Google Scholar 

  20. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9.

    Article  PubMed  Google Scholar 

  21. Junger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706.

    Article  PubMed  Google Scholar 

  22. Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. 1st ed. UK: Wiley; 2011.

    Book  Google Scholar 

  23. Havers SM, Martin E, Wilson A, Hall L. Implementation of government-directed policy in the hospital setting: a modified Delphi study. Health Res Policy Syst. 2019;17(1):91.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Fergie L, Campbell KA, Coleman-Haynes T, Ussher M, Cooper S, Coleman T. Stop smoking practitioner consensus on barriers and facilitators to smoking cessation in pregnancy and how to address these: a modified Delphi survey. Addict Behav Rep. 2019;9:100164.

    PubMed  PubMed Central  Google Scholar 

  25. Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract assess res. Evaluation. 2007;12(10).

  26. Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol. 2016;16:56.

    Article  PubMed  PubMed Central  Google Scholar 

  27. van Vliet DC, van der Meij E, Bouwsma EV, Vonk Noordegraaf A, van den Heuvel B, Meijerink WJ, et al. A modified Delphi method toward multidisciplinary consensus on functional convalescence recommendations after abdominal surgery. Surg Endosc. 2016;30(12):5583–95.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Schneider P, Evaniew N, Rendon JS, McKay P, Randall RL, Turcotte R, et al. Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology. BMJ Open. 2016;6(5):e011780.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Woodcock T, Adeleke Y, Goeschel C, Pronovost P, Dixon-Woods M. A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Med Res Methodol. 2020;20(1):8.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf Manag. 2004;42(1):15–29.

    Article  Google Scholar 

  31. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Fisher RJ, Walker MF, Golton I, Jenkinson D. The implementation of evidence-based rehabilitation services for stroke survivors living in the community: the results of a Delphi consensus process. Clin Rehabil. 2013;27(8):741–9.

    Article  PubMed  Google Scholar 

  33. Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):887–97.

    Article  PubMed  Google Scholar 

  34. Vogel C, Zwolinsky S, Griffiths C, Hobbs M, Henderson E, Wilkins E. A Delphi study to build consensus on the definition and use of big data in obesity research. Int J Obes. 2019;43(12):2573–86.

    Article  Google Scholar 

  35. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

    Article  Google Scholar 

  36. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods. 2017;16(1):1609406917733847.

    Article  Google Scholar 

  37. Li S-A, Jeffs L, Barwick M, Stevens B. Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: a systematic integrative review. Syst Rev. 2018;7(1):72.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Sarkies M, Long JC, Pomare C, Wu W, Clay-Williams R, Nguyen HM, et al. Avoiding unnecessary hospitalisation for patients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes. Implement Sci. 2020;15(1):91.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Bunn F, Goodman C, Corazzini K, Sharpe R, Handley M, Lynch J, et al. Setting priorities to inform assessment of care homes’ readiness to participate in healthcare innovation: a systematic mapping review and consensus process. Int J Environ Res Public Health. 2020;17(3):987.

    Article  PubMed Central  Google Scholar 

  40. Nurjono M, Shrestha P, Ang IYH, Shiraz F, Yoong JS-Y, Toh S-AES, et al. Implementation fidelity of a strategy to integrate service delivery: learnings from a transitional care program for individuals with complex needs in Singapore. BMC Health Serv Res. 2019;19(1):177.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35(1):255–74.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Gifford WA, Squires JE, Angus DE, Ashley LA, Brosseau L, Craik JM, et al. Managerial leadership for research use in nursing and allied health care professions: a systematic review. Implement Sci. 2018;13(1):127.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Yip O, Huber E, Stenz S, Zullig LL, Zeller A, De Geest SM, et al. A contextual analysis and logic model for integrated care for frail older adults living at home: the INSPIRE project. Int J Integr Care. 2021;21(2):9.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Yeager VA, Wharton MK, Monnette A, Price-Haywood EG, Nauman E, Angove RSM, et al. Non-face-to-face chronic care management: a qualitative study assessing the implementation of a new CMS reimbursement strategy. Popul Health Manag. 2018;21(6):454–61.

    Article  PubMed  Google Scholar 

  45. Williams MV, Li J, Hansen LO, Forth V, Budnitz T, Greenwald JL, et al. Project BOOST implementation: lessons learned. South Med J. 2014;107(7):455–65.

    Article  PubMed  Google Scholar 

  46. Brody AA, Arbaje AI, DeCherrie LV, Federman AD, Leff B, Siu AL. Starting up a hospital at home program: facilitators and barriers to implementation. J Am Geriatr Soc. 2019;67(3):588–95.

    Article  PubMed  Google Scholar 

  47. Goodman MS, Sanders Thompson VL. The science of stakeholder engagement in research: classification, implementation, and evaluation. Transl Behav Med. 2017;7(3):486–91.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Norris JM, White DE, Nowell L, Mrklas K, Stelfox HT. How do stakeholders from multiple hierarchical levels of a large provincial health system define engagement? A qualitative study. Implement Sci. 2017;12(1):98.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Gesell SB, Bushnell CD, Jones SB, Coleman SW, Levy SM, Xenakis JG, et al. Implementation of a billable transitional care model for stroke patients: the COMPASS study. BMC Health Serv Res. 2019;19(1):978.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Groenvynck L, Fakha A, de Boer B, Hamers JPH, van Achterberg T, van Rossum E, et al. Interventions to improve the transition from home to a nursing home: a scoping review. Gerontologist. 2021. https://doi.org/10.1093/geront/gnab036.

  51. Olsen CF, Debesay J, Bergland A, Bye A, Langaas AG. What matters when asking, “what matters to you?” - perceptions and experiences of health care providers on involving older people in transitional care. BMC Health Serv Res. 2020;20(1):317.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Cochrane. Effective practice and organisation of care (EPOC). EPOC Taxonomy. 2015; [Available from: epoc.cochrane.org/epoc-taxonomy].

  53. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Waltz TJ, Powell BJ, Fernandez ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Bartholomew LK, Parcel GS, Kok G, Gottlieb NH, Schaalma HC, Markham CC, et al. Planning health promotion programs: an intervention mapping approach. San Francisco: Jossey-Bass; 2006.

  56. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Cammer A, Morgan D, Stewart N, McGilton K, Rycroft-Malone J, Dopson S, et al. The hidden complexity of long-term care: how context mediates knowledge translation and use of best practices. Gerontologist. 2014;54(6):1013–23.

    Article  PubMed  Google Scholar 

  59. Birken SA, Bunger AC, Powell BJ, Turner K, Clary AS, Klaman SL, et al. Organizational theory for dissemination and implementation research. Implement Sci. 2017;12(1):62.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank and acknowledge each member of the expert panel for their valuable participation, time, and commitment to be part of this study.

Funding

The current project is part of the research program TRANS-SENIOR Innovative Training Network funded by the European Union (more information is available at https://www.trans-senior.eu/). “This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 812656.”

Author information

Authors and Affiliations

Authors

Contributions

AF developed the study, recruited the participants, developed the surveys, managed the administrative tasks and communication with the participants, distributed the surveys online and collected the data, planned and conducted the final round using a video-conferencing platform, analyzed and interpreted the data, and wrote the manuscript. BdB coordinated the study, participated in developing the survey, analyzed and interpreted the data, and edited the manuscript. TvA co-supervised the study, advised on the recruitment of the participants, analyzed and interpreted the data, and edited the manuscript. HV supervised the study, advised on the recruitment of the participants, interpreted the data, and edited the manuscript. JH advised on the overall study concept and edited the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Amal Fakha.

Ethics declarations

Ethics approval and consent to participate

Ethical approval was granted by the Maastricht University Faculty of Health, Medicine, & Life Sciences Ethics Committee (approval no. FHML-REC/2020/088). Informed consent was obtained from all participants prior to the study and by including a consent statement in each survey round as the initial question, whereby participants needed to agree in order to progress.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

CREDES checklist for reporting Delphi studies.

Additional file 2.

Modified Delphi survey - Round 1.

Additional file 3.

Modified Delphi survey - Round 2.

Additional file 4.

List of factors that reached consensus on direction of influence on the implementation of TCIs, from round one.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fakha, A., de Boer, B., van Achterberg, T. et al. Fostering the implementation of transitional care innovations for older persons: prioritizing the influencing key factors using a modified Delphi technique. BMC Geriatr 22, 131 (2022). https://doi.org/10.1186/s12877-021-02672-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12877-021-02672-2

Keywords