- Open Access
Defining ICD-10 surrogate variables to estimate the modified frailty index: a Delphi-based approach
BMC Geriatrics volume 22, Article number: 422 (2022)
There are currently no validated globally and freely available tools to estimate the modified frailty index (mFI). The widely available and non-proprietary International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) coding could be used as a surrogate for the mFI. We aimed to establish an appropriate set of the ICD-10 codes for comorbidities to be used to estimate the eleven-variable mFI.
A three-stage, web-based, Delphi consensus-building process among a panel of intensivists and geriatricians using iterative rounds of an online survey, was conducted between March and July 2021. The consensus was set a priori at 75% overall agreement. Additionally, we assessed if survey responses differed between intensivists and geriatricians. Finally, we ascertained the level of agreement.
A total of 21 clinicians participated in all 3 Delphi surveys. Most (86%, 18/21) had more than 5-years’ experience as specialists. The agreement proportionately increased with every Delphi survey. After the third survey, the panel had reached 75% consensus in 87.5% (112/128) of ICD-10 codes. The initially included 128 ICD-10 variables were narrowed down to 54 at the end of the 3 surveys. The inter-rater agreements between intensivists and geriatricians were moderate for surveys 1 and 3 (κ = 0.728, κ = 0.780) respectively, and strong for survey 2 (κ = 0.811).
This quantitative Delphi survey of a panel of experienced intensivists and geriatricians achieved consensus for appropriate ICD-10 codes to estimate the mFI. Future studies should focus on validating the mFI estimated from these ICD-10 codes.
There was an 87.5% consensus in the quantitative Delphi survey to define the ICD-10 variables required to estimate the modified frailty index (mFI) from the 21-panel members that comprised of geriatricians and intensivists, with moderate to a strong inter-rater agreement between the geriatricians and intensivists in all three surveys.
54 ICD-10 codes were considered necessary to estimate the 11 mFI variables.
These ICD-10 codes can be further investigated for their validity in estimating frailty in the geriatric or intensive care setting.
Why does this paper matter?
In this study, we used a quantitative Delphi consensus process from a panel of twenty-one members comprising experienced geriatricians and intensivists, to define the 54 ICD-10 variables required as a surrogate to estimate the modified frailty index (mFI). We also demonstrated moderate to strong inter-rater agreement between geriatricians and intensivists in all three surveys. These ICD-10 codes could be further investigated for their validity in estimating frailty in the geriatric or intensive care setting. As ICD-10 coding is globally available and non-proprietary, the potential impact of this research on clinical care or health policy means that these 54 ICD-10 could be used to estimate an mFI equivalent to provide global frailty data in large populations at a national or multinational level.
Frailty is a clinically recognisable state of increased vulnerability due to aging-associated decline in reserve and function across multiple physiologic systems. Such people with frailty have a reduced ability to cope with acute stressors . Patients with frailty are commonly admitted to intensive care units (ICU), and patients with frailty generally have poorer outcomes [2, 3]. As a result, frailty assessment is often recommended upon ICU admission , with a variety of frailty screening tools. Regrettably, frailty assessments are difficult to perform by critical care teams on a routine basis . Administrative data have therefore been used as surrogates to assess the epidemiology of frailty retrospectively . The modified frailty index (mFI) is one such administrative data-based surrogate.
The mFI was originally designed using data from the National Surgical Quality Improvement Program (NSQIP) database mapped to variables contained within the original 70-item Canadian Study of Health and Aging Frailty Index . The score is based on eleven variables which encompass comorbidities, previous medical events, and functional capacity . The mFI then categorizes patients as non-frail if score = 0; pre-frail if score = 1–2, or frail if the score is ≥3 [8,9,10], allowing for rapid derivation and automation. Although the mFI was originally created for patients having surgery [10, 11], it recently predicted an increased risk of hospital mortality, length of stay, and higher health resource utilization in a large Brazilian cohort of critically ill patients .
The mFI, however, has some limitations. In particular, it cannot be freely obtained outside of specific proprietary databases. Moreover, there are currently no validated globally and freely available tools to estimate the mFI. In theory, however, the widely available and non-proprietary International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) coding could be used as a surrogate for the mFI. The frailty score identified, as a result, could be widely used on an international scale. However, the selection of which ICD-10 codes reflect the mFI items and can be used to estimate them remains challenging. Although several Delphi consensus processes have been performed to quantify frailty, no study to our knowledge, there were no studies with regards to the mFI. To address this issue, we undertook a Delphi consensus process to identify ICD-10 codes that geriatricians and intensivists would consider as representative of the eleven mFI items. Additionally, we assessed if survey responses differed between intensivists and geriatricians. Finally, we ascertained the level of agreement. We hypothesized that there would be a consensus amongst intensivists and geriatricians in determining the ICD-10 codes for the eleven mFI variables as an approximation.
Materials and methods
This study was approved by The Research Governance of Peninsula Health Ethics Committee (HREC reference number AM/47502/PH-2021-251,553(v2)).
Source of ICD-10 codes
Diagnosis Related Groups (DRG) is an admitted patient classification system that provides a clinically meaningful way of relating the number and type of patients treated in a hospital to the resources required by the hospital. Each DRG represents a class of patients with similar clinical conditions requiring similar hospital services. The ICD-10 codes were obtained from the Australian-Refined Diagnosis Related Groups (AR-DRG) upon patient discharge. Although a recent systematic review identified that hospital given that the 11-item mFI (listed in Supplementary Table 1), was derived from the original 70-item Canadian Study of Health and Aging Frailty Index, we used the NSQIP frailty index. The AR-DRG was screened for pertinent ICD-10 codes by a single author (AS). This was discussed and agreed upon with three other authors (RT, JD, and DP). The relevant ICD-10 codes that encompass the nine comorbidities of the mFI were included. However, the remaining two mFI variables (‘not independent of functional status’ and ‘impaired sensorium’) did not have readily available ICD-10 codes. Therefore, all possible ICD-10 codes for these variables were included for the Delphi consensus (Supplementary Table 2).
The Delphi panel
The purpose of the Delphi panel was to reach a consensus on the ICD-10 variables required to calculate the mFI. The panel members comprised a combination of intensivists and geriatricians. The participation was voluntary and agreeing to participate was taken to indicate informed consent. No incentives were offered. All the panel members participated in this multi-step process. We collected basic demographic information on the type of specialist and their years of clinical experience as a specialist.
The Delphi process
Input from the panel was obtained using a 3-step Delphi Consensus-building process. Each step was comprised of a web-based survey. We also provided the weblink to the NSQIP database and definitions that were used for every variable , with results discussed in web-based meetings. The variables that reached consensus were removed in the subsequent Delphi survey. The details of the Delphi process are summarised in Fig. 1. The round 1 Delphi survey questionnaire comprised of 128 items, where the panel members were expected to mark the items on a 3-point Likert scale of ‘Yes’, ‘Maybe’, and ‘No’ (Supplementary Appendix 1). In Delphi round 2, the panel members were requested to mark the items on a two-point Likert scale, either as “Yes” or “No” (Supplementary Appendix 2). In round 3, the panel members were asked to rate the importance of the remaining ICD-10 variables using a 5-point Likert scale: ‘strongly disagree’, ‘disagree’, ‘neutral’, ‘agree’, and ‘strongly agree’ (Supplementary Appendix 3). We grouped the responses ‘strongly agree’, ‘agree’ and ‘neutral’ into one outcome and ‘disagree’ and ‘strongly disagree’ into another.
The consensus was set a priori at 75% overall agreement by the panel members for these ICD-10 codes. In cases where the panel did not reach at least 75% consensus to keep or remove an ICD-10 code, we included it in the subsequent survey. This level of agreement has been considered appropriate in previous Delphi studies . This process was continued until at least 85% of the possible ICD-10 codes for each of the eleven variables reached a minimum of 75% consensus.
The primary outcome was to establish an appropriate set of ICD-10 codes as a surrogate for the eleven items used in the mFI. The secondary outcome included group comparison to assess any differences between intensivists and geriatricians and to ascertain a level of agreement amongst them.
Descriptive statistics were used to report participants’ demographic characteristics. Dichotomous and categorical data were described using frequencies and percentages. Fisher’s exact test was used for group comparisons between intensivists and geriatricians. We not only reported an overall combined comparison between intensivists and geriatricians but also comparisons for individual mFI variables. The measure of agreement between the two groups was analyzed using the Kappa statistic and was reported for each survey. We defined the Kappa (κ statistic) measure of an agreement value to be 0–0.20 as no agreement, 0.21–0.39 as minimal agreement, 0.40–0.59 as weak agreement, 0.60–0.79 as moderate agreement, 0.80–0.90 as strong agreement, and > 0.90 as almost perfect agreement . All p-values reported were two-tailed and the threshold for statistical significance was set at p < 0.05. SPSS (version 27, IBM) was used for all analyses.
A total of 21 senior clinicians participated in all the 3 Delphi surveys as panel members. Most (86%, 18/21) had more than 5-years’ experience as a specialist, 43% had more than 10 years’ experience (Fig. 2).
Agreement proportionately increased with every Delphi survey. After the first survey, the panel had reached 75% consensus in 23.4% (30/128) of the ICD-10 codes. Following the second survey, a consensus was reached for 70.3% (90/128) of ICD-10 codes. This increased to 87.5% of codes (112/128) after the third survey. Figure 3 illustrates the three rounds of the Delphi consensus survey that demonstrated how the 128 ICD-10 variables initially included, were narrowed down to 54 (42.2%) to estimate the mFI (Table 1, Supplementary Tables 3, 4 and 5). The 16 ICD-10 variables (12.6%) that did not reach 75% consensus were also removed (Supplementary Table 6).
Of the 21 Delphi panel members, 13 were intensivists and 8 were geriatricians. There was no difference in the levels of experience between the 2 groups (84.6% [11/13] vs. 87.5% [7/8]; p = 0.74). The overall inter-rater agreement between the intensivists and geriatricians was moderate for survey 1 (κ = 0.728); strong for survey 2 (κ = 0.811) and moderate agreement for survey 3 (κ = 0.780) (Table 2). When the individual variables were compared, the inter-rater agreement was consistently between moderate and almost perfect agreement (Table 2).
We conducted a quantitative Delphi survey to define, by consensus, the ICD-10 variables required to estimate the modified frailty index (mFI). Overall, the Delphi survey reached a consensus for 87.5% of the ICD-10 variables from the 21-panel members who completed all three rounds. There was moderate to a strong inter-rater agreement between intensivists and geriatricians between the three surveys. From an original total of 128 codes, we were able to identify 54 ICD-10 codes, which intensivists and geriatricians considered necessary to estimate the mFI. These ICD-10 codes can be further investigated for their validity in estimating frailty in the intensive care or geriatric setting.
Relationship to previous studies
The consensus methodology has been used to define the components of frailty assessment using the Delphi process in previous studies [14,15,16]. However, to our knowledge, this was the first study that attempted to obtain a consensus between experienced intensivists and geriatricians in identifying the most appropriate ICD-10 codes which could be used to estimate the mFI.
The issue of finding the appropriate ICD-10 codes needed to estimate the mFI is important, to potentially make an mFI equivalent globally available. Brazilian ICUs incorporated the mFI directly into their commercial ICU system (Epimed Monitor). This system is a database with a specific structured library of diagnoses and comorbidities. It has the capability of recording previous functional capacity based on Eastern Cooperative Oncology Group (ECOG) Performance Status and previously impaired sensorium [8, 17, 18]. This approach is not only impractical in real-time but also expensive, and therefore not generalizable on a global scale. Another recent post-hoc study, from a multicentre study , used a relatively small sample size from a single-centre , mapped the mFI variables from their large ICU Frailty database, and found that the mFI predicted hospital mortality . However, these databases are not readily available and, therefore, not globally applicable, highlighting the need for an ICD-10 codes-based system and the rationale for our study.
The use of ICD-10 codes to screen frailty is a reasonably well-researched area. A recent systematic review  listed five models that have demonstrated validity, namely: electronic frailty index , hospital frailty risk score (HFRS) , frailty risk score , preoperative frailty index , and Dr. Foster global frailty score (FGFS) . Two of these, HFRS and FGFS have been externally validated . Although the ICD-10 codes used in our study may not cover all the concepts of the HFRS and FGFS, the mFI is a validated frailty screening tool. In our study, we selected all the pertinent ICD-10 codes to reflect the mFI items. The use of ICD-10 codes has several caveats. There may be a geographic and temporal variation in the coding . Furthermore, these models should consider the amount of historic data that is required . Consequently, it is possible that despite the Delphi consensus that was demonstrated in our study, there may be uncertainty if this can effectively identify frailty. Future studies should aim to validate the diagnostic and predictive ability to use ICD-10 codes to estimate an mFI equivalent.
Our findings imply that intensivists and geriatricians could achieve consensus in determining which ICD-10 codes can be used to estimate the mFI. These 54 ICD-10 codes could act as surrogates and be used to estimate the mFI. As ICD-10 coding is globally available and free, its use could then provide global frailty data in large populations at a national or multinational level.
Strengths and limitations
The study has several strengths. First, an expert consensus was used to select the items. Second, the response rate was 100% from all panel members in all three surveys. This response rate limits responder bias. Third, the Delphi panel included senior experienced clinicians. Fourth, the anonymity of the Delphi panel members was preserved throughout. Fifth, the inter-rater agreement was consistently between moderate and almost perfect agreement.
We acknowledge several limitations. The use of Delphi techniques of surveying experts may be considered inferior to evidence-based methods . However, the ICD-10 quantification to estimate the 11 mFI variables is a novel and logical approach. Equally, consensus does not necessarily mean correctness and there is a lack of consensus on the optimum panel size or criteria for the termination . These models built on these consensus findings will require robust evaluation and validation on clinical data sets  and compared with validated scores frailty screening tools such as the CFS [19, 30]. Also, even though clinical coders or medical statisticians are very familiar with ICD-10 coding, we only chose clinicians as panel members. Furthermore, the selection of panel ‘experts’ has been challenged as being subjective . We did not provide an option for the Delphi panel members to suggest any relevant ICD-10 codes for the included comorbidities. This may be a source of bias; however, the ICD-10 codes are related to comorbidities for these eleven mFI variables and therefore appear reasonable. Finally, the DRG coding may sometimes be inaccurate with complications that occur during a hospital stay incorrectly coded as comorbidities [32, 33].
This quantitative Delphi survey achieved consensus for which ICD-10 codes are appropriate as surrogates to estimate the mFI. There was moderate to a strong inter-rater agreement amongst intensivists and geriatricians participating in the study. Future studies should focus on validating the diagnostic and predictive value of using ICD-10 codes to estimate an mFI equivalent.
Availability of data and materials
Not applicable. This was a Delphi survey. The 3 Delphi surveys are attached in the Supplementary appendix.
Xue QL. The frailty syndrome: definition and natural history. Clin Geriatr Med. 2011;27(1):1–15.
Darvall JN, Bellomo R, Paul E, Bailey M, Young PJ, Reid A, et al. Routine frailty screening in critical illness: a population-based cohort study in Australia and New Zealand. Chest. 2021;160(4):1292–303.
Darvall JN, Bellomo R, Paul E, Subramaniam A, Santamaria JD, Bagshaw SM, et al. Frailty in very old critically ill patients in Australia and New Zealand: a population-based cohort study. Med J Aust. 2019;211(7):318–23.
Robert R, Skrifvars MB, Ranzani OT. Is this critically ill patient elderly or too old? Intensive Care Med. 2017;43(12):1884–6.
Pugh RJ, Ellison A, Pye K, Subbe CP, Thorpe CM, Lone NI, et al. Feasibility and reliability of frailty assessment in the critically ill: a systematic review. Crit Care. 2018;22(1):49.
Gilbert T, Neuburger J, Kraindler J, Keeble E, Smith P, Ariti C, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet. 2018;391(10132):1775–82.
Tsiouris A, Hammoud ZT, Velanovich V, Hodari A, Borgi J, Rubinfeld I. A modified frailty index to assess morbidity and mortality after lobectomy. J Surg Res. 2013;183(1):40–6.
Zampieri FG, Iwashyna TJ, Viglianti EM, Taniguchi LU, Viana WN, Costa R, et al. Association of frailty with short-term outcomes, organ support and resource use in critically ill patients. Intensive Care Med. 2018;44(9):1512–20.
Vermillion SA, Hsu FC, Dorrell RD, Shen P, Clark CJ. Modified frailty index predicts postoperative outcomes in older gastrointestinal cancer patients. J Surg Oncol. 2017;115(8):997–1003.
Farhat JS, Velanovich V, Falvo AJ, Horst HM, Swartz A, Patton JH Jr. Rubinfeld IS: are the frail destined to fail? Frailty index as predictor of surgical morbidity and mortality in the elderly. J Trauma Acute Care Surg. 2012;72(6):1526–30 discussion 1530-1521.
Wahl TS, Graham LA, Hawn MT, Richman J, Hollis RH, Jones CE, et al. Association of the Modified Frailty Index with 30-day surgical readmission. JAMA Surg. 2017;152(8):749–57.
Woodcock T, Adeleke Y, Goeschel C, Pronovost P, Dixon-Woods M. A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Med Res Methodol. 2020;20(1):8.
McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82.
Soong JT, Poots AJ, Bell D. Finding consensus on frailty assessment in acute care through Delphi method. BMJ Open. 2016;6(10):e012904.
Rodriguez-Manas L, Feart C, Mann G, Vina J, Chatterji S, Chodzko-Zajko W, et al. Searching for an operational definition of frailty: a Delphi method based consensus statement: the frailty operative definition-consensus conference project. J Gerontol A Biol Sci Med Sci. 2013;68(1):62–7.
Morley JE, Vellas B, van Kan GA, Anker SD, Bauer JM, Bernabei R, et al. Frailty consensus: a call to action. J Am Med Dir Assoc. 2013;14(6):392–7.
Zampieri FG, Soares M, Borges LP, Salluh JIF, Ranzani OT. The Epimed monitor ICU database(R): a cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29(4):418–26.
Zampieri FG, Bozza FA, Moralez GM, Mazza DD, Scotti AV, Santino MS, et al. The effects of performance status one week before hospital admission on the outcomes of critically ill patients. Intensive Care Med. 2017;43(1):39–47.
Bagshaw SM, Stelfox HT, McDermid RC, Rolfson DB, Tsuyuki RT, Baig N, et al. Association between frailty and short- and long-term outcomes among critically ill patients: a multicentre prospective cohort study. CMAJ. 2014;186(2):E95–102.
Utino Taniguchi L, Ibrahim Q, Azevedo LCP, Stelfox HT, Bagshaw SM. Comparison of two frailty identification tools for critically ill patients: a post-hoc analysis of a multicenter prospective cohort study. J Crit Care. 2020;59:143–8.
Nghiem S, Sajeewani D, Henderson K, Afoakwah C, Byrnes J, Moyle W, et al. Development of frailty measurement tools using administrative health data: a systematic review. Arch Gerontol Geriatr. 2020;89:104102.
Clegg A, Bates C, Young J, Ryan R, Nichols L, Ann Teale E, et al. Development and validation of an electronic frailty index using routine primary care electronic health record data. Age Ageing. 2016;45(3):353–60.
Bertini F, Bergami G, Montesi D, Veronese G, Marchesini G, Pandolfi P. Predicting frailty condition in elderly using multidimensional Socioclinical databases. Proc IEEE. 2018;106(4):723–37.
McIsaac DI, Wong CA, Huang A, Moloo H, van Walraven C. Derivation and validation of a generalizable preoperative frailty index using population-based health administrative data. Ann Surg. 2019;270(1):102–8.
Soong JTY, Kaubryte J, Liew D, Peden CJ, Bottle A, Bell D, et al. Dr Foster global frailty score: an international retrospective observational study developing and validating a risk prediction model for hospitalised older persons from administrative data sets. BMJ Open. 2019;9(6):e026759.
Soong JTY, Ng SH, Tan KXQ, Kaubryte J, Hopper A. Variation in coded frailty syndromes in secondary care administrative data: an international retrospective exploratory study. BMJ Open. 2022;12(1):e052735.
Street A, Maynou L, Gilbert T, Stone T, Mason S, Conroy S. The use of linked routine data to optimise calculation of the hospital frailty risk score on the basis of previous hospital admissions: a retrospective observational cohort study. Lancet Healthy Longev. 2021;2(3):e154–62.
Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Health. 2020;8:457.
Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):1–88.
Bagshaw SM, Stelfox HT, Johnson JA, McDermid RC, Rolfson DB, Tsuyuki RT, et al. Long-term association between frailty and health-related quality of life among survivors of critical illness: a prospective multicenter cohort study. Crit Care Med. 2015;43(5):973–82.
McKenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs. 1994;19(6):1221–5.
Quan H, Parsons GA, Ghali WA. Assessing accuracy of diagnosis-type indicators for flagging complications in administrative data. J Clin Epidemiol. 2004;57(4):366–72.
Cadarette SM, Wong L. An introduction to health care administrative data. Can J Hosp Pharm. 2015;68(3):232–7.
The investigators would like to acknowledge the time and effort volunteered by the following Delphi panel members listed in alphabetical order: Drs Navin Amarasinghe, Peter Chan, Rahul Costa-Pinto, Amelia Crabtree, Caleb Fisher, Sachin Gupta, Dharshi Karalapillai, Nick Kokotsis, Angel Lee, Maurice Leguen, Margot Lodge, Chris Moran, Max Moser, Prakash Nayagam, Seema Parikh, Rafat Parkar, Sacha Richardson, Haran Sathinathan, and Stephen Warrillow.
No financial support, including any institutional departmental funds, was used for the study.
Ethics approval and consent to participate
All experimental protocols were approved by Peninsula Health ethics committee (HREC reference number AM/47502/PH-2021-251553(v2)).
No written informed consent was obtained. The consent was however implied, which means, those who agreed to participate in the surveys, consented to be Delphi Panel Members for all 3 surveys.
All methods were carried out in accordance with the relevant guidelines and regulations of the Declaration of Helsinki.
Consent for publication
All authors declare no support from any organization for the submitted work.
All authors declare no financial or non-financial competing interests with regards to the submitted work.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Subramaniam, A., Ueno, R., Tiruvoipati, R. et al. Defining ICD-10 surrogate variables to estimate the modified frailty index: a Delphi-based approach. BMC Geriatr 22, 422 (2022). https://doi.org/10.1186/s12877-022-03063-x
- modified frailty index
- International Statistical Classification of Diseases and Related Health Problems Tenth Revision
- ICD-10 codes