- Research article
- Open Access
- Open Peer Review
A cross-sectional study examining convergent validity of a frailty index based on electronic medical records in a Canadian primary care program
BMC Geriatrics volume 19, Article number: 109 (2019)
The Correction to this article has been published in BMC Geriatrics 2019 19:133
An electronic frailty index (eFI) has been developed and validated in the UK; it uses data from primary care electronic medical records (EMR) for effective frailty case-finding in primary care. This project examined the convergent validity of the eFI from Canadian primary care EMR data with a validated frailty index based on comprehensive geriatric assessment (FI-CGA), in order to understand its potential use in the Canadian context.
A cross-sectional validation study, using data from an integrated primary care research program for seniors living with frailty in Edmonton, AB. Eighty-five patients 65 years of age and older from six primary care physicians’ practices were recruited. Patients were excluded if they were under 65 years of age, did not provide consent to participate in the program, or were living in a long term care facility at the time of enrolment. We used scatter plots to assess linearity and Pearson correlation coefficients to examine correlations.
Results indicate a strong statistically significant correlation between the eFI and FI-CGA (r = 0.72, 95% CI 0.60–0.81, p < 0.001). A simple linear regression showed good ability of the eFI scores to predict FI-CGA scores (F (1,83) = 89.06, p < .0001, R2 = 0.51). Both indices were also correlated with age, number of chronic conditions and number of medications.
The study findings support the convergent validity of the eFI, which further justifies implementation of a case-finding tool that uses routinely collected primary care data in the Canadian context.
Approximately 1 in 4 older adults presenting to primary care are living with frailty and face the threat of declining health, poor quality of life, loss of independence, and greater reliance on higher levels of care . Frailty is a state of increased vulnerability to stressors involving loss of reserves in interrelated biological, psychological and social domains [2,3,4]. Due to the detrimental impact of frailty and the potential to mitigate its adverse health outcomes with targeted interventions , international consensus guidelines recommend case-finding of frailty in primary care as part of routine clinical practice [6,7,8]. However, a key difficulty in widespread implementation of this recommendation has been the reliance on opportunistic case-finding in clinical practice using bedside instruments and questionnaires. Many of these tools require additional time, training, use of specialized equipment, or clinical resources, thus hindering efficiency and consistency in a busy primary care setting.
A recent breakthrough in the United Kingdom (UK) has been the development, validation and national implementation of an electronic frailty index (eFI) for frailty case-finding in primary care using data from electronic medical records . The eFI is based on the deficit accumulation approach to frailty. This approach, developed first by Rockwood, Mitnitski and colleagues , identifies frailty based on a range of variables (e.g., signs, symptoms, diseases, disabilities, impairments, abnormal test values) collectively referred to as health deficits . According to this model, frailty can be measured by calculating a frailty index (FI) that can be generated from any appropriately populated healthcare database [12,13,14] provided that there are a sufficient number of health deficits that satisfy certain criteria [13, 15]. Primary care electronic medical records (EMRs) contain rich data on a patient’s health and psychosocial context that make it a promising dataset to generate a FI score. The eFI in UK has been developed and validated using routinely available primary care EMR data from around 900,000 patients . A pilot study in primary care in England demonstrated that the eFI was simple and quick to use (i.e. it is fully automated and scores are available at point of care), acceptable to practice staff, and was able to discriminate older patients referred for comprehensive geriatric assessment (CGA) from the total practice population .
While there is a number of frailty indices developed so far, the eFI represents an interesting approach of using routinely available data in the primary care EMR for frailty identification. However, there is limited knowledge of the applicability and validity of such FI in the Canadian context. In this project, the convergent validity—a core component of construct validity of a test —of the eFI (calculated manually) from Canadian primary care EMR data with a validated frailty index tool, a frailty index based on a comprehensive geriatric assessment (FI-CGA) was examined.
Study design, setting and subjects
This cross-sectional study included 85 patients from the Seniors’ Community Hub (SCH), located in Edmonton, Alberta, Canada. The SCH is a team-based, integrated primary care research program guided by a geriatric specialist that aims to improve care for seniors living with frailty in the community. Patients 65 years of age and older from six primary care physicians’ practices were assessed in the SCH. Patients were excluded if they were under 65 years of age, did not provide consent to participate in the SCH, or were living in a long term care facility at the time of enrollment. These 85 patients received a CGA by a geriatric assessment nurse who was trained by the geriatric specialist. Twelve participants were seen by the geriatric specialist but did not consent to participation in the study and hence no data on them were collected.
The eFI includes 36 health deficits derived from the EMR (comorbidities, physical impairments, clinical signs, symptoms, abnormal test values, and social circumstances; see Appendix 1)  EMRs in Canada were developed for transactional patient management rather than reporting and much of the data is in narrative/open text form. Multiple studies have shown that the combination of structured and unstructured data (International Classification of Diseases 9th revision codes, information about medications, laboratory values, visit notes) in EMR results in the best performance for disease identification [18, 19]. Therefore, a trained research assistant manually calculated the eFI scores from patient EMR using all available data sources (e.g. billing and diagnostic codes, problem list, medication list, and free text (visit notes)). As such, this process differs from the eFI validated in the UK where the 36 deficits are linked to over 2000 Read codes and retrieved automatically if present within the primary care EMR system. The research assistant coded deficits as 1 if present in the EMR data for that patient; and 0 if absent, whether the patient does not have that deficit or this information is missing in the EMR. All structured and unstructured data available starting 2012 (i.e. when the EMR was implemented in the participating clinic) was used for eFI calculation. For example, if chronic kidney disease was mentioned in 2013, the corresponding eFI deficit was checked off. However, for temporary conditions (e.g. anemia) the “look-back” period was one year. The eFI score was calculated by dividing the total number of deficits present by 36.
CGA is the current criterion standard for frailty identification and management [20, 21]. CGA is defined here as a thorough interdisciplinary and multidimensional assessment process used to determine the medical, functional, social, and psychological aspects of an older adult living with frailty and guide individualized care and support planning . As with a primary care derived FI, a frailty index can also be derived ‘a posteriori’ from the content of the CGA (FI-CGA).
FI-CGA has been previously developed and validated based on its ability to predict individuals at higher risk of adverse health outcomes . We constructed an FI-CGA that included 41 variables (see Appendix 2) following a standard protocol  from the CGA completed as part of the SCH process of care. The health deficits included in the FI-CGA comprised ordinal and nominal variables: all continuous variables were transformed into categorical variables. Missing data could occur with a variable like gait speed if the patient was unable to walk. Any patient who was missing 20% or more of the variables were excluded from the study. A family physician (member of the research team who is well versed in the CGA) independently calculated FI-CGA scores for all patients based on the content of the completed CGAs. FI-CGA scores were calculated by taking the sum of the deficits present and dividing it by 41.
Descriptive statistics were used for the sample population. We used scatter plots to assess linearity and Pearson correlation coefficients to examine convergent validity between eFI and FI-CGA, as well as to examine correlation between both indices and age, number of chronic conditions, and number of medications. We used the following interpretation of correlation coefficients: 0 to 0.19 = very weak; 0.20 to 0.39 = weak; 0.40 to 0.59 = moderate; 0.60 to 0.79 = strong; 0.80 to 1.0 = very strong . The correlation analysis was followed by regression analysis to construct a best fit model to predict FI-CGA scores based on eFI scores. We used independent-samples t-test to assess differences in eFI and FI-CGA scores between different groups of participants (with grouping factors being sex, living status, number of chronic conditions, etc.). The level of statistical significance was set at an alpha level of 0.05. All statistical analysis was conducted using SAS 9.4 (SAS institute Inc., Cary NC, USA).
The Health Research Ethics Board, University of Alberta approved the study, and each participant signed informed consent forms. Funding was received from the Covenant Health Network of Excellence in Seniors’ Health and Wellness, as specified in the Acknowledgment. The funding agency did not have any role in formulating the research question and objectives, conducting analysis or preparing the manuscript.
Description of the sample
The sample (n = 85) consisted of 51 (60%) females and the mean age was 81.1 (Me = 82, SD = 7.6), with 60% being 80 years and older. Table 1 shows the main characteristics of the sample. None of the patients had more than 20% missing values. There were only two variables with missing values: body mass index (BMI) and 4 m walk test were missing in 3 (3.5%) and 8 (9.4%) patients, respectively. Although the frailty index is meant to be used as a continuous score , to describe different frailty levels as defined by the FI-CGA and eFI, we used proposed cut-off scores identified using stratum specific likelihood ratios by Hoover et al.  that had been validated in a sample of community dwelling seniors in Canada: non-frail (0 to ≤0.1), vulnerable (> 0.1 to ≤0.21), frail (> 0.21 to < 0.45), and most frail (≥ 0.45) . However, due to low frequency of scores of 0.1 and less (only one person), we merged non-frail and vulnerable categories as following: non-frail (0 to ≤0.21), frail (> 0.21 to < 0.45), and most frail (≥0.45). According to the eFI and FI-CGA scores, 12 (14.1%) and 15 (17.6%) were considered non-frail, 66 (77.6%) and 51 (60%) – frail, and 7 (8.2%) and 19 (22.4%) – most frail, respectively.
Distribution of eFI and FI-CGA
Mean scores (and SD) for eFI and FI-CGA were 0.30 (0.10) and 0.35 (0.11), respectively. The difference in the mean scores was statistically significant (p < 0.001). The distribution of the FI-CGA and eFI scores approximates normal distribution (see Figs. 1 and 2, respectively). No ceiling or floor effects for FI-CGA (max = 0.69, min = 0.10) and eFI (max = 0.58, min = 0.08) were observed.
Correlation and regression analysis
The scatterplot depicting the relationship between the eFI and FI-CGA, which was best described using a linear model, is shown in Fig. 3. Results indicate strong statistically significant correlation between the eFI and FI-CGA (r = 0.72, 95% CI 0.60–0.81, p < 0.001).
A simple linear regression was calculated to predict FI-CGA scores based on eFI scores. A significant regression equation was found (F (1,83) = 89.06, p < .0001). Participants’ predicted FI-CGA score is equal to 0.994*eFI + 0.052. So, on average, eFI is FI-CGA shifted down by 0.052—of course there is variance from patient to patient, but the fit of the model was significant.
Both indices were also correlated with age, number of chronic conditions and number of medications (see Table 2). Overall, the degree of correlation between these indices and included factors was weak to moderate. However, the degree of correlation between eFI and aforementioned factors was higher compared to correlation between FI-CGA and these factors.
Description of eFI and FI-CGA scores in the examined sample
The difference in mean scores of eFI and FI-CGA in independent groups (depending on various grouping factors) is shown in Table 3. As measured by FI-CGA, women had higher levels of frailty than men (0.378 vs 0.315, respectively, p = 0.045); however, no significant difference was found in the levels of frailty as measured by eFI. Higher scores of both eFI and FI-CGA indices were observed in patients with 3 and more chronic conditions, polypharmacy, history of falls in the past 12 months and urinary incontinence. No significant difference in scores was found in those who live and do not live alone.
This is one of the first studies to investigate the relationship between the eFI and FI-CGA as different instruments designed to assess the construct of frailty as a state, and both applied to the Canadian primary care setting. The analysis demonstrated a linear relationship and strong correlation between the eFI and FI-CGA scores, with both lower and upper limits of the 95% confidence interval supporting this strong correlation. Thus, the study findings support the convergent validity of the eFI in relation to the FI-CGA, a core component of its construct validity.
The distribution of both scores approximated a normal distribution, which is expected in a population of oldest old (> 80 years old) who warranted CGA in a primary care setting. The distribution of the FI becomes less skewed as the mean age of the sample increases, and its relative heterogeneity diminishes . In our study both indices were free of ceiling and floor effects, which is consistent with the results reported in other studies [28, 29] and might indicate that both indices were constructed with consideration of the selection criteria for health deficits outlined by Searle et al.  Proper deficit selection is crucial in establishing the consistent ability of the FI to determine frailty levels .
While strong correlation between two indices can be explained by the fact that they are based on the same theoretical framework of the cumulative deficit model of frailty, the results of the two indices were not in complete agreement. For example, the mean score for the eFI was significantly lower than that of the FI-CGA. This could be due to low prevalence or reporting of deficits derived from routine primary care data as reported elsewhere . Reasons for this may include suboptimal data entry, tendency of patients not to discuss all of their health and social concerns during a clinic visit, and the greater emphasis on comorbidities rather than function, mobility and health attitudes in this dataset. As such, if frailty-related data is systematically missing from the record, it may be assumed incorrectly to be absent . In our SCH cohort, the deficits were clearly defined and recorded as present if charted in the patient EMR, while eFI deficits not found in EMR were treated as absent. In contrast, the FI derived from the CGA intentionally explores and records challenging cognitive, psychosocial, and functional aspects of frailty and geriatric syndromes thus enriching the FI-CGA.
An important limitation was that the study sample was small; nevertheless, the 95% confidence interval was narrow, which indicates a statistically significant correlation even in such a small sample size. In addition, the sample consisted of community-dwelling older adults, not living in long-term care facilities, that were identified by family physicians as having ongoing concerns (in many cases, multiple concerns), and thus received an assessment by the SCH. This may limit the generalizability of the study findings to the very fit/robust or more functionally dependent or severely frail older adult populations. Other limitation is that the cross sectional design does not allow to compare the predictive performance of the eFI and FI-CGA. Future research may consider the predictive ability of the eFI generated from Canadian Primary Care EMR before being widely implemented. Another important limitation of this study was that the eFI calculation relied on labour-intensive review of medical records, which defeats the fundamental purpose of the eFI for rapid frailty case-finding in primary care. Since much of the data is in narrative/open text form in Canadian primary care EMRs, innovative technologies in computer science such as Natural Language Processing and Machine Learning could facilitate the future automation of the eFI in primary care.
Results indicated that measured by FI-CGA, women had higher levels of frailty than men. This finding is not new and confirmed by numerous studies [1, 26, 31, 32]. Moreover, Herr et al. , in their study of life expectancy in the state of frailty, also point to the fact that women live longer despite bearing a larger burden of health deficits than men. The authors explain this sex difference as an interplay of various social, behavioral, and biological factors. There was no difference in the eFI scores based on sex, which could be explained by the deficits included in the eFI; the comorbidities are more prevalent than functional measures.
Regardless of these differences, both indices correlated with age, number of chronic conditions and number of medications. In both the eFI and the FI-CGA, the strength of association was weak to moderate. However, the strength of these correlations was higher in the eFI, which may reflect its greater dependence on these deficits compared to the FI-CGA which includes additional assessment information, reflecting the complex and rich nature of frailty. Similar to other studies based on the FI [30, 34], higher scores of both eFI and FI-CGA indices were observed in patients with 3 and more chronic conditions, polypharmacy, history of falls in the past 12 months and urinary incontinence. No significant difference in scores was found in those living alone. However, this finding should be interpreted with caution, as living alone does not mean having no social support, nor does it exclude the possibility of ‘assets’ that make such an individual more resilient.
This study supports the convergent validity of the eFI with the FI-CGA. Health deficits that comprise the eFI in the UK were captured in the Canadian primary care EMR. The findings demonstrated a strong, linear association with the validated frailty index, FI-CGA. Therefore, the eFI has the potential to be implemented in Canada as a case-finding instrument. More research is needed to understand its performance in Canadian EMR data and in a more representative community sample.
Body mass index
Comprehensive geriatric assessment
Electronic frailty index
Electronic medical record
Frailty index based on a comprehensive geriatric assessment
Seniors’ Community Hub
Song X, Mitnitski A, Rockwood K. Prevalence and 10-year outcomes of frailty in older adults in relation to deficit accumulation. J Am Geriatr Soc. 2010;58(4):681–7.
Espinoza S, Walston JD. Frailty in older adults: insights and interventions. Cleve Clin J Med. 2005;72(12):1105–12.
Lally F, Crome P. Understanding frailty. Postgrad Med J. 2007;83(975):16–20.
Pel-Littel R, Schuurmans M, Emmelot-Vonk M, Verhaar H. Frailty: defining and measuring of a concept. J Nutr Health Aging. 2009;13(4):390–4.
Harrison JK, Clegg A, Conroy SP, Young J. Managing frailty as a long-term condition. Age Ageing. 2015;44(5):732–5.
British Geriatrics Society. Fit for frailty. Consensus best practice guidance for the care of older people living with frailty in community and outpatient settings. 2014. https://www.bgs.org.uk/sites/default/files/content/resources/files/2018-05-23/fff_full.pdf. Accessed 30 July 2018.
Dent E, Lien C, Lim WS, Wong WC, Wong CH, Ng TP, et al. The Asia-Pacific clinical practice guidelines for the management of frailty. J Am Med Dir Assoc. 2017;18(7):564–75.
National Institute for Health and Care Excellence. Multimorbidity: clinical assessment and management. 2016. https://www.nice.org.uk/guidance/ng56/chapter/Recommendations. Accessed 30 July 2018.
Clegg A, Bates C, Young J, Ryan R, Nichols L, Teale EA, et al. Development and validation of an electronic frailty index using routine primary care electronic health record data. Age Ageing. 2016;45(3):353–60.
Mitnitski A, Mogilner AJ, Rockwood K. Accumulation of deficits as a proxy measure of aging. Sci World J. 2001;1:323–36.
Rutenberg AD, Mitnitski AB, Farrell SG, Rockwood K. Unifying aging and frailty through complex dynamical networks. Exp Gerontol. 2018;107:126–9.
Kim DH, Schneeweiss S. Measuring frailty using claims data for pharmacoepidemiologic studies of mortality in older adults: evidence and recommendations. Pharmacoepidemiol Drug Saf. 2014;23(9):891–901.
Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K. A standard procedure for creating a frailty index. BMC Geriatr. 2008;8(24).
Rockwood K, Mitnitski A. Frailty defined by deficit accumulation and geriatric medicine defined by frailty. Clin Geriatr Med. 2011;27(1):17–26.
Jackson TA, Wilson D, Greig CA. Frailty and ageing. In: Bueno V, Lord J, Jackson T, editors. The ageing immune system and health. Switzerland: Springer; 2017. p. 141–59.
Lansbury LN, Roberts HC, Clift E, Herklots A, Robinson N, Sayer AA. Use of the electronic frailty index to identify vulnerable patients: a pilot study in primary care. Br J Gen Pr. 2017;67(664):e751–6.
Duckworth AL, Kern ML. A meta-analysis of the convergent validity of self-control measures. J Res Personal. 2011;45(3):259–68.
Wilson J, Bock A. The benefit of using both claims data and electronic medical record data in health care analysis. 2012. https://www.optum.com/content/dam/optum/resources/whitePapers/Benefits-of-using-both-claims-and-EMR-data-in-HC-analysis-WhitePaper-ACS.pdf. Accessed 21 July 2018.
Royal College of Physicians and Surgeons of Canada. National Physician Survey. 2014. http://nationalphysiciansurvey.ca/surveys/2014-survey/. Accessed 19 July 2018.
NHS Lanarkshire. Models of care for frailty: a systematic review. AdvantAGE; 2017 [cited 2018 May 18]. http://www.advantageja.eu/images/WP7-Models-of-Care-a-Systematic-Review.pdf
Turner G, Clegg A. Best practice guidelines for the management of frailty: a British geriatrics society, age UK and Royal College of general practitioners report. Age Ageing. 2014;43(6):744–7.
Parker S, McCue P, Phelps K, McCleod A, Arora S, Nockels K, et al. What is comprehensive geriatric assessment (CGA)? An umbrella review. Age Ageing. 2017;47(1):149–55.
Jones DM, Song X, Rockwood K. Operationalizing a frailty index from a standardized comprehensive geriatric assessment. J Am Geriatr Soc. 2004;52(11):1929–33.
Evans JD. Straightforward statistics for the behavioral sciences. Pacific Grove: Brooks/Cole; 1996.
Blodgett J, Theou O, Kirkland S, Andreou P, Rockwood K. Frailty in NHANES: comparing the frailty index and phenotype. Arch Gerontol Geriatr. 2015;60(3):464–70.
Hoover M, Rotermann M, Sanmartin C, Bernier J. Validation of an index to estimate the prevalence of frailty among community-dwelling seniors. Health Rep. 2013;24(9):10. https://pdfs.semanticscholar.org/f95b/585c9114d5351f17293280acd004fd583804.pdf. Accessed December 24 2018.
Rockwood K, Mogilner A, Mitnitski A. Changes with age in the distribution of a frailty index. Mech Ageing Dev. 2004;125(7):517–9.
Drubbel I, de Wit NJ, Bleijenberg N, Eijkemans RJ, Schuurmans MJ, Numans ME. Prediction of adverse health outcomes in older people using a frailty index based on routine primary care data. J Gerontol A Biol Sci Med Sci. 2013;68(3):301–8.
Kanters DM, Griffith LE, Hogan DB, Richardson J, Patterson C, Raina P. Assessing the measurement properties of a frailty index across the age spectrum in the Canadian longitudinal study on aging. J Epidemiol Community Health. 2017;71(8):794–9.
Drubbel I, Numans ME, Kranenburg G, Bleijenberg N, de Wit NJ, Schuurmans MJ. Screening for frailty in primary care: a systematic review of the psychometric properties of the frailty index in community-dwelling older people. BMC Geriatr. 2014;14:1.
Castell M-V, Sánchez M, Julián R, Queipo R, Martín S, Otero Á. Frailty prevalence and slow walking speed in persons age 65 and older: implications for primary care. BMC Fam Pract. 2013;14:1.
Aguayo GA, Donneau A-F, Vaillant MT, Schritz A, Franco OH, Stranges S, et al. Agreement between 35 published frailty scores in the general population. Am J Epidemiol. 2017;186(4):420–34.
Herr M, Arvieu J-J, Ankri J, Robine J-M. What is the duration of life expectancy in the state of frailty? Estimates in the SIPAF study. Eur J Ageing. 2017;15(2):165–73.
Jones D, Song X, Mitnitski A, Rockwood K. Evaluation of a frailty index based on a comprehensive geriatric assessment in a population based study of elderly Canadians. Aging Clin Exp Res. 2005;17(6):465–71.
The SCH research program was funded by the Covenant Health Network of Excellence in Seniors’ Health and Wellness.
Availability of data and materials
The dataset used and analysed during the current study will be submitted as a supplementary file upon acceptance of the manuscript.
Ethics approval and consent to participate
The Health Research Ethics Board, University of Alberta approved the study, and each participant signed informed consent forms.
Consent for publication
AC developed and validated the electronic frailty index in the United Kingdom. OT acts as Associate Editor on the BMC Geriatrics Editorial Board. Other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original version of this article was revised: the original text contained duplicated text due to a typesetting error.
About this article
Cite this article
Abbasi, M., Khera, S., Dabravolskaj, J. et al. A cross-sectional study examining convergent validity of a frailty index based on electronic medical records in a Canadian primary care program. BMC Geriatr 19, 109 (2019) doi:10.1186/s12877-019-1119-x
- Frailty case-finding
- Electronic frailty index
- Primary care