Skip to content


  • Research article
  • Open Access
  • Open Peer Review

Comparing frailty measures in their ability to predict adverse outcome among older residents of assisted living

  • 1, 2,
  • 2,
  • 3,
  • 2,
  • 1,
  • 4 and
  • 2, 5, 6Email author
BMC Geriatrics201212:56

  • Received: 23 April 2012
  • Accepted: 4 September 2012
  • Published:
Open Peer Review reports



Few studies have directly compared the competing approaches to identifying frailty in more vulnerable older populations. We examined the ability of two versions of a frailty index (43 vs. 83 items), the Cardiovascular Health Study (CHS) frailty criteria, and the CHESS scale to accurately predict the occurrence of three outcomes among Assisted Living (AL) residents followed over one year.


The three frailty measures and the CHESS scale were derived from assessment items completed among 1,066 AL residents (aged 65+) participating in the Alberta Continuing Care Epidemiological Studies (ACCES). Adjusted risks of one-year mortality, hospitalization and long-term care placement were estimated for those categorized as frail or pre-frail compared with non-frail (or at high/intermediate vs. low risk on CHESS). The area under the ROC curve (AUC) was calculated for select models to assess the predictive accuracy of the different frailty measures and CHESS scale in relation to the three outcomes examined.


Frail subjects defined by the three approaches and those at high risk for decline on CHESS showed a statistically significant increased risk for death and long-term care placement compared with those categorized as either not frail or at low risk for decline. The risk estimates for hospitalization associated with the frailty measures and CHESS were generally weaker with one of the frailty indices (43 items) showing no significant association. For death and long-term care placement, the addition of frailty (however derived) or CHESS significantly improved on the AUC obtained with a model including only age, sex and co-morbidity, though the magnitude of improvement was sometimes small. The different frailty/risk models did not differ significantly from each other in predicting mortality or hospitalization; however, one of the frailty indices (83 items) showed significantly better performance over the other measures in predicting long-term care placement.


Using different approaches, varying degrees of frailty were detected within the AL population. The various approaches to defining frailty were generally more similar than dissimilar with regard to predictive accuracy with some exceptions. The clinical implications and opportunities of detecting frailty in more vulnerable older adults require further investigation.


  • Frailty
  • Predictive accuracy
  • Agreement
  • Assisted living


Assisted living (AL) is typically defined as a setting that provides health and personal services within a secure residential environment for older adults not requiring the continuous monitoring and more intensive professional care of a long-term care facility [1, 2]. It is a growing and in many ways attractive housing option for older persons often described as frail. AL residents are prone to a number of adverse outcomes in the short term. Over a year approximately a sixth will die with a similar proportion moving to a higher level of care [3, 4]. Acute care hospitalizations also occur frequently among this group of older adults [4, 5].

There is consensus that the core feature of frailty is a heightened vulnerability to stressors (for a given chronological age and sex), which leads to an increased risk for multiple adverse health-related outcomes [6]. As a group, AL residents will have relatively higher levels of vulnerability than similarly aged community-dwelling individuals, but within the AL population there will be varying degrees of frailty. Gradients of frailty are found in long-term care populations with higher levels associated with an increased risk of mortality, cognitive decline, and new onset disability [7]. Identifying AL residents at a higher risk for adverse health outcomes would offer opportunities to both maintain remaining independence and enhance quality of life if this could be linked to effective interventions.

Few studies have directly compared the competing approaches to identifying frailty in more vulnerable older populations. Two frequently used measures are the frailty index, which is based on a count of accumulated deficits divided by the number of potential deficits [8], and the Cardiovascular Health Study (CHS) frailty criteria. With the CHS approach, the determination of frailty requires the presence of three or more characteristics felt to capture what has been termed the phenotype of frailty [9].

Data arising from the implementation of the interRAI family of assessment instruments have increasingly been used for aging-related research, including recent work on frailty [1013]. For example, Armstrong and colleagues developed a 50-item frailty index based on items in the home care instrument [10]. Embedded within each interRAI instrument are various scales. One of them, the Changes in Health, End-stage disease and Signs and Symptoms (CHESS) scale, is a measure of health stability designed to identify individuals at high risk for clinically significant decline [11]. In nursing home populations, higher CHESS scores are predictive of mortality and hospitalization [11, 12]. Some have suggested that the CHESS scale is a frailty measure [11, 13].

The primary objective of this study was to compare the ability of two versions of a frailty index derived from interRAI assessment items and the CHS frailty criteria to accurately predict the occurrence of three outcomes relevant to an AL population (mortality, hospitalization, and transfer to a long-term care facility) over a year. This builds on work previously done by our group in trying to operationalize the assessment of frailty among older residents of AL facilities [14]. A secondary objective was to compare these frailty measures to the CHESS scale in their ability to predict these three outcomes.


Study design

This was a sub-study of the Alberta Continuing Care Epidemiological Studies (ACCES), which investigated the health status of residents and quality of care issues within AL and nursing home facilities across the Province of Alberta, Canada. The AL cohort of ACCES consisted of residents aged 65 years and older residing in designated (i.e., publicly-funded) assisted living and supportive housing facilities (DAL) in two urban and three rural health regions within Alberta. The details of this study have been reported elsewhere [3, 14]. A total of 1,089 AL participants were enrolled and assessed. This sub-study includes 1,066 residents who provided consent for linkage with Provincial administrative data and had a known outcome status during the one-year follow-up.

Ethics approval was obtained from the University of Calgary Conjoint Health Research Ethics Board, the University of Alberta Health Research Ethics Board and the University of Lethbridge Human Subject Research Committee. Administrative approvals from the health regions and/or facilities were also obtained. Informed consent was received from all participants or their legally appointed proxy decision-maker.

Trained research nurses administered the Resident Assessment Instrument for Assisted Living (interRAI-AL) and additional performance-based frailty measures at baseline (2006–2007) and at 1-year (when possible). The interRAI-AL tool is a comprehensive, standardized assessment of the sociodemographic characteristics, physical and cognitive status, health conditions, behavioural problems, and use of medications and services of residents [15].

Frailty measures and CHESS score

Three frailty measures were examined. Their specific composition is described in Appendix A. The first was a frailty index based on the report of Armstrong et al. [10] that we designated as the Armstrong Index (see Appendix A). We employed a modified version with 43 of the original 50 items, as not all items were available with the interRAI-AL tool. An alternative frailty index consisting of 83-items derived from the interRAI-AL was created using the procedure described by Searle et al. [16], which we designated as the Full Frailty Index (see Appendix A). At least 30–40 items, fulfilling recommended criteria, are suggested for inclusion in a frailty index. In general, the more items included, the more precise the estimates of frailty obtained [16]. Our intent with the Full Frailty Index was to make maximal use of the interRAI-AL tool to develop a precise, practical index that was also substantially different from the Armstrong Index (i.e., 44 of the 83 items were not contained in the Armstrong Index). Both indices are calculated as the proportion of accumulated to potential deficits present in a given individual. For most items included in both indices, there was less than 5 percent missing data (with ≥ 80% of items having no missing values) [17]. For three variables (bathing, walking, self-reported health), there were higher percentages with missing values (6.9%, 11.6%, 11.6%, respectively) because either the activity did not occur during the three-day assessment time frame (bathing, walking) or the resident could/would not respond (self-reported health). These variables were retained in the respective indices (bathing and self-reported health in both indices; walking in the Full Frailty Index as well) as each individual variable makes a small contribution to the indices and prior work indicated that doing this would not significantly affect our results [16]. Any variable where there was a missing value was removed from both the numerator and denominator of the ratio used to calculate the frailty index for an individual participant (i.e., if one of our research subjects had one variable with a missing value, when calculating their Full Frailty Index the denominator was 82). In accord with other studies [16, 18], a resident with a score of less than 0.2 was considered “robust” or “not frail” for both indices while scores between 0.2-0.3 (inclusive) indicated a pre-frail state and a score greater than 0.3 would lead to the resident being categorized as frail.

The determination of frailty using the CHS approach is based on the presence of five criteria: slow gait, muscle weakness, low physical activity, unintentional weight loss, and exhaustion. If none are present, the person is categorized as not frail [9]. The presence of three or more denotes the presence of frailty while one or two signifies pre-frail status. Two of these criteria are performance-based. Please see Appendix A. Nearly 40% of AL participants could not complete the CHS frailty assessment as originally intended. By using responses to observed items from the interRAI-AL assessment instrument we were able to reduce the proportion with missing data to 15% [14]. For all analyses using CHS criteria our sample size was reduced to 927.

The derivation of the CHESS scale followed standard interRAI procedures using items from the interRAI-AL (see Appendix B) [11]. Two symptoms (i.e., dehydration, decline in food/fluid intake) of the six included in the original CHESS scale were unavailable on the interRAI-AL form and not used in calculating the symptom component of the total score. Based on the distribution of the scores in our study population and our desire to keep the number of categories comparable to the frailty measures, we collapsed the typical five point scale and categorized the scores as follows: 0 indicating a low risk for serious decline, 1 as intermediate, and 2 or more denoting a high risk.

Outcome measures and baseline characteristics

Primary outcomes were mortality, one or more hospitalizations and transfer to a long-term care facility over the 12 months following the subject’s baseline assessment. For mortality and long-term care admission, information on event occurrence was determined from concurrent reviews of facility discharge records and family interviews (at the time of death or discharge) as well as data collected at the time of resident and family follow-up assessments. For hospitalization, residents’ data were linked with the Alberta Inpatient Discharge Abstract Database for 2002–03 to 2008–09. This administrative database captures essentially 100% of all hospital admissions in the Province of Alberta.

A baseline co-morbidity score was calculated based on the Charlson co-morbidity index [19] and a validated coding algorithm [20] using relevant diagnostic codes (any occurrence during 3 years prior to baseline) from the Alberta Inpatient Discharge Abstract database. Other baseline characteristics examined included an additional co-morbidity measure (based on the sum of recorded diagnoses on the interRAI-AL) and three validated scales derived from items on the tool: the Cognitive Performance Scale (CPS) [21], Depression Rating Scale (DRS) [22], and Activities of Daily Living (ADL) Self-Performance Hierarchy Scale [23].


A weighted kappa statistic (with 95% confidence interval) was used to measure the agreement between the Armstrong Index, the Full Frailty Index, the CHS criteria, and the CHESS scale, using the 3-level risk categorization [24, 25]. Generalized linear models with a binomial distribution and log link were used to estimate risk ratios in analyses. Multivariable models adjusting for age (in years), sex, and co-morbidity were examined to assess the prognostic significance of each approach to defining frailty and the CHESS scale. The risks of one-year mortality, hospitalization (one or more), and transfer to long-term care for those categorized as frail or pre-frail by the Armstrong Index, the Full Frailty Index, and CHS criteria (compared with non-frail residents) were assessed in multivariable models. For the CHESS scale, high and intermediate risk levels were compared to the low risk group. The models considered potential confounding by age, sex and co-morbidity (using the Charlson co-morbidity index score). Predictive accuracy was assessed by comparing the area under the receiver operating characteristic curve (AUC) of a baseline model with only age and sex with that of models with age/sex/co-morbidity and age/sex/co-morbidity/frailty (or CHESS), respectively. To facilitate model comparisons, 95% confidence intervals for differences in AUC estimates (with associated p-values) were calculated [26]. Combined outcomes, death/hospitalization and death/move to long-term care, were also explored to investigate the possibility of competing risks. As there were no appreciable differences in our final conclusions, the results were not included.

We previously examined the effects of clustering (i.e., violation of the independence assumption due to nesting by AL-institution) on the outcomes and determined that the level of clustering was minimal [14]. We have therefore presented the results of our main analysis without adjustment for clustering to permit simpler statistical approaches without the loss of information. SAS Version 9.2 (SAS Institute, Inc., Cary, NC) was used for analyses.


The enrolled cohort was predominantly female (818/1066 or 76.7%) with an average age of 84.9 (standard deviation 7.3) years (see Table 1). Multiple morbidities were common. Most subjects (n = 632, 59.3%) had at least mild cognitive impairment, nearly a fifth (n = 203, 19%) had significant depressive symptoms, and over a third (n = 426, 40%) had limited or greater impairment in the performance of activities of daily living.
Table 1

Baseline characteristics for ACCES – AL Cohort

Characteristicn (%), unless otherwise noted

Main sample (n = 1,066)

CHS measurement (n = 927)1

Age, mean ± SD

84.9 ± 7.3

84.9 ± 7.3


818 (76.7)

700 (75.5)

Charlson co-morbidity index, mean ± SD

1.8 ± 2.0

1.8 ± 2.0

interRAI-AL co-morbidity count, mean ± SD

4.65 ± 2.0

4.65 ± 1.95

Cognitive Performance Scale (CPS)


  Intact/borderline intact (score 0–1)

434 (40.7)

399 (43.0)

  Mild impairment (score 2)

336 (31.5)

305 (32.9)

  Moderate impairment (score 3–4)

183 (17.2)

151 (16.3)

  Severe/very severe impairment (score 5–6)

113 (10.6)

72 (7.8)

Significant depressive symptoms (DRS 3+)

203 (19.0)

158 (17.0)

Activities of Daily Living (ADL) Scale


  Independent (score 0)

454 (42.6)

426 (46.0)

  Supervision required (score 1)

186 (17.4)

160 (17.3)

  Limited impairment (score 2)

126 (11.8)

106 (11.4)

  Extensive assistance required (score 3–4)

243 (22.8)

197 (21.2)

  Dependent (score 5–6)

57 (5.4)

38 (4.1)

Armstrong Index


  Not frail, score <; 0.2

99 (9.3)


  Pre-frail, score ≥ 0.2 and ≤ 0.3

410 (38.5)


  Frail, score > 0.3

557 (52.3)


Full Frailty Index


  Not frail, score <;0.2

344 (32.3)


  Pre-frail, score ≥ 0.2 and ≤ 0.3

416 (39.0)


  Frail, score > 0.3

306 (28.7)


CHS Frailty Criteria


  Not frail, score = 0


32 (3.5)

  Pre-frail, score = 1,2


451 (48.7)

  Frail, score = 3+


444 (47.9)



  Low risk of decline, score = 0

496 (46.5)


  Intermediate risk, score = 1

312 (29.3)


  High risk, score = 2+

258 (24.2)


ACCES Alberta Continuing Care Epidemiological Studies, AL assisted living, SD standard deviation, CHS Cardiovascular Health Study.

1Excluded observations due to missing frailty criteria, not linking to hospital data, loss to follow-up.

On the Armstrong Index, 9.3% (n = 99) were categorized as not frail, 38.5% (n = 410) pre-frail, and 52.3% (n = 557) as frail. The Full Frailty Index categorized 32.3% as not frail (n = 344), 39% pre-frail (n = 416), and 28.7% (n = 306) as frail. Using CHS criteria, 3.5% (n = 32) were categorized as not frail, 48.7% (n = 451) pre-frail and 47.9% (n = 444) as frail. On the CHESS scale, nearly half (46.5%, n = 496) were identified as being at a low risk for a serious decline, with 29.3% (n = 312) and 24.2% (n = 258) categorized as intermediate and high risk respectively.

A weighted kappa statistic showed the highest level of agreement between the Armstrong Index and the Full Frailty Index (kappa = 0.41, 95% CI 0.38-0.45) with just over half (51.7%) having identical categorizations as non-frail, pre-frail and frail on both criteria (see Table 2). The poorest agreement was between the CHS frailty measure and the CHESS scale (kappa = 0.11, 95% CI 0.08-0.14) with only 30.4% assigned to equivalent categories (non-frail/low risk, pre-frail/intermediate risk, and frail/high risk).
Table 2

Comparison and assessment of agreement among the frailty measures, based on 3-level risk categorization


Percent of residents with concordance in categorization

Weighted kappa

95% CI

Armstrong Index – Full Frailty Index




Armstrong Index – CHS Frailty Criteria




Armstrong Index – CHESS Scale




Full Frailty Index – CHS Frailty Criteria




Full Frailty Index – CHESS Scale




CHS Frailty Criteria – CHESS Scale




1All comparisons based on n = 1066 residents except for comparisons involving the CHS Frailty Criteria where n = 927 residents.

Over a year 15.9% (n = 170) of residents died, 39.8% (n = 424) were hospitalized at least once, and 19.1% (n = 204) moved to long-term care (see Table 3). Frail subjects defined by the three approaches as well as those at a high risk for decline on the CHESS scale showed a statistically significant increase in their risk of dying compared to those categorized as either not frail or at low risk for decline. The risk ratios in models adjusted for age, sex, and co-morbidity ranged from 1.74 (95% CI 1.07-2.81) with CHS frailty criteria to 2.35 (95% CI 1.56-3.54) for the Full Frailty Index. Only the Full Frailty Index was associated with a statistically significant increase in the risk of death for pre-frail subjects in our adjusted models (RR 2.00, 95% CI 1.33-3.00).
Table 3

One-year outcomes* associated with frailty measures, Risk Ratios (95% Confidence Intervals)

Frailty model



Move to Long-Term care


(170 events)1

(424 events)1

(204 events)1

Armstrong Index



1.23 (0.63-2.44)

0.93 (0.69-1.24)

2.25 (1.00-5.08)

N = 1,066



2.27 (1.09-4.32)

1.23 (0.94-1.63)

4.21 (1.91-9.25)




1.17 (0.59-2.29)

0.91 (0.68-1.22)

2.21 (0.98-4.99)



1.94 (1.02-3.70)

1.16 (0.87-1.53)

4.14 (1.87-9.14)

Full Frailty Index



2.22 (1.47-3.34)

1.44 (1.19-1.74)

1.85 (1.26-2.72)

N = 1,066



2.69 (1.78-4.07)

1.33 (1.09-1.63)

3.30 (2.30-4.75)




2.00 (1.33-3.00)

1.37 (1.13-1.66)

1.87 (1.27-2.75)



2.35 (1.56-3.54)

1.28 (1.04-1.57)

3.30 (2.29-4.76)

CHS Criteria 1



1.04 (0.59-1.81)

1.10 (0.84-1.43)

1.54 (0.95-2.51)

N = 927



2.25 (1.40-3.61)

1.60 (1.27-2.02)

2.21 (1.41-3.46)




0.88 (0.51-1.54)

1.06 (0.82-1.38)

1.49 (0.91-2.43)



1.74 (1.07-2.81)

1.45 (1.15-1.83)

2.17 (1.38-3.41)




1.48 (1.04-2.09)

1.31 (1.10-1.56)

1.33 (0.98-1.81)

N = 1,066



2.13 (1.53-2.96)

1.32 (1.10-1.58)

1.84 (1.38-2.47)




1.38 (0.98-1.94)

1.27 (1.07-1.51)

1.33 (0.98-1.82)



1.87 (1.35-2.59)

1.25 (1.05-1.50)

1.87 (1.39-2.50)

*Robust (not frail or low risk for decline) used as reference category.

1For CHS analysis the number of events for death, hospitalization, and move to long-term care were 142, 375, and 172 respectively.

2Adjusted for age, sex, and co-morbidity (Charlson co-morbidity index score).

Frail and pre-frail individuals as defined by the Armstrong Index had no statistically significant increase in their risk for hospitalization (please see Table 3). Those meeting CHS criteria for frailty were at greater risk for hospitalization (risk ratio in adjusted model was 1.45, 95% CI 1.15-1.83), but there was no statistically significant increase in the risk ratio for the pre-frail group. Risk ratios for hospitalization were significantly increased for both pre-frail and frail subjects on the Full Frailty Index, but no gradient was observed and confidence intervals for the risk ratios for pre-frail (1.37, 95% CI 1.13-1.66) and frail (1.28, 95% CI 1.04-1.57) subjects overlapped. A similar increased risk for hospitalization was observed for those categorized as intermediate or high risk for decline on the CHESS scale in adjusted analyses.

Residents categorized as frail on all approaches considered and the high risk group for decline on the CHESS scale showed a statistically significant greater risk of transfer to long-term care (please see Table 3). The adjusted risk ratio was strongest for the Armstrong Frailty Index (4.14, 95% CI 1.87-9.14) followed by the Full Frailty Index (3.30, 95% CI 2.29-4.76). Only the Full Frailty Index showed a statistically significant increased risk of long term care admission for pre-frail individuals (1.87, 95% CI 1.27-2.75) in adjusted analyses.

To compare the abilities of the three frailty approaches and the CHESS scale to predict our outcomes of interest, we examined areas under the ROC curve (AUC). As potential confounders we considered sex, age, and co-morbidity index scores (please see Table 4). The AUCs obtained for the frailty measures and the CHESS scale ranged from 0.683-0.701 for mortality, 0.609-0.629 for hospitalization, and 0.602-0.667 for admission to long-term care within a year. For death, the addition of frailty (however operationalized) or CHESS-defined risk for decline significantly improved on the AUC obtained with sex, age, and co-morbidity (p-values less than 0.03 for pair-wise comparisons), though the magnitude of improvement seen was relatively small (0.031-0.049). None of the AUCs with a frailty measure or the CHESS scale differed significantly from each other. As for hospitalization, only the CHS frailty criteria significantly improved on the model based on age, sex, and co-morbidity (AUC 0.629, 95% CI 0.592-0.665 versus AUC 0.598, 95% CI 0.561-0.635, p = 0.003 for the comparison). The magnitude of improvement was relatively small (0.031), and none of the AUCs with the addition of frailty or the CHESS scale differed significantly from each other. For transfer to long-term care, the addition of any of the frailty measures or the CHESS-defined risk of decline improved on the model with sex, age and co-morbidity (p-values less than 0.03 for pair-wise comparisons). The magnitude of improvement seen ranged from 0.048 to 0.113. The AUC for the Full Frailty Index (0.667) differed significantly from the AUCs for the CHESS scale (0.602, p = 0.016) and CHS frailty criteria (0.610, p = 0.003). The difference between the AUCs of the Full Frailty Index and the Armstrong Index (0.638) was of borderline significance (p = 0.087).
Table 4

Area under the ROC curve (95% Confidence Intervals) for models of one-year outcomes, comparing frailty measures




Move to Long-Term care1


Sex and age

0.638 (0.592-0.683)

0.531 (0.495-0.566)

0.552 (0.510-0.593)


Sex, age, and co-morbidity

0.652 (0.607-0.698)

0.592 (0.558-0.627)

0.554 (0.512-0.596)


Sex, age, co-morbidity, and Armstrong Index

0.683 (0.639-0.728)

0.609 (0.575-0.643)

0.638 (0.598-0.678)


Sex, age, co-morbidity, and Full Frailty Index

0.691 (0.648-0.733)

0.610 (0.576-0.644)

0.667 (0.625-0.707)3


Sex, age, co-morbidity, and CHS Frailty Criteria 4

0.701 (0.655-0.747)

0.629 (0.592-0.665)

0.610 (0.564-0.656)


Sex, age, co-morbidity, and CHESS Scale

0.683 (0.640-0.725)

0.610 (0.576-0.645)

0.602 (0.558-0.646)


1. Models 3–6 differed from models 1 and 2 for outcomes of death and long-term care. All p-value less than 0.03.

2. For hospitalization outcome, models 3, 4, and 6 differed from model 1 (p <; .001 for all), and model 5 differed from both models 1 and 2 (p <; .001, p = .003), respectively).

3. For long-term care, model 4 differed from models 5 (p = .003), 6 (p = .016), and marginally with model 3 (p = .087).

4. All pair-wise comparisons involving model 5 are based on n = 927 residents; otherwise, n = 1066 residents. The relevant AUC (95%CIs) estimates for Models 1 and 2 for n = 927 residents are as follows:

Death: Model 1, 0.649 (0.600-0.698); Model 2, 0.670 (0.622-0.719).

Hospitalization: Model 1, 0.520 (0.481-0.558); Model 2, 0.598 (0.561-0.635).

Long-Term Care: Model 1, 0.552 (0.506-0.598); Model 2, 0.552 (0.506-0.598).

As a sensitivity analysis, we looked also at two composite outcomes (hospitalizations + deaths and long-term care transfers + deaths) to see if censoring affected our results. As they were essentially unchanged, we have not reported the specific estimates for these composite outcomes.


In this population of AL residents, both the presence of frailty defined by any of the three approaches examined and a higher CHESS score were associated with an increased risk of dying or being transferred to a long-term care facility. The magnitude of the ability of these measures to correctly classify subjects (relative to models including only age, sex and co-morbidity) was modest for mortality but better for long-term care placement. The AUCs obtained were in the 0.6 to 0.7 range. This generally indicates low accuracy in correctly differentiating risk [27], but our objective was not to develop a comprehensive prediction model for these outcomes in our AL population. The current study comparing these three frailty measures and the CHESS score builds directly on our prior work examining frailty in the AL setting [14].

The different approaches to detecting frailty were more similar than dissimilar with regard to predictive accuracy with a few exceptions. The Full Frailty Index performed significantly better in predicting a move to long-term care and was the only approach that showed higher mortality, hospitalization, and institutionalization risks among those categorized as pre-frail. Only the addition of CHS frailty criteria significantly improved on age, sex, and co-morbidity in predicting hospitalization. While our study did not directly address practicality, acceptability to practitioners, or clinical utility, a number of observations can be made. Both the Armstrong Index and CHS frailty criteria categorized relatively few AL residents as not frail and approximately half as frail. In comparison to the Full Frailty Index and the CHESS scale, they were less successful in creating a more circumscribed group of AL residents for potential interventions. This would be an important consideration when clinical resources are limited and have to be targeted. The Armstrong and Full Frailty Indices (and the CHESS scale) were constructed using data available on the interRAI-AL instrument, one of a suite of similar instruments being implemented across continuing care settings in Canada. While many items were included in the indices, their calculation could be automated. This could make the use of multi-item indices practical from a clinical standpoint. The CHS frailty criteria did not perform significantly better than the frailty indices (other than possibly in predicting hospitalization) and required the use of performance measures, which can be difficult to obtain on AL residents [14, 28]. The latter issue raises major feasibility concerns with this approach in an AL population. The ability to derive the CHESS scale and two frailty indices from previously collected data highlights the practical utility of these measures while the inclusion of a wider set of potentially relevant domains in operationalizing frailty represents a conceptual strength of the two frailty indices [6].

As shown in Table 2, the best agreement among the measures was between the Full Frailty and Armstrong indices. Although moderate agreement between these two indices was observed [29], it is surprising that this was not greater given the degree of commonality in items (see Appendix A) and their similar operational approaches. Interestingly, Rockwood et al. found that random combinations of items making up frailty indices led to little overlap across quartiles [30]. The lower than expected level of agreement between these two indices may also reflect the inclusion of several Instrumental ADLs (e.g., meal preparation, housework, managing medications) in the Armstrong (but not the Full Frailty) index. As these areas are generally managed or provided for residents by the AL facility (resulting in most residents being assessed as impaired in these areas) they would be considered poor criteria for a frailty measure [16]. This may also explain the low proportion of residents categorized as not frail by the Armstrong index. Fair agreement was observed between the Full Frailty Index and the CHESS and between the Armstrong Frailty Index and the CHS. The remaining kappa statistics indicated only slight agreement between the measures. Armstrong et al. had previously reported a low correlation between the CHESS and their frailty index (r = 0.35) [10].

Whether the CHESS scale is a frailty measure cannot be answered by our study. Compared with the other measures examined, the CHESS scale generally performed similarly in predicting our outcomes of interest, although associations were weaker for long term care placement. This was most evident for comparisons between CHESS and the two frailty indices. In settings where interRAI instruments are widely implemented, the CHESS scale offers the advantage of being relatively simple and easy to assess across care settings.

The reasons underlying the relatively weaker associations between frailty and the outcome of hospitalization are not entirely clear. When added to the model with age and sex, the Charlson co-morbidity index score improved the AUC for hospitalization (see Table 4) and has been previously shown to predict hospitalization (and mortality) in older institutionalized residents [31]. The further addition of frailty and/or health instability may offer relatively less predictive gain for this outcome after the addition of a co-morbidity index. Another possibility is competing risk though we found no evidence of this in the analyses performed for this study. In earlier work we failed to observe a statistically significant increase in the risk of hospitalization among frail men that we felt was due to their high mortality rate during follow-up [14]. Other considerations would include the inherent difficulty of predicting hospitalizations especially for catastrophic (e.g., fall with a hip fracture) changes in health, the modifying effects of factors such as advance planning and the resources available within AL facilities [32], and the obscuring influence of variability in local hospitalization rates [33].

There are some limitations in our study that would raise concerns about the generalizability of our findings to the entire AL population. First, over four hundred eligible residents did not enroll in the study [14]. While the age and sex distribution of those not enrolled was comparable to our enrolled cohort, we do not have other information on them. Second, we restricted eligibility to residents of publicly-subsidized AL facilities in Alberta. Our findings may not apply to residents in private AL or AL-type facilities in other jurisdictions that may have different admission criteria, staffing, and institutional policies. Hospitalizations were determined using provincial data, and we may have missed the rare event that occurred outside Alberta. Further research examining these and other relevant outcomes (e.g., functional decline) over longer periods of time is also warranted. Nearly 40% of AL participants could not complete the CHS frailty assessment as originally intended. By using responses to observed items from the interRAI-AL assessment instrument we were able to reduce the proportion with missing data to 15%. As noted previously this raises questions about the feasibility of this particular approach in an AL population.


Using a number of approaches, varying degrees of frailty can be detected within AL populations. Residents with more severe levels of frailty are more likely to die, be hospitalized, and require admission to long-term care. Whether the detection of frailty will turn out to be clinically useful will require further work, in particular intervention studies.

Appendix A

The following items were included in the cumulative deficit index meant to replicate that created by Armstrong et al. [10], (Table 5). Three items which were not exactly replicable using the interRAI-AL were, ”severe malnutrition”, “problem chewing” and “head trauma”. Also “Alzheimer” and “Dementia other than Alzheimer’s” were combined into one category. “Hip fracture”, “other fractures”, and “osteoporosis” were combined into one category. Also, “Feeling of sadness” and “Sad, pained worried facial expressions” were combined into one category. This reduced the index from 50 items to 43 items.

The presence of each condition added “1” to the person’s Index score (unless otherwise indicated).

The following index was created using the criteria based on Searle et al. [16] using information from the interRAI-AL, (Table 6). The presence of each condition added “1” to the person’s index score (unless otherwise indicated).

The following details the frailty criteria from the Cardiovascular Health Study (CHS) [9], (Table 7).
Table 5

Armstrong frailty index (43 items)

Name of frailty item by subject header


  Persistent anger

  Unrealistic fears

  Repetitive health complaints

  Sad, pained, worried facial expressions

  Withdrawal from activities of interest

  Reduced social interactions


  Moderate/severe vision problems

Functional status

  Limited help (0.5), extensive help (1) with meal preparation

  Limited help (0.5), extensive help (1) with ordinary housework

  Limited help (0.5), extensive help (1) with managing finances

  Limited help (0.5), extensive help (1) with managing meds

  Limited help (0.5), extensive help (1) with stair climbing

  Limited help (0.5), extensive help (1) with shopping

  Limited help (0.5), extensive help (1) with bathing

  Limited help (0.5), extensive help (1) with personal hygiene

  Limited help (0.5), extensive help (1) with dressing upper body

  Limited help (0.5), extensive help (1) with dressing lower body

  Limited help (0.5), extensive help (1) with locomotion

  Limited help (0.5), extensive help (1) with transferring

  Minimal help (0.5), extensive help (1) with toilet use

  Minimal help (0.5), extensive help (1) with eating


  Some (0.5) or daily (1) bladder incontinence

Disease diagnosis

  Hip fracture, other fractures, osteoporosis


  Alzheimer’s disease/dementia


  Multiple sclerosis

  Parkinson's disease

  Stroke or CVA


  Coronary heart disease

  Congestive heart failure

  Chronic Obstructive Pulmonary Disease (COPD)/Emphysema/Asthma


  Renal disease

  Peripheral vascular disease

  Cardiac arrhythmias

  Thyroid disease

Health conditions

  Balance - unsteady gait

  Poor self-reported health

  Unstable health


  BMI 30–40 (0.5), BMI > 40 (1)

  Weight loss 5% or more in last 30 days or 10% in last 180 days

Table 6

Full frailty index (83 items)

Name of frailty item by subject header

Psychosocial well-being

  Not close to someone in facility

  No strong supportive relationship with family

  Infrequent participation in longstanding social activities

  Infrequent visits from friends/family

  Infrequent interaction with friends/family


  Makes negative statements

  Persistent anger

  Unrealistic fears

  Repetitive health complaints

  Repetitive anxiety

  Sad, pained, worried facial expressions

  Crying, tearfulness

  Withdrawal from activities of interest

  Reduced social interactions

  Lack of pleasure in life


  Minimally impaired (0.5) or moderate/severely impaired (1) decision-making skills

  Short-term memory problems

  Procedural memory problems

  Situational memory problems

  Easily distracted

  Episodes of disorganized speech

  Declined decision-making last 90 days


  At least some difficulty to make self understood

  At least some difficulty understanding

  Moderate/severe hearing problems

  Moderate/severe vision problems

Functional status

  Limited help (0.5), extensive help (1) with phone use

  Limited help (0.5), extensive help (1) with stair climbing

  Limited help (0.5), extensive help (1) with shopping

  Limited help (0.5), extensive help (1) with bathing

  Limited help (0.5), extensive help (1) with personal hygiene

  Limited help (0.5), extensive help (1) with dressing upper body

  Limited help (0.5), extensive help (1) with dressing lower body

  Limited help (0.5), extensive help (1) with walking

  Limited help (0.5), extensive help (1) with locomotion

  Limited help (0.5), extensive help (1) with transferring

  Minimal help (0.5), extensive help (1) with toilet use

  Minimal help (0.5), extensive help (1) with bed mobility

  Minimal help (0.5), extensive help (1) with eating

  Less than 1 hour of physical activity in last 3 days

  Did not go out within a 3 day period

  Declined in ADL over last 90 days


  Some (0.5), daily (1) bladder incontinence

  Some (0.5), daily (1) bowel incontinence

Disease diagnosis

  Hip fracture, other fractures, osteoporosis


  Alzheimer’s disease/dementia


  Multiple sclerosis


  Parkinson's disease

  Stroke or CVA


  Coronary heart disease

  Congestive heart failure




  Renal disease

  Peripheral vascular disease

  Cardiac arrhythmias

  Thyroid disease

Health conditions

  At least 1 fall in last 30 days

  Balance - turning around

  Balance - dizziness

  Balance - unsteady gait

  Chest pain

  Abnormal thought process





  Non-restful sleep/insomnia

  Too much sleep

  Peripheral edema

  Shortness of breath

  Fatigue - cannot complete day-to-day activities

  Pain present

  Poor self-reported health


  BMI 30–40 (0.5), BMI > 40 (1)

  Weight loss 5% or more in last 30 days or 10% in last 180 days

  Ten or more medications

  Allergy to any drug

Table 7

Cardiovascular Health Study Frailty Criteria1



Frailty criteria

Slow gait

Determined by taking the better of two timed 3-meter walks.

≥7 seconds2, men ≤ 173 cm ≥ 7 seconds, women ≤ 159 cm ≥ 6 seconds, men > 173 cm

≥ 6 seconds, women > 159 cm

Muscle weakness

Average of three grip strength readings using a handheld dynamometer.3

BMI-specific thresholds: ≤ 29-32 kg, men

≤ 17-21 kg, women

Low physical activity

Reported minutes over two weeks per activity type - from the interRAI-AL “Exercise or Leisure Activities” 4

Activities were mapped to Minnesota Leisure Time Activity Questionnaire [34]. Kcals per week calculated based on the intensity codes: < 383 Kcals/week, men

< 270 Kcals/week, women

Unintentional weight loss

Answer to question: “In the past year have you lost more than 10 pounds unintentionally” 5

Response of “Yes”


Answers to 3 questions: “In the past month, on average, have you been: 1) Feeling unusually tired during the day?; 2) Feeling unusually weak?; and/or, 3) Feeling an unusually low energy level?”6

Response of Yes to any of the 3 questions

CHS Cardiovascular Health Study, AL assisted living, cm centimeters, BMI body mass index, kg kilograms, kcals kilocalories.1As detailed in Fried et al. 2001 [9].2Sex and height-specific thresholds.3JAMAR®, Sammons Preston Rolyan, Bolingbrook, IL.4Include: aquasize/swimming; bowling; dancing; exercise bike/treadmill; exercise program; floor curling/lawn bowling; gardening; household chores; shuffleboard/pool; Tai chi/yoga; walking/wheeling indoors & outdoors.5CHS also allowed actual unintentional 5% weight loss over 1-year (not assessed in ACCES).6CHS used 2 items from the CES-D Scale [35]: “I feel that everything I do is an effort” and “I cannot get going” (those reporting feeling this way at least 3–4 days/previous week fulfilled the criterion).

Appendix B


CHESS stands for Changes in Health, End-stage disease and Symptoms and Signs of medical problems [11]. It provides a measure of instability in health (which may be a consequence of frailty) and is believed to be a marker of imminent decline in health. The score is based on the following:

Symptoms: A summary count is first derived from the following symptoms (coded as 0 = no symptoms present; 1 = 1 symptom present; 2 = 2+ symptoms present).

  • vomiting

  • dehydration (insufficient fluid)*

  • decline in food/fluid intake*

  • weight loss

  • shortness of breath

  • edema

To this summary count variable, 1 point is added for “worsening of decision making over previous 90 days”, 1 point for “decline in activities of daily living over previous 90 days”, and 1 point for “end-stage disease”.

The range of values for the CHESS is 0 to 5, where 0 represents stability, and 5 represents highly unstable health.

*Note that two items from the original CHESS, were unavailable on the interRAI-AL form, and were not included in the calculation of symptoms:

  • dehydration

  • decline in food/fluid intake



Special thanks are given to Deanna Wanless, Anna Charlton, Cheri Komar (Study Coordinators), Dr. Misha Eliasziw (Study Advisor), our research staff, and the facilities, residents and their family members who participated in ACCES.


This study was funded by the Alberta Heritage Foundation for Medical Research (#200400893), the Canadian Institutes of Health Research (CIHR) (MOP81216) and CIHR-Institute of Aging Northern and Rural Health Research Initiative (HAS-63179).

Authors’ Affiliations

Division of Geriatric Medicine, Faculty of Medicine, University of Calgary, HSC 3330 Hospital Drive NW, Calgary, AB, Canada
Department of Community Health Sciences, University of Calgary, 3rd Floor TRW, 3280 Hospital Drive NW, Calgary, AB, Canada
Department of Sociology, University of Alberta, 5-21 HM Tory Building, Edmonton, AB, Canada
Division of Geriatric Medicine, Faculty of Medicine, University of Alberta, 1-108, 11350-83 Avenue, Edmonton, AB, Canada
Schools of Pharmacy and Public Health & Health Systems, University of Waterloo, 200 University Avenue West, Waterloo, ON, Canada
Institute for Clinical Evaluative Sciences, Toronto, ON, Canada


  1. Golant SM: Do impaired older persons with health care needs occupy US assisted living facilities? An analysis of six national studies. J Gerontol B Psychol Sci Soc Sci. 2004, 59B (2): S68-S79.View ArticleGoogle Scholar
  2. Canadian Centre for Elder Law: Discussion paper on assisted living: Past, present and future legal trends in Canada.], Accessed April 13, 2012
  3. Strain LA, Maxwell CJ, Wanless D, Gilbart E: Designated Assisted Living (DAL) and Long-term Care in Alberta (LTC): Selected Highlights from the Alberta Continuing Care Epidemiological Studies (ACCES). 2011, Edmonton, AB: ACCES Research Group, University of Alberta,], Accessed April 13, 2012Google Scholar
  4. Zimmerman S, Sloane PD, Eckert JK, Gruber-Baldini AL, Morgan LA, Hebel JR, Magaziner J, Stearns SC, Chen CK: How good is assisted living? Findings and implications from an outcomes study. J Gerontol B Psychol Sci Soc Sci. 2005, 60 (4): S195-S204. 10.1093/geronb/60.4.S195.View ArticlePubMedGoogle Scholar
  5. Becker M, Boaz T, Andel R, Demuth A: Predictors of avoidable hospitalizations among assisted living residents. J Am Med Dir Assoc. 2012, 13 (4): 355-359. 10.1016/j.jamda.2011.02.001.View ArticlePubMedGoogle Scholar
  6. Bergman H, Ferrucci L, Guralnik J, Hogan DB, Hummel S, Karunananthan S, Wolfson C: Frailty: an emerging research and clinical paradigm-issues and controversies. J Gerontol A Biol Sci Med Sci. 2007, 62A (7): 731-737.View ArticleGoogle Scholar
  7. Rockwood K, Abeysundera MJ, Mitnitski A: How should we grade frailty in nursing home patients?. J Am Med Dir Assoc. 2007, 8 (9): 595-603. 10.1016/j.jamda.2007.07.012.View ArticlePubMedGoogle Scholar
  8. Mitnitski AB, Mogilner AJ, Rockwood K: Accumulation of deficits as a proxy measure of aging. The Scientific World. 2001, 1: 323-336.View ArticleGoogle Scholar
  9. Fried LP, Tangen CM, Walston J, Newman AB, Hirsch C, Gottdiener J, Seeman T, Tracy R, Kop WJ, Burke G, McBurnie MA, Cardiovascular Health Study Collaborative Research Group: Frailty in older adults: evidence for a phenotype. J Gerontol A Biol Sci Med Sci. 2001, 56 (3): M146-M156. 10.1093/gerona/56.3.M146.View ArticlePubMedGoogle Scholar
  10. Armstrong JJ, Stolee P, Hirdes JP, Poss JW: Examining three frailty conceptualizations in their ability to predict negative outcomes for home-care clients. Age Ageing. 2010, 39 (6): 755-758. 10.1093/ageing/afq121.View ArticlePubMedGoogle Scholar
  11. Hirdes JP, Frijters D, Teare G: The MDS CHESS Scale: a new measure to predict mortality in the institutionalized older people. J Am Geriatr Soc. 2003, 51 (1): 96-100. 10.1034/j.1601-5215.2002.51017.x.View ArticlePubMedGoogle Scholar
  12. Mor V, Intractor O, Unruh MA, Cai S: Temporal and geographic variation in the validity and internal consistency of the Nursing Home Resident Assessment Minimum Data Set 2.0. BMC Health Serv Res. 2011, 11: 78-10.1186/1472-6963-11-78.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Tjam EY, Heckman GA, Smith S, Arai B, Hirdes J, Poss J, McKelvie RS: Predicting heart failure mortality in frail seniors: comparing the NYHA functional classification with the Resident Assessment Instrument (RAI) 2.0. Int J Cardiol. 2012, 155: 75-80. 10.1016/j.ijcard.2011.01.031.View ArticlePubMedGoogle Scholar
  14. Freiheit EA, Hogan DB, Strain LA, Schmaltz HN, Patten SB, Eliasziw M, Maxwell CJ: Operationalizing frailty among older residents of assisted living facilities. BMC Geriatr. 2011, 11: 23-10.1186/1471-2318-11-23.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Hawes C, Phillips C, Morris JN, Nonemaker S, Berg K, Bjorkgren M, Fries BE, Frijters D, Gilgen R, Gray L, Henrard JC, Hirdes JP, Ljunggren G, Steel K, Zimmerman D: User’s guide for the interRAI-AL. 2006, Washington, DC: interRAIGoogle Scholar
  16. Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K: A standard procedure for creating a frailty index. BMC Geriatr. 2008, 8: 24-10.1186/1471-2318-8-24.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Rockwood K, Mitnitski A: Frailty in relation to the accumulation of deficits. J Gerontol A Biol Sci Med Sci. 2007, 62 (7): 722-727. 10.1093/gerona/62.7.722.View ArticlePubMedGoogle Scholar
  18. Kulminski AM, Ukraintseva SV, Kulminskaya IV, Arbeev KG, Land K, Yashin AI: Cumulative deficits better characterize susceptibility to death in elderly people than pheonotypic frailty: lessons from the Cardiovascular Health Study. J Am Geriatr Soc. 2008, 56: 898-903. 10.1111/j.1532-5415.2008.01656.x.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Charlson ME, Pompei P, Ales KL, MacKenzie CR: A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987, 40 (5): 373-383. 10.1016/0021-9681(87)90171-8.View ArticlePubMedGoogle Scholar
  20. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, Saunders LD, Beck CA, Feasby TE, Ghali WA: Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005, 43: 1130-1139. 10.1097/01.mlr.0000182534.19832.83.View ArticlePubMedGoogle Scholar
  21. Hartmaier SL, Sloane PD, Guess HA, Koch GG, Mitchell CM, Phillips CD: Validation of the minimum data set cognitive performance scale: agreement with the mini-mental state examination. J Gerontol A Biol Sci Med Sci. 1995, 50A (2): M128-M133. 10.1093/gerona/50A.2.M128.View ArticleGoogle Scholar
  22. Burrows AB, Morris JN, Simon SE, Hirdes JP, Phillips C: Development of a Minimum Data Set-based depression rating scale for use in nursing homes. Age Ageing. 2000, 29: 165-172. 10.1093/ageing/29.2.165.View ArticlePubMedGoogle Scholar
  23. Morris JN, Fries BE, Morris SA: Scaling ADLs with the MDS. J Gerontol A Biol Sci Med Sci. 1999, 54 (11): M546-M553. 10.1093/gerona/54.11.M546.View ArticlePubMedGoogle Scholar
  24. Fleiss JL, Cohen J, Everitt BS: Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969, 72 (5): 323-327.View ArticleGoogle Scholar
  25. Fleiss JL, Levin B, Paik MC: Statistical Methods for Rates and Proportions. 2003, New York: John Wiley & Sons, 3View ArticleGoogle Scholar
  26. Delong ER, Delong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988, 44 (3): 837-845. 10.2307/2531595.View ArticlePubMedGoogle Scholar
  27. Akobeng AK: Understanding diagnostic tests 3: receiver operating characteristic curves. Acta Paediatr. 2007, 96: 644-647. 10.1111/j.1651-2227.2006.00178.x.View ArticlePubMedGoogle Scholar
  28. Giuliani CA, Gruber-Baldini AL, Park NS, Schrodt LA, Rokoske F, Sloane PD, Zimmerman S: Physical performance characteristics of assisted living residents and risk for adverse health outcomes. Gerontologist. 2008, 48: 203-212. 10.1093/geront/48.2.203.View ArticlePubMedPubMed CentralGoogle Scholar
  29. McGinn T, Wyler PC, Newman TB, Keitz S, Leipzig R, Guyatt G, for the Evidence-Based Medicine Teaching Tips Working Group: Tips for learners of evidence-based medicine: 3. measures of observer variability (kappa statistic). CMAJ. 2004, 171 (11): 1369-1373.View ArticlePubMedPubMed CentralGoogle Scholar
  30. Rockwood K, Mitnitski A, Song X, Steen B, Skoog I: Long-term risks of death and institutionalization of elderly people in relation to deficit accumulation at age 70. J Am Geriatr Soc. 2006, 54: 975-979. 10.1111/j.1532-5415.2006.00738.x.View ArticlePubMedGoogle Scholar
  31. Buntinx F, Niclaes L, Suetens C, Jans B, Mertens R, Van den Akker M: Evaluation of Charlson’s comorbidity index in elderly living in nursing homes. J Clin Epidemiol. 2002, 55 (11): 1144-1147. 10.1016/S0895-4356(02)00485-7.View ArticlePubMedGoogle Scholar
  32. Dobbs D, Meng H, Hyer K, Volicer L: The influence of hospice use on nursing home and hospital use in assisted living among dual-eligible enrollees. J Am Med Dir Assoc. 2012, 13 (2): 189.e9-189.e13. 10.1016/j.jamda.2011.06.001.View ArticleGoogle Scholar
  33. Epstein AM, Jha AK, Orav EJ: The relationship between hospital admission rates and rehospitalizations. N Engl J Med. 2011, 365: 2287-2295. 10.1056/NEJMsa1101942.View ArticlePubMedGoogle Scholar
  34. Taylor HL, Jacobs DR, Shucker B, Knudsen J, Leon AS, Debacker G: A questionnaire for the assessment of leisure time physical activities. J Chronic Dis. 1978, 31: 741-755. 10.1016/0021-9681(78)90058-9.View ArticlePubMedGoogle Scholar
  35. Orme JG, Reis J, Herz EJ: Factorial and discriminate validity of the Center for Epidemiological Studies Depression (CES-D) scale. J Clin Psychol. 1986, 42 (1): 28-33. 10.1002/1097-4679(198601)42:1<28::AID-JCLP2270420104>3.0.CO;2-T.View ArticlePubMedGoogle Scholar
  36. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:


© Hogan et al.; licensee BioMed Central Ltd. 2012

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.