An evaluation of the psychometric properties of the Indicator of Relative Need (IoRN) instrument

Abstract

Background

The Indicator of Relative Need (IoRN) instrument is designed for both health and social care services to measure function and dependency in older people. To date, the tool has not undergone assessment of validity. We report two studies aimed to evaluate psychometric properties of the IoRN.

Methods

The first study recruited patients receiving social care at discharge from hospital, those rehabilitating in intermediate care, and those in a rehabilitation at home service. Participants were assessed using the IoRN by a single researcher and by the clinical team at baseline and 8 weeks. Comparator instruments (Barthel ADL, Nottingham Extended ADL and Townsend Disability Scale) were also administered. Overall change in ability was assessed with a 7 point Likert scale at 8 weeks. The second study analysed linked routinely collected, health and social care data (including IoRN scores) to assess the relationship between IoRN category and death, hospitalisation and care home admission as a test of external validity.

Results

Ninety participants were included in the first study, mean age 77.9 (SD 12.0). Cronbach’s alpha for IoRN subscales was high (0.87 to 0.93); subscales showed moderate correlation with comparator tools (r = 0.43 to 0.63). Cohen’s weighted kappa showed moderate agreement between researcher and clinician IoRN category (0.49 to 0.53). Two-way intraclass correlation coefficients for IoRN subscales in participants reporting no change in ability were high (0.88 to 0.98) suggesting good stability; responsiveness coefficients in participants reporting overall change were equal to or better than comparator tools. 1712 patients were included in the second study, mean age 81.0 years (SD 7.7). Adjusted hazard ratios for death, care home admission and hospitalisation in the most dependent category compared to the least dependent IoRN category were 5.9 (95 % CI 2.0–17.0); 7.2 (95 % CI 4.4–12.0); 1.1 (95 % CI 0.5–2.6) respectively. The mean number of allocated hours of care 6 months after assessment was higher in the most dependent group compared to the least dependent group (5.6 vs 1.4 h, p = 0.005).

Conclusions

Findings from these analyses support the use of the IoRN across a range of clinical environments although some limitations are highlighted.

Background

The numbers of people with complex health conditions are increasing globally [1]. The United Kingdom (UK) is no exception to this growth, meaning that demands for health and social care systems are also rising [2]. In response, the UK Government has made health and social care integration a priority to improve outcomes for people who use their services and to maximize finite resources [3]. Joined-up services are required to co-ordinate person-centred and holistic care needs across more than one agency or discipline [4]. However, for health and social care integration to prove successful, shared knowledge, resources, and learning are required [5]. One way to help facilitate this is through shared assessment tools.

The Indicator of Relative Need (IoRN) was originally developed in Scotland by the Scottish Executive and the Information Services Division of the Scottish Government with the active involvement of social work teams and is already in widespread use by many health and social care teams. The tool was originally developed to classify older people into groups according to their level of dependency in order to inform decisions on the provision of care packages appropriate to their need. The IoRN questionnaire is administered by a social worker or other healthcare professional when conducting a full adult assessment. The tool is divided into four main domains to measure ability to mobilise, ability to attend to personal care, mental health, and bowel care management. Scores from IoRN are combined via an algorithm into a final category ranging from A1 to I, with A1 being the least dependent and I the most dependent category (http://www.jitscotland.org.uk/action-areas/data/iorn/) [6].

IoRN has since been further applied to support the Single Shared Assessment (SSA) process or its local equivalent, which addresses health and social care need of all adults, not just older people. In addition, the IoRN has recently been piloted for a range of additional purposes, as a tool to measure improvement after reablement or rehabilitation. A different version of the IoRN is available for use in care homes as an index of dependency and to support a staffing model [7].

A key obstacle to the use of the IoRN across health and social care services is the lack of detailed validation data. The psychometric properties of the IoRN tool have not been subjected to independent scrutiny to date. Such data are important to ensure that the tool is capable of measuring what it claims to measure, that the measurements are reliable, and that if improvements or deterioration in function are produced, the tool is capable of detecting these.

In order for agencies to be able to adopt the IoRN with confidence, the instrument requires rigorous examination to provide evidence of suitability whilst also making clear any limitations and boundaries to its appropriate use. This study therefore aimed to evaluate the psychometric properties of the IoRN: 1) inter-rater variability; 2) reliability; 3) responsiveness to change; 4) external (criterion) validity.

Methods

Two studies are reported in this paper. Study 1 was a prospective analyses involving a range of clients from different settings who provided sociodemographic details alongside information from the IoRN instrument and other comparable tools, [Barthel Activities of Daily Living (ADL) Index; Nottingham Extended ADL (NEADL); Townsend Disability Scale (TDS)] [8–10]. Study 1 addressed the first 3 aims; inter- rater variability, reliability and responsiveness to change in the IoRN. Study 2 was retrospective in nature and analysed linked routinely collected health and social care data including IoRN assessments. Study 2 focused on the fourth objective: external validity.

Study 1 (prospective)

Participants

Participants aged 18 and over from NHS Tayside were recruited to the study. Clients who were admitted to an intermediate care unit, care at home scheme, or discharged from hospital with a Dundee City Council care package were invited to participate. A single research nurse liaised with participating units and provided potential participants with information sheets and consent forms in advance for their consideration. Mutually convenient appointments were arranged by the research nurse with participants who expressed an interest in taking part. Participants unable to give written informed consent, those who previously participated in the study (i.e. during a previous rehabilitation admission), and those who were not expected to live to discharge from units were excluded from the study.

Data collection

A single research nurse gathered socio-demographic information from each participant once written informed consent was obtained i.e. age, gender, number of medications, medication dispensing aids, meals on wheels, living arrangements, district nurse use, informal care, package of care.

The following assessment tools were administered both at the beginning of the study and after an 8 week follow-up period.

• 1. IoRN scores were completed independently by the research nurse and the clinical team. Both measures were obtained by using information already recorded in medical, nursing and social work notes. A comparison of how the researcher and the clinical team interpreted the same data to score the IoRN was made in order to test inter-rater variability.

• 2. The 10 point Barthel Index [11] measured basic activities of daily living as a comparator to IoRN and was calculated by the research nurse based on information obtained from clinical notes.

• 3. Nottingham Extended ADL Scale (NEADL) [12, 13] and Townsend Disability Scale (TDS) [14, 15] were administered to each participant on a one-to-one basis by the research nurse as supplementary comparators to IoRN.

Overall change in function between baseline and follow-up was measured using a 7 point Likert scale ranging from 1 to 7, with higher scores indicating greater improvement, i.e. much worse; worse; a little worse; no change; a little better; better; much better. This was completed by the participant, and an additional Likert measure was completed by the care team. Data from all instruments were entered and analysed using SPSS v21. A 2 sided p value of <0.05 was classed as significant for all analyses.

Analyses

Cronbach’s alpha was calculated to measure internal consistency between IoRN subscales from the research nurse and care teams over the three different settings. Construct validity was calculated by performing Pearson’s correlation with IoRN subscores and other measures of function (Barthel, NEADL, Townsend and mental health scores) alongside intensity of care package. Cohen’s Kappa (weighted) examined associations between IoRN subscores calculated by the research nurse and the clinical team to test Inter-rater variability.

Cohen’s d (effect size) and Guyatt’s responsiveness coefficient were used to test responsiveness to change [16, 17]. For Cohen’s d, all the ‘worse’ categories on the Likert scale were aggregated due to small numbers. Similarly all of the improvement categories were aggregated into a single ‘better’ category. Cohen’s d was calculated as: (Mean difference between baseline and follow up) / (pooled SD of baseline and follow up. Where pooled SD was calculated as the square root of: ((baseline n-1 × SD baseline) + (followup n-1 × SD followup)) / (baseline n + followup n).

Guyatt’s responsiveness coefficient was calculated using the minimum clinically important difference (MCID), taken as those noting either slight improvement or slight worsening on the Likert scale. The coefficient is calculated as; mean change in score in group showing slight improvement or slight worsening on Likert / SD of change score in group showing ‘no change’ on Likert.

Study 2 (retrospective)

Participants

Routinely collected health and social care data from NHS Tayside and Dundee City Council were linked and analysed through the Health Informatics Centre (HIC) Safe Haven at the University of Dundee. Databases were probabilistically linked using name, date of birth and postcode alongside clerical review procedures to obtain >95 % accuracy of patient matching [18]. Clients who had been assessed with IoRN by Dundee Social Work Department between 2008 and 2012 were selected from the matched data. Details of the linked data sets used in this analysis has been published previously [19].

Data collection and analyses

Analyses examined external validity of the IoRN by testing associations between IoRN categories and outcomes that would be expected to be related to IoRN score, namely death, hospitalization and care home admission. Associations between IoRN scores and time to death, time to hospital admission and time to care home admission using Cox regression models were performed. Models were adjusted for age, gender, number and length of stays in hospital in the previous year; all factors known to be important predictors of death and rehospitalisation and used in other scores e.g. Scottish Patients at Risk of Readmission and Admission (SPARRA) [20]. IoRN subscores were entered into Cox regression analyses (forced entry) without other adjustment.

Care allocation data (number of hours of home care allocated per person per week) recorded by Dundee Social Work Department were also examined in association with IoRN scores. Student’s t-test was performed to compare the mean weekly hours of allocated care at 6 months after the IoRN assessment, excluding clients who had died or been admitted to a care home. All analyses were performed using SPSS version 21. A 2 sided p value of <0.05 was taken as significant for all analyses.

Results

Study 1 - prospective

Baseline study details

Ninety participants were recruited between October 2014 and September 2015. Table 1 shows baseline details. Group A comprised those admitted to rehabilitation or intermediate care; group B comprised those admitted to an intermediate care at home scheme, and Group C comprised those discharged from hospital with a package of care.

Table 1 Study 1 baseline details by group

Questionnaire completion rates

Completion rates for questionnaires varied considerably between baseline and follow-up for the research nurse and care teams (Fig. 1). The analysis revealed overall completion rates at baseline and follow-up respectively: Research Nurse IoRN (100 and 61 %); Barthel/TDS/NEADL (100 and 62 %), Care Teams IoRN (72 and 24 %). There was also a marked difference in completion rates between group care teams at baseline and follow-up respectively: Group A (66 and 2 %); Group B (81 and 67 %); and Group C (73 and 14 %). From 90 participants, 3 % died before follow up (n = 3) and 10 % declined follow-up (n = 9).

Fig 1

Flowchart showing completion rates for questionnaires

Internal consistency

Overall Cronbach’s alpha for ADL and personal care subscales from the research nurse and care teams indicated a high degree of internal consistency. IoRN ADL and personal care for the research nurse (0.87 and 0.92) respectively, IoRN ADL and personal care for care teams (0.93 and 0.88) respectively. Mental health scores were almost universally normal, therefore lacked sufficient variation to calculate Cronbach’s alpha. These results are consistent with Cronbach’s alpha from stage 2 (retrospective); IoRN ADL (0.80); IoRN personal care (0.91); IoRN mental health (0.71).

Construct validity

Table 2 shows how IoRN subscales relate to two measures of activities of daily living (Barthel and NEADL) and one measure of disability (TDS). The IoRN ADL and personal care scores show moderate correlations with these measures, as well as with the intensity of planned care package at the time of assessment, suggesting that these IoRN scales are measuring a related construct (convergent validity). The IoRN mental health score is only weakly correlated with the other scores – this is again to be expected as this is a very different construct, suggesting that this subscale has discriminant validity.

Table 2 Correlations between IoRN sub-scores and other measures (baseline IoRN data collected by research nurse)

Inter-rater variability

Cohen’s kappa (weighted) was used to quantify inter-rater variability in IoRN categorisation. When treating each subcategory within A and B as separate categories (i.e. A1, A2, A3 all separate), weighted kappa was 0.49. If subcategories were aggregated (i.e. treating A1, A2, A3 all as category A), kappa was 0.53. These values suggest moderate agreement between raters. Agreement between the researcher and the clinical team was much better for groups B and C than it was for group A. Excluding group A yielded somewhat better kappa values of 0.58 and 0.64.

Change in scores over time

To analyse whether changes in IoRN scores correlated with changes in overall function, we asked both the clinical team and the client how their overall function had changed between baseline and follow up, using a seven-point Likert scale. This dual approach was used as the perceptions of the client may differ from the perceptions of the care team. Changes in both IoRN subscales and changes in Barthel, NEADL and TDS scores are shown subdivided by Likert category from the care team and from the client (Table 3). Inter- rater agreement between the care team and client Likert categories (weighted kappa) was moderate at 0.64 (p < 0.001).

Table 3 Likert categories from care team (overall change)

These categorisations then formed the basis for assessing the reliability (stability) of each score in clients who selected ‘no change’ on the Likert scale, and assessing the responsiveness to change for those clients perceiving that they had improved or worsened.

Reliability

To estimate reliability (also known as stability), the change in scores between baseline and follow up for those reporting ‘no change’ on the overall Likert scales (i.e. those who remained stable in perceived function) were compared using intra-class correlation coefficients (ICC). Results are given in Table 4. The ICC values for both the ADL and personal care subscales were very high, suggesting a high degree of stability of these measures in stable clients. The mental health subscale lacked sufficient variability to allow testing.

Table 4 Intra-class correlation coefficients for those reporting ‘no change’ on Likert scale

Responsiveness

Two indices of responsiveness were calculated – Cohen’s d (effect size) and Guyatt’s responsiveness coefficient. Results are shown in Table 5.

Table 5 Cohen’s d and Guyatt’s responsiveness coefficient

These results suggest that the IoRN scores are no less responsive than the Barthel, NEADL or TDS scores, and indeed the IoRN ADL subscale showed considerably better responsiveness on both the Guyatt’s and Cohen’s measures. Almost all of the scores appeared more responsive to worsening than to improvement.

Study 2 - retrospective study

Baseline study details

A total of 1712 patients were included in the analyses. Mean age was 81.0 years (SD 7.7), and 1200 (70 %) were female. 142 (8 %) died during follow up. Cox proportional hazard regression analyses for time to death, care home admissions and next unscheduled hospital admission are shown in Table 6.

Table 6 Cox regression analysis for time to death, care home, hospital admission, and hours of care per week at 30 days post-assessment, adjusted for age, gender, number and length of stays in hospital in the previous year

The mean number of hours of care received per week was compared between IoRN categories. The only significant finding occurred between the least dependent category (A1) and the most dependent category I. A1 (mean = 1.4 h, 95 % CI 0.5–2.3 h) and I (mean = 5.6 h, 95 % CI 1.9–9.2 h) p <0.05 by ANOVA.

To test the contribution that each individual sub-score from the IoRN made to prediction of death hospitalisation or care home admission, all sub-scores were entered into Cox regression analyses (forced entry) without other adjustment. Results are shown in Table 7.

Table 7 Using sub-scores from IoRN to predict death, hospitalisation or care home admission

Discussion

The results show IoRN performed well on a range of psychometric measures. Internal consistency was good, subscales correlated with comparable instruments, and did not show any association with unrelated tools (e.g. mental health) demonstrating convergent and discriminant validity. IoRN demonstrated stability in clients whose overall function did not change. Responsiveness to change was not high, however it was no worse than the performance of other commonly used measures of disability and function in this cohort. Responsiveness to deterioration appeared to be stronger than for improvement. Inter-rater reliability was good for subscale scores, but only moderate for Individual IoRN category scores. Performance seemed to be highly dependent upon training and engagement of clinical teams. The importance of consistent training of teams cannot be emphasized enough as well as taking note that attempts to impose the IoRN on reluctant teams are likely to generate poor quality data.

External validity was demonstrated indicating IoRN as a very good predictor of future care home admission, a good predictor of death, but IoRN did not predict future hospital admission in this analysis. The lack of association between IoRN category and risk of future hospital admission is likely to reflect that there are multiple other factors that impact on the risk of hospital admission (e.g. undiagnosed conditions, frailty, medication adherence, body mass index), which the current analysis could not measure. The IoRN was not explicitly designed to predict hospitalisation and so it is perhaps unsurprising that it does not do so. It is also possible that those in higher IoRN categories receive more care which may have reduced the risk of future hospitalization, thus weakening any association. Similarly, incorporating previous hospitalisations into the predictive model is likely to have weakened any added predictive value that the IoRN might have for future hospitalisation.

The IoRN tool has not undergone published psychometric assessment before, and there is thus no existing literature with which to compare our results. There are many tools in use within healthcare that have been validated for assessment of impairment, disability, daily function and care needs; how then does the IoRN tool fit into this already crowded landscape?

Assessment tools can be divided into 1st, 2nd and 3rd generation instruments that measure function and dependency in older people. 1st generation tools (of which IoRN is one) focus within a single or limited set of domains, for example, Barthel activities of daily living, Nottingham Extended ADL Scale and Townsend disability scale [8, 12, 14]. 2nd generation tools gather information from multidimensional fields for instance, Easy Care health assessments [21], and FACE (functional analyses of care environments) [22]. 3rd generations extend 2nd generation models by applying supporting software to fully integrate collections of measurements to enable transfer of information across different health and social care settings, as exemplified by the Inter-RAI suite [23].

Implementing 2nd and 3rd generation instruments into current health and social care systems, at least in the UK, presents significant challenges. For example, a series of case studies from 8 countries illustrated the difficulties associated with employing Inter-RAI across different health and social systems and cultures who hold conflicting approaches to gathering client information [24]. This was particularly relevant to the UK where reports of ‘too lengthy’ and ‘too clinical’ were recorded [24]. Little flexibility was provided to incorporate free-text information to capture important nuanced and contextual information making results difficult to interpret [24]. Thus although the IoRN tool captures data around a more limited set of domains, it has been designed for use by social care practitioners, and sits within existing assessment processes. Social care practitioners may be more likely to accept the IoRN because they may be unfamiliar with many of the tools already used in healthcare assessment. Conversely, although healthcare practitioners already have many tools to choose from, the results of this assessment show that IoRN is a viable alternative tool for many assessment uses within healthcare - and support its use as a tool acceptable to both health and social care practitioners [25]. Whilst the comprehensive, consistent approach across multiple care settings of second and third generation tools are advantages, such benefits cannot be realised if practitioners are unwilling to use such tools; in these circumstances, a less-perfect first generation tool in widespread use is preferable to a more sophisticated tool that is not used. The widespread uptake of the IoRN within Scotland suggests that such first generation tools are perceived as having advantages by both health and social care practitioners.

Limitations to study

A number of limitations to the studies merit discussion. Completion rates for the IoRN by some clinical teams in the prospective study were not as high as anticipated, and in the case of one team, significant concerns existed about the quality and completeness of the IoRN tool data collection by the clinical team. This was despite standardized training given to all participating teams prior to the start of data collection. Although we cannot provide empirical data to give insight into why this might be, levels of engagement with the IoRN and with the project varied across teams, and was noticeably weaker in the group with poor results. Staff in this group were a mixture of public-sector healthcare staff and private-sector care home staff, with reactive, intermittent input from senior medical staff. This group was also characterised by short length of stay, rapid turnover of patients and high bed occupancy. It is therefore possible that the low quality of data collected in this group is due to a combination of a high-pressure care environment where the emphasis is on discharge, together with training, cultural and leadership factors affecting performance of the team. These findings serve as a reminder that simply providing training and resources to teams is not sufficient to ensure high-quality data collection, even with relatively simple tools.

A further issue limiting the power of the prospective study was the relatively low rate of successful follow- up. For some teams, this was due to short stays in the intermediate care unit, without a clear mechanism for follow-up in routine clinical care. This was a particular problem for patients enrolled at the point of hospital discharge with a package of care. For example, hospitals involved did not allocate a designated social worker to individuals when returning patients to the community. Tracing patients through social work was difficult and even when they were located, social workers, who incidentally were not affiliated to the study, and already under considerable work pressure, did not see an IoRN assessment as a priority.

Numbers with successful follow-up by the researcher were higher, but even here a combination of deaths, illness, client reluctance and an inability to contact clients at their usual address contributed to attrition of follow-up. Whilst this led to some loss of power for the prospective study, it did not prevent the project from being able to draw conclusions. Our inability to analyse Mental Health sub-scores was as a result of limited range in the results. Almost all participants scored the lowest possible mental health score. This is difficult to avoid in a prospective study requiring client capacity to consent. Clients who would score high on the mental health sub-scores would likely hold significant cognitive impairment which would prevent them from taking part in the study.

The retrospective study used routinely collected data from clients aged 65 years and over as this was standard practice of the time. Whereas the prospective study IoRN analysed data with clients aged 18 years and over. Mean ages between the two studies however scored similar (81 versus 79.9 years respectively) and were unlikely to affect findings. Finally, data on the relationship between IoRN category and care needs are limited by the fact that the social work dataset used for this analysis captured formal, but not informal care (i.e. care by relatives, friends). This is likely to have weakened any relationship between IoRN category and the amount of care provided.

Conclusions

Findings from these analyses support the use of the IoRN across a range of clinical environments. Some caveats around different uses of the IoRN are worthy of note. IoRN is less responsive to improvement but has a better responsiveness to deterioration and may be useful for detecting this. IoRN is suitable for predicting mortality, future care home admission, and future need for care home provision at service level. It is unlikely to be useful to predict risk of future hospitalization. Alternative tools (e.g. Scottish Patients at Risk of Readmission and Admission – SPARRA) exist to examine this risk.

Abbreviations

ADL, activities of daily living; HIC, Health Informatics Centre; IoRN, indicator of relative need; MCID, minimum clinically important difference; NEADL, Nottingham extended activities of daily living; SPARRA, Scottish patients at risk of readmission and admission; SSA, single shared assessment; TDS, Townsend Disability Scale; UK, United Kingdom; UN, United Nations; WHO, World Health Organisation.