This article has Open Peer Review reports available.
Agreement between administrative data and the Resident Assessment Instrument Minimum Dataset (RAI-MDS) for medication use in long-term care facilities: a population-based study
© Lix et al.; licensee BioMed Central. 2015
Received: 9 October 2014
Accepted: 25 February 2015
Published: 11 March 2015
Prescription medication use, which is common among long-term care facility (LTCF) residents, is routinely used to describe quality of care and predict health outcomes. Data sources that capture medication information, which include surveys, medical charts, administrative health databases, and clinical assessment records, may not collect concordant information, which can result in comparable prevalence and effect size estimates. The purpose of this research was to estimate agreement between two population-based electronic data sources for measuring use of several medication classes among LTCF residents: outpatient prescription drug administrative data and the Resident Assessment Instrument Minimum Data Set (RAI-MDS) Version 2.0.
Prescription drug and RAI-MDS data from the province of Saskatchewan, Canada (population 1.1 million) were linked for 2010/11 in this cross-sectional study. Agreement for anti-psychotic, anti-depressant, and anti-anxiety/hypnotic medication classes was examined using prevalence estimates, Cohen’s κ, and positive and negative agreement. Mixed-effects logistic regression models tested resident and facility characteristics associated with disagreement.
The cohort was comprised of 8,866 LTCF residents. In the RAI-MDS data, prevalence of anti-psychotics was 35.7%, while for anti-depressants it was 37.9% and for hypnotics it was 27.1%. Prevalence was similar in prescription drug data for anti-psychotics and anti-depressants, but lower for hypnotics (18.0%). Cohen’s κ ranged from 0.39 to 0.85 and was highest for the first two medication classes. Diagnosis of a mood disorder and facility affiliation was associated with disagreement for hypnotics.
Agreement between prescription drug administrative data and RAI-MDS assessment data was influenced by the type of medication class, as well as selected patient and facility characteristics. Researchers should carefully consider the purpose of their study, whether it is to capture medication that are dispensed or medications that are currently used by residents, when selecting a data source for research on LTCF populations.
Accurate measures of prescription medication use are important for understanding the health and healthcare use of long-term care facility (LTCF) residents. A large percentage of LTCF residents have complex healthcare needs for which one or more medications may be prescribed  and there is potential for inappropriate prescribing in this vulnerable population . Measures of medication use in previous studies about the quality of care and health outcomes of LTCF residents include anti-psychotic use in the absence of a psychotic disorder or related condition, anti-depressant therapy in residents without depression, and repeated use of hypnotic medications .
Data sources that capture medication information for LTCF residents include surveys, medical charts, administrative health databases, and patient clinical assessment records. Resident surveys are sensitive to self-report or recall biases, while medical chart reviews are time- and labour-intensive to conduct, particularly for multiple facilities. Population-based sources, including administrative health databases and assessment data like the Resident Assessment Instrument Minimum Data Set (RAI-MDS) [4,5], are ideal for making cross-facility or inter-jurisdictional comparisons . However, these data sources were not originally intended for developing facility-specific or jurisdiction-wide quality of care indicators or for studying health outcomes. Administrative data were developed for health system management and monitoring, while the RAI-MDS clinical assessment data were developed to improve resident care. Administrative data do not contain the rich clinical and functional information that is available in clinical assessments, but assessment data contains less information about disease diagnoses, which are essential to characterize patient risk, than administrative data. Thus, there are advantages and disadvantages to both data sources. Accordingly, studies about the comparability of the data in administrative and clinical assessment data, including data on medication use, are important to aid decisions about which source to use in studies about LTCF residents.
Given this context, the purpose of this study was to estimate agreement between medication information recorded in population-based RAI-MDS and prescription drug administrative data. This agreement analysis was conducted for three medication classes that are frequently prescribed for LTCF residents: anti-psychotic, anti-depressant, and anti-anxiety/hypnotic medications.
Data for this study were from the province of Saskatchewan, Canada, which has a population of approximately 1.1 million according to the 2011 Statistics Canada Census, and which has a universal healthcare program and therefore captures virtually all healthcare contacts for the entire population. The province maintains comprehensive health care databases in electronic format and these can be anonymously linked via a unique personal health number .
RAI-MDS Version 2.0, prescription drug, and person registry system records from the 2010/11 fiscal year (a fiscal year extends from April 1 to March 31), the most current year available at the time of the study, were used to conduct the research. The RAI-MDS, originally developed in the US by the Centers for Medicare and Medicaid Services, captures information about care and functioning of LTCF residents. This information is collected by trained assessors, usually nurses, who use interviews with the person and family members, consultation with other clinicians, and chart review, to complete a form. Forms are required to be completed within 14 days of LTCF admission, quarterly and annually thereafter, and whenever there is a major change in a resident’s health status. The RAI-MDS also captures dates of LTCF admission and discharge and some characteristics of the LTCF itself. Saskatchewan was the first Canadian province to make the RAI-MDS mandatory in all LTCFs; this requirement was introduced in April 2001, although full implementation was not achieved until 2004.
The prescription drug database contains records of outpatient drugs dispensed to provincial residents eligible for insurance coverage. It does not capture inpatient medications, medications for residents of a small number of LTCFs with in-house pharmacies, and approximately 10% of the population who are covered by a federally funded pharmacare program (e.g., military, federal police, federal prisoners, First Nations residents). Each available record includes the date of dispensation and national drug identification number (DIN). DINs are linked to codes in the American Hospital Formulary System (AHFS) Pharmacologic-Therapeutic Classification System (www.ashp.org), which is used to group drugs with similar pharmacologic, therapeutic, and/or chemical characteristics.
The person registry system captures dates of health insurance coverage, demographic information, and location of residence. The accuracy and completeness of Saskatchewan’s administrative health data for research has been well documented [8-12] although not specifically for LTCF populations. Ethics approval for database access was received from the University of Saskatchewan Biomedical Research Ethics Board. Data were accessed and analyzed at the provincial Health Quality Council in accordance with a standing data sharing agreement between that organization and the Ministry of Health.
The study cohort inclusion criteria were: (a) resident in a LTCF for at least 60 days, (b) at least one admission, quarterly, or annual RAI-MDS assessment during the residency period, and (c) at least one record for a prescription drug during the residency period. The latter criterion was used to ensure that the cohort included individuals eligible to receive prescription drug benefits.
The assessment date of the first admission, quarterly, or annual RAI-MDS assessment in 2010/11 was the study index date. The observation period extended 30 days before and 30 days after the index date. LTCF residents who did not have continuous health insurance coverage and were not eligible for prescription drug benefits during the study observation period were excluded.
Medication information was extracted from both the RAI-MDS and prescription drug databases. Socio-demographic variables were defined using the person registry data. Additional information on resident and facility characteristics were obtained from the RAI-MDS data.
The RAI-MDS captures the number of days of medication use in the seven-day period prior to the assessment reference date. The following medication classes are included in Section O of the assessment form: anti-psychotic, anti-depressant, anti-anxiety, hypnotic, and diuretic. Based on previous research, anti-anxiety and hypnotic medication classes were combined into a single category because they often have a similar indication and diuretics were excluded because they tend to have lower prevalence . In the prescription drug administrative data, anti-psychotic, anti-depressant, and anti-anxiety/hypnotic medications were identified using generic drug names (see Additional file 1) and AHFS codes from the provincial formulary (i.e., drugs covered by prescription drug benefits). The prescription drug database was searched for dispensations of the three medications in the study observation window. We selected a 60-day observation window because the medications under investigation are commonly dispensed in one-month quantities . A sensitivity analysis was conducted using a 28-day observation window (i.e., 14 days before and 14 days after the index date).
LTCF residents were described on the socio-demographic characteristics of age, sex, region of residence, and income quintile. All measures were defined at the index date. Urban or rural region of residence was assigned based on the postal code contained in the person registry. Urban residents were those living in one of the two health regions in the province that contain major urban centres, while rural residents were those individuals living in the remaining 11 health regions. Income quintile was an area-level measure assigned based on average household income from the Statistics Canada Census. Each individual’s postal code from the person registry was assigned to a dissemination area (DA), the smallest geographic unit for which Census data are reported. The entire Saskatchewan population was then divided into five approximately equal groups according to the DA average household income . Preliminary analysis of the data showed that for three quarters (i.e., 74.0%) of cohort members, the index date corresponded to the initial assessment; accordingly, the resident’s postal code would correspond to the place of resident prior to admission.
Selected chronic diseases identified in the RAI-MDS were also captured for each member of the study cohort, including Alzheimer’s disease, dementia, mood disorders (i.e., anxiety, depression, bipolar), and schizophrenia, which are potential indications for the medications under investigations. These conditions were defined at the study index date using admission and annual assessments, or the most recent admission or annual assessment if the index date was the date of a quarterly assessment because not all diagnoses (e.g., schizophrenia) were captured in quarterly assessments.
Facility characteristics captured in the RAI-MDS data in Saskatchewan include the type of facility and its affiliation. These characteristics were defined at the study index date. LTCFs were classified as special care home or other; the former are public facilities for which residence is determined based on need, while the latter are typically private facilities. LTCF affiliation includes amalgamate, affiliate, and contract. Health regions may operate facilities on their own (amalgamate), or the facility may be operated by an independent health care organisation (affiliate), or through a contract for services with an independent organization (contract).
The study cohort was described using frequencies, percentages, means, and standard deviations (SDs). Crude prevalence estimates (percentages) were calculated for each type of prescription medication. Cohen’s κ was used to estimate agreement between the RAI-MDS and administrative data; 95% confidence intervals (CIs) were also computed. The interpretation of κ adopted in this study was : κ < 0.20 is poor agreement, 0.20 ≤ κ ≤ 0.39 is fair agreement, 0.40 ≤ κ ≤ 0.59 is moderate agreement, 0.60 ≤ κ ≤ 0.79 is good agreement, and κ ≥ 0.80 is very good agreement.
Mixed-effects multiple logistic regression models were fit separately to the data for each medication to test resident and facility variables associated with disagreement between the two data sources . All individuals who were identified as medication users in one data source but not in the other data source were included in the disagreement category, while individuals who were either identified as medication users or as medication non-users in both data sources were included in the agreement category. Two models were fit to the data: (a) null model, which contained a random facility intercept only, to account for clustering of patients within facilities, and (b) full model which contained a random intercept as well as resident and facility fixed-effect covariates. Facility type was excluded because it was collinear with affiliation. The intra-class correlation (ICC) was computed for the null and full models  using a latent variable method . The Akaike Information Criterion (AIC) was used to compare model fit between the null and full models . The c-statistic, which is equal to the area under the receiver operating characteristic (ROC) curve for dichotomous outcomes, was used to assess discriminative performance ; it was estimated for the full model when the clustering effect.
All regression coefficients were exponentiated and reported as odds ratios (ORs), along with 95% CIs. Analyses were conducted using SAS software, version 9.2 .
A total of 9,913 individuals were identified as residents in a Saskatchewan LTCF for at least 60 days in 2010/11. Of this number, 350 (3.5%) did not have an RAI-MDS admission, annual, or quarterly assessment, 477 (4.8%) residents did not meet health insurance coverage or prescription drug benefit requirements and 220 residents (2.2%) did not have at least one prescription drug record during the observation period, leaving 8,866 LTCF residents in the study cohort.
Characteristics of long-term care facility residents in the study cohort (N = 8,866)
65 – 74 years
75 – 84 years
A total of 163 facilities were identified in the data; the majority were amalgamates and special care homes. The average number of study cohort members in each LTCF was 54.4 (SD = 53.5; median = 37.0).
Cohen’s κ and frequency (%) of agreement and disagreement of RAI-MDS and prescription drug data on medication class use
Anti-Anxiety or Hypnotic
Cohen’s κ (95% Confidence Interval)
0.85 (0.83, 0.86)
0.83 (0.82, 0.85)
0.40 (0.38, 0.42)
0.81 (0.77, 0.86)
0.84 (0.79, 0.88)
0.46 (0.39, 0.54)
0.85 (0.84, 0.86)
0.83 (0.82, 0.85)
0.39 (0.37, 0.41)
0.82 (0.80, 0.84)
0.79 (0.76, 0.81)
0.39 (0.35, 0.43)
0.86 (0.84, 0.87)
0.85 (0.84, 0.87)
0.40 (0.38, 0.43)
Frequency (%) of Agreement/Disagreement
Positive Agreement: In RAI-MDS and Prescription Drug Data
Negative Agreement: In neither RAI-MDS nor Prescription Drug Data
Disagreement: In RAI-MDS & Not in Prescription Drug Data
Disagreement: Not in RAI-MDS & in Prescription Drug Data
Odds ratios (ORs) and 95% confidence intervals (95% CIs) and model characteristics for disagreement between RAI-MDS and prescription drug data
OR (95% CI)
0.90 (0.72, 1.12)
75 – 84 years
0.97 (0.77, 1.21)
65 – 74 years
0.94 (0.72, 1.22)
0.94 (0.84, 1.06)
0.94 (0.84, 1.04)
Mood disorders: present
1.37 (1.23, 1.54)*
Mood disorders: absent
1.23 (0.91, 1.65)
Income Quintile: Q1 (lowest)
0.95 (0.79, 1.15)
Income Quintile: Q2
1.03 (0.85, 1.24)
Income Quintile: Q3
1.08 (0.90, 1.29)
Income Quintile: Q4
1.04 (0.86, 1.26)
Income Quintile: Q5 (highest)
0.93 (0.81, 1.07)
1.22 (0.92, 1.62)
1.33 (1.01, 1.77)*
Clustering Effect & Model Discrimination
Intra-class correlation (ICC)
The sensitivity analysis conducted using the 28-day observation window, produced similar results. Specifically, Cohen’s κ ranged from 0.34 for anti-anxiety/hypnotic medications to 0.78 for anti-psychotic medications.
Deterministic record linkage techniques were used to compare LTCF resident medication information captured in prescription drug administrative data and RAI-MDS data for the entire LTCF population of a Canadian province. Agreement ranged from moderate to very good for three medication classes captured in RAI-MDS data, and was lowest for anti-anxiety/hypnotic medications. For this latter medication class, a diagnosis of a mood disorder and the regional affiliation of the facility were associated with disagreement, although the discriminative performance of resident and facility characteristics was modest.
When compared to prescription drug administrative data, which captures information on medication dispensations, it appears that the RAI-MDS clinical assessment data, which captures information on utilization of medications, contain comparable information for anti-psychotic and anti-depressant medication classes . The small size of the ICC for all medications indicates that there was little variation across facilities in the lack of capture of medication class information in either data source. However, for anti-anxiety and hypnotic medication classes, it appears that there may be some differences across facility affiliations, with those being operated by health regions having slightly higher odds of disagreement compared with those contracted to provide services by the health region. This finding suggests that requirements around completion of the RAI-MDS assessment form may not be the same for different types of facilities. Higher disagreement for residents with mental health conditions may, in part, arise because it is more time consuming to complete the assessment form for residents with complex needs, leading to a higher potential for error or incomplete capture of medication class information.
The high rate of disagreement between the RAI-MDS and prescription drug data for anti-anxiety and hypnotic medication classes may have arisen for a number of reasons that are not captured in the study variables. Not all anti-anxiety or hypnotic medications may be covered by provincial health benefits, although this number will be small . The RAI-MDS assessment form does not contain generic names or DINs, therefore LTCF staff may not be able to correctly identify anti-anxiety or hypnotic medications. A patient prescription might not always be filled through a community pharmacy for anti-anxiety or hypnotic medication classes and therefore might not be captured in the prescription drug database; some medication stocks may be kept in-house in LTCFs if they are used on an infrequent or intermittent basis. As well, for medications used on an infrequent basis, the observation window may need to be substantially widened in order to achieve a high estimate of agreement if consumption does not occur shortly after dispensation. However, we found that moving from a 28-day to 60-day window had almost no impact on prevalence estimates for this type of medication.
There are a number of considerations when choosing between prescription drug administrative data and RAI-MDS data for measuring medication use in residents of LTCFs. Databases that contain prescription drug dispensation records do not capture individuals who are not covered by prescription drug benefit programs; this may represent substantial portions of the population in some jurisdictions. In Canada for example, prescription drug dispensations are not routinely captured in some provincial electronic database, or they are captured only for individuals 65+ years of age . At the same time, some jurisdictions may not mandate collection of RAI-MDS data for all LTCF residents, which will result in less than complete population coverage and affect accuracy and generalizability of prevalence estimates. While administrative data capture all drugs prescribed on an outpatient basis, inpatient medications are not typically captured; residents of LTCFs affiliated with acute care inpatient facilities may be missed. While only a small number of medication classes are captured in Section O of the RAI-MDS form, medications administered in the last seven days are captured; this will include medications prescribed while in hospital for individuals who are transferred from an acute care facility to a LTCF.
This study has some limitations. An external validation of each data source was not conducted, although a chart abstraction study or patient medication review would be difficult to undertake given the large number of facilities and patients that would have to be sampled to produce generalizable results; as well such a study would be very expensive to undertake. The discriminative performance of the model covariates was moderate for all of the medications, suggesting that other covariates should be considered as predictors of disagreement. As well, there may be unmeasured confounding in the model due to the limited set of covariates available for consideration. For example, facility characteristics such as number and type of staff might be useful to consider, but are not currently captured in electronic databases.
This study of agreement between medication classes captured in prescription drug administrative data and clinical assessment data from the RAI-MDS suggests that the medication information contained in both data sources is consistent for anti-psychotics and anti-depressants. For anti-anxiety and hypnotic medication ascertainment, low agreement between the two data sources was not associated with resident socio-demographic characteristics and was only weakly associated with resident disease diagnoses or facility type. While we hypothesize that agreement may be associated with the pattern or frequency of use in LTCF residents, further research is required to draw conclusions about these effects.
Research about the optimal source of data to measure medication use in LTCF populations is important for a number of reasons. Medication use is a risk factor for adverse health outcomes such as hip fracture. Accordingly, failure to include measures of medication use in prediction models represents a potential source of bias due to unmeasured confounding that could lead to erroneous conclusions . Accurate measurement of medication use is also critical to minimize variations in prescribing practices.
This study is based in part on de-identified data provided by the Saskatchewan Ministry of Health. The interpretation and conclusions contained herein do not necessarily represent those of the Government of Saskatchewan or the Ministry of Health. The research team is funded by the Ministry of Health. This research was also supported in part by career awards to the first author from the University of Saskatchewan Centennial Chair Program and the Manitoba Health Research Council Research Chair Program.
- Doshi JA, Shaffer T, Briesacher BA. National estimates of medication use in nursing homes: Findings from the 1997 Medicare Current Beneficiary Survey and the 1996 Medical Expenditure Survey. J Am Geriatr Soc. 1997;2005(53):438–43.Google Scholar
- Lustenberger I, Schupbach B, von Gunten A, Mosimann U. Psychotropic medication use in Swiss nursing homes. Swiss Med Wkly. 2011;141:w13254.PubMedGoogle Scholar
- Zimmerman DR. Improving nursing home quality of care through outcomes data: the MDS quality indicators. Int J Geriatr Psychiatry. 2003;18:250–7.View ArticlePubMedGoogle Scholar
- Mor V, Intrator O, Unruh MA, Cai S. Temporal and geographic variation in the validity and internal consistency of the Nursing Home Resident Assessment Minimum Data Set 20. BMC Health Serv Res. 2011;11:78.View ArticlePubMedPubMed CentralGoogle Scholar
- Metge C, Grymonpre R, Dahl M, Yogendran M. Pharmaceutical use among older adults: Using administrative data to examine medication-related issues. Can J Aging. 2012;24:81–95.View ArticleGoogle Scholar
- Richter T, Mann E, Meyer G, Haastert B, Kopke S. Prevalence of psychotropic medication use among German and Austrian nursing home residents: A comparison of 3 cohorts. J Am Med Dir Assoc. 2012;13:187.e7–187.e13.View ArticleGoogle Scholar
- Downey W, Beck C, McNutt M, Stang M, Osei W, Nichol J. Health databases in Saskatchewan. In: Strom BL, editor. In Pharmacoepidemiology. 3rd ed. New York: Wiley; 2000. p. 325–45.View ArticleGoogle Scholar
- Edouard L, Rawson NS. Reliability of the recording of hysterectomy in the Saskatchewan health care system. Br J Obstet Gynaecol. 1996;103:891–7.View ArticlePubMedGoogle Scholar
- Liu L, Reeder B, Shuaib A, Mazagri R. Validity of stroke diagnosis on hospital discharge records in Saskatchewan, Canada: implications for stroke surveillance. Cerebrovasc Dis. 1999;9:224–30.View ArticlePubMedGoogle Scholar
- Rawson NS, Malcolm E. Validity of the recording of ischaemic heart disease and chronic obstructive pulmonary disease in the Saskatchewan health care datafiles. Stat Med. 1995;14:2627–43.View ArticlePubMedGoogle Scholar
- Rawson NS, D’Arcy C. Assessing the validity of diagnostic information in administrative health care utilization data: experience in Saskatchewan. Pharmacoepidemiol Drug Saf. 1998;7:389–98.View ArticlePubMedGoogle Scholar
- Rawson NS, Malcolm E, D’Arcy C. Reliability of the recording of schizophrenia and depressive disorder in the Saskatchewan health care datafiles. Soc Psychiatry Psychiatr Epidemiol. 1997;32:191–9.View ArticlePubMedGoogle Scholar
- Hutchinson AM, Milke DL, Maisey S, Johnson C, Squires JE, Teare G, et al. The Resident Assessment Instrument-Minimum Data Set 20 quality indicators: a systematic review. BMC Health Serv Res. 2010;10:166.View ArticlePubMedPubMed CentralGoogle Scholar
- Ministry of Health. Government of Saskatchewan: Ministry of Health Formulary, 60th edition, April 2010 - March 2011. Regina, SK: Saskatchewan Ministry of Health; 2010.Google Scholar
- Roos NP, Mustard CA. Variation in health and health care use by socio-economic status in Winnipeg, Canada: the system works well? Yes and no. Milbank Q. 1997;75:89–111.View ArticlePubMedPubMed CentralGoogle Scholar
- Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall; 2001.Google Scholar
- Gibbons RD. Mixed-effects models for mental health services research. Health Serv Outcomes Res Meth. 2000;1:91–129.View ArticleGoogle Scholar
- Merlo J, Chaix B, Ohlsson H, Beckman A, Johnell K, Hjerpe P, et al. A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health. 2006;60:290–7.View ArticlePubMedPubMed CentralGoogle Scholar
- Harrell Jr FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.View ArticlePubMedGoogle Scholar
- Steyerberg EWEMJC, Harrell FE, Habbema JDF. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 2000;19:1059–79.View ArticlePubMedGoogle Scholar
- SAS Institute Inc. SAS/STAT 9.2 User’s Guide. Cary, NC: SAS Institute Inc; 2008.Google Scholar
- Canadian Institute for Health Information. Data quality documentation, continuing care reporting system, 2011–2012. Ottawa, ON: Canadian Institute for Health Information; 2012.Google Scholar
- Miller E, Blatman B, Einarson TR. A survey of population-based drug databases in Canada. Can Med Assoc J. 1996;154:1855–64.Google Scholar
- Kannegaard PN, van der Mark S, Eiken P, Abrahamsen B. Excess mortality in men compared with women following a hip fracture. National analysis of comedications, comorbidity and survival. Age Ageing. 2010;39:203–9.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.