Development of toileting behaviour evaluation for Japanese older patients using wheelchairs in a hospital setting: a validation study

Abstract

Background

Ageing limits the activities of daily living (ADLs). Among ADLs, a lack of toileting independence causes a decline in the quality of life, mental health, and social participation. Therefore, occupational therapists spend considerable time assessing toileting disability based on various assessment methods for toileting behaviour. However, these assessment methods have issues with the grading levels, number of items, and diseases covered, and they fail to evaluate toileting behaviour accurately and sensitively. Hence, this study developed a Toileting Behaviour Evaluation (TBE) on a 6-point ordinal scale for patients using wheelchairs, with 22 activity components for various diseases.

Methods

This study examined the reliability and validity of the TBE in acute and subacute hospitals in Japan. To this end, two occupational therapists assessed 50 patients for inter-rater reliability at different times and one assessed them twice within 7–10 days for intra-rater reliability using the TBE. Furthermore, occupational therapists assessed 100 patients for internal consistency using the TBE and for concurrent validity using the TBE and Functional Independence Measure (FIM). The patients had been diagnosed with various diseases. This study used the weighted kappa coefficient for statistical analysis of the inter-rater and intra-rater reliability, Cronbach’s alpha coefficient for internal consistency, and Spearman’s rank correlation coefficient for concurrent validity. We performed all statistical analyses using the IBM SPSS Statistics ver. 25 for Windows. All P-values < 0.05 were considered statistically significant.

Results

The minimum weighted kappa coefficients for the inter-rater and intra-rater reliability for each item were 0.67 and 0.79, respectively. Cronbach’s alpha was 0.98 for the 22 items. The Spearman’s rank correlation coefficient between the mean scores on the TBE and FIM for toilet-related items was 0.74 (P < .01).

Conclusions

The TBE demonstrated good reliability and validity. This means that therapists can use it to identify impaired toileting behaviour. However, the relationship between impairments and each item of toileting behaviour should be explored in future studies. Additionally, studies should examine the creation of a specific index of functions of independence in each toileting behaviour.

Background

Globally, the number and proportion of people aged 60 years and older have increased rapidly. Indeed, this number is expected to increase from 1 billion in 2019 to 1.4 billion and 2.1 billion by 2030 and 2050, respectively [1].

In 2020, 28.6 per cent of the population in Japan was aged 65 years old and over, indicating that the ageing society in Japan is progressing quite rapidly compared to the US and the European countries [2].

Ageing limits the activities of daily living (ADLs) [3]. Specifically, it decreases physical function and increases morbidity [4], thereby impairing ADLs [5]. In Japan, health insurance plans that cover acute and recovery phase rehabilitation services were started in 2000. These services provide intensive rehabilitation to patients to help them regain the ability to perform ADLs [6, 7].

Among ADLs, toileting independence in particular impacts the length of stay in hospitals and the time of discharge [8]. For instance, previous studies have documented that independent toilet activity in hospitalized patients was highlighted as a key factor in determining the time of discharge [9,10,11]. A lack of toileting independence, notably, causes a decline in the quality of life [12], mental health, and social participation [13]. Studies have also shown that toileting dependence results in economic, mental, and physical burdens for not only the patients but also their families and society [14, 15]. Therefore, occupational therapists providing ADL support must assess and assist patients with toileting activities in acute and recovery phase rehabilitation services (subacute hospitals). The Functional Independence Measure (FIM) and the Barthel Index (BI) are international assessment methods that are widely used by occupational therapists to evaluate toileting behaviour. However, these instruments do not provide a breakdown of the individual component activities that comprise toileting behaviour and indicate poor responsiveness [16]. These component activities include putting on and taking off pants, wiping the buttocks, and tearing off toilet paper [17, 18]. An assessment of the individual components of each daily activity has been shown to be effective for determining rehabilitation goals and treatment planning [17]. Therefore, an evaluation of the component activities of toileting behaviour is necessary.

The following three methods are used to evaluate these component activities: the Toileting Performance Assessment Test (TPAT) [16], Toileting Tasks Assessment Form (TTAF) [17], and Toileting Assessment Test (TAT) [18]. The TPAT classifies the toileting activities of patients using wheelchairs into 10 component activities, such as transferring from bed to wheelchair, standing up, and shifting weight to one side while maintaining a standing position. It has three grading levels— (0) not possible, (1) possible with partial assistance, and (2) possible, and its reliability, validity, and responsiveness have been verified [16]. The TTAF classifies the toileting behaviour of patients using wheelchairs into 24 component activities, including opening and closing the toilet door, standing up from the wheelchair, and pulling the lower garments down. It has three grading levels—(A) independent, (B) requires supervision or verbal assistance, and (C) requires assistance, and research has verified its reliability and validity [17]. The TAT classifies the toileting behaviour of patients using wheelchairs into 10 component activities, such as standing up from the toilet, taking off pants, and wiping the buttocks. It has five grading levels: (0) no instruction, (1) verbal instruction, (2) modelling, (3) physical cueing, and (4) physical guidance. The validity of the TAT has been partially examined [18].

Although these evaluation methods can be used to achieve a sensitive and accurate evaluation of patients’ toileting behaviour, the limited number of grading levels in the TPAT and TTAF continue to pose a challenge for capturing small changes in patients. This highlights the need to incorporate more detailed grading levels to achieve a more precise evaluation. The TPAT and the TAT are also limited in that they classify toileting behaviour into only 10 component activities while omitting several major activities (“Open the door”, “Manoeuvre the wheelchair to the appropriate place for transfer to the toilet seat”,” Lock the wheelchair brakes”, “Place feet on the footrest”), so they do not cover the important components of toileting behaviour. Another common limitation of these three assessment methods is that they are designed to assess only stroke patients. These limitations highlight the need for an evaluation method that includes detailed grading levels, a larger number of component activities of toileting behaviour, and a wide range of diseases. Therefore, we developed the Toileting Behaviour Evaluation (TBE) using a 6-point ordinal scale comprising 22 component activities for patients using wheelchairs. This study examined the reliability and validity of the TBE for various diseases frequently managed by therapists.

Methods

Study sample and setting

To study the inter-rater and intra-rater reliability (test-retest reliability), we conducted the study at two hospitals in an urban area of Japan where we belong; these included acute and subacute hospitals that have an average number of beds and are staffed by rehabilitation specialists and occupational therapists. We enrolled participants who were admitted to these two hospitals between April 2020 and March 2021.

For internal consistency and concurrent validity as well, we conducted the study at three acute and subacute hospitals in an urban area of Japan where we belong. The selection criteria were the same as those of the reliability study. We enrolled participants who were admitted to these three hospitals between April 2020 and March 2021.

Participants were selected if they were aged 60 years or above and able to sit in a wheelchair. We excluded participants if they were physically unable to perform toileting activities (e.g. people with bladder catheters), could walk to the toilet, or could not give consent. For inter- and intra-rater reliability, we also excluded those whose FIM toilet-related behaviour scores changed during the study period. Within each hospital, therapists randomly selected subjects who met the above criteria. Following the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) Study Design checklist, we aimed to have a sample size of 50 for inter-rater and intra-rater reliability and 100 for internal consistency and concurrent validity [19]. The COSMIN Study Design checklist is recommended for use by researchers and clinicians or other professionals designing studies to evaluate the measurement properties of existing patient-reported outcome measures (PROMs). In this checklist, each standard is accompanied by a 4-point rating scale. This 4-point rating scale is added for illustrative purposes to better understand the consequences of choices made in a study design for the methodological quality of the study.

Assessment

Toileting behaviour evaluation (TBE)

Between April 2019 and January 2020, we conducted five meetings with six occupational therapists holding a master’s degree or higher, where we reviewed items from preceding studies and analysed the toileting behaviour of patients using wheelchairs. Initially, a scale consisting of 22 items on a 4-point scale (4: independence, 3: supervision, 2: verbal assistance, 1: physical assistance) was developed. It was then reviewed for clinical utility, and at the meeting with the six experts, the scale was revised to 22 items on a 6-point scale (6: independence, 5: modified independence, 4: supervision, 3: verbal assistance, 2: physical assistance, and 1: total assistance). This scale was again used in clinical practice with several patients to confirm its practicality, and all six experts agreed on this scale at a meeting. Finally, we developed the TBE with 22 items on a 6-point ordinal scale (Appendix 1). The evaluation procedure involved observing the sequence of the participants’ toileting behaviour. Following an earlier study [18], we scored the participants by the level of assistance they required for toileting to evaluate their maximum ability. In other words, we provided the scoring in the following order: ‘independence’, ‘modified independence’, ‘supervision’, ‘verbal assistance’, ‘physical assistance’, and ‘total assistance’. Physical assistance does not precede verbal assistance, except in emergency situations. Verbal assistance includes gestures (e.g. pointing gestures) and is different from supervision.

Appendix 1 Toileting Behaviour Evaluation (TBE)

Functional independence measure (FIM)

Developed by Granger in 1983, the FIM is an ADL assessment method measuring a person’s level of disability in terms of the burden of care. It covers two broad functional domains: motor and cognition. The FIM-Motor comprises 13 items based on 4 domains (self-care, sphincter control, transfers, locomotion). The FIM-Cognition comprises 5 items based on 2 domains (communication and social cognition). Each FIM item is scored on a 7-point ordinal scale ranging from total assistance (or complete dependence) to complete independence. There are two toilet-related items—toilet transfer (involving movement between the wheelchair and the toilet seat) and toileting (involving maintaining perineal hygiene and adjusting clothing before and after using a toilet).

Reliability

We adopted the following procedure to conduct the inter-rater reliability assessment. First, we randomly chose two occupational therapists on each occasion to assess the participants at different times using the TBE (the second occupational therapist assessed the participant within 10 days of the assessment by the first occupational therapist). Second, we examined the rate of agreement between the results of each assessment.

For intra-rater reliability, we chose one occupational therapist at random to assess the participants using the TBE twice within a period of 7 to 10 days. Subsequently, we examined the rate of agreement between the results of each assessment. To analyse the assessments, we calculated the weighted kappa coefficients for each of the 22 items for both inter-rater and intra-rater reliability.

Internal consistency

The participants were evaluated using the TBE, and the Cronbach’s alpha was calculated based on the item scores of all the participants.

Concurrent validity

We conducted two assessments to examine concurrent validity. Occupational therapists assessed the participants using the TBE and the FIM within 10 days of each other. For the analysis, we calculated the correlation coefficients between the mean scores of the TBE and the FIM toilet-related item scores (e.g. toilet movement and toilet transfer). We also calculated the correlation coefficients between the scores of each item of the TBE and the FIM toilet-related item scores.

Participants’ information

We collected participants’ demographic characteristics (e.g. age, gender), diagnosis, and FIM scores from their medical records.

Statistical analysis

We used the weighted kappa coefficient for each of the 22 items for statistical analysis of the inter-rater and intra-rater reliability, Cronbach’s alpha coefficient for internal consistency based on the item scores of all the participants, and Spearman’s rank correlation coefficient for concurrent validity. We performed all statistical analyses using the IBM SPSS Statistics ver. 25 for Windows. All P-values < 0.05 were considered statistically significant.

Results

Reliability

For inter-rater reliability, we sampled 50 (18 men, 32 women) patients with a mean age of 79.1 years (SD = 8.4). Table 1 shows the participant characteristics, including their FIM-Motor and FIM-Cognition information (Table 2). A total of 34 occupational therapists (range of experience: 1–17 years; mean = 5.7 years, SD = 4.6) participated in this study, and the mean difference in years of experience between two paired occupational therapists was 5.4 years (SD = 3.9). The mean weighted kappa coefficient for each item was 0.74 (range: 0.68–0.85; Table 1), and all the coefficients were significant.

Table 1 Weighted kappa coefficient of each item of the Toileting Behaviour Evaluation

For intra-rater reliability, we sampled 50 participants (20 men, 30 women) with a mean age of 78.2 years (SD = 8.4) (Table 2). A total of 22 occupational therapists (range of experience: 1–17 years; mean = 6.0 years, SD = 4.7) participated in this study. The mean weighted kappa coefficient for each item was 0.90 (range: 0.78–0.97) (Table 1), and all the coefficients were significant.

Table 2 Participants’ characteristics

Internal consistency

We sampled 100 participants (50 men, 50 women) with a mean age of 79.0 years (SD = 8.00). Table 1 shows the participant characteristics, including their FIM-Motor and FIM-Cognition information (Table 2). A total of 39 occupational therapists (range of experience: 1–19 years; mean = 6.2 years, SD = 4.9) participated in this study. Cronbach’s alpha was 0.98 for the 22 items.

Concurrent validity

The participants and the occupational therapists were the same as that for the internal consistency study (Table 2). The Spearman’s correlation coefficient between the mean scores on the TBE and the FIM toilet-related item scores was 0.74 (P < .01). All the correlations showed significance. The correlation coefficients between each item score and the FIM toilet-related item scores were 0.57–0.79 (Table 3). All correlations were significant.

Table 3 Correlation between each item of the Toileting Behaviour Evaluation and Functional Independence Measure ‘toileting’ scores

Discussion

In this study, the TBE was developed to assess the toileting ability of patients with various diseases. The target sample size was reached for all analyses, and the COSMIN checklist criteria [19] were met adequately or higher. We believe that the method and procedures were followed rigorously and were appropriate.

For reliability research, kappa coefficients were interpreted as follows: ≤ 0.20, 0.21–0.40, 0.41–0.60, 0.61–0.80, and ≥0.81 as slight, fair, moderate, substantial, and almost perfect agreement, respectively [20]. The minimum kappa coefficient for inter-rater reliability in this study was 0.68, indicating at least substantial inter-rater reliability for each TBE item. The inter-rater reliability of the mean score was also substantial at 0.74. This value suggests that similar results were obtained when two occupational therapists with different years of experience evaluated the same patients. The intra-rater reliability of the mean score was almost perfect at 0.90, and the minimum kappa coefficient was 0.78, indicating at least substantial intra-rater reliability for each TBE item. It means the TBE was a reproducible assessment. Cronbach’s alpha was 0.98, indicating almost perfect internal consistency. The TBE could measure the ability to perform toilet activities as a whole. The correlation coefficients between the mean score of the TBE and the FIM toilet-related item scores were more than 0.70, indicating a high correlation. It indicated that the TBE was a valid assessment method for toileting abilities; it had good inter- and intra-rater reliability, internal consistency, and concurrent validity.

Previous studies reported the minimum weighted kappa coefficient value for inter-rater reliability of the TPAT to be the same as that of the TTAF (0.61) [16, 17]. For intra-rater reliability, minimum weighted kappa coefficient values were reported at 0.83 [16] and 0.60 [17] for the TPAT and the TTAF, respectively, similar to those in this study. They also reported Cronbach’s alpha values of 0.97 for the TPAT [16] and 0.95 or 0.94 for the TTAF [17], which are also similar to the current results. The previous concurrent validity study reported a correlation coefficient value of 0.86 between the TPAT and the FIM total score [16] and correlation coefficients of 0.88–0.93 and 0.91–0.93 between the TTAF and the FIM toileting scores and toilet transfer scores, respectively [17]. These results are also similar to the current findings. Additionally, this study found a moderate and higher correlation between the scores of each TBE item and the FIM toilet-related item scores. We found that the correlation coefficients were below 0.60 for ‘open the door’, ‘close the door’, ‘turn on the light’, ‘manoeuvre the wheelchair to the appropriate place for transfer to the toilet seat’, ‘maintain a seating position on the toilet seat’, and ‘open the door and exit the toilet room’. These items might show low correlations because they were not assessed under the transfer and toileting items of the FIM. However, the items covered by the FIM showed correlations of 0.70 or higher. In other words, the validity of each item of the TBE was confirmed.

With 22 items on a 6-point ordinal scale, the TBE could be used to evaluate in detail the toileting behaviour of patients with various diseases, who are frequently evaluated by therapists in the hospital for toileting. Despite an increase in the total number of items as well as the values on the rating scale, inter- and intra-rater reliabilities were high. The inter-rater reliability study yielded particularly interesting results. The mean difference in the raters’ years of therapist experience was 5.4 years, showing that, despite the difference in the years of experience, therapists can provide similar evaluations of the participants’ toileting behaviour using the TBE. As mentioned in the method to assess participants’ maximum ability, the assistance provided was scored in the following order: independence, modified independence, supervision, verbal assistance, physical assistance, and total assistance. Physical assistance did not precede verbal assistance. The verbal assistance included gestural prompting (e.g. pointing gesture), which was distinguished from supervision. Since these criteria were clear and simple, it was easy for the therapists to conduct the evaluation.

Limitations and future studies

The use of the TBE could help therapists identify which part of the toileting sequence poses a challenge to the participants. However, the study’s target population comprised those admitted to acute or subacute hospitals. Therefore, further research is needed before it can be used for patients with other diseases admitted to nursing homes and other facilities. Investigating the causes hindering the independence of each item of toileting behaviour would be critical to provide more effective support. Hence, in future studies, we plan to examine the relationship between impairments and each item of toileting behaviour using the TBE.

Conclusion

This study confirmed the reliability and validity of the TBE, designed to help identify impaired toileting behaviour. Future studies should explore the relationship between functional disability and each item of toileting behaviour and discuss the creation of a specific index of functions for achieving independence in each toileting behaviour.