Skip to main content

Table 1 Weighted Kappa and percentage agreement scores for each session by round of pilot coding, set of transcripts and coding pair

From: Measuring fidelity of delivery of the Community Occupational Therapy in Dementia-UK intervention

Linear weighted Kappa (percentage agreement)

Set of transcripts

Coding pair

Introduction (1)

OPHI (2)

Ethnographic interview (3)

Summary and Goal-setting (4&5)

Consultation and advice (6)

Evaluation (7)

Initial piloting

 1

1a

0.79 (87)

0.66 (68.8)

0.48 (56.3)

0.5 (58)

6: 0.22 (35)

6 (2): 0.44 (65)

0.37 (44)

 1

1b

0.93 (87)

0.40 (56.3)

0.2 (43.8)

0.23 (41.7)

6: 0.11 (58.8)

6 (2): 0.01 (41)

0.11 (33.3)

 2

1b

0.65 (66.7)

0.27 (43.8)

0.57 (62.5)

0.28 (50)

6: 0.16 (47)

6 (2): 0.13 (35.3)

0.36 (55.6)

 1

2

0.40 (60)

0.50 (62.5)

0.49 (56.3)

0.15 (37.5)

6: 0.5 (64.7)

6 (2): 0.08 (47)

0.16 (33.3)

 2

2

0.50 (53.3)

0.68 (75)

0.81 (81.3)

0.42 (54)

6: 0.32 (41)

6 (2): 0.34 (47)

0.54 (66)

After initial piloting

 3

3

0.70 (80)

0.75 (81.3)

0.66 (75)

4: 0.78 (82.6)

5: 0.37 (60.9)

0.58 (82.4)

0.33 (55.6)

 4

3

0.71 (73.3)

0.87 (87.5)

1.00 (100)

0.56 (59.6)

0.3 (58.8)

0.71 (87.5)

 5

3

0.57 (66.7)

0.56 (68.8)

0.48 (62.5)

0.50 (69.6)

0.42 (56.3)

0.5 (50)

 5

4

0.56 (60)

0.69 (81)

0.67 (75)

0.43 (65)

0.61 (68.8)

0.67 (75)

 5

5

0.49 (66.7)

0.87 (87.5)

0.67 (75)

0.4 (56.5)

0.31 (62.5)

0.27 (37.5)

 6

4

0.67 (73.3)a

0.87 (87.5)a

0.81 (87.5)

0.37 (52)

0.54 (81.3)

1.00 (100)

 81

4

0.7 (80)a

0.94 (93.8)a

0.55 (68.8)

0.46 (60.9)

0.63 (75)

0.73 (75)

 12

4

0.66 (80)a

0.81 (87.5)a

0.87 (87.5)

0.77 (88.2)

0.93 (93.3)a

0.58 (75)

 1

4

–

–

0.57 (70.6)

0.59 (70.6)

0.66 (73.3)a

1.00 (100)

 2

4

–

–

0.81 (87.5)a

0.34 (58.8)

0.615 (80)a

0.39 (62.5)

 9

4

–

–

0.66 (75)a

0.64 (76.5)a

–

0.74 (75)a

 10

4

–

–

0.81 (81)a

0.76 (76.5)a

–

0.73 (75)a

 11

4

–

–

–

0.49 (70.6)

–

0.77 (87.5)a

 3

4

–

–

–

4: 0.94 (94.1)a

–

–

5: 0.82 (88.2)a

 4

4

–

–

–

0.44 (82.4)a2

–

–

  1. Note: Missing responses were accounted for in the analysis
  2. aIndicates agreement > 0.61 was reached | 1Coding guidelines not changed after coding this set | 2Weighted kappa did not reach > 0.61 however > 70% agreement achieved five times in a row and > 0.6 kappa 3 times in last five sets Kappa low due to distribution of responses: lots of ‘done’ responses, despite only three disagreements
  3. -: Agreement had already been reached and no further sessions needed to be coded until the 10% checks