Table 7 Reported inter-rater reliability of GAS in included studies
First author | Year | Drug study | N | Methods and results | Quality |
|---|---|---|---|---|---|
Bovend’Eert | 2011 | No | 29 | Mixed model ICC(a, k) between therapist and masked assessor scoring procedures is 0.478 (low); LoA −1.52 +/− 24.54. | - ICC ≤0.7 |
Brown | 1998 | No | 24 | The Pearson’s r correlations and inter-rater ICCs (2,1) between the scores of the treating therapist and the independent raters were r = 0.84 (p <0.0001, n = 360, r2 = 70.90/0) and ICC = 1.00 (between raters: (IF = 1, SS = 0.01; within raters: df = 695, SS = 1, 172.65), respectively. The coefficients between scores of the 2 independent raters were r = 0.81 (p <0.0001, n = 135, rZ = 66.2 %) and ICC = 0.997 (between raters: dl = 1, SS = 1.48; within raters: f = 245, SS = 433,39). The results support acceptable inter-rater reliability of the scores for the goals in this study. | + ICC ≥0.7 |
De Beurs | 1993 | Yes | 40 | Agreement on the content of the chosen goals was measured between the intakers, in other words the people who performed the first session before the patients were randomized, and therapists was measured. Also, the agreement between the therapists and the people who objectively set the goals, or the goal setters, was measured. Agreement between goal setters and therapists and between goal setters and intakers was 43 and 57 % respectively. The calculations used to establish the agreement were not reported. | - Unclear design or method, agreement ≤0.7 |
Palisano | 1993 | No | 21 | Before data collection, an inter rater reliability was measured between the author and an examiner (Kappa = 0.89, agreement 90 %). During the study 16 goals were simultaneously scored. The agreement was 88 % (Kappa = 0.75). | + ICC ≥0.7 |
Rockwood | 1993 | No | 45 | A primary nurse and a multidisciplinary team scored GAS, ICC = 0.91. | + ICC ≥0.7 |
Rockwood | 1997 | No | 44 | ICC = 0.95 for admission scoring, ICC = 0.95 for discharge scoring, ICC = 0.93 for change score. | + ICC ≥0.7 |
Ruble | 2012 | No | 35 + 44 (reference to previous study) | Two raters independently coded 20 % of the GAS forms for the three features of agreement in sample 1 and 2. ICC for average agreement in sample 1 on measurability (0.96, 95 % CI [.87, .99]), difficulty (0.59, 95 % CI [−.18, .81]) and equidistance (0.96, 95 % CI [.74, .99); ICC for average agreement in sample 2 on measurability (1.0), difficulty (0.96, 95 % CI [.83, .99]) and equidistance (0.96, 95 % CI [.84, .99]). | + Only ICC for difficulty is lower than 0.7 |
Ruble | 2013-a | No | 49 | Two coders independently coded 39 % of the goals, ICC for social skills = 0.82, ICC for communication skills = 0.86, ICC for learning skills = 0.91. | + ICC ≥0.7 |
Ruble | 2013-b | No | Not stated (reference to previous study) | Excellent inter-rater reliability was achieved for both study 1 (ICC = 0.99) and study 2 (ICC = 0.90). | + ICC ≥0.7 |
Steenbeek | 2005 | Yes | 11 | A video scoring and scoring by a physiotherapist were compared, gaining a Kappa of 0.63. 5 out of 33 of the goal scores differed significantly (tested with a Wilcoxon signed rank test). | - k ≤0.7 |
Stolee | 1999 | No | 173 | ICC (N = 61) = 0.93 of GAS follow-up score. ICC (N = 61) = 0.89 of the separate goals, when checked whether the goals have been attained. | + ICC ≥0.7 |
Woodward | 1978 | No | 279 | Correlation of two goal attainment scores: 0.84. 33 % scored identical, 78 % within one level, 95 % within two levels. GAS scores did not differ significantly (F(6,268) = 1.25, P >0.10). | - Non-standard way of measuring inter-rater reliability |