Skip to main content

Table 1 Multi-rater reliability for reviewers rating of 30 studies of diagnostic reliability using QAREL

From: The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)

Item Item description (abbreviated) Subsequent evaluation
   κ 95% CI
1 Was the sample of subjects representative? 0.27 (0.11, 0.42)
2 Was the sample of raters representative? 0.59 (0.43, 0.74)
3 Were raters blinded to the findings of other raters? 0.92 (0.76, 1.00)
4 Were raters blinded to their own prior findings? 0.78 (0.62, 0.94)
5 Were raters blinded to the accepted reference standard? 0.66 (0.49, 0.82)
6 Were raters blinded to clinical information not part of test 0.51 (0.37, 0.64)
7 Were raters blinded to additional non-clinical cues? 0.59 (0.39, 0.78)
8 Was the order of examination varied? 0.71 (0.58, 0.84)
9 Was the time interval between repeated measures appropriate? 0.69 (0.50, 0.88)
10 Was the test applied correctly and interpreted appropriately? 0.35 (0.18, 0.51)
11 Were appropriate statistical measures of agreement used? 0.73 (0.54, 0.92)
  1. κ = multi-rater kappa. 95% CI = 95% confidence interval.