Skip to main content

Advertisement

Table 1 Empirical coverage probability and bias in % of Krippendorff’s alpha and Fleiss’ K for simulated data with varying percentage of missing values

From: Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate?

  Missing values Krippendorff’s alpha Fleiss’ K
  Coverage probability (%) Bias (%) Coverage probability (%) Bias (%)
I 10 % 95.4 - 0.82 94.4 - 0.78
25 % 94.3 - 0.54 94.3 - 1.40
50 % 93.9 - 0.67 40.8 - 25.93
II 10 % 92.9 0.04 95.2 - 0.16
25 % 94.7 0.03 67.7 8.27
50 % 93.6 0.01 13.3 - 25.72
III 10 % 95.1 0.01 93.8 - 0.26
25 % 95.2 −0.02 65.5 - 7.76
50 % 94.8 −0.13 33.3 - 23.72
  1. The scenarios are defined as: I. N = 100, n = 5, k = 2, low agreement; II. N = 100, n = 5, k = 5, high agreement; III. N = 100, n = 10, k = 3, medium agreement (with N as number of observations, n as number of raters and k as number of categories)