Skip to main content

Table 10 Weighted inter-rater agreement measures for the motivating example

From: The impact of grey zones on the accuracy of agreement measures for ordinal tables

   Weight
Measure Table Unweighted Linear Quadratic Radical Ordinal
Scott’s π Original 0.461 0.564 0.671 0.510 0.629
  Scenario 1 0.258 0.367 0.503 0.308 0.447
  Scenario 2 0.504 0.582 0.671 0.541 0.635
  Scenario 3 0.469 0.552 0.648 0.508 0.610
  CV 0.263 0.194 0.130 0.229 0.154
Gwet’s AC2 Original 0.529 0.719 0.850 0.629 0.809
  Scenario 1 0.407 0.673 0.840 0.550 0.790
  Scenario 2 0.576 0.750 0.869 0.668 0.832
  Scenario 3 0.550 0.737 0.863 0.649 0.825
  CV 0.145 0.047 0.015 0.083 0.023
Krippendorff’s α Original 0.464 0.566 0.673 0.513 0.631
  Scenario 1 0.262 0.370 0.506 0.312 0.450
  Scenario 2 0.507 0.584 0.673 0.543 0.637
  Scenario 3 0.472 0.554 0.650 0.510 0.612
  CV 0.260 0.192 0.129 0.226 0.153
Brennan-Prediger’s S Original 0.514 0.680 0.812 0.599 0.768
  Scenario 1 0.376 0.600 0.774 0.491 0.716
  Scenario 2 0.560 0.711 0.830 0.637 0.790
  Scenario 3 0.532 0.694 0.821 0.615 0.779
  CV 0.165 0.073 0.030 0.111 0.043
  1. CV: coefficient of variation
  2. Coefficient of variation is calculated for each pair of agreement measure and weight