Table 6 Effects of category, skew, and difficulty on observed chance agreement, reliability, and index estimations (average scores)
From: Interrater reliability estimators tested against true interrater reliabilities
| Â | Â | A. | B. | C. | D. | E. | F. | G. | H. | I. | J. | K. | L. | M. | N | O | P | Q | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Reliability Observation or Estimation | Chance Agreement Observation or Estimation | Â | |||||||||||||||||
1 | Author/ Source | Observed | %-Agreement | Bennett et al. | Perreault & Leigh | Gwet | Scott | Cohen | Krippen-dorff | Observed | %-Agreement | Bennett et al. | Perreault & Leigh | Gwet | Scott | Cohen | Krippen-dorff | Â | |
2 | Estimator: | ori | ao | S | Ir | AC1 | π | κ | α | oac | aoac | Sac | Irac | ACac | πac | κac | αac | Nc | |
3 | Ground 0 | .555 | .685 | .370 | .608 | .371 | .369 | 370 | .373 | .130 | 0 | .500 | .500 | .499 | .501 | .500 | .498 | 32 | |
4 | Category (C) | 2 | .537 | .701 | .402 | .584 | .470 | .230 | .232 | .234 | .164 | 0 | .500 | .500 | .401 | .598 | .597 | .596 | 96 |
5 | 4 | .550 | .678 | .571 | .747 | .621 | .226 | .230 | .230 | .128 | 0 | .250 | .250 | .142 | .573 | .571 | .571 | 96 | |
6 | 6 | .557 | .676 | .612 | .777 | .644 | .239 | .241 | .242 | .119 | 0 | .167 | .167 | .087 | .562 | .561 | .561 | 96 | |
7 | 8 | .578 | .686 | .641 | .796 | .664 | .254 | .257 | .257 | .108 | 0 | .125 | .125 | .062 | .564 | .563 | .562 | 96 | |
8 | Skew (sk) | .50 | .550 | .688 | .560 | .732 | .592 | .370 | .372 | .374 | .138 | 0 | .260 | .260 | .203 | .501 | .500 | .498 | 128 |
9 | .75 | .556 | .678 | .547 | .722 | .588 | .302 | .304 | .305 | .122 | 0 | .260 | .260 | .186 | .545 | .543 | .543 | 128 | |
10 | .99 | .560 | .690 | .561 | .723 | .619 | .040 | .044 | .045 | .130 | 0 | .260 | .260 | .132 | .678 | .676 | .676 | 128 | |
11 | Difficulty (df) | .000 | .824 | .844 | .782 | .884 | .810 | .482 | .484 | .485 | .020 | 0 | .260 | .260 | .152 | .630 | .629 | .628 | 48 |
12 | .143 | .783 | .805 | .728 | .852 | .761 | .404 | .406 | .407 | .021 | 0 | .260 | .260 | .158 | .616 | .615 | .615 | 48 | |
13 | .286 | .721 | .757 | .659 | .808 | .697 | .341 | .343 | .344 | .036 | 0 | .260 | .260 | .164 | .599 | .598 | .600 | 48 | |
14 | .429 | .659 | .721 | .600 | .765 | .643 | .273 | .275 | .277 | .062 | 0 | .260 | .260 | .169 | .591 | .589 | .588 | 48 | |
15 | .571 | .543 | .659 | .518 | .706 | .563 | .196 | .199 | .200 | .116 | 0 | .260 | .260 | .180 | .565 | .563 | .563 | 48 | |
16 | .714 | .439 | .606 | .444 | .647 | .495 | .117 | .121 | .121 | .168 | 0 | .260 | .260 | .182 | .548 | .546 | .546 | 48 | |
17 | .857 | .331 | .567 | .387 | .591 | .440 | .068 | .071 | .072 | .236 | 0 | .260 | .260 | .189 | .534 | .533 | .532 | 48 | |
18 | 1.00 | .142 | .523 | .332 | .552 | .389 | .018 | .022 | .022 | .380 | 0 | .260 | .260 | .194 | .514 | .512 | .511 | 48 | |
19 | Â | Mean | .555 | .685 | .556 | .726 | .600 | .237 | .240 | .241 | .130 | 0 | .260 | .260 | .173 | .575 | .573 | .572 | 384 |
20 | Â | Nd | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |