From: Kappa statistic to measure agreement beyond chance in free-response assessments
Simulation parameters | Mean observed KFR | Degenerate samplea(d = 0 or d = N) | Coverage of 95% confidence interval | Mean width of 95% confidence interval | |||||
---|---|---|---|---|---|---|---|---|---|
N | KFR | Logit delta method (Equation 3) | Agresti-Coull method | Clopper-Pearson method | Logit delta method (equation 3) | Agresti-Coull method | Clopper-Pearson method | ||
20 | 0.3 | 0.291 | 0.020 | 0.932 | 0.952 | 0.966 | 0.446 | 0.444 | 0.473 |
0.5 | 0.491 | <0.001 | 0.944 | 0.944 | 0.969 | 0.426 | 0.419 | 0.471 | |
0.7 | 0.693 | 0 | 0.957 | 0.957 | 0.976 | 0.354 | 0.345 | 0.392 | |
0.9 | 0.897 | 0.019 | 0.964 | 0.981 | 0.964 | 0.224 | 0.218 | 0.235 | |
50 | 0.3 | 0.297 | <0.001 | 0.962 | 0.962 | 0.962 | 0.293 | 0.294 | 0.314 |
0.5 | 0.497 | 0 | 0.949 | 0.949 | 0.965 | 0.284 | 0.281 | 0.305 | |
0.7 | 0.697 | 0 | 0.953 | 0.936 | 0.968 | 0.230 | 0.227 | 0.246 | |
0.9 | 0.899 | <0.001 | 0.958 | 0.958 | 0.974 | 0.134 | 0.134 | 0.142 | |
100 | 0.3 | 0.298 | 0 | 0.954 | 0.954 | 0.954 | 0.211 | 0.212 | 0.223 |
0.5 | 0.498 | 0 | 0.945 | 0.945 | 0.968 | 0.204 | 0.203 | 0.215 | |
0.7 | 0.698 | 0 | 0.946 | 0.946 | 0.966 | 0.164 | 0.163 | 0.172 | |
0.9 | 0.899 | 0 | 0.948 | 0.948 | 0.963 | 0.093 | 0.093 | 0.098 | |
200 | 0.3 | 0.299 | 0 | 0.947 | 0.947 | 0.959 | 0.151 | 0.151 | 0.157 |
0.5 | 0.499 | 0 | 0.948 | 0.948 | 0.957 | 0.146 | 0.145 | 0.151 | |
0.7 | 0.699 | 0 | 0.952 | 0.952 | 0.952 | 0.116 | 0.116 | 0.120 | |
0.9 | 0.900 | 0 | 0.957 | 0.957 | 0.957 | 0.065 | 0.065 | 0.068 |