Skip to main content

Table 1 Simulations of the coverage and mean width of 95% confidence intervals for the free-response kappa at selected sample sizes (20, 50, 100, 200) and values of kappa (0.3, 0.5, 0.7, 0.9), using three methods: delta method (Eq. 3), Agresti-Coull confidence limits, and Clopper-Pearson confidence limits

From: Kappa statistic to measure agreement beyond chance in free-response assessments

Simulation parameters Mean observed KFR Degenerate samplea(d = 0 or d = N) Coverage of 95% confidence interval Mean width of 95% confidence interval
N KFR Logit delta method (Equation 3) Agresti-Coull method Clopper-Pearson method Logit delta method (equation 3) Agresti-Coull method Clopper-Pearson method
20 0.3 0.291 0.020 0.932 0.952 0.966 0.446 0.444 0.473
0.5 0.491 <0.001 0.944 0.944 0.969 0.426 0.419 0.471
0.7 0.693 0 0.957 0.957 0.976 0.354 0.345 0.392
0.9 0.897 0.019 0.964 0.981 0.964 0.224 0.218 0.235
50 0.3 0.297 <0.001 0.962 0.962 0.962 0.293 0.294 0.314
0.5 0.497 0 0.949 0.949 0.965 0.284 0.281 0.305
0.7 0.697 0 0.953 0.936 0.968 0.230 0.227 0.246
0.9 0.899 <0.001 0.958 0.958 0.974 0.134 0.134 0.142
100 0.3 0.298 0 0.954 0.954 0.954 0.211 0.212 0.223
0.5 0.498 0 0.945 0.945 0.968 0.204 0.203 0.215
0.7 0.698 0 0.946 0.946 0.966 0.164 0.163 0.172
0.9 0.899 0 0.948 0.948 0.963 0.093 0.093 0.098
200 0.3 0.299 0 0.947 0.947 0.959 0.151 0.151 0.157
0.5 0.499 0 0.948 0.948 0.957 0.146 0.145 0.151
0.7 0.699 0 0.952 0.952 0.952 0.116 0.116 0.120
0.9 0.900 0 0.957 0.957 0.957 0.065 0.065 0.068
  1. Each simulation based on 50′000 replicates
  2. aLogit delta method not applicable. These simulations were treated as cases of non-coverage, and were not used for computation of the width of the confidence interval for this method