Kappa statistic to measure agreement beyond chance in free-response assessments

Table 1 Simulations of the coverage and mean width of 95% confidence intervals for the free-response kappa at selected sample sizes (20, 50, 100, 200) and values of kappa (0.3, 0.5, 0.7, 0.9), using three methods: delta method (Eq. 3), Agresti-Coull confidence limits, and Clopper-Pearson confidence limits

Simulation parameters		Mean observed K_FR	Degenerate sample^a(d = 0 or d = N)	Coverage of 95% confidence interval			Mean width of 95% confidence interval
N	K_FR	Mean observed K_FR	Degenerate sample^a(d = 0 or d = N)	Logit delta method (Equation 3)	Agresti-Coull method	Clopper-Pearson method	Logit delta method (equation 3)	Agresti-Coull method	Clopper-Pearson method
20	0.3	0.291	0.020	0.932	0.952	0.966	0.446	0.444	0.473
	0.5	0.491	<0.001	0.944	0.944	0.969	0.426	0.419	0.471
	0.7	0.693	0	0.957	0.957	0.976	0.354	0.345	0.392
	0.9	0.897	0.019	0.964	0.981	0.964	0.224	0.218	0.235
50	0.3	0.297	<0.001	0.962	0.962	0.962	0.293	0.294	0.314
	0.5	0.497	0	0.949	0.949	0.965	0.284	0.281	0.305
	0.7	0.697	0	0.953	0.936	0.968	0.230	0.227	0.246
	0.9	0.899	<0.001	0.958	0.958	0.974	0.134	0.134	0.142
100	0.3	0.298	0	0.954	0.954	0.954	0.211	0.212	0.223
	0.5	0.498	0	0.945	0.945	0.968	0.204	0.203	0.215
	0.7	0.698	0	0.946	0.946	0.966	0.164	0.163	0.172
	0.9	0.899	0	0.948	0.948	0.963	0.093	0.093	0.098
200	0.3	0.299	0	0.947	0.947	0.959	0.151	0.151	0.157
	0.5	0.499	0	0.948	0.948	0.957	0.146	0.145	0.151
	0.7	0.699	0	0.952	0.952	0.952	0.116	0.116	0.120
	0.9	0.900	0	0.957	0.957	0.957	0.065	0.065	0.068

Each simulation based on 50′000 replicates
^aLogit delta method not applicable. These simulations were treated as cases of non-coverage, and were not used for computation of the width of the confidence interval for this method

ISSN: 1471-2288