Detection of grey zones in inter-rater agreement studies

Table 9 Sample size, \(\rho\), true \(\kappa\), TP, TN, FP, and FN, sensitivity, specificity, and MCC when the grey zone is in cell (1, 2) of \(3\times 3\) and \(4\times 4\) agreement tables

R	Case	n	\(\rho\)	True \(\kappa\)	TP	FP	FN	TN	Sens	Spec	MCC
3	GZ at cell (1,2)	50	0.960	0.639	9126	874	385	9615	0.913	0.962	0.875
			0.980	0.756	7352	2648	58	9942	0.735	0.994	0.755
			0.986	0.817	4017	5983	80	9920	0.402	0.992	0.488
		100	0.930	0.639	9906	94	168	9832	0.991	0.983	0.974
			0.965	0.744	8432	1568	98	9902	0.843	0.990	0.843
			0.985	0.835	9338	662	749	9251	0.934	0.925	0.859
		250	0.925	0.633	10000	0	777	9223	1.000	0.922	0.925
			0.963	0.753	9973	27	2141	7859	0.997	0.786	0.801
			0.977	0.838	10000	0	3288	6712	1.000	0.671	0.711
		500	0.910	0.630	10000	0	846	9154	1.000	0.915	0.919
			0.960	0.754	10000	0	733	9267	1.000	0.927	0.929
			0.984	0.838	10000	0	2225	7775	1.000	0.778	0.797
		1000	0.900	0.632	10000	0	1424	8576	1.000	0.858	0.866
			0.960	0.767	10000	0	1430	8570	1.000	0.857	0.866
			0.980	0.832	10000	0	212	9788	1.000	0.979	0.979
4	at cell (1,2)	50	0.911	0.624	3958	6042	72	9928	0.396	0.993	0.484
			0.969	0.731	5224	4776	12	9988	0.522	0.999	0.593
			0.982	0.839	4839	5161	13	9987	0.484	0.999	0.563
		100	0.935	0.612	9694	306	993	9007	0.969	0.901	0.872
			0.982	0.746	9804	196	727	9273	0.980	0.927	0.909
			0.992	0.840	9959	41	229	9771	0.996	0.977	0.973
		250	0.945	0.616	10000	0	933	9067	1.000	0.907	0.911
			0.975	0.755	10000	0	623	9377	1.000	0.938	0.940
			0.987	0.824	10000	0	1065	8935	1.000	0.894	0.899
		500	0.945	0.613	10000	0	1381	8619	1.000	0.862	0.870
			0.977	0.747	10000	0	562	9438	1.000	0.944	0.945
			0.987	0.824	10000	0	522	9478	1.000	0.948	0.949
		1000	0.940	0.617	10000	0	757	9243	1.000	0.924	0.927
			0.975	0.740	10000	0	1450	8550	1.000	0.855	0.864
			0.985	0.828	10000	0	563	9437	1.000	0.944	0.945

TP: GZ+ TGZ+; FP: GZ+ TGZ-; NF: GZ- TGZ+; TN: GZ- TGZ-; TGZ+: There is a grey zone in the table; TGZ-: There is no grey zone in the table; GZ+: A grey zone is identified; in the table; GZ-: No grey zone is identified in the table; Sens: Sensitivity; Spec: Specificity; MCC: Mathew’s correlation coefficient

ISSN: 1471-2288