Kappa statistic to measure agreement beyond chance in freeresponse assessments
 Marc Carpentier^{1},
 Christophe Combescure^{1},
 Laura Merlini^{2} and
 Thomas V. Perneger^{1}Email author
DOI: 10.1186/s1287401703406
© The Author(s). 2017
Received: 21 October 2016
Accepted: 7 April 2017
Published: 19 April 2017
Abstract
Background
The usual kappa statistic requires that all observations be enumerated. However, in freeresponse assessments, only positive (or abnormal) findings are notified, but negative (or normal) findings are not. This situation occurs frequently in imaging or other diagnostic studies. We propose here a kappa statistic that is suitable for freeresponse assessments.
Method
We derived the equivalent of Cohen’s kappa statistic for two raters under the assumption that the number of possible findings for any given patient is very large, as well as a formula for sampling variance that is applicable to independent observations (for clustered observations, a bootstrap procedure is proposed). The proposed statistic was applied to a reallife dataset, and compared with the common practice of collapsing observations within a finite number of regions of interest.
Results
The freeresponse kappa is computed from the total numbers of discordant (b and c) and concordant positive (d) observations made in all patients, as 2d/(b + c + 2d). In 84 fullbody magnetic resonance imaging procedures in children that were evaluated by 2 independent raters, the freeresponse kappa statistic was 0.820. Aggregation of results within regions of interest resulted in overestimation of agreement beyond chance.
Conclusions
The freeresponse kappa provides an estimate of agreement beyond chance in situations where only positive findings are reported by raters.
Keywords
Reproducibility of results Reliability (Epidemiology) Methodological Study BiostatisticsBackground
Good agreement between raters is a desirable property of any diagnostic method. Agreement is usually assessed by the kappa statistic [1], which quantifies by how much the observed agreement between raters exceeds agreement due to chance alone. The assessment of the kappa statistic requires the numbers of evaluations, both positive (or abnormal) and negative (or normal), to be known for all raters. This is not the case when raters report only positive findings and do not notify the number of negative findings. This situation can be referred to as the freeresponse paradigm [2]. It is a common situation in imaging procedures, where raters typically report positive findings, but do not list all negative observations for a given patient.
To date, the methods used to estimate the agreement corrected for chance of freeresponse assessments have all required a simplification of the data, so as to make negative findings explicit. One possibility is to analyze the data at the level of a patient, by rating a patient “positive” if at least one lesion is detected, but this causes an important loss of information. Another approach is to split the radiograph into regions of interest. Each region of interest is then assessed by all raters. Since negative ratings are explicitly notified, the number of regions of interest rated as negative by all raters is known and the standard kappa statistic can be computed. This approach reduces the loss of information compared with a single dichotomous rating per patient, but the regions of interest must be small and numerous enough to preserve clinical relevance. For instance, in a diagnostic study, Mohamed et al [3] defined 68 regions of interest per patient. Generally, constraining a freeresponse paradigm to a finite number of ratings (patientlevel or regionlevel) causes a loss of information and may lead to overestimation of the agreement, because disagreements below the selected level of granularity are ignored.
The objective of the present paper is to propose a kappa statistic for freeresponse dichotomous ratings that does not require the definition of regions of interest or any other simplification of the observed data. This kappa statistic also takes into account withinpatient clustering [4–6] of multiple observations made for the same patient.
Methods
Derivation of the freeresponse kappa
When patients can contribute more than one observation, data are clustered. Yang et al [7] proposed a kappa statistic obtained from the usual formula (P_{o}P_{e})/(1P_{e}) where P_{o} is a weighted average of the proportions of agreement over clusters (patients) and P_{e} is obtained from weighted averages of marginal proportions of ratings of each rater. With this approach, the kappa for clustered data has the same estimate as when clustering is ignored. Therefore the basic 2 × 2 table is also appropriate for the estimation of agreement for clustered data.
For freeresponse assessments, each rater reports only positive findings and the number a is unknown. It would be wrong to replace a by 0, as if the raters had not agreed on any negative observation; both the observed agreement and kappa would be underestimated. It would also be incorrect to simply replace a by the number of patients without any positive finding, because several potential lesion sites exist in each patient. Typically, a can be assumed to be high in imaging examinations, because each output displays a large number of anatomical or functional structures or substructures, each potentially positive or negative. Therefore, the number of positive findings in a given patient is usually small in comparison with the potential number of abnormalities that might occur.
Properties of freeresponse kappa
K_{FR}has several interesting properties. It does not depend on a, but only on the positive observations b, c, and d. Therefore the uncertainty about a does not preclude the estimation of agreement beyond chance if the number of negative findings can be considered very large.
When interpreting K_{FR}, it is helpful to consider the numbers of ratings made by each rater individually. The first rater made c + d positive observations, and the second rater made b + d positive observations. Therefore the denominator b + c + 2d is the total number of positive individual observations made by the 2 raters, 2d is the number of positive observations made by either rater that were confirmed by the other, and b + c is the number of positive observations made by either rater that were not confirmed by the other. K_{FR} is thus the proportion of confirmed positive individual observations among all positive individual observations. A K_{FR} statistic of 0.5 means that half of the positive findings were confirmed by the other rater, which may be considered average, whereas 0.8 might be considered very good. This is in line with published interpretation guidelines for Cohen’s kappa [8].
It follows that patients without any detected lesions do not contribute to the estimate of K_{FR}; their weight is zero. Therefore patientlevel clustering does not need to be taken into account to compute K_{FR}, and patients without positive finding can be ignored.
Of note, the equation for K_{FR} corresponds to the proportion of specific (positive) agreement as described by Fleiss [9]. While the equation is identical, the purpose and interpretation are different. For Fleiss, specific positive agreement (and also specific negative agreement) is a complementary statistic that enhances the interpretation of overall agreement. The omission of double negative observations is an a priori decision. Importantly, Fleiss is interested in observed agreement, not in agreement corrected for chance. Finally, Fleiss does not address the freeresponse context.
Variance of the freeresponse kappa
Thus a confidence interval can be obtained for logit (K_{FR}), and the lower and upper confidence bounds backtransformed to the original scale.
An alternative approach is to make use of the direct relationship between K_{FR} and the proportion of congruent pairs of observations among all available observations, p = d/(b + c + d). It is easily shown that K_{FR} = 2p/(1 + p). Therefore a 95% confidence interval can be obtained for p, using any available method for binomial proportions including exact methods, and the confidence bounds can be then backtransformed to the K_{FR} scale.
We have simulated the performance of three confidence interval methods for independent observations at K_{FR} values of 0.3, 0.5, 0.7, and 0.9, and for sample sizes (N = b + c + d) of 20, 50, 100, and 200. For each condition we generated 50’000 random samples from a binomial distribution with parameters N and p, where p was defined by K_{FR}/(2K_{FR}), which is the inverse of the equation K_{FR} = 2p/(1 + p). For each sample we computed a 95% confidence interval using Eq. (3) for the logit of K_{FR}, and also using 2 methods for the binomial parameter p that are appropriate for small samples in which asymptotic estimation methods may yield incorrect results: the AgrestiCoull method [10], and the ClopperPearson method [11]. For each situation we report the mean simulated value of K_{FR}, the proportion of confidence intervals that include the true value, and the mean width of the confidence intervals.
Simulations of the coverage and mean width of 95% confidence intervals for the freeresponse kappa at selected sample sizes (20, 50, 100, 200) and values of kappa (0.3, 0.5, 0.7, 0.9), using three methods: delta method (Eq. 3), AgrestiCoull confidence limits, and ClopperPearson confidence limits
Simulation parameters  Mean observed K_{FR}  Degenerate sample^{a}(d = 0 or d = N)  Coverage of 95% confidence interval  Mean width of 95% confidence interval  

N  K_{FR}  Logit delta method (Equation 3)  AgrestiCoull method  ClopperPearson method  Logit delta method (equation 3)  AgrestiCoull method  ClopperPearson method  
20  0.3  0.291  0.020  0.932  0.952  0.966  0.446  0.444  0.473 
0.5  0.491  <0.001  0.944  0.944  0.969  0.426  0.419  0.471  
0.7  0.693  0  0.957  0.957  0.976  0.354  0.345  0.392  
0.9  0.897  0.019  0.964  0.981  0.964  0.224  0.218  0.235  
50  0.3  0.297  <0.001  0.962  0.962  0.962  0.293  0.294  0.314 
0.5  0.497  0  0.949  0.949  0.965  0.284  0.281  0.305  
0.7  0.697  0  0.953  0.936  0.968  0.230  0.227  0.246  
0.9  0.899  <0.001  0.958  0.958  0.974  0.134  0.134  0.142  
100  0.3  0.298  0  0.954  0.954  0.954  0.211  0.212  0.223 
0.5  0.498  0  0.945  0.945  0.968  0.204  0.203  0.215  
0.7  0.698  0  0.946  0.946  0.966  0.164  0.163  0.172  
0.9  0.899  0  0.948  0.948  0.963  0.093  0.093  0.098  
200  0.3  0.299  0  0.947  0.947  0.959  0.151  0.151  0.157 
0.5  0.499  0  0.948  0.948  0.957  0.146  0.145  0.151  
0.7  0.699  0  0.952  0.952  0.952  0.116  0.116  0.120  
0.9  0.900  0  0.957  0.957  0.957  0.065  0.065  0.068 
Of note, the mean values of observed K_{FR} were slightly below the parameter values, especially at low sample sizes. This is because we simulated with a fixed parameter p, and K_{FR} = 2p/(1 + p) is a concave function. By Jensen’s inequality, the expectation of a concave function of p (i.e., the mean observed K_{FR}) will be then less than the function of the expectation of p (i.e., the K_{FR} that corresponds to the parameter p).
To be valid, these estimation methods require observations to be mutually independent. This may apply in some circumstances: e.g., if a paired screening test is applied to a large population, and only those with at least one positive result are referred for further investigation. But for most imaging procedures data are naturally clustered within patients. Then the proposed asymptotic variance of K_{FR} would be biased. In presence of clustering, a bootstrap procedure can be used to obtain a confidence interval (see Appendix 2).
Results: case study
Contingency table of matched ratings in the Magnetic Resonance Imaging study
Second Rater  

Negative  Positive  Total  
First Rater  Negative  unspecified  19  
Positive  57  173  230  
Total  192  249 
Contingency tables of matched ratings for three different levels of regions of interest
Patient level  Second Rater  

Negative  Positive  Total  
First Rater  Negative  26  1  27 
Positive  2  55  57  
Total  28  56  84  
Intermediate level (9 regions per ratient)  Second Rater  
Negative  Positive  Total  
First Rater  Negative  640  8  648 
Positive  21  87  108  
Total  661  95  84×9  
Detailed level (95 regions per ratient)  Second Rater  
Negative  Positive  Total  
First Rater  Negative  7743  18  7761 
Positive  53  166  219  
Total  7796  184  84×95 
At the patient level, of 84 mutually independent ratings, 81 are concordant (55 patients for whom each rater found at least one abnormality, and 26 double negatives), and 3 discordant (patients for whom one rater found a lesion and the other found none). In this case the kappa statistic is 0.919 (0.816, 1.00). At the intermediate level (9 regions of interest per patient), there are 29 discordant ratings, 727 concordant ratings, and the kappa statistic equals 0.835 (0.763, 0.901). Finally at the detailed level (95 regions of interest per patient), there are 71 discordant findings, 7909 concordant findings, and the kappa statistic is 0.819 (0.738, 0.889). This result is virtually indistinguishable from the freeresponse kappa of 0.820, which assumed an infinite number of potential findings.
Discussion
Situations in which only positive findings are explicit are frequent in imaging procedures. Images can cover large areas or even the whole body, and identify multiple abnormalities, such as metastases, plaques in multiple sclerosis, or stenoses along the coronary system. In many cases the universe of possible findings (abnormalities or lesions) is very large and cannot be enumerated. The lack of a specific number of doublenegative observations precludes the use of the classic formulation of the kappa statistic.
In this paper, we propose a variant of the kappa statistic that relies on the properties of the classic kappa statistic when the number of negative ratings can be considered large. In that case, agreement does not depend on the unknown data and can be estimated from positive findings only. This freeresponse kappa corresponds to the proportion of all confirmed individual positive ratings (2d) among all positive individual ratings (b + c + 2d).
Unlike simplifications that circumvent the freeresponse paradigm, the freeresponse kappa statistic only uses the available data at the level at which ratings – and specifically, the decisions about what constitutes agreement or disagreement – were made. It requires neither an enumeration of all possible lesion sites, nor a reduction of the data by defining regions of interest. On the contrary, for the freeresponse kappa, the more precise the ratings are, the more they conform to the assumption a nonfinite universe of lesions.
The validity of the freeresponse kappa relies upon an accurate definition of concordant and discordant findings. This is true for any agreement study, but for Cohen’s kappa, e.g. when regions of interest are defined, pairing is straightforward because it follows the definition of the regions or objects of study. The freeresponse paradigm requires that observations from 2 raters be classified as concordant or discordant. This must be planned carefully when designing the study and defining the rating procedures. Typically, such a study is done in two steps: first, 2 independent raters assess the images, and then a third independent rater identifies concordant pairs. Therefore the concordance of the 2 descriptions is determined by a human observer, who may be prone to error. For this reason the descriptive system used by the raters should be as detailed as required for clinical management, and fully standardized to facilitate the decisions regarding agreement.
Assumption of infinity
The notion of an infinite number of potential lesions may appear excessive or unrealistic. However, when one considers the number of anatomical structures in the human body, multiplied by the number of study participants, this is not far fetched. Furthermore, once the possible number of double negative observations in the study (i.e., in all participants) exceeds a few thousand, K_{FR} has reached its asymptote and does not change meaningfully if this number is further increased. Nevertheless K_{FR} can be considered as an upper bound on agreement corrected for chance.
The requirement of a large number of potential lesions is not fulfilled in all imaging studies. If one is interested in measuring agreement on the chest Xray performed to rule out iatrogenic pneumothorax after a central venous catheter insertion, there is one diagnosis and only a few radiologic signs to consider. In this case, the number of clinically relevant normal findings is limited and the freeresponse kappa would not be appropriate. Then, and more generally when it is reasonable to specify the number X of potential abnormalities that can be identified, it is reasonable to use X to infer the number of double negatives, as a = Xbcd, and to obtain the standard kappa statistic.
Clustering of observations
For most imaging procedures, each patient can contribute several positive findings, and data are naturally clustered within patients. Clustering does not influence the computation of the freeresponse kappa, but must be taken into account for the computation of the standard error. Importantly, the global freeresponse kappa is a weighted average of withincluster kappa statistics, with weights proportional to b _{ k } + c _{ k } + 2d _{ k }, the total number of positive ratings in a cluster (ignoring pairing). This decomposition holds for any partition of the data and could be done for any covariate, e.g., to compare agreement beyond chance in obese versus nonobese patients, or for skeletal lesions versus lesions in soft tissues.
When observations are independent, confidence intervals can be computed using several methods, compared in Table 1. For clustered data, a common situation in radiology, we propose a bootstrapbased approach. We sampled patients (with replacement), and used all observations from any selected patient [13, 14]. We reasoned that this represented best the role of sampling variability in imaging studies: a patient is a “random” factor, but a lesion within a patient is not. Nevertheless, alternative methods for the estimation of K_{FR} should be explored in future studies. Future developments should also address the generalization of freeresponse kappa to multiple raters, and to ordinal ratings.
Conclusions
We have proposed a kappa statistic that is appropriate for freeresponse assessments, and discussed its properties. This statistic may be particularly useful for imaging studies.
Abbreviations
 CI:

Confidence interval
 K_{FR} :

Freeresponse kappa
 Logit (x):

Ln (x/(1x))
 MRI:

Magnetic Resonance Imaging
Declarations
Acknowledgments
None
Funding
This research did not receive any specific grant.
Availabity of data and materials
Data from the casestudy can be requested from Laura Merlini.
Authors’ contributions
MC coordinated the project, derived the formula for the kappa statistic, collaborated on other statistical aspects, performed analyses of the casestudy, wrote the paper. CC discussed the statistical aspects of the kappa statistic, derived variance formulas, provided critical revisions on the paper. LM brought the problem to attention, provided the data for the casestudy, discussed clinical implications, provided critical revisions on the paper. TP advised on the derivation of the kappa statistic and related variance formulas, performed confidence interval simulations, discussed implications, wrote the paper, supervised the project. All authors have read and approve the final version of the manuscript.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not obtained (see above).
Ethics approval and consent to participate
Not applicable to the methodologic development. The clinical case study (reference 12) was conducted in compliance with local institutional governance. Ethics approval was granted by the Research Ethics Commission of Canton Geneva (CER:13073R), which also waived the requirement to obtain patient consent for the research use of MRI images originally obtained for clinical purposes.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
 Kundel HL, Polansky M. Measurement of observer agreement. Radiology. 2003;228:303–8.View ArticlePubMedGoogle Scholar
 Chakraborty DP. A brief history of freeresponse receiver operating characteristic paradigm data analysis. Acad Radiol. 2013;20:915–9.
 Mohamed ASR, Ruangskul MN, Awan MJ, et al. Quality assurance assessment of diagnostic and radiation therapy–simulation CT image registration for head and neck radiation therapy: anatomic region of interest–based comparison of rigid and deformable algorithms. Radiology. 2014;274:752–63.
 Gönen M, Panageas KS, Larson SM. Statistical issues in analysis of diagnostic imaging experiments with multiple observations per patient. Radiology. 2001;221:763–7.
 Genders TSS, Spronk S, Stijnen T, Steyerberg EW, Lesaffre E, Hunink MGM. Methods for calculating sensitivity and specificity of clustered data: a tutorial. Radiology. 2012;265:910–6.
 Levine D, Bankier AA, Halpern EF. Submissions to Radiology: our Top 10 list of statistical errors. Radiology. 2009;253:288–90.
 Yang Z, Zhou M. Kappa statistic for clustered matchedpair data. Stat Med. 2014;33:2612–33.View ArticlePubMedGoogle Scholar
 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.View ArticlePubMedGoogle Scholar
 Fleiss JL. Statistical Methods for Rates and Proportions. Second ed. New York: Wiley, John and Sons, Incorporated; 1981. p. 214–5.
 Agresti A, Coull BA. Approximate is better than “exact” for interval estimation of binomial proportions. Am Stat. 1998;52:119–26.Google Scholar
 Clopper CJ, Pearson ES. The use of confidence or fiduacial limits illustrated in the ase of the binomial. Biometrika. 1934;26:404–13.View ArticleGoogle Scholar
 Merlini L, Carpentier M, Ferrey S, Anooshiravani M, Poletti PA, Hanquinet S. Wholebody MRI in children: Would a 3D STIR sequence alone be sufficient for investigating common paediatric conditions? A comparative study. Eur Radiol. 2017;88:155–62.
 Kang C, Qaqish B, Monaco J, Sheridan SL, Cai J. Kappa statistic for clustered dichotomous responses from physicians and patients. Stat Med. 2013;32:3700–19.View ArticlePubMedGoogle Scholar
 Field CA, Welsh AH. Bootstrapping clustered data. J R Stat Soc Ser B Stat Methodol. 2007;69:369–90.View ArticleGoogle Scholar
 Casella G, Berger RL. Statistical Inference. Belmonte: Brooks/Cole Publishing Company; 1990. p. 328–31.Google Scholar
 Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Boca Raton: CRC Press; 1994. p. 178–201.Google Scholar