Skip to main content

Table 3 Inter-rater agreement (percentage agreement) and reliability (kappa coefficients) of the items from the COSMIN checklist (COSMIN step 4)

From: Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

Item nr Item N (minus articles with 1 rating)a % agreement N Kappa
Generalisability Box (n = 866) b c    
Was the sample in which the HR-PRO instruments was evaluated adequately described? In terms of:     
1d median or mean age (with standard deviation or range)? 733 86 865 0.36
2d distribution of sex? 735 88 863 0.38e
3 important disease characteristics (e.g. severity, status, duration) and description of treatment? 746 80 862 0.39f
4d setting(s) in which the study was conducted? e.g. general population, primary care or hospital/rehabilitation care 735 89 863 0.30e
5d countries in which the study was conducted? 733 90 861 0.40e
6d language in which the HR-PRO instrument was evaluated? 733 86 861 0.41e
7d Was the method used to select patients adequately described? e.g. convenience, consecutive, or random 729 81 857 0.40
8 Was the percentage of missing responses (response rate) acceptable? 724 82 849 0.48
  1. a When calculating percentage agreement, articles that were only scored once on the particular item were not taken into account;b number of times a box was evaluated;c sample sizes of Generalisability box are much higher that other items, because scores of the items on the Generalisability box for all measurement properties were combined;d dichotomous item;e Items with low dispersal i.e. more than 75% of the raters who responded to an item rated the same response category;f Combined kappa coefficient calculated because of nominal response scale in a one-way design; printed in bold indicates Kappa > 0.70 or % agreement >80%.