Skip to main content

Table 3 Inter-individual comparison of rating scales

From: Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability

Statistic three-point scale five-point scale nine-point scale Sensitivity analysis
five-point scalea nine-point scalea
Overallb
 Changes in 2nd survey (in %) 12.48 24.73 32.26 20.96 8.57
 Class imbalancea 1st survey (in %) 79.16 64.93 63.62 64.93 88.25
 Test-retest agreement (in %) 87.52 75.27 67.74 79.04 91.43
 Weighted kappa [95% CI] 0.63 [0.62; 0.64] 0.47 [0.07; 0.86] 0.78 [0.78; 0.78] 0.54 [0.50; 0.58] 0.58 [0.55; 0.62]
Mean [range] over the 19 proposed treatment goals
 Changes in 2nd survey (in %) 12.60 [2.41; 25.61] 24.75 [16.05; 38.82] 32.43 [17.07; 55.13] 20.96 [16.05; 28.24] 8.69 [0.00; 24.00]
 Class imbalancec 1st survey (in %) 0.80 [49.38; 95.35] 66.05 [35.71; 83.13] 63.46 [21.25; 81.18] 68.65 [45.78; 83.13] 88.19 [37.50; 100.00]
 Test-retest agreement (in %) 87.40 [74.39; 97.59] 75.25 [61.18; 83.95] 67.57 [44.87; 82.93] 79.04 [71.76; 83.95] 91.31 [76.00; 100.00]
 Weighted kappa 0.55 [0.18; 0.87] 0.44 [0.29; 0.62] 0.61 [0.17; 0.81] 0.49 [0.35; 0.67] 0.40 [0.00; 0.80]
  1. aRating scale mapped onto three categories
  2. bOverall refers total ratings of all participants of all treatment goals, e.g., the number of participants times 19 goals times ratings of the respective scale five-point/nine-point scale
  3. cClass imbalance is highlighted by the percentage of the most frequently used rating category (e.g. in the first survey, the rating categories main goal/secondary goal/no goal scored 79%/11%/10% across all participants’ ratings of all goals, hence, the imbalance is 79%)