Table 4 Tendency of the individual raters to score above or below the mean score

Rater 1

17.6

7.4

8.3

5.6

-0.45 ± 1.53

0.003

Rater 2

12

7.4

10.2

7.4

0.03 ± 1.43

0.839

Rater 3

15.9

8.4

6.5

5.6

-0.25 ± 1.50

0.084

Rater 4

4.6

2.8

10.2

21.3

0.68 ± 1.52

< 0.001

Formal quality

% of answers 2 or more points below mean score

% of answers > 1 and < 2 points below mean score

% of answers > 1 and < 2 points above mean score

% of answers 2 or more points above mean score

Bias (score-mean score) (mean ± SD)

P (comparison score with mean score)

Rater 1

24.1

17.6

0.9

0

-0.97 ± 1.13

< 0.001

Rater 2

2.8

3.7

2.8

0.9

0.00 ± 0.71

1.000

Rater 3

9.3

9.3

4.7

0.9

-0.25 ± 0.98

0.012

Rater 4

1.9

0

22.2

6.5

0.66 ± 0.89

< 0.001

Rater 5

2.8

2.8

19.4

6.5

0.56 ± 0.94

< 0.001