Table 3 Effects of estimation targets, category, skew & difficulty on observed or estimated chance agreement and reliability (dr²)

Effects on Intcdr Reliability Obsv & Ests

2

Right: Obsd / Estd Interrater Reliability as Dependent Variables

Down: Independent Variables

o_ri

a_o

S

I_r

AC₁

π

κ

α

3

Observed Reliability (o_ri)

1.00^***

.841^***

.691^***

.599^***

.721^***

.312^***

.312^***

.312^***

4

Category (C)

.003

−.002

.175^***

.185^***

.123^***

.001

.001

.001

5

Distribution Skew (s_k)

.000

.000

.000

−.000

.003

−.293^***

−.292^***

−.293^***

6

Difficulty (d_f)

−.774^***

−.778^***

−.566^***

−.434^***

−.554^***

−.389^***

−.389^***

−.389^***

Effects on Chance Agrt Obsv & Ests

7

Right: Obsd / Estd. Chance Agreement as Dependent Variables

Down: Independent Variables

o_ac

ao_ac = 0^a

S_ac

Ir_ac

AC_ac

π_ac

κ_ac

α_ac

8

Observed Chance Agreement (o_ac)

1.00^***

–

.021^**

.021^**

.075^***

−.151^***

−.152^***

−.151^***

9

Category (C)

−.019^**

–

−.863^***

−.863^***

−.661^***

−.013^*

−.014^*

−.013^*

10

Distribution Skew (s_k)

−.001

–

.000

.000

−.039^***

.437^***

.434^***

.437^***

11

Difficulty (d_f)

.585^***

–

.000

.000

.009

−.123^***

−.125^***

−.123^***

N

12

N_c (number of rating sessions)

384

384

384

384

384

384

384

384

13

N_d (number items within each session)

100

100

100

100

100

100

100

100