Table 10 Weighted inter-rater agreement measures for the motivating example

Measure

Table

Unweighted

Linear

Quadratic

Radical

Ordinal

Scott’s π

Original

0.461

0.564

0.671

0.510

0.629

Scenario 1

0.258

0.367

0.503

0.308

0.447

Scenario 2

0.504

0.582

0.671

0.541

0.635

Scenario 3

0.469

0.552

0.648

0.508

0.610

CV

0.263

0.194

0.130

0.229

0.154

Gwet’s AC2

Original

0.529

0.719

0.850

0.629

0.809

Scenario 1

0.407

0.673

0.840

0.550

0.790

Scenario 2

0.576

0.750

0.869

0.668

0.832

Scenario 3

0.550

0.737

0.863

0.649

0.825

CV

0.145

0.047

0.015

0.083

0.023

Krippendorff’s α

Original

0.464

0.566

0.673

0.513

0.631

Scenario 1

0.262

0.370

0.506

0.312

0.450

Scenario 2

0.507

0.584

0.673

0.543

0.637

Scenario 3

0.472

0.554

0.650

0.510

0.612

CV

0.260

0.192

0.129

0.226

0.153

Brennan-Prediger’s S

Original

0.514

0.680

0.812

0.599

0.768

Scenario 1

0.376

0.600

0.774

0.491

0.716

Scenario 2

0.560

0.711

0.830

0.637

0.790

Scenario 3

0.532

0.694

0.821

0.615

0.779

CV

0.165

0.073

0.030

0.111

0.043