Fig. 6

The contour plots show the Krippendorff’s alpha (left) and percent error (right) values for the simulated evaluators, for the prescribed grid of systematic error and random error. The values for the human evaluators are plotted in bold numerals (experts in lighter font), with their approximate position corresponding to their calculated systematic error and random error compared to the reference dataset. This figure provides insight into the types of mistakes being made by evaluators. Whereas the evaluators were positioned in this figure based on their calculated systematic error and random error, positioning them based on the Krippendorff’s alpha and percent error contours would still provide reasonably accurate estimates of the true amount of systematic and random error