Skip to main content
Fig. 4 | BMC Medical Research Methodology

Fig. 4

From: Reliability in evaluator-based tests: using simulation-constructed models to determine contextually relevant agreement thresholds

Fig. 4

This scatterplot shows the average Krippendorff’s alpha and percent error values for each of the 49 simulated evaluators. The curve was fit to data points of simulated evaluators with systematic error only as these points provide the worst-case percent error for a given value of Krippendorff’s alpha. The results of the expert human evaluators are shown as “E’s” and trained evaluators are shown as “X’s”. The darker vertical lines represent theoretical levels of percent error and the lighter horizontal lines are corresponding Krippendorff’s alpha thresholds for enforcing those error limits. This demonstrates how a Krippendorff’s alpha threshold can be used to limit the amount of error from evaluators based on the observed relationship from the model

Back to article page