Skip to main content

Table 1 The agreement algorithm used for the Crowd task. Breaks in the consecutive chain or any ‘unsure’ classification sends the records to resolvers to make the final decision

From: Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study

Decision 1 Decision 2 Decision 3 Final decision
Potentially relevant Potentially relevant Potentially relevant Potentially relevant
Not relevant Not relevant Not relevant Not relevant
Potentially relevant Potentially relevant Not relevant Resolver decision
Potentially relevant Not relevant Not applicable Resolver decision
Not relevant Not relevant Potentially relevant Resolver decision
Not relevant Potentially relevant Not applicable Resolver decision
Unsure Not applicable Not applicable Resolver decision