Table 1 The agreement algorithm used for the Crowd task. Breaks in the consecutive chain or any ‘unsure’ classification sends the records to resolvers to make the final decision
From: Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study
Decision 1 | Decision 2 | Decision 3 | Final decision |
|---|---|---|---|
Potentially relevant | Potentially relevant | Potentially relevant | Potentially relevant |
Not relevant | Not relevant | Not relevant | Not relevant |
Potentially relevant | Potentially relevant | Not relevant | Resolver decision |
Potentially relevant | Not relevant | Not applicable | Resolver decision |
Not relevant | Not relevant | Potentially relevant | Resolver decision |
Not relevant | Potentially relevant | Not applicable | Resolver decision |
Unsure | Not applicable | Not applicable | Resolver decision |