Table 3 Recommendations regarding data extraction

Data extraction should be conducted by at least two people

Data extraction by at least two people independently (parallel extraction)

Data extracted by one individual accuracy checks by a second (double-checking)

Parallel extraction for critical items and double-checking for non-critical items

Validation of a random sample of data by a third investigator

Data extraction by reviewers with complementary expertise

Data extraction by at least one reviewer with expertise in systematic review methods

Quantify agreement using a reliability measure such as Cohen’s kappa

If agreement is quantified, do this only for critical items

Informally consider reliability of coding throughout the review process

Explicit procedures or rules for resolving disagreements

Report who was involved in data extraction

Document disagreements and how they were resolved