Table 2 Recommendations regarding the piloting of data collection forms

Train data extractors in using the form

Pilot test form using a sample of studies

(Partially) repeat piloting if major changes are made during the review

In case of modifications to the data collection form, re-check reports that have already undergone data extraction

Involve reviewers with complementary expertise

Involve reviewer with experience in systematic review methods

Quantify agreement using a reliability measure such as Cohen’s kappa

If agreement is quantified, do this only for critical items

Repeat the piloting process until a specified agreement is reached

Informally consider reliability of coding while piloting