Table 4 Interrater agreement for R-AMSTAR

1. Was an ‘a priori’ design provided

A

−0.20-0.24

B

0.88–0.99

C

0.15–0.54

2. Was there duplicate study selection and data extraction

A

0.66–0.90

B

0.48–0.79

C

0.45–0.78

3. Was a comprehensive literature search performed

A

0.93–1.00

B

0.89–0.99

C

0.03–0.47

D

0.38–0.73

E

0.73–0.93

4. Was the status of publication (i.e. grey literature) used as an inclusion criterion

A

0.38–0.72

B

−0.52-(−0.12)

C

0.04–0.44

D

0.61–0.88

5. Was a list of studies (included and excluded) provided

A

0.89–1

B

0.67–0.92

C

0.21–0.60

D

0.30–0.66

6. Were the characteristics of the included studies provided

A

0.68–0.91

B

0.28–0.64

C

0.43–0.76

7. Was the scientific quality of the included studies assessed and documented

A

0.77–0.95

B

0.50–0.81

C

0.45–0.78

D

0.59–0.86

8. Was the scientific quality of the included studies used appropriately in formulating conclusions

A

0.50–0.81

B

0.03–0.43

C

0.07–0.48

D

0.93–1

9. Were the methods used to combine the findings of studies appropriate

A

0.57–0.85

B

0.60–0.88

C

0.53–0.83

D

0.52–0.82

E

0.28–0.66

10. Was the likelihood of publication bias assessed

A

0.41–0.74

B

0.69–0.92

C

0.90–1

11. Was the conflict of interest included

A

0.56–0.84

B

0.44–0.76

C

0.76–0.95

0.53–0.70