Table 2 Positive responses and inter-rater reliability per AMSTAR question, for Cochrane and non-Cochrane systematic reviews

1. Was an “a priori” design provided?

29 (96.7%)^a

10 (15.4%)

<0.001^b

0.93 (0.82, 1.00)

0.78 (0.63, 0.92)

“Almost perfect” vs. “substantial”^c

2. Was there duplicate study selection and data extraction?

24 (80.0%)

21 (32.3%)

<0.001^b

0.65 (0.36, 0.93)

0.75 (0.59, 0.91)

“Substantial” vs. “substantial”

3. Was a comprehensive literature search performed?

30 (100.0%)

42 (64.6%)

<0.001^b

0.96 (0.89, 1.00)

0.64 (0.44, 0.83)

“Almost perfect” vs. “substantial”^c

4. Did the authors search for reports regardless of their publication type?

27 (90.0%)

23 (35.4%)

<0.001^b

0.85 (0.68, 1.00)

0.72 (0.55, 0.89)

“Almost perfect” vs. “substantial”^c

5. Was a list of studies (included and excluded) provided?

30 (100.0%)

24 (36.9%)

<0.001^b

1.00 (1.00, 1.00)

0.65 (0.47, 0.84)

“Perfect” vs. “substantial”^c

6. Were the characteristics of the included studies provided?

30 (100.0%)

57 (87.7%)

0.045^b

1.00 (1.00, 1.00)

0.91 (0.82, 0.99)

“Perfect” vs. “almost perfect”^c

7. Was the scientific quality of the included studies assessed and documented?

29 (96.7%)

39 (60.0%)

<0.001^b

0.97 (0.89, 1.00)

0.62 (0.43, 0.82)

“Almost perfect” vs. “substantial”^c

8. Was the scientific quality of the included studies used appropriately in formulating conclusions?

26 (86.7%)

47 (72.3%)

0.12

0.79 (0.59, 0.99)

0.60 (0.40, 0.80)

“Substantial” vs. “moderate”^c

9. Were the methods used to combine the findings of studies appropriate?

28 (93.3%)

55 (84.6%)

0.23

0.84 (0.66, 1.00)

0.69 (0.52, 0.87)

“Almost perfect” vs. “substantial”^c

10. Was the likelihood of publication bias assessed?

16 (53.3%)

30 (46.2%)

0.52

0.67 (0.39, 0.95)

0.71 (0.53, 0.88)

“Substantial” vs. “substantial”

11. Was the conflict of interest stated?

19 (63.3%)

9 (13.9%)

<0.001^b

0.75 (0.51, 1.00)

0.65 (0.46, 0.84)

“Substantial” vs. “substantial”