Skip to main content

Table 1 Comparison of matched AMSTAR-2 and ROBIS items

From: Similarities, reliability and gaps in assessing the quality of conduct of systematic reviews using AMSTAR-2 and ROBIS: systematic survey of nutrition reviews

AMSTAR-2 ROBIS Level of agreement; Gwet’s
AC1 (95% CI)
Domain 1: Study eligibility criteria
1. Did the research questions and inclusion criteria for the review include the components of PICO? 1.3 Were eligibility criteria unambiguous? Almost perfect;
0.87 (0.78, 0.96)
2. Did the report of the review contain an explicit statement that the review methods were established before the conduct of the review and did the report justify any significant deviations from the protocol? 1.1 Did the review adhere to predefined objectives and eligibility criteria? (protocol) Almost perfect;
0.99 (0.97, 1)a
3. Did the review authors explain their selection of the study designs for inclusion in the review? Not considered
Not considered 1.2 Were the eligibility criteria appropriate for the review question?
Not considered 1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate?
Not considered 1.5 Were any restrictions in eligibility criteria based on sources of information appropriate?
Domain 2: Identification and selection of studies
4. Did the review authors use a comprehensive literature search strategy? 2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
Substantial;
0.79 (0.74, 0.85)*b
5. Did the review authors perform study selection in duplicate? 2.5 Were efforts made to minimize error in selection of studies? Almost perfect;
0.87 (0.77, 0.96)
Not considered 3.5 Were efforts made to minimize error in risk of bias assessment?
Domain 3: Data collection and study appraisal
6. Did the review authors perform data extraction in duplicate? 3.1 Were efforts made to minimize error in data collection? Almost perfect;
0.88 (0.79, 0.98)
7. Did the review authors provide a list of excluded studies and justify the exclusions? Not considered
8. Did the review authors describe the included studies in adequate detail? 3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results? Moderate;
0.6 (0.44, 0.76)a
Not considered 3.3 Were all relevant study results collected for use in the synthesis?
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? 3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria? Almost perfect;
0.88 (0.79, 0.98)c
10. Did the review authors report on the sources of funding for the studies included in the review? Not considered
Domain 4: Synthesis and findings
Not considered 4.1 Did the synthesis include all studies that it should?
Not considered 4.2 Were all predefined analyses reported or departures explained?
11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results? 4.3 Was the synthesis appropriate, given the nature and similarity in the research questions, study designs, and outcomes across included studies?
4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?d
Almost perfect;
0.81 (0.69, 0.92)e
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?
13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review?
4.6 Were biases in primary studies minimal or addressed in the synthesis? Substantial;
0.77 (0.64, 0.89)f
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? 4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?d Substantial;
0.73 (0.59, 0.86)
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? 4.5 Were the findings robust, for example, as demonstrated through funnel plot or sensitivity analyses? Slight;
0.18 (−0.03, 0.38)g,h
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? Not considered
  1. 95% CI - 95% confidence interval
  2. Gwet’s AC1 - Gwet’s first-order agreement coefficient
  3. ROBIS ‘Y’ and ‘PY’ corresponded to AMSTAR-2 ‘Y’, while ROBIS ‘N’, ‘PN’, and ‘NI’ corresponded to AMSTAR-2 ‘N’
  4. (a) ‘Y’ and ‘PY’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS. ‘N’ in AMSTAR-2 equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS
  5. (b) ‘PY’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS provided these were assessed accordingly both in ROBIS items 2.3 and 2.4. ‘Y’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS provided these were assessed accordingly in ROBIS items 2.1, 2.2, 2.3, and 2.4. Otherwise, ROBIS judgements equalled ‘N’ in AMSTAR-2
  6. (c) If the study included randomized controlled studies (RCTs) only or non-randomised studies of interventions/exposures (NRSI) only: ‘Y’ and ‘PY’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS; ‘N’ in AMSTAR-2 equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS. If the study included both RCTs and NRSI: ‘Y’ or ‘PY’ for both RCTs and NRSI in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS. If RCTs and/or NRSI scored ‘N’ in AMSTAR-2 it equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS
  7. (d) ROBIS item 4.4 was used for two comparisons
  8. (e) Not considered if no MA in the study. If there was MA including RCTs only or NRSI only: ‘Y’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS. ‘N’ in AMSTAR-2 equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS. If the study included MA of RCTs and NRSI: ‘Y’ for both RCTs and NRSI in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS. If RCTs and/or NRSI scored ‘N’ in AMSTAR-2 it equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS
  9. (f) If there was no MA only AMSTAR-2 item 13 was compared with ROBIS item 4.6: ‘Y’ in AMSTAR-2 equalled ‘Y’ and ‘PY’ in ROBIS. ‘N’ in AMSTAR-2 equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS. If the study included MA: ‘Y’ in both AMSTAR-2 item 12 and 13, this equalled ‘Y’ and ‘PY’ in ROBIS. If AMSTAR-2 item 12 or 13 scored ‘N’, this equalled ‘N’, ‘PN’, and ‘NI’ in ROBIS
  10. (g) Not considered if no MA in the study
  11. (h) This comparison was considered partially overlapping
  12. * quadratic weights