Table 1 Comparison of matched AMSTAR-2 and ROBIS items
AMSTAR-2 | ROBIS | Level of agreement; Gwet’s AC1 (95% CI) |
|---|---|---|
Domain 1: Study eligibility criteria | ||
1. Did the research questions and inclusion criteria for the review include the components of PICO? | 1.3 Were eligibility criteria unambiguous? | Almost perfect; 0.87 (0.78, 0.96) |
2. Did the report of the review contain an explicit statement that the review methods were established before the conduct of the review and did the report justify any significant deviations from the protocol? | 1.1 Did the review adhere to predefined objectives and eligibility criteria? (protocol) | Almost perfect; 0.99 (0.97, 1)a |
3. Did the review authors explain their selection of the study designs for inclusion in the review? | Not considered | – |
Not considered | 1.2 Were the eligibility criteria appropriate for the review question? | – |
Not considered | 1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate? | – |
Not considered | 1.5 Were any restrictions in eligibility criteria based on sources of information appropriate? | – |
Domain 2: Identification and selection of studies | ||
4. Did the review authors use a comprehensive literature search strategy? | 2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports? 2.2 Were methods additional to database searching used to identify relevant reports? 2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? 2.4 Were restrictions based on date, publication format, or language appropriate? | Substantial; 0.79 (0.74, 0.85)*b |
5. Did the review authors perform study selection in duplicate? | 2.5 Were efforts made to minimize error in selection of studies? | Almost perfect; 0.87 (0.77, 0.96) |
Not considered | 3.5 Were efforts made to minimize error in risk of bias assessment? | – |
Domain 3: Data collection and study appraisal | ||
6. Did the review authors perform data extraction in duplicate? | 3.1 Were efforts made to minimize error in data collection? | Almost perfect; 0.88 (0.79, 0.98) |
7. Did the review authors provide a list of excluded studies and justify the exclusions? | Not considered | – |
8. Did the review authors describe the included studies in adequate detail? | 3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results? | Moderate; 0.6 (0.44, 0.76)a |
Not considered | 3.3 Were all relevant study results collected for use in the synthesis? | – |
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? | 3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria? | Almost perfect; 0.88 (0.79, 0.98)c |
10. Did the review authors report on the sources of funding for the studies included in the review? | Not considered | – |
Domain 4: Synthesis and findings | ||
Not considered | 4.1 Did the synthesis include all studies that it should? | – |
Not considered | 4.2 Were all predefined analyses reported or departures explained? | – |
11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results? | 4.3 Was the synthesis appropriate, given the nature and similarity in the research questions, study designs, and outcomes across included studies? 4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?d | Almost perfect; 0.81 (0.69, 0.92)e |
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? 13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review? | 4.6 Were biases in primary studies minimal or addressed in the synthesis? | Substantial; 0.77 (0.64, 0.89)f |
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? | 4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?d | Substantial; 0.73 (0.59, 0.86) |
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? | 4.5 Were the findings robust, for example, as demonstrated through funnel plot or sensitivity analyses? | Slight; 0.18 (−0.03, 0.38)g,h |
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? | Not considered | – |