Table 3 Description of challenges identified when using AMSTAR in overviews, with corresponding recommendations

Many non-Cochrane SRs provided limited detail when reporting the characteristics and quality of their included primary studies.

Q6: Were the characteristics of the included studies provided?

Q7: Was the scientific quality of the included studies assessed and documented?

Q6: Some SRs presented only aggregate study characteristics; others provided insufficient detail about the populations, interventions, comparators, outcome assessments, and/or study settings.

Q7: Non-Cochrane SRs used various quality assessment tools including the Cochrane Risk of Bias tool [51], the Jadad tool [75], and additional lesser-known tools. The Risk of Bias tool was often applied inconsistently across SRs, with different SRs assessing and reporting different domains.

Award point(s) only if the amount and quality of information reported in the SR is sufficient for use at the overview level.

Overview authors rely upon the primary study information as it is reported in the included SRs, and overview quality may be compromised due to inadequate reporting of SRs.

Some SRs that examined the same interventions for the same disorder analyzed outcome data differently and/or came to different conclusions.

Q8: Was the scientific quality of the included studies used appropriately in formulating conclusions?

Q9: Were the methods used to combine the findings of studies appropriate?

There were several instances where one SR conducted meta-analyses while another SR examining the same intervention for the same disorder presented narrative summaries only. In several instances, these SRs also reached different conclusions.

Award point(s) if authors provide appropriate justification for why they chose their method of data analysis and/or how they came to a particular conclusion.

It may not be possible to objectively determine whether the conclusions or methods of analysis of one SR were more appropriate or valid than those in another, similar SR. It is more objective to examine the authors’ justification for whichever decisions were made.

Some SRs were broader in scope than the overview’s clinical question, meaning that some primary studies included in the SR were excluded from the overview.

Q5: Was a list of studies (included and excluded) provided?

Q6, Q7, Q8, Q9: See above.

Q10: Was the likelihood of publication bias assessed?

For both Cochrane and non-Cochrane SRs, there were many instances where the scope of the SR was broader than that of the overview. For example, an overview that is restricted to children only will aim to exclude adult data contained within relevant SRs.

Assess quality of the SRs overall; do not try to “piece apart” the SRs to assess only the relevant studies.

It is important to capture information about the conduct of the SR as a whole; attempting to isolate only the primary studies of interest is unnecessarily difficult.

Some non-Cochrane SRs, such as those produced for government or research organizations, searched for and included other SRs as well as primary studies. It was very difficult to assess the quality of non-Cochrane (‘original’) SRs that also included other (‘embedded’) SRs.

Q5, Q6, Q7, Q8, Q9, Q10: See above.

The original non-Cochrane SRs often did not provide sufficient information about their embedded SRs (or the studies contained within their embedded SRs). This scenario also raises additional questions for which there are no adequate answers. For example, would the original SR be awarded a point for Q5 if it did not contain a list of the studies that were included and excluded in each of its embedded SRs?

Assess the embedded SRs for inclusion into the overview. If any of them meet the inclusion criteria, obtain and refer to the full-text of these SRs and treat them as independent publications (in place of using the descriptions provided in the original SR).

It is likely not possible, nor desirable, to integrate the embedded SRs with the primary studies included in the original SR.