Skip to main content

Advertisement

Table 2 Judgments Involved in Systematic Reviews

From: A system for rating the stability and strength of medical evidence

Judgments pertaining to study quality assessment
• What method will be used to assess study quality?
• If a quality scale is to be used, what scoring method will apply?
• What is the threshold for excluding a study from analysis due to poor quality?
• How will the individual study quality ratings be summarized to yield a single overall rating of quality to the evidence base (High, Moderate, or Low)?
Judgment pertaining to sufficient evidence for quantitative estimate
• What is the minimum number of studies required to permit a quantitative estimate?
• What is the minimum percentage of studies reporting accurate information (i.e., calculable effect sizes) required to permit a quantitative estimate?
• What imputation methods will be used for studies that did not report sufficient information for a calculable effect size?
Judgments pertaining to initial meta-analysis
• What effect size measure will be used?
• How will heterogeneity be measured (Q or I2)?
• What is the threshold for considering an evidence base heterogeneous?
• Will the summary effect size estimate be derived from a fixed-effects or random-effects meta-analysis?
Judgments pertaining to quantitative robustness testing
• What robustness tests will be used?
• If a cumulative meta-analysis is one of the robustness tests, will it be a fixed-effects or random-effects model?
• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation?
• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness?
• If a cumulative meta-analysis is one of the robustness tests, what threshold for a change in the summary effect size will be used to determine robustness?
• If publication bias testing is one of the robustness tests, which method of testing for publication bias will be used?
• If confidence interval width is one of the robustness tests, how narrow must the interval be for the summary estimate to be considered robust?
• Will overall robustness be judged based on passing all of the robustness tests, or simply a majority, or what percentage?
Judgments pertaining to meta-regression
• What is the minimum number of studies required to perform meta-regression?
• Which covariates will be included in multiple regression models?
• How many covariates are permitted in any given regression model?
• Does "explaining heterogeneity" require a statistically significant covariate, or the lack of resultant heterogeneity, or both of these?
• What robustness tests will be performed for the meta-regression?
Judgments pertaining to qualitative robustness testing
• What robustness tests will be performed?
• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation?
• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness?
Other judgments
• What is the size of a lowest possible effect that is still clinically important?
• What is the p value for statistical significance?
• What is the definition of a large magnitude of effect?
• How will qualitative consistency be defined (based on point estimates, confidence intervals, some percentage of studies, etc)?