From: A system for rating the stability and strength of medical evidence
Judgments pertaining to study quality assessment |
• What method will be used to assess study quality? |
• If a quality scale is to be used, what scoring method will apply? |
• What is the threshold for excluding a study from analysis due to poor quality? |
• How will the individual study quality ratings be summarized to yield a single overall rating of quality to the evidence base (High, Moderate, or Low)? |
Judgment pertaining to sufficient evidence for quantitative estimate |
• What is the minimum number of studies required to permit a quantitative estimate? |
• What is the minimum percentage of studies reporting accurate information (i.e., calculable effect sizes) required to permit a quantitative estimate? |
• What imputation methods will be used for studies that did not report sufficient information for a calculable effect size? |
Judgments pertaining to initial meta-analysis |
• What effect size measure will be used? |
• How will heterogeneity be measured (Q or I^{2})? |
• What is the threshold for considering an evidence base heterogeneous? |
• Will the summary effect size estimate be derived from a fixed-effects or random-effects meta-analysis? |
Judgments pertaining to quantitative robustness testing |
• What robustness tests will be used? |
• If a cumulative meta-analysis is one of the robustness tests, will it be a fixed-effects or random-effects model? |
• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation? |
• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness? |
• If a cumulative meta-analysis is one of the robustness tests, what threshold for a change in the summary effect size will be used to determine robustness? |
• If publication bias testing is one of the robustness tests, which method of testing for publication bias will be used? |
• If confidence interval width is one of the robustness tests, how narrow must the interval be for the summary estimate to be considered robust? |
• Will overall robustness be judged based on passing all of the robustness tests, or simply a majority, or what percentage? |
Judgments pertaining to meta-regression |
• What is the minimum number of studies required to perform meta-regression? |
• Which covariates will be included in multiple regression models? |
• How many covariates are permitted in any given regression model? |
• Does "explaining heterogeneity" require a statistically significant covariate, or the lack of resultant heterogeneity, or both of these? |
• What robustness tests will be performed for the meta-regression? |
Judgments pertaining to qualitative robustness testing |
• What robustness tests will be performed? |
• If a cumulative meta-analysis is one of the robustness tests, in what order will studies be entered into the cumulation? |
• If a cumulative meta-analysis is one of the robustness tests, how many steps (i.e., study removals) will be examined to determine robustness? |
Other judgments |
• What is the size of a lowest possible effect that is still clinically important? |
• What is the p value for statistical significance? |
• What is the definition of a large magnitude of effect? |
• How will qualitative consistency be defined (based on point estimates, confidence intervals, some percentage of studies, etc)? |