Table 1 Criteria for investigating clinical heterogeneity in systematic reviews [5]

Review team

It is recommended to have at least one or two individuals with clinical expertise, and at least one or

two individuals with methodological expertise in systematic reviews/meta-analyses and on the type of study designs that are included.

The team should recognize their own biases and attempt to compensate by including members with a wide range of (potentially conflicting) beliefs.

Quantitative synthesis

Did the review perform a quantitative synthesis?

Clinical heterogeneity variables

Did the authors choose characteristics for exploring that can be considered aspects of "clinical heterogeneity"?

Patient level: Age, baseline disease severity, sex, gender, ethnicity, comorbidities, genetic, other

psychosocial variables, and other important features of the disease.

Intervention level: Dose/strength/intensity of treatment, duration of treatment, brand/manufacturer, cointerventions, timing, route of administration, compliance, clinician training, implementation, other.

Outcome level: Event type, outcome measure type, outcome definition, length of follow-up, timing of outcome measurement(s).

Other: Research setting, geographical issues, length of follow-up.

Planned clinical heterogeneity exploration

Did the authors describe how they planned to investigate differences between studies?

All investigations of clinical heterogeneity should ideally be pre-planned a priori and not be driven by observing the data. But methods for looking at data to identify unanticipated variables of interest (i.e., post-hoc investigations) need to be pre-specified as well (e.g., looking at summary tables, graphical displays). Describe the following: which variables you will investigate, how this will be done, when you will perform these investigations, and how results will be interpreted and incorporated into your results and conclusions.

A priori vs. post-hoc

Was it a priori or post-hoc?

Reviewers should think through all potentially relevant variables to explore and not rely on statistical measures of heterogeneity to justify such investigations.

Individual patient data (IPD) vs. aggregate patient data (APD)

Was IPD, APD or a combination used?

APD = Summary or aggregate data from trials only. This is subject to ecological bias, that is, investigations of trial-level variables are valid (e.g., dose, duration), while investigations of patient-level variables are not (e.g., age, baseline severity).

IPD = Original individual data on each patient. This type of data is valid for both trial-level and patient-level variables. But, one must control for baseline difference between the patients across trials. Consider contacting authors and reviewing protocols of primary studies where available. Obtaining IPD for investigating clinically related patient-level variables is ideal.

Parsimony—number of investigations to perform and variables to explore

Was parsimony used in choosing variables to explore?

Use parsimony as a guide to such investigations. A rule of thumb for the number of trials is that there should be close to ten trials when working with summary or aggregate patient data (APD) or ten individuals per variable, when working with pooled or individual patient data (IPD). Consider making a hierarchy of clinically related variables and investigate only those variables for which your rationale and power are sufficient.

Clinical heterogeneity variables not later investigated

Were there characteristics that were chosen that were not eventually investigated?

Variables that were mentioned to potentially contribute to clinical heterogeneity which were ultimately never investigated (or reported) in the analysis.

Outliers/sensitivity analysis

When there are individual trials that are clear outliers, was there an attempt to determine why? (e.g., was a sensitivity analysis done, where these trials are eliminated and effect estimate changes?

When there are individual trials that are clear outliers, attempt to determine why and consider a

sensitivity analysis to eliminate these trials and observe how the effect estimate changes. One may also consider an influence analysis, in which the effect of deleting individual studies from the analysis on the overall estimate is explored.

Statistical heterogeneity

Was statistical heterogeneity assessed?

Statistical heterogeneity as prerequisite to investigate clinical heterogeneity

Was statistical heterogeneity used as a prerequisite for investigating clinical variables?

Reviewers should think through all potentially relevant variables to explore and not rely on statistical measures of heterogeneity to justify such investigations. Clinical heterogeneity related to specific individual factors could be present even in the absence of a significant statistical test for the presence of heterogeneity (e.g., Cochran’s Q test)

Plots/visuals

Were plotting or other visual aids used to explore reasons for clinical heterogeneity?

Consider using graphical displays of data from trials to help identify potential clinical reasons for

heterogeneity. Examples of plotting and visual aids of the data include: summary data sheets, forest plots, L’Abbé plots, funnel plots, Galbraith plots/radial plots, influence plots, dose/response curves, multidimensional scaling, and heat maps.

Cautious inferences

Was caution used in making inferences from the findings of investigations of heterogeneity?

Results are generally observational and thus hypothesis generating only. Authors should express the validity of and confidence in their findings. When interpreting results of these investigations it is suggested to consider: confounding, other sources of bias (e.g., publication, misclassification, dilution, selection), magnitude and direction of effect and CI, and thinking through the plausibility of causal relationships. It may not be appropriate to conclude that there is consistency of effect if subgroup effects are not found [20]. Authors should use their findings to make specific recommendations about how future research could proceed or build upon these results (not just conclude that “more research is needed”).

Sufficient reporting

When there was insufficient information, were the study authors contacted for more information?

Reporting at a limitation

Was the reporting in the included studies assessed and commented on as a potential problem for investigating clinical heterogeneity?

Consider the potential for lack of reporting of data or information relating to clinical variables in the

primary studies. Consider contacting the authors for missing or additional data on important clinical

variables. Reviewers must be careful to report all of their proposed and actual investigations of clinical heterogeneity. The PRISMA statement should be adhered to when reporting their reviews.