Skip to main content

Table 3 Recommendations regarding the methods of choosing or identifying clinical covariates for investigation and interpretation of the findings

From: Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature

General Category of Recommendation Specific Recommendation Number of Resources1 Citations
When to identify covariates in the review process A priori (e.g., in protocol) 17 76, 92,93,95, 100, 98, 18, 26, 39, 40, 30, 59, 29, 31, 46, 94, 114
How to find important clinical covariates from trial information Looking at forest plots (variation in point estimates/CI overlap/ adding a vertical line for levels of some clinical variable) 6 92, 98, 93, 97, 98, 94
  Proceed regardless of formal testing of statistical heterogeneity 5 35, 92, 97, 98, 29
  Looking at L’Abbe plots 4 98, 45, 93, 98
  Influence plot 3 98, 54, 85
  Looking at summary tables 2 92, 24
  Looking at funnel plots 2 49, 98
  Use conceptual frameworks to facilitate choice of covariates (i.e., using taxonomies for active ingredients) 2 98, 112
  I2 (look at the change in statistical heterogeneity by adding subgroups) 2 87, 100
  Plot of effect size against individual covariates 1 48
  Using an adaptation of multidimensional scaling (CoPlot) 1 55
  Plot of normalized z-scores 1 93
  Radial/Galbraith plot 1 93
  Frequency distributions 1 98
  Dose-response graph 1 3?
  Use P.I.C.O. model to guide choice of characteristics 1 115
  Use causal mediating processes 1 113
  Treat strata within trials as separate studies; these subgroups if similar across studies can be combined 1 46
Rationale for choice of covariate Scientific (e.g., pathophysiological, pharmacologic argument) 10 7,76,92,93, 100, 18, 26, 59, 31, 115
  Previous research (e.g., large RCT) 3 76, 68, 100
  Clinical grounds 2 96, 100
  Indirect evidence 1 59
Personnel Use of clinical experts 2 21, 115
  Blind to results of trials 1 35
Number of covariates/trials needed Small number of covariates 7 92, 95, 100, 18, 26, 31, 94
  Each covariate investigation should be based on an adequate number of studies (e.g., 10 for every moderator) 6 100, 59, 50, 94, 115
  Investigators must report actual number of covariates investigated for reader to determine the potential for false-positives 1 115
Number of outcomes to investigate Restrict investigations to small number of outcomes (e.g., primary) 1 26
  Limit to central question in the analysis 1 94
Interpretation of results of investigations Use caution (4 resources note especially with post hoc testing) 12 100, 18, 29, 31, 85, 16, 20, 23, 25, 61, 32, 35
  Observational only 6 59, 23, 94, 98, 100, 114
  Exploratory or hypothesis generating only 4 32, 100, 40, 94
  Consider confounding between covariates 4 100, 50, 115, 59
  Consider artifactual causes of between-study variation 2 6, 98
  Consider biases (e.g., misclassification, dilution, selection) 2 93, 115
  Look at magnitude of the effect and the 95% CI; not just effect and p-value; consider precision of the subgroup effects (e.g., sample sizes in the studies dictate precision of the subgroup effects) 2 100, 115
  Seek evidence to justify claims of subgroup findings 1 26
  Identify knowledge gaps in the investigations 1 24
  Consider effect of variability within studies 1 19
  Consider if the magnitude is clinically important (i.e., differences in effect between subgroups) 1 100
  Think through causal relationships, especially directionality 1 113
  Use caution with variables grouped after randomization 1 23
  Consider parabolic relationships (i.e., beyond linear regression) 1 115
  Be cautious not to say there is a consistency of effect if no subgroup effects are found 1 115
Descriptive methods Perform a narrative synthesis of these investigations 4 115, 98, 27, 100
  Other: 1. idea webbing, 2. qualitative case descriptions, 3. investigator/methodological/conceptual triangulation 1 98
Use of types of data Aggregate patient data for trial level covariates 4 23, 25, 118, 46
  Only group characteristics derived prior to randomization (e.g., stratifying) 2 23, 46
  Individual patient data for participant level covariates 1 59
  Individual patient data only for all covariates where possible 1 59
  1. 1. The number (N) of resources equals the percentage of resources since we include 101 total resources.