 Research article
 Open access
 Published:
Charting the landscape of graphical displays for metaanalysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis
BMC Medical Research Methodology volume 20, Article number: 26 (2020)
Abstract
Background
Datavisualization methods are essential to explore and communicate metaanalytic data and results. With a large number of novel graphs proposed quite recently, a comprehensive, uptodate overview of available graphing options for metaanalysis is unavailable.
Methods
We applied a multitiered search strategy to find the metaanalytic graphs proposed and introduced so far. We checked more than 150 retrievable textbooks on research synthesis methodology cover to cover, six different software programs regularly used for metaanalysis, and the entire content of two leading journals on research synthesis. In addition, we conducted Google Scholar and Google image searches and citedreference searches of prior reviews of the topic. Retrieved graphs were categorized into a taxonomy encompassing 11 main classes, evaluated according to 24 graphfunctionality features, and individually presented and described with explanatory vignettes.
Results
We ascertained more than 200 different graphs and graph variants used to visualize metaanalytic data. One half of these have accrued within the past 10 years alone. The most prevalent classes were graphs for network metaanalysis (45 displays), graphs showing combined effect(s) only (26), funnel plotlike displays (24), displays showing more than one outcome per study (19), robustness, outlier and influence diagnostics (15), study selection and pvalue based displays (15), and forest plotlike displays (14). The majority of graphs (130, 62.5%) possessed a unique combination of graph features.
Conclusions
The rich and diverse set of available metaanalytic graphs offers a variety of options to display many different aspects of metaanalyses. This comprehensive overview of available graphs allows researchers to make betterinformed decisions on which graphs suit their needs and therefore facilitates using the metaanalytic tool kit of graphs to its full potential. It also constitutes a roadmap for a goaldriven development of further graphical displays for research synthesis.
Background
Data visualization is essential for the exploration of any empirical data and for the communication of statistical results in science in general [1,2,3]. Graphical displays allow to present complex statistical information in a comprehensive way. They are especially suited for the illustration of data comparisons, patterns, trends, and relationships [4].
Metaanalysis is the statistical approach for quantitatively combining and synthesizing the results of two or more empirical studies with identical or comparable research questions [5, 6]. Its principal aim is to critically assess and to summarize the available data answering to a specific research hypothesis. Metaanalysis is highly relevant across all fields of empirical science, which invariably depend on the accumulation of empirical evidence over time, in order to support or to reject hypotheses and theories.
Metaanalytic data and results represent complex data structures. Their interpretation relies on the evaluation and integration of a multitude of statistical information, for example, whole collections of effect sizes, their respective confidence intervals, metaanalytic study weights, the influence of single studies on the summary effect, or associations of effect sizes with study characteristics. For these combined reasons, metaanalysis may be considered a prime candidate domain for the application of datavisualization methods. Visualization may facilitate insight into, and allow drawing firmer conclusions from, complex metaanalytic data.
As a matter of fact, a considerable number of graphical displays is available, which have been designed and introduced with the purpose of visualizing key topics of interest in metaanalysis. These include the estimation of summary effects and their statistical uncertainty, outlier and sensitivity analysis, the exploration of betweenstudy effect heterogeneity, and the assessment of publication bias and related forms of evidence distortion. Some examples of widely known, and most frequently used, options for displaying metaanalytic data are shown in Fig. 1.
Several reviews of general graphing options available for metaanalysis have been published over the years, partly as book chapters [7,8,9], partly as journal articles [10,11,12]. In addition, two reviews about specific graphical displays for network metaanalysis are available [13, 14]. The currently most comprehensive of these general reviews covers about 50 data graphical display variants, with a focus on just four wellknown metaanalytic displays, namely, the forest plot, the funnel plot, the L’Abbé plot, and the Galbraith plot [11].
Data visualization for metaanalysis, as part of metaanalytic methodology, is subject to ongoing research and rapid development. Consequently, a multitude of novel datavisualization methods have been proposed during the past few years, in the fashion of a “graphics explosion” in the field of research synthesis. We estimate that the number of distinct graphical displays, including variants of these, certainly has more than tripled since the year 2005, and likely has almost doubled since the year 2010. Figure 2 shows several examples of such just recently proposed displays for metaanalysis. Hence, a comprehensive, uptodate, general overview, accounting for these most recent developments, is overdue. In addition, previous reviews of graphical displays for metaanalysis were not conducted in a systematic manner, but rather were foremost based on expert opinion and mere awareness of available metaanalytic graphical displays.
Here, we aim to provide an uptodate, and systematically gathered, compilation of available graphical displays, and to categorize and to describe this large and diverse body of datavisualization methods specific to metaanalysis and systematic reviews. Thereby, the present review serves two purposes: first, it represents a comprehensive and structured overview for researchers on available visualization methods and their utility to visualize different aspects of metaanalytic data. Second, it reveals possible gaps in the current landscape of datavisualization methods and therefore, in a way of a roadmap, may supply impulses for future developments of visualization methods in the context of research synthesis approaches.
Methods
Evidence search strategy: graphical displays
For the inclusion of a graphical display in our collection, three criteria had to be fulfilled. First, the graphical display depicts statistical information from empirical metaanalytic data. Second, the graph is not merely an aesthetic variant of a display already included in the corpus of graphical displays (i.e., differences that do not alter the statistical information conveyed). Third, the graph had to be used in the past in a published reference to depict metaanalytic data. For assembling the collection of graphical displays for metaanalytic data, we utilized a systematic, multitiered evidence search strategy, as described in the following.
First, we checked all retrievable monographs on metaanalysis methodology (as detailed in section 2.2) cover to cover. Second, we checked Google Scholar (end of May 2018) for relevant scientific publications, using the search string allintitle: “visualization “OR “display “OR “graph “OR “plot ““metaanalysis “OR “meta analyses“. All retrieved 134 results were screened whether they contained graphical displays eligible for inclusion in our corpus. Third, we conducted a Google Image search (end of May 2018) with the same search string as above and checked all retrieved results. Fourth, we investigated all plotting options for metaanalytic data in three widely used specialized metaanalysis software programs: CMA (Comprehensive MetaAnalysis; version 3) [15], Revman (version 5.3) [16], and Mix 2.0 [17]. Relatedly, three widely used multipurpose statistical software programs were checked: Stata [18], NCSS (version 12) [19], and R [20]. These latter searches included all 102 R packages dedicated to metaanalysis, as listed in the CRAN (Comprehensive R Archive Network) Task View: MetaAnalysis [21]. Fifth, we conducted citedreference searches (end of April 2018) in Google Scholar for the two most comprehensive and most cited review articles on graphical displays for metaanalysis [10, 11] and checked all resulting citing references. Sixth and finally, two authors (MK, MV) handsearched all articles in all issues of the journal Research Synthesis Methods (from 2010 onwards) and screened all abstracts of the journal BMC Medical Research Methodology (from 2001 onwards) containing the search string (metaanalysis OR metaanalyses) AND (display OR plot OR graph* OR visual*) up to May 2018. As of this writing, Research Synthesis Methods is the only journal exclusively dedicated to the methodology of metaanalysis and systematic reviews, and BMC Medical Research Methodology has a long tradition in publishing methodological approaches to metaanalysis and systematic reviews in the health sciences. Both journals regularly publish papers on new datavisualization methods in this context (e.g., [22,23,24,25,26]).
For each graphical display included in our corpus, we ascertained the year in which the graph was first introduced in print or, alternatively, was used in the context of metaanalysis, along with the corresponding published source reference (if applicable and retrievable).
Evidence search strategy: monographs on metaanalysis
We exclusively considered monographs mainly concerned with metaanalytic methods. We therefore excluded textbooks containing merely single chapters on metaanalysis (such as broadranging textbooks on quantitative research methods), as well as books not primarily concerned with metaanalytic methods, but rather describing the results of metaanalytic applications (with the exception of the earliest ones of this type, which typically comprise method development and application conjointly; e.g., [27]). If there was more than one version of a book over the years, we additionally considered any later (revised or expanded) editions. Importantly, we also considered nonEnglish sources, with the only languagebased restriction (i.e., noneligibility) being that the monograph was in a language written in a nonLatin alphabet (Arabic, Chinese, Hebrew, Japanese, Korean, or Russian). Of metaanalytic software manuals, we included early and influential (mostly, commercial) ones, but did not add the manuals and documentations of the now more than 100 R packages related to metaanalysis (see above) to the list of monographs. Finally, we also did not consider journal special issues on metaanalytic methodology. Superficially, these might be deemed as rather similar to edited textbooks. A main difference is, however, that edited books generally are planned ahead (in terms of their focus, scope, coverage, and contributors) to a greater extent, as well as more centrally, than usually is the case for topical issues of journals. One might therefore expect less overlap and redundancy across chapters of edited books than across individual articles of a journal special issue.
The starting point for the literature search for monographs on metaanalysis was an existing corpus of such textbooks held in possession of one author (MV). This corpus was complemented and updated by the following two search strategies (up to May 2018): first, by systematically searching Amazon.com, using the search string allinanchor:“Meta analysis” OR “Meta analyses” site:amazon.com in Google; second, by searching WorldCat, the largest online metacatalogue of library catalogues from all over the globe, for books with the word metaanalysis appearing in the title (in ten languages: English, Danish, Dutch, French, German, Italian, Norwegian, Portuguese, Spanish, and Swedish).
These multitiered search strategies combined resulted in a corpus of 153 textbooks, published between 1980 and 2018, and totalling about 38,000 book pages. The earliest book in the corpus was the pioneering metaanalytic monograph on the efficiency of psychotherapeutic treatment [27], for which Glass and colleagues had developed core methods of metaanalysis, published in the same book. The most recent book in the corpus was a textbook on network metaanalysis published in March 2018 [28]. All books in the final corpus were checked cover to cover for content regarding graphical representations in metaanalysis and systematic reviews, and the relevant information was independently extracted by two authors (MK, MV). This mainly included which graphs were displayed, which graphs were explicitly discussed, and which suggestions were provided regarding their use.
The complete bibliography of textbooks on metaanalytic methodology can be found in Additional File 1. It is, to our knowledge, the most comprehensive one of its kind. For this reason, apart from documenting a main information source for our review work, it constitutes a basic, and readily usable, bibliographic resource of its own, for future scholarly investigations (e.g., research on the history of metaanalysis, or the evolution and propagation of its methodology during the past four decades).
Taxonomy of datavisualization methods in research synthesis (metaanalysis and systematic reviews)
For the sake of clarity, we grouped all displays according to a derived classification system, or taxonomy (Table 1). This classification system was developed by using a bottomup strategy; that is, the graphical displays were classified into categories which, in the end, contained similar plots. The categories were derived, and graphical displays assigned to categories, by all three authors in an iterative, consensual process. There is a plethora of possible ways to construct such classification systems, and the one proposed here is one of many imaginable ones. In fact, some plots might arguably be assigned to more than one category (e.g., a cumulative metaanalysis plot showing the summary effect and its confidence interval for additional studies added over time is a “cumulative metaanalysis and time trends” plot, and at the same time is a “forest plotlike” display). However, for the sake of a clear and structured presentation of the multitude of available graphical displays, a categorization appeared practical.
Within each category, we present different variants of the same display together. Variants of the same display were defined as conveying the same information, but, in addition, graphically showing some further, or alternative, statistical information. Aesthetic differences alone were not counted as distinct variants. Moreover, to avoid redundancies, we did not consider variants of variants. For example, the rainforest plot is a recently proposed variant of the forest plot [23] and, as such, was added to the graph collection. However, variants of the rainforest plot (e.g., a subgroup rainforest plot) were not added to the collection, because the rainforest plot itself is already a variant, and a subgroup forest plot (as a variant of the forest plot) was already included.
On the lowest level of the (twolevel or threelevel) taxonomy, graphical displays are presented in chronological order, using the publication year of the reference in which they were first proposed.
Description (feature analysis) of metaanalytic visualization methods
The full set of metaanalytic displays was described according to a variety of different functionality dimensions by two authors (MK, MV). For this purpose, we iteratively and consensually derived and used 24 functionality features (Table 2). Each graph feature in this functionality space was rated as either present, partly present, or not present (coded on an ordinal scale: 2, 1, 0) for a specific plot or plot variant (in all cases, “not present” was equivalent to “not applicable”). In inconclusive cases, the plot or plot variant shown in Additional File 2 formed the basis for the description. After completion of the initial rating process, cases of rater disagreement were jointly resolved in discussion.
Results
The compilation of graphical displays for metaanalytic or systematicreview data totalled 208 plots. These 208 plots can be further divided into 114 (54.8%) distinct stem displays vs. 94 (45.2%) variants of these. Table 3 lists these graphical displays for metaanalytic data in their entirety, including their categorization (Section 3.2), source reference (if applicable and retrievable), and the year of introduction. Graph vignettes, with complete presentations and short descriptions for all 208 graphical displays, can be found in Additional File 2.
In the following sections, the compilation of the datavisualization methods available for research synthesis approach is ordered, grouped, and described. In Section 3.1, the historical development of these datavisualization methods and their coverage in metaanalytic textbooks over time is illustrated. In Section 3.2, an overview of the graph collection is provided, using the derived categorization system (taxonomy). In Section 3.3, the entirety of graphs is described with respect to a suite of different functionality features (graphfeature analysis).
Historical development of data visualization in metaanalysis
Data visualization in metaanalysis is a topic of ongoing development, and many graphs have been proposed rather recently (Fig. 3). For 156 of all 208 display a year of introduction could be identified with a reasonable degree of confidence (Table 3) and these form the basis for the following historical descriptions of graph development. The number of available metaanalytic graphs has grown exponentially in the past. On average, the number of graphs (N) has grown roughly by 9% annually (N = e^{1.17 + 0.094*(Year – 1975)}). However, as is evident from Fig. 3, approximately starting in 2007, the number of newly proposed graphical displays has steeply increased. Up to 2006, the graph compilation has grown close to linear, with on average 1.5 novel displays introduced each year. Since 2007, this figure now is 6 times larger, with an average of 9 new displays per year.
Looking at the growth of the graph compilation by different graph categories, it is apparent that one – but not exclusively – driving factor for the graphics explosion in metaanalysis in the last decade was the quite large number of novel graphical displays particularly developed for the framework of network metaanalysis (Fig. 4).
However, despite this large number of newly proposed graphs in recent years, most of the graphs actually used in published metaanalyses [22] date back to the very beginnings of metaanalysis in the 1970s and 1980s (e.g., the forest plot was introduced not later than 1982, funnel plots in 1984, the L’Abbé plot in 1987, and the radial plot in 1988).
To assess the popularity of graphs and data visualization in metaanalysis in a novel way, we looked at their implicit and explicit coverage in all textbooks on metaanalytic methodology. A graph was deemed as explicitly covered, if there was a dedicated presentation and explanation of the graph in the book, whereas for implicit coverage it would be sufficient when the graph was used to show metaanalytic data without any graphspecific explanations. Of all 153 books, 20 (13.1%) show a metaanalytic graphical display on their cover. Overall, 95 (62.1%) of the books at least cover one graph explicitly (Fig. 5), while 129 (84.3%) cover one or more plots at least implicitly.
By far the most prevalent explicitly covered displays (Fig. 5) are the funnel plot and its variants (50.3%) and the forest plot and its variants (43.1%), followed by univariate displays illustrating the distribution of effect sizes (16.3%; e.g., boxplots, histograms, or stemandleaf plots), the Galbraith plot (a.k.a. radial plot) and its variants (14.4%), the L’Abbé plot (9.8%), bivariate scatter plots or metaregression plots showing the association of effect sizes with a continuous covariate (9.8%), and the (normal) quantilequantile plot (4.6%).
Explicit coverage has not been constant over time (Fig. 5). While explicit graph coverage in textbooks was rare in the first years of metaanalysis (up to the mid1990s), coverage quickly increased to beyond 50% of all books available up to a specific year since the early 2000s. Descriptively, as indicated by their explicit coverage, the popularity of funnel and forest plots rose in the second half of the 1990s. Coverage then quickly increased from 15.8 and 10.5% (of all books available up to the year 1995) to 43.5 and 31.8% (of all books available in 2005), and to 48.6 and 38.6% (in 2015) for the funnel and forest plot, respectively. Therefore, the number of books covering these two iconic displays relatively grew at a much faster rate than the book corpus itself, illustrating their dissemination and propagation. The proportion of books explicitly covering any of the other most prevalent displays stayed rather constant or even declined; that is, the number of books covering these other plots relatively grew at a similar (or even slower) rate than the book corpus itself.
Compared to explicit coverage, by also considering implicit coverage, the prevalence of forest plots increased greatly from 43.1 to 62.7%, whereas the prevalence of funnel plots essentially stayed the same (50.3% vs. 52.3%). This indicates that funnel plots were hardly displayed in textbooks without being explained and covered explicitly at the same time, whereas this seemed not to be the case for forest plots. Implicit coverage was clearly more common than explicit coverage for bivariate displays of effect sizes and covariates (i.e., scatter plots: 26.1%) and univariate displays of effectsize distributions (e.g., histograms: 21.6%), which is less surprising, when considering their more general, not genuinely metaanalytic, nature.
A taxonomy of available metaanalytic graphs
To arrive at a structured and ordered presentation of the graph compilation, each graph was categorized into one of 11 distinct graph categories (see Methods section). The most prevalent categories were network metaanalysis (45 displays), combined effect(s) only (26), and funnel plotlike (24), followed by more than one outcome per study (19), robustness, outlier, and influence diagnostics (15), study selection and pvalue based (15), forest plotlike (14), effectsize distribution (13), study or subgroup characteristics (13), continuous effect moderators (12), and cumulative metaanalysis and time trends (12). An overview and summary of the graph compilation using these graph categories is given in the following. For presentations and brief descriptions of all the 208 graphical displays retrieved, see Additional File 2.
Forest plotlike graphical displays
The forest plot is probably the most iconic of genuine displays for metaanalytic data. Key characteristics are the depiction of summary and studylevel effects, as well as interval estimates and a clear labelling of each study. Showing study effect sizes and their confidence intervals in a confidence interval plot (a.k.a. caterpillar plot) dates back at least to 1978 [128], while the first actual forest plot additionally depicting a metaanalytic summary estimate was first used not later than 1982 (for a historical overview, see [129]). Classic variations of the forest plot are the subgroup forest plot and the summary forest plot, showing and comparing additional or exclusive summary estimates of groups of studies. For Bayesian metaanalysis, a forest plot variant depicting posterior distributions or posterior intervals (a.k.a. shrinkage plots) for each study has regularly been used. An early, nowadays seldomused, forest plotlike graph is the odd man out plot, visualizing effectsize areas for which at least a specified number of study confidence intervals overlap [33]. Forest plots with vertical lines indicating userspecified limits of equivalence have been used [30], which allow drawing conclusions regarding noninferiority and equivalence testing on the study and summaryeffect level [130]. Examples of recently proposed variants of the forest plot are the rainforest plot and the thick forest plot, which allocate more visual emphasis on those study effects which have been estimated with higher precision [23]. A novel, rather atypical, forest plotlike display is the fuzzy number plot, which shows study and summary effects and respective interval estimates using fuzzy numbers and which has specifically been proposed for largescale metaanalyses with numerous studies, for which traditional forest plots are less suited [34].
Funnel plotlike graphical displays
Apart from the forest plot, the funnel plot is probably the most widely known genuine metaanalytic plot. Funnel plotlike displays can be seen as specialized scatter plots showing effect sizes (or functions thereof) on one axis and the studies’ standard error (or functions thereof) on the other axis. Typical plots in this category are the eponymous funnel plot [35] and the Galbraith plot (a.k.a. radial plot), essentially conveying the same information [45].
Remarkably, the funnel plot is the display in the graph compilation with the most distinct variants (15). Initially proposed for the assessment of potential publication bias, indicated through smallstudy effects, early variants include visual depictions of statistical methods concerned with publication bias, e.g., by showing studies imputed by the trimandfill algorithm [38], or fitted lines of regression tests (e.g., Egger regression test [36]). Specifically, in the last decade a large number of variants in the form of different contourenhanced funnel plots have been introduced. The significance contourenhanced funnel plot [39] allows incorporating information about the nominal (statistical) significance of studies into funnel plot assessment. Additional evidence contours [40] show the robustness of the nominal significance (or lack thereof) of the metaanalytic summary effect and the robustness of the magnitude of the estimated betweenstudy heterogeneity with respect to a hypothetical additionally observed study. Further variants show the effect of a hypothetical additional study on the width, or upper and lower bounds, of the summary effect’s confidence interval [41], or on the magnitude of the summary effect [42].
Graphical displays for continuous effect moderators
One key aspect of metaanalysis is to explore the role of study covariates (or moderators) on the metaanalytic summary effect. Not surprisingly, scatter plots of study effect sizes and metaregression plots were one of the first plots used in published metaanalyses [6]. Modern metaregression plots include differentlysized symbols with respect to study precision or the metaanalytic study weight, and a fitted line and confidence bands for the estimated metaanalytic summary effect. Novel developments within this category came along with methodological advancements. A generalization of the trimandfill algorithm to metaregression has been proposed, along with visualization of the corresponding corrected line of fit [49]. Machinelearning methods have recently been applied to metaanalysis and have led to the visualization of (meta)regression trees [50] and illustrations of functional associations of single predictors with outcomes in metaanalytic random forests, using partial dependence plots [51].
Graphical displays for robustness, outlier, and influence diagnostics
The assessment of the sensitivity of metaanalytic results is another important field of application of metaanalytic graphs. One of the first genuine diagnostic plots has been the τ^{2} sensitivity plot [53], showing the trajectory of the metaanalytic summary effect for increasing values of τ^{2} (i.e., from the fixedeffect model, implying τ^{2} = 0, to a next to unweightedaverage model for very large τ^{2} values). Graphs showing the metaanalytic summary effect for single studies being left out have been proposed as line charts [37] and, more commonly, as leaveoneout sensitivity forest plots [54]. The Baujat plot is a genuine metaanalytic plot to detect outliers and influence points by plotting the change of the summary effect for systematically leaving out one study at a time against the contribution of this study to the betweenstudy heterogeneity statistic Q [55]. Widely known diagnostic plots for detecting outliers, leverage, and influence points in regression analysis have been proposed in the context of metaanalysis and metaregression models in particular [57]. These displays include, among others, scatter and line plots of studentized deleted residuals, Cook’s distance values, and hat values.
The GOSH (Graphical Display of Study Heterogeneity) plot [59, 131] shows the results of combinatorial metaanalyses; i.e., metaanalyses of either all 2^{k} – 1 possible subsets of k studies, or a random sample of these. For each combination, the resulting metaanalytic summary effect and the I^{2} value are shown in a scatter plot, and study subsets including a certain study can be highlighted, thus revealing their influence on the summary effect or the estimated betweenstudy heterogeneity. Forward plots accompany newly proposed methods to detect outlier studies, which monitor the effect on metaanalytic estimates by iteratively adding individual studies to increasingly heterogeneous sets of studies [61].
Graphical displays for cumulative metaanalysis and time trends
Questions regarding the development of evidence over time are typical for research synthesis. Time series of published effect sizes were displayed not later than in the mid1980s [35]. Quality control charts, namely, x bar charts and CUMSUM (cumulative sum) charts, were proposed to identify changes in temporal trends and unusual observations in effectsize timeseries data [63]. Cumulative metaanalysis plots show the development of the metaanalytic summary effect point and interval estimate over time in a classic forest plotlike display [64]. Sequential monitoring boundaries have been used and displayed in cumulative metaanalysis plots to assess whether additional evidence is needed [65]. While graphs showing the development of the metaanalytic summary effect have been used predominantly, variants showing metaanalytic heterogeneity statistics over time have been proposed as well [67]. In addition, the trajectory of evidence over time has been shown, using cumulative Bayes factors [68].
Graphical displays for effectsize distribution
Standard statistical graphs have primarily been used for the visualization of observed univariate effectsize distributions. These include histograms, boxplots, dot plots, stemandleaf displays, and kernel density plots. Weighted variants exist for histograms, boxplots, and density plots, to readily incorporate different precision and therefore different metaanalytic weights of studies. The (normal) quantilequantile plot has been proposed as a suitable display to check statistical assumptions in the context of metaanalysis, including normality and homogeneity of effects and absence of publication bias [72].
Graphical displays for study or subgroup characteristics
Study characteristics other than effect sizes or precision have been displayed using standard statistical graphs. For continuous characteristics, the same plots have been used as to show effectsize distributions (see above), and, to visualize categorical study characteristics, bar or pie charts have been repeatedly used. Genuine metaanalytic plots within this category are the Cochrane risk of bias plot and the risk of bias summary plot [73], visualizing the overall and studylevel risk of bias on several dimensions. The PRISMA (Preferred Reporting Items for Systematic Reviews and MetaAnalyses) flow chart [74] informs about literature search and study inclusion and exclusion details in the course of systematic reviews or metaanalyses. The veritas plot is a tool to compare several studies or study subgroups with respect to five different dimensions of relevance arranged in a pentagon (such as betweenstudy heterogeneity, publication bias, evidence and quality gradings) [75]. Specialized displays to visualize the qualitative evidence and characteristics of a potentially diverse set of studies are the harvest plot [24], the error matrix display [76], the effectdirection plot [77], and the evidencemap bubble display [78].
Graphical displays for more than one outcome per study (multivariate)
Displays for more than one outcome per study were predominantly developed for visualizing two potentially dependent outcomes per study. Bivariate metaanalyses of two outcomes per study have been visualized with bivariate scatter plots no later than in the early 1990s, including a metaanalytic summary effect and confidence ellipses on the study or summary level [71]. A novel variant of these multivariate displays additionally shows the studylevel confidence intervals in both outcomes simultaneously in a socalled multivariate crosshairs plot [83].
Several multivariate displays were proposed for the visualization of metaanalyses of dichotomous outcomes. The L’Abbé plot is a genuine and classic metaanalytic plot, showing for each study the risk for an event in the treatment and control group in a scatter plot [80]. Variants with superimposed effect contours allow depicting studylevel results and the metaanalytic summary effect either as risk ratio, odds ratio, or risk difference [81].
ROC (Receiver Operating Characteristic) plots and their variants are used to simultaneously display the specificity and sensitivity and the ROC curve on the study or the summary level [84]. Crosshairs plots were proposed as an enhancement, showing the studylevel confidence intervals for sensitivity and specificity [85]. For studies reporting sensitivity and specificity values for more than one threshold, recently proposed methods include visualizations of the estimated metaanalytic summary and studylevel sensitivities and specificities for different diagnostic thresholds [88].
The Olliaro display was proposed to visualize absolute, as well as relative, effects of a treatment at the same time, showing the absolute failure rate of a treatment on one axis and the difference of failure rates with comparator treatments on the other axis [87].
Graphical displays for combined effect(s) only
As a rather heterogeneous category, displays exclusively showing metaanalytic summary or subgroup effects visualize quite different aspects of metaanalyses. The perhaps first genuine metaanalytic display visualized a single metaanalytic summary effect size by two overlapping normal distributions in 1976 [5]. Similarly, Hattie visualized the magnitude of single summary effects with a barometertype infographic [92]. Fishbone diagrams [95] and evidence flowers [96] have recently been proposed as infographics to enable an overview of several summary findings concurrently (e.g., for different endpoints of interest).
Other typical graphs in this category show distributionlike displays of metaanalytic key parameters. Likelihood functions of metaanalytic parameters, prior, posterior, and posterior predictive distributions have been used to summarize Bayesian metaanalytic results. Likelihood functions or posterior densities for two parameters at the same time (predominantly, the summary effect and heterogeneity estimates) have been visualized, using twodimensional contour plots or threedimensional surface plots.
Summary survival curves have been displayed in metaanalyses of timetoevent data [82], whereas the summary results of metaanalyses of path and structural equation models have been visualized via path diagrams [89] not later than in the early 1990s.
Finally, there are several graphs for the depiction of metaanalyses of genetic data, displaying a large number of summary effects for different gene loci at the same time. Adopted displays from visualizing the results of primary studies include the metaanalytic Manhattan and Miami plots, the regional association plot, the volcano plot, and (summary) heat maps of gene expressions. A display genuinely proposed for metaanalysis of genetic data is the circos plot which shows metaanalytically derived summary estimates of downregulated or upregulated gene expressions for certain diseases in a circular display [94].
Graphical displays for study selection and p values
The majority of displays based on the p value of studies are related to methods for publicationbias assessment. A contourline plot has been used to illustrate the sensitivity of the summary result to the parameters used in a selection model [99]. The test of excess significance [100] has been supplemented by a sensitivity display, showing the trajectory of the test result for different significance thresholds α. Formann used plots of truncated normal distributions to visualize the likely region of unpublished effects due to publication bias [102]. The caliper test display shows the distribution of p values associated with test statistics and highlights an abundance of justsignificant results in a specific histogram [101]. Similarly, the pcurve display shows peculiarities of distributions of p values in the significance region and allows assessing the likely presence of phacking and the evidential value of a set of studies with a specific line plot [104]. The PM display was proposed for genetic data, showing the p values of studies on one axis and the posterior probability that the effects exist in each study on the other axis [103].
A few further displays exist which focus on the presentation of study p values. One early account is the SchwederSpjøtvoll display introduced in 1982, essentially showing the empirical distribution function of observed p values of a set of studies [97]. A recently proposed display based on p values is the albatross plot, showing the p values and sample sizes of studies in a scatter plotlike display. In addition, effectsize contours are overlain, showing for a specific effect size the resulting p values for all possible sample sizes, thereby allowing to assess the probable magnitude of the underlying effect, as well as an excess of betweenstudy heterogeneity [107].
Graphical displays for network metaanalysis
Graphs specifically proposed for network or mixedtreatment comparison metaanalysis constitute the most recent, and already largest, category in the graph compilation. Basically, within this category four main types of network graphs can be distinguished.
First, there are graphs, showing which treatments are directly compared in the network. Examples for this type of graphs are network graphs, with vertices visualizing treatments and edges visualizing the number observed comparisons [108], and the flowofevidence graph, showing in a network graph for a certain treatment comparison which direct and indirect paths contribute to the network estimate [109]. Threedimensional network plots, showing comparisonspecific covariate values on a third axis within a network graph have recently been proposed [110].
Second, for the presentation of the results from a network metaanalysis, forest plots [111, 112] and funnel plots [14] have been adapted and enhanced for depicting network results on the treatmentcontrast level.
Third, several displays exist for the visualization of estimated treatment rankings. Rankograms show for each treatment the estimated (absolute or cumulative) probability for each treatment ranking [119]. For two outcomes, a bivariate ranking scatter plot shows their ranking metrics simultaneously for each treatment [14]. Also, rank heat plots were proposed to compare treatment rankings on more than one outcome in a circular heat display [124]. Hasse diagrams were introduced to visualize rankings of treatments in a network graph with respect to more than one outcome, using partial ordering of treatments [125].
Fourth, there are a number of graphs which primarily visualize inconsistencies between directly and indirectly estimated treatment comparisons (analogously to effect heterogeneity in directevidence, univariate metaanalysis), as well as the contribution of direct and indirect treatment comparisons to the network estimates (analogously to study weights in directevidence, univariate metaanalysis). The network indirect path decomposition forest plot shows the contribution of indirect evidence and the resulting summary effects, considering only direct evidence, as compared to indirect evidence [26]. The netheat plot visualizes the contribution of different direct and indirect treatment comparisons, as well as inconsistencies related to specific comparisons in a matrix display [25]. Recently, several displays for network metaregression were proposed, visualizing the contribution of single studies and ranges of covariate values to the network metaregression estimates [121].
Description of metaanalytic graphical displays by their functionality (feature analysis)
In the following, the inventory of datavisualization methods in metaanalysis is described with respect to the 24 graphfunctionality features (Table 2; Fig. 6). Feature assessments for the entire collection of graphs can be found in Additional File 3.
Whereas all graphical displays are suitable to display smallsized metaanalyses (say, 10 studies), 76.9 and 49.5% of graphical displays remain fully suited for mediumsized (say, 50 studies) and largesized metaanalyses (say hundreds of studies), respectively. The most common further (fully present) functionality features were depiction of summary effect(s) (51.0% of all displays), depiction of individual study effect sizes (38.0%), depiction of further study features (37.0%), and depiction of study weight/sample size/standard error (25.0%).
Features that allow assessing the trustworthiness, sensitivity, and robustness of metaanalytic results were less common: 14.9% of all displays are suitable to assess publication bias and other forms of biases (7.7% partly), 13.0% are suitable to assess the robustness of the summary effect (4.8% partly), 4.3% the robustness of heterogeneity summary effects (0.5% partly), 6.2% are suitable to assess distributional assumptions of effect sizes (8.2% partly), and 6.2% are fully suited to identify influential studies (15.4% are partly suited).
Despite the prevalence of displays which depict study and summary effects, those which also show confidence intervals of effect sizes (10.1%) and confidence intervals of summary effects (22.6%) were less frequent. The likelihood or posterior distribution of metaanalytic parameter estimates was conveyed by 4.8% of all graphs. In addition, while nearly 40% of graphs showed study effect sizes, only 13.9% allowed identifying studies with study identifiers; 10.6% allowed for a categorical classification of studylevel significance (i.e., significant vs. not), and 3.8% (7.7% partly) for a continuous classification. Of all displays, 10.1% show more than one effect size per study.
Remarkably, despite heterogeneity being one of the key topics of metaanalysis, only 5.3% of displays visualize summary heterogeneity statistics, and 7.2% displays were suited to assess betweenstudy heterogeneity (19.2% of displays were partly suited). Taken together, this suggests that surprisingly few specialized plots for heterogeneity assessment are available. For the explanation of betweenstudy heterogeneity, 22.1% of all displays allow examining the association of study effect sizes with categorical (10.6%) and continuous (8.2, 5.3% partly) study covariates, while 5.3% depict time trends in metaanalytic estimates (1% partly).
On average, graphs had 5.4 functionality features fully present (Mdn = 5, SD = 1.7, Min = 2, Max = 11) and 6.6 at least partly present (Mdn = 6, SD = 2.6, Min = 3, Max = 15). The graphical displays with the most features fully present, and therefore potentially conveying the most information, were a Galbraith plot variant, which additionally showed subgroup information (11 features, 15 at least partly), the subgroup forest plot (10 features, 14 at least partly), and the rainforest plot, a novel forest plot variant (10 features, 14 at least partly).
Of all 208 plots or plot variants in the compilation, 130 (62.5%) possessed a unique combination of graph features. When only fully present features were considered and compared to features partly present or not present combined, still 116 graphs (55.8%) showed a combination of features that no other graph in the compilation possessed. Arguably, this further attests to the heterogeneous, nonredundant, and specialized nature of the landscape of graphs available for metaanalysis.
Of particular interest is that the presence or absence of functionality features in a specific graph is not random (Fig. 7). Exploring features that often or seldom occur together in the same graph might help identifying potential gaps in the current graph inventory for metaanalysis and may serve as a roadmap for future development of graphical displays for research synthesis.
There is a strong negative association of a graph showing, on the one hand, summary outcome interval estimates, individual studylevel effects, studyeffect interval estimates, study weights, or study identifiers, and, on the other hand, being suitable for larger or mediumsized metaanalyses. Although naturally hard to combine, displays for mediumsized to largesized metaanalyses, which still allow identifying each study and its effects, apparently are rare and thus a fruitful avenue for future graph development.
Graphs suitable for the assessment of publication bias or other forms of bias tend to show neither a metaanalytic summary effect nor effectsize confidence intervals, and seldom are suited for showing more than one effect size per study. In addition, displays showing more than one effect size per study (multivariate metaanalysis), influential or outlier studies, and displays suitable for the assessment of distributional assumptions of effect sizes, tend to show no metaanalytic summary outcomes. Moreover, showing some kind of metaanalytic summary estimate (summary effect estimate, heterogeneity summary statistics) is negatively related to displaying any additional study features. The most prevalent combinations of graph features are as expected: graphs showing a summary effect tend to show a confidence interval (or some other form of interval estimator) as well; graphs suitable for mediumsized metaanalyses are often suited for largesized metaanalysis, too (e.g., by showing only summary, not studylevel, estimates); and graphs often allow to depict nominal statistical significance on the study level categorically, as well as continuously at the same time.
Discussion
We collected, structured, classified, and described the landscape of metaanalytic graphs in unprecedented scope and detail. The introduction of new graphical displays for research synthesis (metaanalysis and systematics reviews) has grown at a remarkable pace: all in all, we collected 208 distinct graphs and graph variations. The availability of such a large number of statistical graphs for metaanalysis may well come as a surprise for many. Previously available general reviews on graphs in metaanalysis covered at most one quarter the size of the present compilation. One driving factor of the graphics explosion in the field of metaanalysis in the mid2000s has been the continuing development of new displays for network metaanalysis. New plotting options have been added recently, however, for practically any other type of metaanalysis as well. Metaanalytic graphs and their variants possess a rich and diverse set of graph features. Thus, the present graph compilation contains a large number of diverse and specialized displays for numerous aspects of metaanalysis.
However, despite the availability and potential of graphical displays for exploring and communicating metaanalytic results, their usage in published metaanalyses was, and still is, rather limited. In an early review, Light, Singer, and Willet reported that for 74 metaanalyses published in Psychological Bulletin between 1985 and 1991, only 19% included graphical displays [7]. This proportion increased to 52% among 60 metaanalyses published in the same journal from 2000 to 2005 [9]. In both these studies, the majority of graphical displays observed were univariate depictions of effectsize distributions (e.g., histograms). Schild and Voracek systematically reviewed graph use in metaanalyses published in top journals in medicine, psychology, and business research over 30 years (1981 to 2011) [22]. Of the total of 993 metaanalyses inspected, only 50% contained any graphical display to communicate their results. The single dominant display was the forest plot; hardly any other graphs were used.
Also, graphical displays are barely covered in existing published guidelines. In APAissued MARS (MetaAnalysis Reporting Standards) [132], graphical displays are not mentioned at all. In PRIMSA, solely the optional use of forest plots for visualizing individual study results is recommended [74]. Relatedly, given the evidence for a graphics explosion in the domain of metaanalysis since the mid2000s, it is perhaps ironic to observe that, while the first two editions (1994 and 2009) of a major textbook resource of research synthesis methodology each had included a dedicated chapter on visual displays for metaanalysis [7, 9], the most recent edition thereof (2019) has none such [133].
We observed consistent results when examining graph use in metaanalysis by looking at both implicit and explicit graph coverage in textbooks. In the available textbooks on metaanalytic methodology (Additional File 1), the forest plot and the funnel plot once more were the most often covered displays, and often the only ones.
Hence, despite the diverse and large number of available graphical displays, it seems that only very few of these are regularly applied in scientific practice. Existing reporting guidelines clearly fail to explicitly encourage their use. The existing repertoire of visualization methods is thus likely not used to its full potential in exploring and presenting metaanalytic results.
As to why many graphical displays are not used on a common basis by metaanalysts, we highlight three possible reasons: first, many of the available graphical displays and their uses might be widely unknown. Second, researchers who publish metaanalyses, as well as editors and reviewers, might not see the additional benefits in using graphical displays towards the goal of communicating metaanalytic results optimally. Third, userfriendly software for creating graphical displays might not be readily available. We hope that the comprehensive survey of currently available graphical displays at hand might successfully counter the first two of these inhibitory reasons.
Reviews on software availability for graphing metaanalytic data have been conducted elsewhere ([22, 134]) and are beyond the intended scope of our account. In short, most of the widely used classic metaanalytic software packages primarily allow to create traditional metaanalytic displays, namely, forest plots (CMA [15], Revman [16], Mix 2.0 [17]), funnel plots (CMA [15], Revman [16], Mix 2.0 [17]), radial plots (Mix 2.0 [17]), L’Abbé plots (Mix 2.0 [17]), and metaregression plots (CMA [15], Mix 2.0 [17]). Many of the more recently proposed and potentially less known graphs can only be created using syntaxbased statistical software and software packages (e.g., R [20] or Stata [18]). Userfriendly statistical software solutions for a large number of the graphs and graph variants described here currently are unavailable.
The primary aim of our account is to give an overview of available graph options for metaanalysis. However, because of the large number of graphs found, it was not feasible to discuss each and every display in more detail other than in the form of a vignette (Additional File 2). For a more elaborated and focused discussion, as well as for suggestions on the use of the most widely known displays for univariate metaanalysis (namely, the forest, funnel, L’Abbé, and Galbraith plots), we recommend to refer to [11]. Likewise, for a focused treatment of a number of graphical displays for network metaanalysis, we refer to [13].
Although much thought and iterative effort was put into the derivation of a useful taxonomy, our choice is only one of many imaginable ones, and thus the membership of a plot to a certain category in this taxonomy should not be overstated. For the description of plots, we used a bottomup derived list of graph features evaluated by two expert raters (Additional File 3). These ratings should be taken as a crude guide as to which plot in principle conveys which statistical information. The ratings are not intended to compete with, or replace, original empirical research on the visual perception of specific statistical information from different metaanalytic graphs (e.g., [10]; for forest plot variants: [23]).
Data visualization in metaanalysis is a field of long tradition and swiftly ongoing development. Typical feature spaces of currently available graphs still show gaps and thus leave ample room for novel visualization methods. Two examples for such gaps identified here are, firstly, graphs allowing to depict more than two effect sizes per study (or, more generally, per level in multilevel metaanalysis), and secondly, suitable displays for mediumsized to largesized metaanalyses, which nevertheless allow to depict studylevel effects and study identifiers. Therefore, despite the large number of already available graphs, in all likelihood the trend of new developments will continue in the foreseeable future, in tandem with advancements in metaanalytic methodology.
There arguably are a number of potentially useful, but currently underused, or at least underreported, graphs. One area of such underreported graphs are most likely diagnostic graphs, which assess the robustness and sensitivity of metaanalytic results to study inclusions and common methodological decisions (e.g., fixedeffect vs. randomeffects model). Given the possibility of providing additional supplemental files online, there remain few, if any, reasons on the side of article authors for not providing more such diagnostic plots, in order to beneficially increase the transparency of their metaanalytic reporting [135].
Conclusion
The present overview took stock of a total of 208 retrievable distinct graphical displays, which so far have been proposed and used for exploring and communicating metaanalytic results. We hope this resource will contribute to utilizing the available tool kit of datavisualization methods in metaanalysis to its full potential and enable researchers to make betterinformed decisions on which graphs to consider for presenting their metaanalytic data. Likewise, the current overview may well constitute a roadmap for goaldriven development of further graphical displays for research synthesis.
Availability of data and materials
All data generated and analysed during this study either are included in this article and its supplementary information files and/or are available at the Open Science Framework repository, https://osf.io/gkpe5/.
Abbreviations
 CMA:

Comprehensive MetaAnalysis
 CRAN:

Comprehensive R Archive Network
 CUMSUM:

Cumulative Sum
 GOSH:

Graphical Display of Study Heterogeneity
 MARS:

MetaAnalysis Reporting Standards
 PRISMA:

Preferred Reporting Items for Systematic Reviews and MetaAnalyses
 ROC:

Receiver Operating Characteristic
References
Tukey JW. Exploratory data analysis. Reading: AddisonWesley; 1977.
Tufte ER. The visual display of quantitative information. 2nd ed. Cheshire: Graphics Press; 2001.
Krause A, O’Connell M. A picture is worth a thousand tables: graphics in life sciences. New York: Springer; 2012.
Chen CH, Härdle WK, Unwin A. Handbook of data visualization. Berlin: Springer; 2008.
Glass GV. Primary, secondary, and metaanalysis of research. Educ Res. 1976;5:3–8.
Glass GV. Integrating findings: the metaanalysis of research. Rev Educ Res. 1977;5:351–79.
Light RJ, Singer JD, Willet JB. The visual presentation and interpretation of metaanalyses. In: Cooper H, Hedges LV, editors. The handbook of research synthesis. New York: Russell Sage; 1994. p. 439–53.
DuMouchel W, Normand SL. Computermodeling and graphical strategies for metaanalysis. In: Stangl DK, Berry DA, editors. Metaanalysis in medicine and health policy. New York: Marcel Dekker; 2000. p. 127–78.
Borman GD, Grigg JA. Visual and narrative interpretation. In: Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and metaanalysis. 2nd ed. New York: Russell Sage; 2009. p. 497–519.
Bax L, Ikeda N, Fukui N, Yaju Y, Tsuruta H, Moons KG. More than numbers: the power of graphs in metaanalysis. Am J Epidemiol. 2009;169:249–55.
AnzuresCabrera J, Higgins JP. Graphical displays for metaanalysis: an overview with suggestions for practice. Res Synth Methods. 2010;1:66–80.
Kiran A, Crespillo AP, Rahimi K. Graphics and statistics for cardiology: data visualisation for metaanalysis. Heart. 2017;103:19–23.
Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from multipletreatment metaanalysis: an overview and tutorial. J Clin Epidemiol. 2011;64:163–71.
Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network metaanalysis in STATA. PLoS One. 2013;8:e76654.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to metaanalysis. Chichester: Wiley; 2009.
The Cochrane Collaboration. Review Manager (RevMan) (Version 5.3) [Computer program]. Copenhagen: The Nordic Cochrane Centre; 2014.
Bax L, Yu LM, Ikeda N, Tsuruta H, Moons KG. Development and validation of MIX: comprehensive free software for metaanalysis of causal research data. BMC Med Res Methodol. 2006;6:50.
Palmer TM, Sterne JAC, editors. Metaanalysis in Stata: an updated collection from the Stata journal. 2nd ed. College Station: Stata Press; 2015.
NCSS 12 statistical software. NCSS, LLC. Kaysville, Utah, USA, ncss.com/software/ncss; 2018.
R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2018. https://www.Rproject.org/
Dewey M. CRAN task view: metaanalysis. 2018. https://cran.rproject.org/web/views/MetaAnalysis.html. Accessed 14 May 2018.
Schild AH, Voracek M. Less is less: a systematic review of graph use in metaanalyses. Res Synth Methods. 2013;4:209–19.
Schild AH, Voracek M. Finding your way out of the forest without a trail of bread crumbs: development and evaluation of two novel displays of forest plots. Res Synth Methods. 2015;6:74–86.
Ogilvie D, Fayter D, Petticrew M, Sowden A, Thomas S, Whitehead M, Worthy G. The harvest plot: a method for synthesising evidence about the differential effects of interventions. BMC Med Res Methodol. 2008;8:1.
Krahn U, Binder H, König J. A graphical tool for locating inconsistency in network metaanalyses. BMC Med Res Methodol. 2013;13:1.
Krahn U, Binder H, König J. Visualizing inconsistency in network metaanalysis by independent path decomposition. BMC Med Res Methodol. 2014;1:131.
Smith ML, Glass GV, Miller TI. The benefits of psychotherapy. Baltimore: Johns Hopkins University Press; 1980.
Dias S, Ades AE, Welton NJ, Jansen JP, Sutton AJ. Network metaanalysis for decisionmaking. Chichester: WileyBlackwell; 2018.
Barrowman NJ, Myers RA. Raindrop plots: a new way to display collections of likelihoods and distributions. Am Stat. 2003;57:268–74.
Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR, Abrams KR. Evidencebased sample size calculations based upon updated metaanalysis. Stat Med. 2007;26:2479–500.
Yang G, Xie MG. gmeta [R software package]. 2010. https://CRAN.Rproject.org/package=gmeta
Nakagawa S, Noble DW, Senior AM, Lagisz M. Metaevaluation of metaanalysis: ten appraisal questions for biologists. BMC Biol. 2017;15:18.
Walker AM, MartinMoreno JM, Artalejo FR. Odd man out: a graphical approach to metaanalysis. Am J Public Health. 1988;78:961–6.
Thompson CG. Graphing effects as fuzzy numbers in metaanalysis. J Mod Appl Stat Methods. 2016;15:49.
Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge: Harvard University Press; 1984.
Egger M, Smith GD, Schneider M, Minder C. Bias in metaanalysis detected by a simple, graphical test. Br Med J. 1997;315:629–34.
Elvik R. Evaluating the statistical conclusion validity of weighted mean results in metaanalysis by analysing funnel graph diagrams. Accid Anal Prev. 1998;30:255–66.
Duval S, Tweedie R. Trim and fill: a simple funnelplotbased method of testing and adjusting for publication bias in metaanalysis. Biometrics. 2000;56:455–63.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contourenhanced metaanalysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61:991–6.
Langan D, Higgins JP, Gregory W, Sutton AJ. Graphical augmentations to the funnel plot to assess the impact of additional evidence on a metaanalysis. J Clin Epidemiol. 2012;65:511–9.
Crowther MJ, Langan D, Sutton AJ. Graphical augmentations to the funnel plot to assess the impact of additional evidence on a metaanalysis. Stata J. 2012;12:605–22.
Chevance A, Schuster T, Steele R, Ternès N, Platt RW. Contour plot assessment of existing metaanalyses confirms robust association of statin use and acute kidney injury risk. J Clin Epidemiol. 2015;68:1138–43.
Radua J. metansue [R package software]. 2015. https://CRAN.Rproject.org/package=metansue
Bowden J, Jackson C. Weighing evidence “steampunk” style via the MetaAnalyser. Am Stat. 2016;70:385–94.
Galbraith RF. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat Med. 1988;7:889–94.
Gee T. Capturing study influence: the concept of ‘gravity’ in metaanalysis. Aust Couns Res J. 2005;1:52–75.
Barendregt JJ, Doi S. MetaXL User Guide. Sunrise Beach: EpiGear International; 2016.
Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet. 1998;351:123–7.
Weinhandl ED, Duval S. Generalization of trim and fill for application in metaregression. Res Synth Methods. 2012;3:51–67.
Dusseldorp E, Van Genugten L, van Buuren S, Verheijden MW, van Empelen P. Combinations of techniques that effectively change health behavior: evidence from MetaCART analysis. Health Psychol. 2014;33:1530–40.
van Lissa CJ. metaforest [R package software]. 2017. S
Wang XV, Cole B, Bonetti M, Gelber RD. MetaSTEPP: subpopulation treatment effect pattern plot for individual patient data metaanalysis. Stat Med. 2016;35:3704–16.
Thompson SG. Controversies in metaanalysis: the case of the trials of serum cholesterol reduction. Stat Methods Med Res. 1993;2:173–92.
Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for metaanalysis in medical research. Chichester: Wiley; 2000.
Baujat B, Mahé C, Pignon JP, Hill C. A graphical method for exploring heterogeneity in metaanalyses: application to a metaanalysis of 65 trials. Stat Med. 2002;21:2641–52.
Barrowman NJ, Fang M, Sampson M, Moher D. Identifying null metaanalyses that are ripe for updating. BMC Med Res Methodol. 2003;3:13.
Viechtbauer W, Cheung MWL. Outlier and influence diagnostics for metaanalysis. Res Synth Methods. 2010;1:112–25.
Poorolajal J, Mahmoodi M, Majdzadeh R, Fotouhi A. Metaplot: a novel stata graph for assessing heterogeneity at a glance. Iran J Public Health. 2010;39:102–4.
Olkin I, Dahabreh IJ, Trikalinos TA. GOSH: a graphical display of study heterogeneity. Res Synth Methods. 2012;3:214–23.
Beath KJ. A finite mixture method for outlier detection and robustness in metaanalysis. Res Synth Methods. 2014;5:285–93.
Mavridis D, Moustaki I, Wall M, Salanti G. Detecting outlying studies in metaregression models using a forward search algorithm. Res Synth Methods. 2017;8:199–211.
Mathur MB, VanderWeele TJ. Sensitivity analysis for unmeasured confounding in metaanalyses. 2017. https://arxiv.org/pdf/1707.09076.pdf. Accessed 31 July 2017.
Kulinskaya E, Koricheva J. Use of quality control charts for detection of outliers and temporal trends in cumulative metaanalysis. Res Synth Methods. 2010;1:297–307.
Lau J, Antman EM, JimenezSilva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative metaanalysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992;327:248–54.
Pogue JM, Yusuf S. Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative metaanalysis. Control Clin Trials. 1997;18:580–93.
Ioannidis JP, ContopoulosIoannidis DG, Lau J. Recursive cumulative metaanalysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. J Clin Epidemiol. 1999;52:281–91.
Villanueva EV, Zavarsek S. Evaluating heterogeneity in cumulative metaanalyses. BMC Med Res Methodol. 2004;4:18.
Scheibehenne B, Jamil T, Wagenmakers EJ. Bayesian evidence synthesis can reconcile seemingly inconsistent results: the case of hotel towel reuse. Psychol Sci. 2016;27:1043–6.
Heck DW, Gronau QF, Wagenmakers EJ. metaBMA [R software package]. 2017. https://CRAN.Rproject.org/package=metaBMA
Viechtbauer W. Conducting metaanalyses in R with the metafor package. J Stat Softw. 2010;36:1–48.
van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to metaanalysis. Stat Med. 1993;12:2273–84.
Wang MC, Bushman BJ. Using the normal quantile plot to explore metaanalytic data sets. Psychol Methods. 1998;3:46–54.
Higgins JP, Green S, editors. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley; 2008.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9.
Panesar SS, Rao C, Vecht JA, Mirza SB, Netuveli G, Morris R, et al. Development of the Veritas plot and its application in cardiac surgery: an evidencesynthesis graphic tool for the clinician to assess multiple metaanalyses reporting on a common outcome. Can J Surg. 2009;52:E137–45.
Keus F, Wetterslev J, Gluud C, van Laarhoven CJ. Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010;10:1.
Thomson HJ, Thomas S. The effect direction plot: visual display of nonstandardised effects across multiple outcome domains. Res Synth Methods. 2013;4:95–101.
Wang DD, ShamsWhite M, Bright OJM, Parrott JS, Chung M. Creating a literature database of lowcalorie sweeteners and health studies: evidence mapping. BMC Med Res Methodol. 2016;16:1.
Hanji MB. Metaanalysis in psychiatry research: fundamental and advanced methods. Toronto: Apple Academic Press; 2017.
L’Abbé KA, Detsky AS, O’Rourke K. Metaanalysis in clinical research. Ann Intern Med. 1987;107:224–33.
Jimenez FJ, Guallar E, MartínMoreno JM. A graphical display useful for metaanalysis. Eur J Pub Health. 1997;7:101–5.
Voest EE, van Houwelingen JC, Neijt JP. A metaanalysis of prognostic factors in advanced ovarian cancer with median survival and overall survival (measured with the log (relative risk)) as main objectives. Eur J Cancer. 1989;25:711–20.
Brannick MT, Gültaş M. Crosshairs: a scatterplot for metaanalysis in R. Res Synth Methods. 2017;8:53–63.
Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: dataanalytic approaches and some additional considerations. Stat Med. 1993;12:1293–316.
Phillips B, Stewart LA, Sutton AJ. ‘Cross hairs’ plots for diagnostic metaanalysis. Res Synth Methods. 2010;1:308–15.
Deeks JJ, Altman DG. Effect measures for metaanalysis of trials with binary outcomes. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: metaanalysis in context. 2nd ed. London: BMJ Books; 2001. p. 313–35.
Olliaro P, Vaillant MT. Alternative visual displays of metaanalysis of malaria treatment trials to facilitate translation of research into policy. Diagn Microbiol Infect Dis. 2010;68:422–31.
Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in metaanalysis of diagnostic test accuracy studies. BMC Med Res Methodol. 2016;16:97.
Shadish WR, Sweeney RB. Mediators and moderators in metaanalysis: there’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. J Consult Clin Psychol. 1991;59:883–93.
Rhodes DR, Barrette TR, Rubin MA, Ghosh D, Chinnaiyan AM. Metaanalysis of microarrays: interstudy validation of gene expression profiles reveals pathway dysregulation in prostate cancer. Cancer Res. 2002;62:4427–33.
Choi JK, Yu U, Kim S, Yoo OJ. Combining multiple microarray studies and modeling interstudy variation. Bioinformatics. 2003;19:i84–90.
Hattie J. Visible learning: a synthesis of over 800 metaanalyses relating to achievement. London: Routledge; 2008.
Bowden J, Tierney JF, Copas AJ, Burdett S. Quantifying, displaying and accounting for heterogeneity in the metaanalysis of RCTs using standard and generalised Q statistics. BMC Med Res Methodol. 2011;11:41.
Feichtinger J, McFarlane RJ, Larcombe LD. CancerMA: a webbased tool for automatic metaanalysis of public cancer microarray data. Database. 2012;2012:bas055.
Gartlehner G, Schultes MT, Titscher V, Morgan LC, Bobashev GV, Williams P, West SL. User testing of an adaptation of fishbone diagrams to depict results of systematic reviews. BMC Med Res Methodol. 2017;17:169.
Babatunde OO, Tan V, Jordan JL, Dziedzic K, ChewGraham CA, Jinks C, et al. Evidence flowers: an innovative, visual method of presenting “best evidence” summaries to health professional and lay audiences. Res Synth Methods. 2018;9:273–84.
Schweder T, Spjøtvoll E. Plots of pvalues to evaluate many tests simultaneously. Biometrika. 1982;69:493–502.
Dear KB, Begg CB. An approach for assessing publication bias prior to performing a metaanalysis. Stat Sci. 1992;7:237–45.
Copas J, Shi JQ. Metaanalysis, funnel plots and sensitivity analysis. Biostatistics. 2000;1:247–62.
Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4:245–53.
Gerber AS, Malhotra N. Publication bias in empirical sociological research: do arbitrary significance levels distort published results? Sociol Methods Res. 2008;37:3–30.
Formann AK. Estimating the proportion of studies missing for metaanalysis due to publication bias. Contemp Clin trials. 2008;29:732–9.
Han B, Eskin E. Interpreting metaanalyses of genomewide association studies. PLoS Genet. 2012;8:e1002555.
Simonsohn U, Nelson LD, Simmons JP. Pcurve: a key to the filedrawer. J Exp Psychol Gen. 2014;143:534–47.
Taylor AE, Munafò MR. Triangulating metaanalyses: the example of the serotonin transporter gene, stressful life events and major depression. BMC Psychol. 2016;4:23.
Schwarzer G, Carpenter JR, Rücker G. Metaanalysis with R. Cham: Springer; 2015.
Harrison S, Jones HE, Martin RM, Lewis SJ, Higgins JP. The albatross plot: a novel graphical tool for presenting results of diversely reported studies in a systematic review. Res Synth Methods. 2017;8:281–9.
Lumley T. Network metaanalysis for indirect treatment comparisons. Stat Med. 2002;21:2313–24.
König J, Krahn U, Binder H. Visualizing the flow of evidence in network metaanalysis and characterizing mixed treatment comparisons. Stat Med. 2013;32:5414–29.
Batson S, Score R, Sutton AJ. Threedimensional evidence network plot system: covariate imbalances and effects in network metaanalysis explored using a new software tool. J Clin Epidemiol. 2017;86:182–95.
Song F, Harvey I, Lilford R. Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol. 2008;61:455–63.
Tan SH, Cooper NJ, Bujkiewicz S, Welton NJ, Caldwell DM, Sutton AJ. Novel presentational approaches were developed for reporting network metaanalysis. J Clin Epidemiol. 2014;67:672–80.
Phillippo DM, Dias S, Ades AE, Didelez V, Welton NJ. Sensitivity of treatment recommendations to bias in network metaanalysis. J R Stat Soc Ser A Stat Soc. 2018;181:843–67.
Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148:544–53.
Chung H, Lumley T. Graphical exploration of network metaanalysis data: the use of multidimensional scaling. Clin Trials. 2008;5:301–7.
Hawkins N, Scott DA, Woods BS, Thatcher N. No study left behind: a network metaanalysis in nonsmallcell lung cancer demonstrating the importance of considering all relevant data. Value Health. 2009;12:996–1003.
Naci H. Communication of treatment rankings obtained from network metaanalysis using data visualization. Circ Cardiovasc Qual Outcomes. 2016;9:605–8.
Salanti G, Marinho V, Higgins JP. A case study of multipletreatments metaanalysis demonstrates that covariates should be considered. J Clin Epidemiol. 2009;62:857–64.
Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, Churchill R, et al. Comparative efficacy and acceptability of 12 newgeneration antidepressants: a multipletreatments metaanalysis. Lancet. 2009;373:746–58.
Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing betweenstudy heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with nonrheumatic atrial fibrillation. Stat Med. 2009;28:1861–81.
Donegan S, Dias S, TudurSmith C, Marinho V, Welton NJ. Graphs of study contributions and covariate distributions for network metaregression. Res Synth Methods. 2018;9:243–60.
Senn S, Gavini F, Magrez D, Scheen A. Issues in performing a network metaanalysis. Stat Methods Med Res. 2013;22:169–89.
Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network metaanalysis. PLoS One. 2014;9:e99682.
Veroniki AA, Straus SE, Fyraridis A, Tricco AC. The rankheat plot is a novel way to present the results from a network metaanalysis including multiple outcomes. J Clin Epidemiol. 2016;76:193–9.
Rücker G, Schwarzer G. Resolve conflicting rankings of outcomes in network metaanalysis: partial ordering of treatments. Res Synth Methods. 2017;8:526–36.
Rücker G, Schwarzer G, Krahn U, König J. netmeta [R software package]. 2017. https://CRAN.Rproject.org/package=netmeta
Cipriani A, Furukawa TA, Salanti G, Chaimani A, Atkinson LZ, Ogawa Y, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network metaanalysis. Lancet. 2018;391:1357–66.
Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: survey of 71 negative trials. N Engl J Med. 1978;299:690–4.
Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. Br Med J. 2001;322:1479–80.
Lakens D, Scheel AM, Isager PM. Equivalence testing for psychological research: a tutorial. Adv Methods Pract Psychol Sci. 2018;1:259–69.
Voracek M, Kossmeier M, Tran US. Which data to metaanalyze, and how? A specificationcurve and multiverseanalysis approach to metaanalysis. Z Psychol. 2019;227:64–82.
APA Publications and Communications Board Working Group on Journal Article Reporting Standards. Reporting standards for research in psychology: why do we need them? What might they be? Am Psychol. 2008;63:839–51.
Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and metaanalysis. 3rd ed. New York: Russell Sage; 2019.
Bax L, Yu LM, Ikeda N, Moons KG. A systematic comparison of software dedicated to metaanalysis of causal studies. BMC Med Res Methodol. 2007;7:40.
Tay L, Parrigon S, Huang Q, LeBreton JM. Graphical descriptives: a way to improve data transparency and methodological rigor in psychology. Perspect Psychol Sci. 2016;11:692–701.
Acknowledgements
None.
Funding
The authors declare that they received no financial support for the research and authorship of this article. The publication of this article was supported by the Open Access Publishing Fund of the University of Vienna.
Author information
Authors and Affiliations
Contributions
Detailed authors’ contributions, according to the CRediT (Contributor Roles Taxonomy) system, are as follows: Conceptualization: MV MK. Data curation: MK. Formal analysis: MK. Funding acquisition: MV. Investigation: MK MV UST. Methodology: MK MV UST. Project administration: MV. Resources: MV. Software: MK. Supervision: MV. Validation: MV UST. Visualization: MK. Writing – original draft: MK. Writing – review & editing: MK MV UST. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Additional file 1.
Chronological bibliography of textbooks, monographs, and software manuals on metaanalysis and systematic reviews.
Additional file 2.
Graph vignettes for the entire collection of 208 graphical displays for metaanalysis and systematic reviews.
Additional file 3.
Feature assessments for the entire collection of 208 graphical displays for metaanalysis and systematic reviews.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Kossmeier, M., Tran, U.S. & Voracek, M. Charting the landscape of graphical displays for metaanalysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. BMC Med Res Methodol 20, 26 (2020). https://doi.org/10.1186/s1287402009119
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s1287402009119