Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data
- Byron C Wallace^{1, 2}Email author,
- Christopher H Schmid^{1, 3},
- Joseph Lau^{1} and
- Thomas A Trikalinos^{1}
DOI: 10.1186/1471-2288-9-80
© Wallace et al; licensee BioMed Central Ltd. 2009
Received: 30 April 2009
Accepted: 4 December 2009
Published: 4 December 2009
Abstract
Background
Meta-analysis is increasingly used as a key source of evidence synthesis to inform clinical practice. The theory and statistical foundations of meta-analysis continually evolve, providing solutions to many new and challenging problems. In practice, most meta-analyses are performed in general statistical packages or dedicated meta-analysis programs.
Results
Herein, we introduce Meta-Analyst, a novel, powerful, intuitive, and free meta-analysis program for the meta-analysis of a variety of problems. Meta-Analyst is implemented in C# atop of the Microsoft .NET framework, and features a graphical user interface. The software performs several meta-analysis and meta-regression models for binary and continuous outcomes, as well as analyses for diagnostic and prognostic test studies in the frequentist and Bayesian frameworks. Moreover, Meta-Analyst includes a flexible tool to edit and customize generated meta-analysis graphs (e.g., forest plots) and provides output in many formats (images, Adobe PDF, Microsoft Word-ready RTF). The software architecture employed allows for rapid changes to be made to either the Graphical User Interface (GUI) or to the analytic modules.
We verified the numerical precision of Meta-Analyst by comparing its output with that from standard meta-analysis routines in Stata over a large database of 11,803 meta-analyses of binary outcome data, and 6,881 meta-analyses of continuous outcome data from the Cochrane Library of Systematic Reviews. Results from analyses of diagnostic and prognostic test studies have been verified in a limited number of meta-analyses versus MetaDisc and MetaTest. Bayesian statistical analyses use the OpenBUGS calculation engine (and are thus as accurate as the standalone OpenBUGS software).
Conclusion
We have developed and validated a new program for conducting meta-analyses that combines the advantages of existing software for this task.
Background
Systematic reviews of randomized controlled trials or epidemiological studies have emerged as a key source of evidence across medical disciplines [1, 2]. A central component of many systematic reviews is meta-analysis, the quantitative synthesis of information across methodologically and epidemiologically similar studies that address the same research question. Meta-analysis increases the statistical power to detect effects for which individual studies may be underpowered. Reciprocally, in the absence of statistically significant effects, it can increase the power to exclude clinically important differences. Most importantly, meta-analytic methodologies-particularly meta-regression, provide the framework to quantify and explore between-study heterogeneity (between-study dissimilarity) [3].
Meta-analysis is usually performed using computer programs. Herein we present a new program for the Microsoft Windows operating system, Meta-Analyst, and report on its testing versus other widely used and accepted software. Meta-Analyst features an easy and intuitive graphical user interface and has a spreadsheet-based layout. The program was developed by the Tufts Evidence-based Practice Center under contract with the US Agency for Healthcare Research and Quality (AHRQ). It is available for use by the AHRQ-designated Evidence-based Practice Centers for performing meta-analyses in their evidence reports. Additionally, the software is now being made available to all interested investigators worldwide at no cost. The latest version can be obtained from http://tuftscaes.org/meta_analyst/ (last accessed 11/12/2009).
Existing software
Meta-analysis can be performed in various general statistical and numerical analysis environments (e.g., Stata, R/Splus, Octave/MATLAB), or in dedicated programs (e.g., the Microsoft DOS version of Meta-Analyst, Comprehensive Meta-Analysis, RevMan, MIX [4]). A recent overview [5] compared the features of 6 graphical user interface packages dedicated to meta-analysis.
Comparison of Meta-Analysis Software
Stata/WinBUGS | R/OpenBUGS | MIX | CMA | RevMan | Meta-Analyst | |
---|---|---|---|---|---|---|
Operating system | Windows, Mac, Linux | Windows, Mac, Linux | Windows | Windows | Windows, Mac, Linux | Windows |
Version | 10 | 2.6 | 1.7 | 2 | 5.0.18 | Beta 1.0 |
Price | $785* | FREE | FREE | $1,295 | FREE | FREE |
Import data | ✓ | ✓ | ✓ | ✓ | ✓ | |
Meta-analysis interface/routines | Macros | Macros | Dedicated | Dedicated | Dedicated | Dedicated |
Meta-regression | ✓ | ✓ | ∅ | ✓ | ∅ | ✓ |
Single group | ✓ | ✓ | ✓^{†} | ✓ | ✓^{†} | ✓ |
Fixed effects | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Random effects | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Multilevel models | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Random effects meta-regression | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Bayesian models | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Cumulative meta-analysis | ✓ | ✓ | ✓ | ✓ | ∅ | ✓ |
Subgroup analysis | ✓ | ✓ | ∅ | ✓ | ✓ | ✓ |
Small study effects (Publication bias tests)** | ✓ | ✓ | ✓ | ✓ | ∅ | ∅ |
Binary data | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Continuous data | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Diagnostic test data | ✓ | ✓ | ∅ | ∅ | ✓ | ✓ |
Multivariate | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Documentation of methods | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Forest plot | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Funnel plot | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
SROC | ✓ | ✓ | ∅ | ∅ | ✓ | ✓ |
HSROC - bivariate meta-analysis | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Point and click plot editing | ✓ | ∅ | ∅ | ∅ | ∅ | ✓ |
Programming capabilities | ✓ | ✓ | ∅ | ∅ | ∅ | ✓ |
Leave one out sensitivity | ✓ | ✓ | ∅ | ✓ | ∅ | ✓ |
Results format | RTF | RTF | MS Excel | RTF, PowerPoint | RevMan | PDF, RTF, image files |
As shown in Table 1, in Meta-Analyst strives to combine the ease-of-use of standalone meta-analysis packages with the advanced analytic capabilities offered by general statistical packages.
Implementation
Meta-Analyst is written primarily in C#, and runs atop Microsoft's .NET framework. The .NET framework allowed rapid development of an intuitive Windows-based user interface. Data entry and management follows the familiar Microsoft Excel^{©}-like spreadsheet layout. We use specialized open-source software libraries to create plots (Zedgraph library for graphs and charts [11]) and reports (iTextSharp [12] document generation toolkit). Although the .NET Common Language Runtime is an open standard, and therefore theoretically platform independent, Meta-Analyst currently runs only on the Windows operating system.
Results
Methods Implemented
Statistical methods
Methods available in Meta-Analyst
Fixed | Random | Bayes | ||||
---|---|---|---|---|---|---|
IV* | MH | Peto | DL | EM^{†} | ||
Binary outcomes | ||||||
Odds ratio (OR) | √ | √ | √ | √ | √ | √^{‡} |
Risk ratio (RR) | √ | √ | - | √ | √ | √^{‡} |
Risk difference (RD) | √ | √ | - | √ | √ | √^{‡} |
Proportion** | √ | - | - | √ | √ | √^{‡} |
Continuous outcomes | - | - | ||||
WMD | √ | - | - | √ | √ | √^{‡} |
Hedge's g | √ | - | - | √ | √ | √^{‡} |
Cohen's d | √ | - | - | √ | √ | √^{‡} |
Glass' δ | √ | - | - | √ | √ | √^{‡} |
Mean** | √ | - | - | √ | √ | √^{‡} |
Diagnostic test data | - | - | ||||
Specificity | √ | - | - | √ | √ | √^{‡} |
Sensitivity | √ | - | - | √ | √ | √^{‡} |
Accuracy | √ | - | - | √ | √ | √^{‡} |
Positive predictive value (PPV) | √ | - | - | √ | √ | √^{‡} |
Negative predictive value (NPV) | √ | - | - | √ | √ | √^{‡} |
Positive likelihood ratio | √ | √ | - | √ | √ | √^{‡} |
Positive likelihood ratio | √ | √ | - | √ | √ | √^{‡} |
Diagnostic odds ratio | √ | √ | - | √ | √ | √^{‡} |
Summary ROC curve | [weighted, unweighted] | [weighted] | ||||
Bivariate | --- | - | √ | |||
Hierarchical SROC | - | - | √ |
Currently (as of version Beta 3.1) Meta-Analyst implements only one Bayesian model for each type of data (binary, continuous and diagnostic; for model details see http://tuftscaes.org/meta_analyst/AppendixA.html, last accessed 11/12/2009). Because of the way we have interfaced Meta-Analyst with OpenBUGS (Figure 2) we can easily add additional models.
For detailed explanation of the statistical routines used, including handling of zero-cells, please see our methods document at: http://tuftscaes.org/meta_analyst/metaanalyst_methods.html (last accessed 1112/2009).
Exploratory and sensitivity analyses
Meta-Analyst automates cumulative meta-analysis, leave-one-out sensitivity analysis and subgroup analysis. In cumulative meta-analysis one reorders the studies according to a covariate (e.g., increasing year of publication) and re-estimates the summary effect at each step, i.e., each time a new study is added. It is typically a graphical analysis that plots the aggregate overall estimate at each step [17, 18]. This elucidates the evolution, or pattern, of evidence over time. Leave-one-out analyses explore the influence of individual studies as follows: If there are n studies in the meta-analysis at hand, plot n summary estimates, each corresponding to leaving one of the n studies out of the calculation. This plot illuminates influential studies, as when they are left out of the analysis, the overall estimate will be notably perturbed. Subgroup analysis is a tool for exploring the effects of a treatment on population subgroups, e.g., females older than 50 years old versus younger women. This is done by conducting separate meta-analyses on the respective subgroups and plotting overall estimates for both.
User Interface
While editing data, the outcome and corresponding confidence intervals are updated dynamically. The outcome metric can be changed via a right-click menu (e.g., from odds ratio to risk difference) and the outcomes will be re-computed automatically. While Figure 5 shows binary data being manipulated, the interface is analogous for continuous and diagnostic data.
The user can also provide additional numeric and string variables that describe characteristics of the analyzed studies. By convention, user-added numeric variables are termed covariates. Typically, covariates are used as explanatory variables in meta-regression analyses. To use covariates in the analyses a user has to activate them (by right clicking on the corresponding covariate name and choosing the respective option). When at least one covariate is activated, fixed and random effects meta-regression becomes available as an analytic option in the program's menus. If the covariate is excluded from the analyses, meta-regression is not available as an option. To perform meta-regression with several explanatory variables, the user simply activates the corresponding covariates.
User-added string variables are termed labels and are typically used to provide textual descriptions, or to specify subgroups for subgroup analyses. When a dataset contains at least one label, the program allows the user to perform subgroup analyses according to the categories defined by the selected label. The subgroups are automatically named according to the contents of the label. Labels are ignored in meta-regression analyses (though displayed in plots when pertinent). For example, suppose the user adds a label "country". Further suppose that studies 1 and 2 are labelled "United States" while studies 3 and 4 are labelled "India". Then a subgroup analysis performed using the "country" label will automatically plot the overall (pooled) effects for the studies that were labelled as being conducted in "India" (1 and 2) and the pooled estimate will also be plotted for those labelled as being conducted in the "United States" (3 and 4).
The left-hand side of the results tab shows a tree populated with collapsible parent nodes for each analysis that has been run ("Analysis 1" and "Analysis 2", in the figure). Each of these parent nodes have child nodes corresponding to the various tabular and graphical outputs associated with the analysis. Clicking on one of these child nodes, e.g., "Forest Plot", scrolls the corresponding graphic into view.
In addition to the interactive output shown in Figures 7 and 8, Adobe^{©} PDF and Microsoft^{©} Word -ready Rich Text Format report files are generated and saved. These include all of the tabular and graphical output. Finally, the graphics themselves are automatically output to separate image (.PNG) files, for use in other programs. Our aim was to provide the output in as many formats as possible, in order to provide flexibility to the user.
Validation
To validate the computational results, we systematically tested Meta-Analyst vs. the results from metan version 1.86 in Stata. We compared the output of the programs in 11,803 meta-analyses of binary outcomes and 6,881 meta-analyses of continuous outcomes from issue 4 of the Cochrane Library of Systematic Reviews, 2005. This database of meta-analyses was described elsewhere [19], and it includes meta-analyses that have very different characteristics.
Over the 11,803 analyses (over all methods and all metrics) for binary data and 6,881 analyses over continuous data, we recorded the minimum of the absolute and normalized differences between the outputs from Stata and Meta-Analyst, where the normalized difference is defined as Δ_{rel} = | Θ_{Stata} - Θ_{MA}|/Θ_{Stata}. Θ is any of the numerical output of the program such as a summary effect size for each meta-analysis metric and method, its variance, and the Q, τ^{2} and I^{2} statistics (for random effects models). We aimed to identify differences that are beyond those introduced by machine (im)precision. If there are such differences, both the absolute and normalized difference between the two numbers will be relatively large. When the numbers in question are very large, the absolute difference might be relatively large (merely because of machine imprecision) whereas their normalized difference will be very small. Reciprocally, for small magnitudes the normalized difference can be relatively large (in the absence of computational errors), while the absolute difference is very small.
Over the binary set of meta-analyses, the maximum discrepancy was 2.9 × 10^{-6}. For continuous data analyses, the maximum discrepancy was 7.4 × 10^{-5}. These maximum discrepancies appeared in meta-analyses with extreme between study heterogeneity, and are ascribed to rounding errors (version 1.86 of metan does not use double precision for all internal calculations as Meta-Analyst does).
As previously discussed, Bayesian analyses are run through OpenBUGS, and so the output is as thoroughly tested as OpenBUGS.
Testing for diagnostic test accuracy analyses is not as extensive, because we have not found a suitable reference scripting environment to test Meta-Analyst output against. However, the simple diagnostic test methods are based on weighted proportions (sensitivity, specificity), relative risks (likelihood ratios), odds ratios and regression (SROC, meta-regression). These methods use the same computational algorithms as those for binary data and so have been tested. Two diagnostic methods remain to be sufficiently tested; bivariate meta-analysis of sensitivity and specificity for diagnostic tests and random effects SROC. These analyses are flagged as not thoroughly checked when they are requested from the user. However, these will soon be reconfigured to run in OpenBUGS so that they will be validated as well.
Discussion
In order to attain widespread use, meta-analysis software must be easy to use. In particular, requiring that users learn an entire language to run their analyses will prohibit general adaptation of a program. Dedicated meta-analysis programs such as MIX, Comprehensive Meta-analysis, and MetaDiSc are appealing due to their small learning curve. On the other hand, by their very nature, such programs are less flexible than general statistical packages. For example, they have no scripting functionality, which precludes their use for large-scale empirical research or simulation studies. Further, they are not able to perform advanced analyses, such as bivariate diagnostic test meta-analyses, because they cannot maximize difficult likelihood functions, and they cannot be readily extended to include additional analytic options.
Conclusion
Meta-Analyst mitigates several of the weaknesses inherent to dedicated meta-analysis packages. It incorporates their ease-of-use, while providing advanced analytic methods that can be implemented in packages such as Stata, R and SAS by a statistical programmer.
The current version of Meta-Analyst is made available free of charge to interested researchers. It runs on any version of Windows that is compatible with the .NET platform (comprising Windows 98, ME, NT 4.0, 2000, XP and Vista). We have already started development of a cross-platform completely open-source version of the software that uses the R statistical language, and will be readily modifiable and extendable by any interested party http://www.github.com/bwallace/OpenMeta-analyst-.
Availability and requirements
An installer file for the latest version of Meta-Analyst has been provided as an additional/supplemental file for the peer-reviewers [Additional File 1]. Alternatively, the latest version can be obtained from http://tuftscaes.org/meta_analyst/ (last accessed 11/12/2009). Meta-Analyst is made readily available to any scientist wishing to use it for non-commercial purposes, without any restriction (including the need for a material transfer agreement).
Declarations
Acknowledgements
Funding: Meta-Analyst has been developed with funding from the National Center for Research Resources (NCRR, grant number R33 RR17109) and the Agency for Healthcare Research and Quality (AHRQ, contract number 290-02-0022) and R01HS018574, a contract from AHRQ (No 290-02-0022),
Authors’ Affiliations
References
- Lau J, Schmid CH, Chalmers TC: Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995, 48: 45-57. 10.1016/0895-4356(94)00106-Z.View ArticlePubMed
- Mosteller F, Colditz GA: Understanding research synthesis (meta-analysis). Annu Rev Public Health. 1996, 17: 1-23. 10.1146/annurev.pu.17.050196.000245.View ArticlePubMed
- Lau J, Ioannidis JP, Schmid CH: Summing up evidence: one answer is not always enough. Lancet. 1998, 351: 123-27. 10.1016/S0140-6736(97)08468-7.View ArticlePubMed
- Bax L, Yu LM, Ikeda N, Tsuruta H, Moons KG: Development and validation of MIX: comprehensive free software for meta-analysis of causal research data. BMC Med Res Methodol. 2006, 6: 50-10.1186/1471-2288-6-50.PubMed CentralView ArticlePubMed
- Bax L, Yu LM, Ikeda N, Moons KG: A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Med Res Methodol. 2007, 7: 40-10.1186/1471-2288-7-40.PubMed CentralView ArticlePubMed
- Zamora J, Abraira V, Muriel A, Khan K, Coomarasamy A: Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol. 2006, 6: 31-10.1186/1471-2288-6-31.PubMed CentralView ArticlePubMed
- van Houwelingen HC, Arends LR, Stijnen T: Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002, 21: 589-624. 10.1002/sim.1040.View ArticlePubMed
- Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA: A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007, 8: 239-51. 10.1093/biostatistics/kxl004.View ArticlePubMed
- Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005, 58: 982-90. 10.1016/j.jclinepi.2005.02.022.View ArticlePubMed
- Rutter CM, Gatsonis CA: A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001, 20: 2865-84. 10.1002/sim.942.View ArticlePubMed
- Champion J: Zedgraph. 2009, Ref Type: Electronic Citation, [http://zedgraph.org]
- iTextSharp. Ref Type: Electronic Citation, [http://itextsharp.sourceforge.net/]
- Gamma E, Helm R, Johnson R, Vlissides J: Design Patterns: Elements of Reusable ObjectOriented Software. 1995, Addison-Wesley
- Thomas A, O Hara B, Ligges U, Sturtz S: Making BUGS Open. R News. 2009, 6: 12-17.
- IronPython. 2009, Ref Type: Electronic Citation, [http://www.codeplex.com/IronPython]
- van Rossum G: Python programming language. 2009, Ref Type: Electronic Citation, [http://www.python.org]
- Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC: Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992, 327: 248-54. 10.1056/NEJM199207233270406.View ArticlePubMed
- Lau J, Schmid CH, Chalmers TC: Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995, 48: 45-57. 10.1016/0895-4356(94)00106-Z.View ArticlePubMed
- Ioannidis JP, Trikalinos TA: The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ. 2007, 176: 1091-96.PubMed CentralView ArticlePubMed
- Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I: The case of the misleading funnel plot. BMJ. 2006, 333: 597-600. 10.1136/bmj.333.7568.597.PubMed CentralView ArticlePubMed
- Terrin N, Schmid CH, Lau J: In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005, 58: 894-901. 10.1016/j.jclinepi.2005.01.006.View ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/9/80/prepub
Pre-publication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.