 Technical advance
 Open Access
 Published:
Comparative efficiency research (COMER): metaanalysis of costeffectiveness studies
BMC Medical Research Methodology volume 14, Article number: 139 (2014)
Abstract
Background
The aim of this study was to create a new metaanalysis method for costeffectiveness studies using comparative efficiency research (COMER).
Methods
We built a new score named total incremental net benefit (TINB), with inverse variance weighting of incremental net benefits (INB). This permits determination of whether an alternative is costeffective, given a specific threshold (TINB > 0 test). Before validation of the model, the structure of dependence between costs and qualityadjusted life years (QoL) was analysed using copula distributions. The goodnessoffit of a Spanish prospective observational study (n = 498) was analysed using the Independent, Gaussian, T, Gumbel, Clayton, Frank and Placket copulas. Validation was carried out by simulating a copula distribution with lognormal distribution for costs and gamma distribution for disutilities. Hypothetical cohorts were created by varying the sample size (n: 15–500) and assuming three scenarios (1costeffective; 2noncosteffective; 3dominant). The COMER result was compared to the theoretical result according to the incremental costeffectiveness ratio (ICER) and the INB, assuming a margin of error of 2,000 and 500 monetary units, respectively.
Results
The Frank copula with positive dependence (−0.4279) showed a goodnessoffit sufficient to represent costs and QoL (pvalues 0.524 and 0.808). The theoretical INB was within the 95% confidence interval of the TINB, based on 15 individuals with a probability > 80% for scenarios 1 and 2, and > 90% for scenario 3. The TINB > 0 test with 15 individuals showed pvalues of 0.0105 (SD: 0.0411) for scenario 1, 0.613 (SD: 0.265) for scenario 2 and < 0.0001 for scenario 3.
Conclusions
COMER is a valid tool for combining costeffectiveness studies and may be of use to health decision makers.
Background
Economic evaluation of health technologies (EEHT) has become a firstorder health policy decision making tool at the European level. EEHT allows both the economic and clinical value of technologies (drugs, devices, health programs, etc.) to be evaluated [1] and, when used for the allocation of health resources, allows decisionmakers to make informed decisions on specific problems by combining the probabilities of all possible outcomes and the health benefits assigned to each of them [2, 3].
In the last decade, there has been a rise in EEHT, as reflected by the large amount of costeffectiveness studies [4, 5] and the interest in the technological advances generated by their results. Although costeffectiveness analyses (CEA) have improved due to the development of methods of synthesizing evidence, systematic revisions and direct, indirect and mixed metaanalyses [6, 7], there is still no consensual method for carrying out metaanalyses of CEA. Proposed methods range from mere qualitative review of CEA [8, 9] to the categorization of CEA according to whether the costs are higher, the same or lower and whether the effects are better, the same or poorer, in order to assess how many CEA there are in each category [10]. We suggest that the method for metaanalyses of CEA proposed here, which is known as COMparative Efficiency Research (COMER), and which groups together studies on a specific topic, may represent reality more closely and allow more evidencebased decision making. As for any metaanalysis, the validity of the COMER method will depend on systematic review of the types of studies supporting the data, the populations assessed and the variability or levels of consistency between centres and/or studies.
Although metaanalyses usually use aggregate study data, recent research has shown that the use of individuallevel data or individual and aggregatelevel data, allows for a better estimate of variability [11, 12]. Therefore, the COMER method should be able to use both individual and aggregate data, although it remains difficult to obtain original data from clinical studies for use by researchers other than the original authors [13].
The aim of this study was to create a method of CEA metaanalysis based on either individual or aggregate data through the generation of a multivariate distribution function which allows the dependence between costs and effects to be modelled. This mathematical dependence is called copula.
Methods
Costeffectiveness studies compare the efficiency of therapies evaluated using the incremental costeffectiveness ratio (ICER), where the numerator is the difference between the estimated cost of the new treatment and that of the reference treatment and the denominator estimates the effectiveness gained by the new treatment compared with the reference treatment [1].
where 1 and 2 correspond to the new treatment and the reference treatment, respectively.
Thus, when the therapy being assessed is more expensive than the reference treatment, and is also less effective, the new treatment is dominated. When the treatment is more effective than the reference treatment, and also less expensive, the new treatment is dominant over the reference treatment. In the remaining cases, the ICER must be compared with the willingness to pay (k) to assess whether the new treatment is costeffective (ICER < = k) or not costeffective (ICER > k) [1, 2]. The choice of the threshold of efficiency remains an unresolved issue in many health systems. Theoretically, the threshold of efficiency should be related to the value that society accords to a health outcome in, for example, life years gained, depending on the available resources [1, 2].
One limitation of the ICER is that, as it is a ratio, the expected value of the ICER represents the difference in the expected cost divided by the difference in the expected effectiveness. Therefore, the main difficulty in making inferences from costeffectiveness studies is that the random variable obtained is not necessarily normal or symmetric [14]. However, even when cost and effectiveness data are distributed normally, there is no guarantee that the ratio between them will also behave normally and, ultimately, it is not possible to calculate the confidence intervals (CI). Thus, there is a need to use alternative methods, such as bootstrapping or incremental net benefits (INB) for this purpose [15].
Therefore, the COMER methodology, both for individual and aggregate data, consists of estimating the total incremental net benefit (TINB) according to the INB of the studies included weighted by the inverse of the variance of each INB.
Where s is the number of the study and 1 and 2 the new treatment and reference treatment, respectively:
and
and the variance is:
According to the central limit theorem, the INB is asymptotically normal [15] and the TINB is also asymptotically normal when the mean and variance is known, and the probability of the TINB being < 0 can be calculated. Therefore, the decision criterion will be that the new treatment is costeffective for the established threshold if the probability is lower than 5% (Table 1).
Equivalently,
with
Validation of the method
The costs and effects of individual patients were modelled for marginal distributions to illustrate the COMER method. The use of the copula distribution, which permits the joint distribution of costs and effects to be obtained, has been suggested [16], since the dependence between costs and effects does not have to be linear. In this way, a cost and effect dependence structure is obtained through the copula and their univariate behaviour.
The process carried out consisted of:

Identifying the copula: the joint distribution that best fits the costs and effects.

Creating a simulated cohort under the marginal and joint distribution for each alternative compared.

COMER estimation.

Method validation.
Identification of marginal distributions and copula
Copulas are bivariate distributions that provide dependent structures for two statistical variables, with any type of univariate distribution [17–19]. In its construction, a C(u,v) copula is a multivariate distribution function defined from U and V random variables with uniform distribution in the interval [0,1], which verifies the following properties:

1.
C(u, 0) = 0 = C(0, v)

2.
C(u, 1) = u and C(1, v) = v

3.
For u1, u2, v1, v2 in [0, 1] so that u1 ≤ u2 and v1 ≤ v2, C(u2, v2) − C(u2, v1) − C(u1, v2) + C(u1, v1) ≥ 0.
Since the relationship between random variables is not based on the distribution, but rather constructed from mathematical structures between random variables, dependence can be evaluated using nonparametric correlation statistics (Spearman’s ρ_{s} and Kendall’s τ), which are independent on marginal distributions.
Copula distributions are little used in health economics, and are mainly applied to specify the distribution of regression models with more than one dependent variable [20–23]. We used copulas to describe the dependence structure of costeffect random variables, allowing random generation of a patient cohort under such a distribution in order to make different simulations. By applying copulas to EEHT, the joint distribution of each treatment can be simulated, since C_{j}(c_{ij},e_{ij}) is a copula for treatment j, where costs (c) and effects (e) are known for each patient i. In addition, Sklar’s theorem [24] shows that, given a copula (the joint distribution), the copula can be reconstructed through the marginal theoretical or empirical inverse distribution function, according to the specific case. However, recent studies show that there is no justification for the claim that a specific copula may be the most appropriate for the combination of costs and effects, not even when costs are broken down into direct and indirect costs, or if effects are represented by the following measurements: therapeutic success, life years, or qualityadjusted life years (QALY) [25].
To create a hypothetical population to test the validity of the COMER methodology, we used data from a Spanish prospective observational study of patients with allergic rhinitis (n = 498) with direct costs (c) and mean utilities from the SF12v1 (e) [26]. The study served to obtain a copula structure which, by adding some theoretical marginal distributions, allowed the generation of a hypothetical population on which to apply the COMER methodology. The empirical copula was compared to the parametric estimate of the potential copula (Additional file 1) to evaluate the goodness of fit of the data with the theoretical copulas [27]. The copula bound to evaluate the goodness of fit was developed using the inversion of Kendall’s τ. The pvalue of the test was calculated by simulation of size100 bootstrap, due to the lack of an analytical construction.
The following copulas were evaluated [19, 25, 28]:
Independent copula: this can be generated automatically since, given U and V random variables, the joint distribution function is C ^{0}(u, v) = u * v. Thus, the association between this copula and some data indicates the stochastic independence of U and V; equivalently, the absence of structure.
Gaussian copula: this is defined as G _{∅}(u, v) = N _{ ρ }(Φ^{− 1}(u), Φ^{− 1}(v)) where N _{ ρ }(x, y) is the normal distribution function of parameters x: mean and y: the standard deviation, Φ^{− 1}(x) is the marginal distribution function N(0,1) and ρ is Pearson’s correlation coefficient. The Gaussian copula is still a multivariate normal distribution.
T copula: this is derived from the Student’s t multivariate distribution and is defined as ${T}_{\theta ,\rho}\left(u,v\right)\phantom{\rule{0.5em}{0ex}}=\phantom{\rule{0.5em}{0ex}}\frac{\mathrm{\Gamma}\left(\left(\mathrm{\theta}\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}2\right)/2\right)}{\mathrm{\Gamma}\left(\mathrm{\theta}/2\right)\sqrt{1\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}{\mathrm{\rho}}^{2}}\mathrm{\pi}\mathrm{\theta}}{\left(1\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}\left({\mathrm{u}}^{2}\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}{\mathrm{v}}^{2}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}2\mathrm{\rho}\mathrm{u}\mathrm{v}\right)/\left(\mathrm{\theta}\left(1\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}{\mathrm{\rho}}^{2}\right)\right)\right)}^{\left(\mathrm{\theta}+2\right)/2}$ where ρ < 1, θ > 0. This copula shows a similar structure to the Gaussian copula, but presents tail dependence, points (0,1) and (1,0). One of its qualities is that it includes the Gaussian copula when θ → ∞.
Gumbel copula: this allows a positive dependence structure to be modelled, region (1,1), and is defined as ${G}_{\theta}\left(u,v\right)\phantom{\rule{0.5em}{0ex}}=\phantom{\rule{0.5em}{0ex}}\mathit{exp}\left({\left({\left(\mathrm{l}\mathrm{n}\mathrm{u}\right)}^{\mathrm{\theta}}\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}{\left(\mathrm{l}\mathrm{n}\mathrm{v}\right)}^{\mathrm{\theta}}\right)}^{\frac{1}{\theta}}\right)$ where θ ∈ [1, ∞]. When θ = 1 it is equivalent to the independent copula and when θ → ∞ it behaves as a comonotonic copula (min(u,v)). Thus, the behaviour of the Gumbel copula is an interpolation between the independent copula and the copula of perfect positive dependence.
Clayton copula: this is defined as $C{O}_{\theta}\left(u,v\right)\phantom{\rule{0.5em}{0ex}}=\phantom{\rule{0.5em}{0ex}}{\left({\mathrm{u}}^{\mathrm{\theta}}\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}{\mathrm{v}}^{\mathrm{\theta}}1\right)}^{\frac{1}{\theta}}$ where θ > 0. Like the Gumbel copula, this copula is an interpolation but lies between the independent copula and the perfect negative dependence (point (0,0)).
Frank Copula: this has the quality of showing symmetric dependencies and not showing dependence at points (0,0) and (1,1). It is defined as ${F}_{\theta}\left(u,v\right)\phantom{\rule{0.5em}{0ex}}=\phantom{\rule{0.5em}{0ex}}\frac{1}{\mathrm{\theta}}ln\left(1\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}\frac{\left(exp\left(\mathrm{\theta}\mathrm{u}\right)\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)\left(exp\left(\mathrm{\theta}\mathrm{v}\right)\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)}{exp\left(\mathrm{\theta}\right)1}\right)$ where θ ∈ ℝ \{0}.
Plackett copula: this is defined as ${P}_{\mathrm{\theta}}\left(u,v\right)\phantom{\rule{0.5em}{0ex}}=\phantom{\rule{0.5em}{0ex}}\frac{1\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}\left(\mathrm{\theta}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)\left(u\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}v\right)\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}\sqrt{{\left(1\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}\left(\mathrm{\theta}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)\left(u\phantom{\rule{0.5em}{0ex}}+\phantom{\rule{0.5em}{0ex}}v\right)\right)}^{2}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}4uv\mathrm{\theta}\left(\mathrm{\theta}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)}}{2\left(\mathrm{\theta}\phantom{\rule{0.5em}{0ex}}\phantom{\rule{0.5em}{0ex}}1\right)}$ where θ ≥ 0. This structure shows both positive and negative maximum dependence in function of parameter θ.
Creation of cohorts
Once the joint distribution was known, some theoretical marginal distributions were associated. To simulate costs, a lognormal distribution was associated, and to simulate life quality, a gamma distribution was associated (disutilities), with respect to a baseline quality of life (0.9 utility) [14] (Table 2). For the two alternatives, the same copula and the same randomization (bivariate uniforms [0,1]) were used. This ensured covariance and correlation between the values generated for the costs and effects. Different random samples per cohort were created with sample sizes between 15 and 500 individuals for each alternative. Individuals in the simulated cohort were randomly assigned to the studies. To ensure that there was a minimum variability per study, the random assignation was conditioned to ensure that, for each study, there were at least 3 individuals.
The COMER methodology was applied to the simulated data for each alternative. For each alternative the mean costs, mean effectiveness, differential variance in costs, differential variance in effects and covariance between the differences in costs and effects, and the INB were estimated, setting an efficiency threshold (k = 30,000 monetary units per QALY gained). For example, of a sample of 15 individuals, the seven first could be assigned to study 1, the following five to study 2, and the remaining three to study 3, and the COMER methodology applied (Additional file 2).
For each sample size generated, 500 replications were made, entailing 25,000 metaanalyses for each scenario, thus allowing the number of the times the methodology agreed with reality to be validated. A 2,000monetaryunit/QALYtolerance was assumed to calculate the ICER, and a 500monetaryunittolerance was assumed for the TINB. Additionally, we estimated the minimum sample size required to obtain an adjusted estimate with a probability >70% and when simulations converged to the original Kendall’s τ.
The following are the scenarios evaluated by modifying the marginal distribution parameters (Table 2):

1.
The second alternative is costeffective.

2.
The second alternative is more costly and more effective, although the ICER is above the willingness to pay threshold.

3.
The second alternative is dominant.
All the possibilities are covered by these scenarios since the remaining potential outcomes derived from an EEHT which have not explicitly been evaluated are obtained assuming that the current alternative is the second alternative instead of the first.
The analysis was made using the R 3.0.1 statistical package (Additional file 3). In addition, the COMER method is included as an Excel file in the supplementary materials (Additional file 4).
Results
Comparison of the copulas used with the real costs and effects data showed pvalues between 0.075 and 0.191 for the independent distribution, 0.443 and 0.769 for the Gaussian copula, 0.309 and 0.633 for the T copula, 0.004 and 0.034 for the Gumbel copula, 0.242 and 0.526 for the Clayton copula, 0.524 and 0.808 for the Frank copula and 0.531 and 0.808 for the Plackett copula (Table 3).Therefore, both the Frank and Plackett copulas were candidates to be the copula for the distribution of the simulated cohort. We opted for the Frank copula, which showed a value of −0.4279 (pvalue: 0.128), indicating a positive dependence between costs and effects (Figure 1).
Evaluation of the convergence from the empirical to the theoretical Kendall’s τ showed a range between −0.505 and 0.543 for 15 simulated individuals (mean: 0.029; SD: 0.192), between −0.135 and 0.228 for 115 individuals (mean: 0.046; SD: 0.062) and between −0.059 and 0.130 for 415 individuals (mean: 0.049; SD: 0.029).

1.
Costeffective scenario
In the costeffectiveness scenario it was observed that, starting from a sample size of 75 individuals per cohort, both the ICER and the INB were within the tolerance. It was also observed that, when the number of simulated individuals was increased, the adjustment increased with probabilities > 90%.Comparison of the INB with the individual data and the TINB showed a minimum of 175, 115 and 75 individuals were necessary to obtain a probability greater than 70%, when tolerances of 6%, 9% and 12%, respectively, were assumed. However, the INB was within the CI of the TINB with a probability > 85% for samples of > 15 individuals and > 90% for samples > 25 individuals. The same happened when the CI of the TINB for each metaanalysis was compared with the theoretical value of the INB. (+80% for 15+ individuals and +90% for 25+ individuals) (Figure 2A).
With respect to the probability of alternative 2 being costeffective, for the sample size of 15 individuals, the pvalue was 0.0105 (SD: 0.0411), for 115 individuals it was 0.015 (SD: 0.0214) and for 415 individuals it was 0.0165 (SD: 0.0128). Thus, for the metaanalyses obtained, alternative 2 was costeffective.

2.
Noncosteffective scenario
The ICER reached values > 70% within the tolerance, from 400 individuals upwards. Because the denominator of the ICER had a mean difference of 0.01 QALYs, the ICER showed values between 19,840 and 103,436 for n: 15. This meant that, in 7% of the simulations with n: 15 and in 1% of those with n:35,the ICER was inferior to the willingnesstopay, and was thus costeffective. However, the probability of the INB being within the tolerance (500 monetary units) was > 95% for any sample size.Comparison of the INB with the individual data with respect to the 95% CI of the TINB showed a probability of 85% for samples > 15 individuals. The same occurred when the 95% CI of the TINB for each metaanalysis was compared with the theoretical value of the INB (+80% for 15+ individuals and +90% for 25+ individuals) (Figure 2B). The mean difference between the mean TINB and the theoretical value showed a range of 15 to 121 monetary units.
With respect to the probability of alternative 2 being costeffective, for the sample size of 15 individuals, the pvalue was 0.613 (SD: 0.265), for 115 individuals it was 0.709 (SD: 0.123) and for 415 individuals it was 0.729 (SD: 0.072). Thus, for the metaanalyses obtained, the outcome showed that alternative 2 was not costeffective.

3.
Dominant scenario
The ICER reached values > 70% within the tolerance, from 35 individuals upwards, while the INB reached these values from 15 individuals upwards. Furthermore, when comparing the INB with the individual data and the theoretical INB with a 6% tolerance, a probability > 70% was obtained from 35 individuals upwards.When the INB of the individual data or the theoretical INB was within the CI, probabilities > 90% were observed for any sample size (Figure 2C). Regarding the probability of alternative 2 being costeffective, scenario 3 showed pvalues < 0.0001 for any sample size.
Discussion
EEHT are a set of tools which aim to examine the short and longterm consequences of using health technologies on individuals and on society. Since there are numerous alternatives for allocating those resources, EEHT tries to make relevant information from the efficiency perspective available to health decision makers, understanding efficiency as the maximization of health gains obtained from the limited resources available. The application of COMER in decisionmaking may allow decisionmakers to move from evidencebased medicine to economic evaluation based on evidence, understanding evidence within a comprehensive framework where both clinical and economic evidence are compatible. COMER may be applied both in studies where individual level data or aggregate level data are available, always provided the covariance matrix of the difference between costs and effects, the mean cost and the mean effects of each alternative are available. When there is access to individual data from clinical studies, these measurement can be obtained through bootstrapping [29–31], and in the case of economic evaluation models, they can be obtained from the Monte Carlo simulation [14]. The advantage of bootstrapping is that it is a non parametric method which does not presuppose any distribution and is based on studying the sample as if it were the population, through resampling [30].
We believe that the fact that the simulated individuals were only assigned to 3 studies did not influence the results. Although many metaanalyses agglutinate more than 3 studies, the number of studies used was the minimum required to enable evaluation of the generalization of the results. In addition, in the assignation of individuals per study we forced the assignation of three or more individuals to avoid studies with minimum or null variability. Although, in reality it is unusual to find studies with 3 cases, analysis of verysmall samples allows a small number of individuals to be assigned, thus generating greater heterogeneity between studies, as occurs in reality. However, the analysis with small simple sizes showed, as expected, greater divergence in the results obtained using the ICER and the TINB. Likewise, using two theoretical distributions such as the lognormal for costs and the gamma for disutilities is sufficient to validate the method. Furthermore, since approximation by copulas is independent of marginal distributions, and because typical theoretical distributions were selected to simulate the uncertainty of the costs and disutilities, we believe that the outcome is not biased by this decision. However, any interpretation of the COMER methodology must take into consideration the field in which it has been implemented and validated. In addition, the metaanalysis constructed has fixed effects (inverse variance weighting), which means it must be assumed that the studies are homogenous. This assumption does not necessarily occur in reality, and even though, to validate the method, different sample sizes and high levels of variance (e.g., a coefficient of variation for the costs >1) were used to generate greater heterogeneity in the study data, this should be taken into account when interpreting the results. Future studies should use the COMER methodology to develop a method to adjust the divergences in the homogeneity of the populations evaluated, similar to that carried out for metaregressions.
A potential limitation to applying the COMER method is that health systems in different countries can generate different resource uses and, hence, different total costs. For this reason, the transfer of economic evaluations must be assessed and costs must be homogenized before using this method [32–34]. When using studies from different countries with potentiallysimilar cost structures, the currencies must be homogenized to US dollar equivalents and costs must be updated to the same year, as is done in comparisons made by the World Health Organization [35], the OECD [36] or the IMF [37].
Some authors suggest that a separate cost and effectiveness metaanalysis is sufficient to make an appropriate value judgment [38], while other suggest assessment of the results of the costs and marginal effects of CEA in order to evaluate the worth of a new treatment [10]. However, we believe that the COMER method has potential applications, even for teams with limited resources. Mere systematic review and metaanalysis, separate from economic evaluation studies, are not an EEHT per se, as they could lead decision makers to draw the wrong conclusion due to the impossibility of balancing global costs and effects and measuring additional value. EEHT should be accompanied by a fuller metaanalysis such as the proposed COMER method, although a full, new EEHT might be necessary if sufficient information were not obtained by the COMER method.
Conclusions
Systematic review of economic analyses requires methods to synthesize and interpret the results of multiple analyses. The COMER methodology is a valid option for the analysis of both individual and aggregate data and may allow decisionmakers to better understand the added value of new therapeutic alternatives by synthesizing all available evidence. The COMER methodology opens the way to further research of this type of metaanalysis with two variables having a known dependence structure [16], both from a frequentist and from a Bayesian perspective [39].
Abbreviations
 CEA:

Costeffectiveness analyses
 CI:

Confidence interval
 COMER:

Comparative efficiency research
 EEHT:

Economic evaluation of health technologies
 ICER:

Incremental costeffectiveness ratio
 INB:

Incremental net benefits
 QALY:

Quality adjusted life years
 SD:

Standard deviation
 TINB:

Total incremental net benefit.
References
 1.
Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL: Methods for the economic evaluation of health care programmes. 2005, Oxford: Oxford University Press, 3
 2.
Gisbert R: Economía y Salud. Economía, gestión económica y evaluación en el ámbito sanitario. 2002, Barcelona: Masson
 3.
Oliva J, Antoñanzas F, RiveroArias O: Evaluación económica y la toma de decisiones en salud. El papel de la evaluación económica en la adopción y difusión de tecnologías sanitarias. Informe SESPAS 2008. Gac Sanit. 2008, 22 (Suppl 1): 13742.
 4.
Neumann PJ, Fang CH, Cohen JT: 30 years of pharmaceutical costutility analyses: growth, diversity and methodological improvement. Pharmacoeconomics. 2009, 27 (10): 86172. 10.2165/1131272000000000000000.
 5.
Rodríguez Barrios JM, Pérez Alcántara F, Crespo Palomo C, González García P, Antón De Las Heras E, Brosa Riestra M: The use of cost per life year gained as a measurement of costeffectiveness in Spain: a systematic review of recent publications. Eur J Health Econ. 2012, 13 (6): 72340. 10.1007/s101980110326y.
 6.
Sutton A, Ades AE, Cooper N, Abrams K: Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics. 2008, 26 (9): 75367. 10.2165/0001905320082609000006.
 7.
Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, Barrett A: Conducting indirecttreatmentcomparison and networkmetaanalysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011, 14 (4): 42937. 10.1016/j.jval.2011.01.011.
 8.
Luce BR, Simpson K: Methods of costeffectiveness analysis: areas of consensus and debate. Clin Ther. 1995, 17 (1): 10925. 10.1016/01492918(95)800123.
 9.
Drummond MF, Shemilt I, Vale L, On behalf of the Campbell and Cochrane Economic Methods group: Should The Cochrane Collaboration be Producing Reviews of Efficiency?. 2013, York, UK: The Cochrane Collaboration
 10.
Nixon J, Khan KS, Kleijnen J: Summarising economic evaluations in systematic reviews: a new approach. BMJ. 2001, 322 (7302): 15968. 10.1136/bmj.322.7302.1596.
 11.
Saramago P, Manca A, Sutton AJ: Deriving input parameters for costeffectiveness modeling: taxonomy of data types and approaches to their statistical synthesis. Value Health. 2012, 15 (5): 63949. 10.1016/j.jval.2012.02.009.
 12.
Signorovitch JE, Sikirica V, Erder MH, Xie J, Lu M, Hodgkins PS, Wu EQ: Matchingadjusted indirect comparisons: a new tool for timely comparative effectiveness research. Value Health. 2012, 15 (6): 9407. 10.1016/j.jval.2012.05.004.
 13.
Van Walraven C: Individual patient metaanalysis: rewards and challenges. J Clin Epidemiol. 2010, 63 (3): 2357. 10.1016/j.jclinepi.2009.04.001.
 14.
Briggs A, Sculpher M, Claxton K: Handbooks in Health Economic Evaluation. Decision Modelling for Health Economic Evaluation. 2006, Oxford UK: Oxford University Press
 15.
Stinnett AA, Mullahy J: Net health benefits: a new framework for the analysis of uncertainty in cost–effectiveness analysis. Med Decis Making. 1998, 18: S6880. 10.1177/0272989X9801800209.
 16.
Naveršnik K, Rojnik K: Handling input correlations in pharmacoeconomic models. Value Health. 2012, 15 (3): 5409. 10.1016/j.jval.2011.12.008.
 17.
Cuadras CM: Probability distributions with given multivariate marginals and given dependence structure. J Multivar Anal. 1992, 42: 5166. 10.1016/0047259X(92)90078T.
 18.
Genest C, MacKay J: The joy of copulas: bivariate distributions with uniform marginals. Am Stat. 1986, 40: 280283.
 19.
Nelsen RB: An Introduction to Copulas. 2006, New York: Springer, 2
 20.
Smith M: Modelling sample selection using Archimedean copulas. Econometrics J. 2003, 6: 99123. 10.1111/1368423X.00101.
 21.
Quinn C: Generalisable Regression Methods for CostEffectiveness using Copulas. HEDG Working Paper 05/13. http://www.york.ac.uk/economics/postgrad/herc/hedg/wps/wp2005/,
 22.
Quinn C: Improving Precision in CostEffectiveness Analysis using Copulas. HEDG Working Paper 07/23. http://www.york.ac.uk/media/economics/documents/herc/wp/07_23.pdf,
 23.
Murteira JMR, Lourenço OD: Health Care Utilization and SelfAssessed Health: Specification of Bivariate Models Using Copulas. HEDG Working Paper 07/27. http://www.york.ac.uk/media/economics/documents/herc/wp/07_27.pdf,
 24.
Sklar A: Fonctions de Répartition à n Dimensions et Leurs Marges. 1959, Publications de l’Institut de Statistique de l’Université de Paris, 8: 229231.
 25.
Diaz W: PhD Thesis. Contribution to Dependence and Dimensionality in Copulas. 2012, Barcelona, Spain: University of Barcelona, Statistical Department
 26.
Colas C, Valero A: Direct and Indirect Cost of Allergic Rhinitis in Spain. Reunión Anual de la Asociación Aragonesa de Alergia 2012 Plenary session, http://www.alergoaragon.org/2012/inaugural1.html,
 27.
Fermanian JD: Goodnessoffit tests for copulas. J Multivar Anal. 2005, 95 (1): 119152. 10.1016/j.jmva.2004.07.004.
 28.
Yan J: Enjoy the joy of copulas: with a package copula. J Stat Softw. 2007, 21 (4): 121.
 29.
Briggs AH, Wonderling DE, Mooney CZ: Pulling costeffectiveness analysis up by its bootstraps: a non parametric approach to confidence interval estimation. Health Econ. 1997, 6 (4): 32740. 10.1002/(SICI)10991050(199707)6:4<327::AIDHEC282>3.0.CO;2W.
 30.
Campbell MK, Torgerson DJ: Bootstrapping: estimating confidence intervals for costeffectiveness ratios. QJM. 1999, 92 (3): 17782. 10.1093/qjmed/92.3.177.
 31.
Hunink MG, Bult JR, de Vries J, Weinstein MC: Uncertainty in decision models analyzing costeffectiveness: the joint distribution of incremental costs and effectiveness evaluated with a nonparametric bootstrap method. Med Decis Making. 1998, 18 (3): 33746. 10.1177/0272989X9801800312.
 32.
Drummond M, Pang F: Transferability of economic evaluation results. Economic Evaluation in Health Care: Merging Theory with Practice. Edited by: Drummond M, McGuire A. 2001, New York (NY): Oxford University Press
 33.
Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, Eastwood A: Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess. 2004, 8 (49): 1192. iiiiv
 34.
Antonanzas F, RodríguezIbeas R, Juárez C, Hutter F, Lorente R, Pinillos M: Transferability indices for health economic evaluations: methods and applications. Health Econ. 2009, 18 (6): 62943. 10.1002/hec.1397.
 35.
TanTorres Edejer T, Baltussen R, Adam T, Hutubessy R, Acharya A, Evans DB, Murray CJL: Making Choices in Health: WHO Guide to CostEffectiveness Analysis. 2003, Geneva: World Health Organization, http://www.who.int/entity/choice/publications/p_2003_generalised_cea.pdf,
 36.
Organisation for Economic Cooperation and Development: Purchasing Power Parities for GDP dataset (1960–2012). http://stats.oecd.org/Index.aspx?datasetcode=SNA_TABLE4#,
 37.
International Monetary Fund: World Economic Outlook Database. 2013, http://www.imf.org/external/pubs/ft/weo/2013/02/weodata/index.aspx,
 38.
Shemilt I, McDaid D, Marsh K, Henderson C, Bertranou E, Mallander J, Vale L: Issues in the incorporation of economic perspectives and evidence into Cochrane reviews. Syst Rev. 2013, 2 (1): 8310.1186/20464053283.
 39.
Moreno E, Girón FJ, VázquezPolo FJ, Negrín MA: Complementing information from incremental net benefit: a Bayesian perspective. Health Serv Outcomes Res Met. 2010, 10: 8699. 10.1007/s107420100059x.
Prepublication history
The prepublication history for this paper can be accessed here:http://www.biomedcentral.com/14712288/14/139/prepub
Acknowledgments
We wish to thank Prof. Carles Cuadras for his invaluable help in providing more information on copula distributions in one of his seminars.
Author information
Additional information
Competing interest
The authors declare that they have no competing interests.
Authors’ contributions
CC developed the idea for the study, conducted the research and data analysis, wrote the COMER program, and drafted the report. WD provided background information based on experience with copula distributions. All investigators contributed to the final version of the report and they agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All authors read and approved the final manuscript.
Electronic supplementary material
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Crespo, C., Monleon, A., Díaz, W. et al. Comparative efficiency research (COMER): metaanalysis of costeffectiveness studies. BMC Med Res Methodol 14, 139 (2014). https://doi.org/10.1186/1471228814139
Received:
Accepted:
Published:
Keywords
 Costeffectiveness analysis
 Metaanalysis
 Incremental net benefit
 Copula distribution