On the probability of costeffectiveness using data from randomized clinical trials
 Andrew R Willan^{1, 2}Email author
DOI: 10.1186/1471228818
© Willan; licensee BioMed Central Ltd. 2001
Received: 23 May 2001
Accepted: 6 September 2001
Published: 6 September 2001
Abstract
Background
Acceptability curves have been proposed for quantifying the probability that a treatment under investigation in a clinical trial is costeffective. Various definitions and estimation methods have been proposed. Loosely speaking, all the definitions, Bayesian or otherwise, relate to the probability that the treatment under consideration is costeffective as a function of the value placed on a unit of effectiveness. These definitions are, in fact, expressions of the certainty with which the current evidence would lead us to believe that the treatment under consideration is costeffective, and are dependent on the amount of evidence (i.e. sample size).
Methods
An alternative for quantifying the probability that the treatment under consideration is costeffective, which is independent of sample size, is proposed.
Results
Nonparametric methods are given for point and interval estimation. In addition, these methods provide a nonparametric estimator and confidence interval for the incremental costeffectiveness ratio. An example is provided.
Conclusions
The proposed parameter for quantifying the probability that a new therapy is costeffective is superior to the acceptability curve because it is not sample size dependent and because it can be interpreted as the proportion of patients who would benefit if given the new therapy. Nonparametric methods are used to estimate the parameter and its variance, providing the appropriate confidence intervals and test of hypothesis.
Introduction
In reporting costeffectiveness analyses alongside clinical trials, authors [1–4] have used various definitions, estimation methods and interpretations for acceptability curves. Acceptability curves provide an excellent means of quantifying the stochastic uncertainty of the estimated incremental costeffectiveness ratio (ICER) in relation to a particular value ascribed to a unit of effectiveness. It is the certainty, expressed as a probability, that the current evidence would lead us to believe that some new therapy is costeffective, insofar as the ICER is less than the value ascribed to a unit of effectiveness. In addition, acceptability curves provide an estimator for the ICER and its confidence limits. However, acceptability curves are often interpreted and expressed as the probability that the new therapy is costeffective. It is argued below that this is subject to misinterpretation, and an alternative definition for a parameter representing the probability that the new therapy is costeffective is introduced. Data from a clinical trial can be used to make statistical inference about this parameter. Furthermore, the inference provides a nonparametric estimator for the ICER and its confidence interval.
In a twoarm randomized control trial let e_{ji} and c_{ji} be the respective measures of effectiveness and cost for patient i on therapy j, where j = T (treatment), S (standard); i = 1, 2, . . .n_{j} ; and n_{j} is the number of patients randomized to therapy j. Typically, e_{ji} is the patient's survival time (perhaps qualityadjusted) from randomization to death or to the end of the period of interest. Let . Define _{c} similarly. Let E( _{e}) = Δ_{e} and E( _{c}) = Δ_{c}, where E is the expectation function. Thus, the incremental costeffectiveness ratio (ICER) is Δ_{c}/Δ_{e}, and is estimated by _{c} / _{e}. In addition, the incremental net benefit [5–9] (INB) is Δ_{e}λΔ_{c}, and is estimated by _{e}λ  _{c}, where λ is the value given to a unit of effectiveness. Typically the INB is expressed as a function of λ, allowing readers to provide the value they consider most relevant.
van Hout et al. [1] define the acceptability curve as "the probability that the [ICER] is under a certain acceptable limit", say λ. The acceptability curve, then, is a function of λ. If one assumes that the authors are referring to the true ICER ratio, the definition is Bayesian. In the same paper they define the acceptability curve in algebraic terms as , where f is the joint probability density function for the random vector ( _{e}, _{c})'. Here the definition is not Bayesian, since it is the probability that, in repeated sampling, the random variable _{e}λ  _{c} (i.e., the observed net benefit) is greater than 0. In an illustration, the authors substitute the sample estimates for the model parameters in f to yield an empirical density function, call it , and refer to the acceptability curve as the probability that the ICER is acceptable. Briggs and Fenn [2] refer to the acceptability curve as "the probability an intervention is costeffective in relation to different values of" λ. For estimation they propose using the integration of , as above, or determining the proportion of bootstrap resamples in which _{e}λ  _{c} is greater than 0.
Briggs[3] provides a purely Bayesian approach by defining the acceptability curve as the probability that Δ_{e}λ  Δ_{c} is greater than 0. In an illustration the author interprets the acceptability curve as "probability of costeffective". Assuming f is the density function for a bivariate normal random vector, and using an uninformative prior, the acceptability curve is given by A(λ)= g(x)dx, where g is the probability density function for a normal random variable with mean _{e}λ  _{c} and variance , where and are estimates of the variance of e_{ji} and c_{ji}, respectively, and is an estimate of the correlation between e_{ji} and c_{ji} . This is exactly the same curve as determined by the integration of , given above, and, due to the symmetry, is equal to 1 minus the pvalue of the test of the hypothesis Δ_{e}λ  Δ_{c} < 0. In reporting the results of a costeffectiveness analysis, Raikou et al. [4] interpret the acceptability curve as the "probability that intervention is cost effective".
By rewriting Pr(Δ_{e}λ  Δ_{c} > 0) as Pr(Δ_{c}/Δ_{e} < λ), assuming Δ_{e} > 0, the acceptability curve can be interpreted as the posterior distribution for the ICER. Defining A(λ_{γ}) = γ, the estimate for the ICER and its (1α/2)100% Baysian limits are given by λ_{0.5}, λ_{α/2} and λ_{1α/2}, respectively.
The interpretation that the acceptability curve is the probability that the intervention is costeffective is not entirely accurate and could easily be misunderstood by policy makers. Consider the situation in which the observed INB for treatment is very small, but due to a very large sample size the acceptability curve at the value of λ of interest is 0.99. Attaching the label "the probability that the intervention is costeffective" to this quantity could mislead policy makers into thinking that treatment is highly beneficial compared to standard. What, in fact, is high is our confidence that the INB, however small, is not zero. A more accurate interpretation of the acceptability curve is that it is a measure of the certainty with which the current evidence would lead us to believe that treatment is costeffective, i.e., Δ_{e}λ  Δ_{c} > 0. For a Bayesian, this is Pr(Δ_{e}λ  Δ_{c}) > 0, and for a frequentist, it is, assuming symmetry, 1 minus the pvalue for the test of the hypothesis Δ_{e}λ  Δ_{c} < 0. This is not just the traditional confusion between statistical and clinical significance. In significance testing as the sample size increases, the variance of the estimator decreases, but the magnitude of the parameter being estimated stays the same. However, as sample size increase the magnitude of the acceptability curve for a given λ increases.
In the next section we provide a more accurate definition for the probability that treatment is costeffective. The definition contains no element of certainty, and is the probability of the "next" patient realizing a larger net benefit if he or she is given treatment rather than standard. Using data from a clinical trial, nonparametric methods can be used to estimate this probability, and uncertainty
Methods
The probability that treatment is costeffective
Let b_{ji}(λ) = e_{ji}λ  c_{ji}. The quantity b_{ji}(λ) is the net benefit, expressed in money, realized by patient i on therapy j. An alternative to the acceptability curve for quantifying the probability that treatment is costeffective is defined as: θ(λ) ≡ the probability, for a given λ, that a patient will receive a larger net benefit with treatment rather than standard. Its definition is not Bayesian because it is a probability statement about random variables, not population parameters. Nonetheless, it relates directly to the notion of the probability of treatment being costeffective. A policy maker can genuinely interpret θ(λ) as the probability of the "next" patient realizing a larger net benefit if he or she is given treatment rather than standard. Such a direct interpretation is not provided by the acceptability curve. The acceptability curve is the probability that the average net benefit of a group of patients of the same size as in the clinical trial will be greater is they receive treatment rather than standard.
To estimate θ(λ) we borrow methodology from receiver operating characteristic curves [10]. An estimate of θ(λ) is given by:
The hypothesis H_{0}: θ (λ) = 0.5 versus H_{1}: θ (λ) > 0.5 can be tested at the level α by rejecting H_{0} if . Rejecting H_{0} leads to the conclusion that the data provide evidence that treatment is costeffective.
Example
Sample sizes and parameter estimates for prostate example
n_{j}  average  average 


 

effectiveness  cost  
Standard  53  28.1  29039  16.4  7,872,681  2,876 
Treatment  61  40.9  27322  24.1  6,466,351  2,771 
Since the lower bound crosses the 0.5 horizontal at 334, for any value greater than 334 the hypothesis θ(λ) < 0.5 can be rejected at the 5% level of significance. Thus, a nonparametric upper bound of the ICER is 334. This compares to 378 using Fieller's theorem [13, 14]. For this example a health policy maker can interpret the results as follows: for any positive λ, the estimated probability that treatment is costeffective is greater than 50%; and for any λ greater than 334 per QALW, the probability that treatment is costeffective is statistically significantly greater than 50%.
Discussion
As an alternative to the acceptability curve, the quantity θ(λ) is proposed as a definition for the probability that treatment is costeffective. One advantage is that it is not sample size dependent, i.e. it is a population parameter. Another is that it has an appropriate interpretation, namely, it is the proportion of patients that realize a larger net benefit if given treatment rather than standard. The acceptability curve does not provide this, although the language often used regarding it, implies that it does. The use of θ(λ) should be helpful to policy makers, since it does not confuse the magnitude of the benefit with the certainty of its estimate.
Analysis regarding the quantity θ(λ) is not proposed as an alternative to traditional costeffectiveness analysis for allocating health care resources. When allocating a fixed amount of resources to one of two new treatments, the proportion of patients receiving an increase in net benefit would be maximized by choosing the treatment with the larger θ(λ) to Δ_{c} ratio. However, this would not maximize net benefit, since the ratio may be larger only because the betweenpatient variability in cost and effectiveness is smaller, resulting in a larger θ(λ).
Nonparametric methods can be used to estimate θ(λ) and its variance. This provides the appropriate confidence intervals and test of hypothesis. In addition, nonparametric estimates of the ICER and its confidence intervals can be determined. This is of particular importance in the presence of highly skewed cost data.
Declarations
Acknowledgements
The author wishes to acknowledge the reviewers whose comments led to a much improved manuscript and to thank Gary Foster for help with the figure.
Authors’ Affiliations
References
 van Hout BA, Al MJ, Gordon GS, Rutten EFH: Cost, effects and C/Eratios alongside a clinical trial. Health Economics. 1994, 3: 309319.View ArticlePubMedGoogle Scholar
 Brigg A, Fenn P: Confidence intervals or surfaces? Uncertainty on the costeffectiveness plane. Health Economics. 1998, 7: 723740. 10.1002/(SICI)10991050(199812)7:8<723::AIDHEC392>3.0.CO;2O.View ArticleGoogle Scholar
 Briggs AH: A Bayesian approach to stochastic costeffectiveness analysis. Health Economics. 1999, 8: 257261. 10.1002/(SICI)10991050(199905)8:3<257::AIDHEC427>3.0.CO;2E.View ArticlePubMedGoogle Scholar
 Raikou M, Gray , Briggs , Stevens A, Cull C, McGuire A, Fenn P, Stratton I, Holman R, Turner R: Cost effectiveness analysis of improved blood pressure control in hypertensive patients with type 2 diabetes: UKPDS 40. British Medical Journal. 1998, 317: 720726.View ArticleGoogle Scholar
 Phelps CE, Mushlin AI: On the (near) equivalence of costeffectiveness and costbenefit analysis. International Journal of Technology Assessment in Health Care. 1991, 7: 1221.View ArticlePubMedGoogle Scholar
 Ament A, Baltussen R: The interpretation of results of economic evaluation: explicating the value of health. Health Economics. 1997, 6: 625635. 10.1002/(SICI)10991050(199711)6:6<625::AIDHEC309>3.0.CO;2O.View ArticlePubMedGoogle Scholar
 Stinnett AA, Mallahy J: Net health benefits: a new framework for the analysis of uncertainty in costeffectiveness analsyis. Medical Decision Making. 1998, 18 Suppl.: S68S80.View ArticleGoogle Scholar
 Tambour M, Zethraeus , Johannesson M: A note on confidence intervals in costeffectiveness analysis. International Journal of Technology Assessment. 1998, 14: 467471.View ArticleGoogle Scholar
 Willan AR, Lin DY: Incremental net benefit in randomized clinical trials. Statistics in Medicine. 2001, 20: 15631574. 10.1002/sim.789.View ArticlePubMedGoogle Scholar
 DeLong ER, DeLong DM, ClakePearson DL: Comparing the area under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988, 44: 837845.View ArticlePubMedGoogle Scholar
 Tannock IF, Osoba D, Stockler MR, Ernst DS, Neville AJ, Moore MJ, Armitage GR, Wilson JJ, Venner PM, Coppin CM, Murphy KC: Chemotherapy with mitoxantrone plus prednisone or prednisone alone for symptomatic hormoneresistant prostate cancer: a Canadian trial with palliative endpoints. Journal of Clinical Oncology. 1996, 14: 17561764.PubMedGoogle Scholar
 Bloomfield DJ, Krahn MD, Neogi T, Panzarella T, Warde P, Willan AR, Ernst S, Moore MJ, Neville A, Tannock IF: Economic evaluation of chemotherapy with mitoxantrone plus prednisone for symptomatic hormoneresistant prostate cancer: based on a Canadian trial with palliative endpoints. Journal of Clinical Oncology. 1998, 16: 22722279.PubMedGoogle Scholar
 Willan AR, O'Brien BJ: Confidence intervals for costeffectiveness ratios: An application of Fieller's theorem. Health Economics. 1996, 5: 297305. 10.1002/(SICI)10991050(199607)5:4<297::AIDHEC216>3.0.CO;2T.View ArticlePubMedGoogle Scholar
 Chaudhary MA, Stearns SC: Confidence intervals for costeffectiveness ratios: an example from a randomized trial. Statistics in Medicine. 1996, 15: 14471458. 10.1002/(SICI)10970258(19960715)15:13<1447::AIDSIM267>3.0.CO;2V.View ArticlePubMedGoogle Scholar
 The prepublication history for this paper can be accessed here:http://www.biomedcentral.com/14712288/1/8/prepub
Prepublication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.