Skip to main content

Table 1 Examples for quantitative approaches for benefit-harm assessment

From: A framework for organizing and selecting quantitative approaches for benefit-harm assessment

NNT and NNH

Examples where researchers consider single benefit and harm outcomes with or without a benefit-harm comparison metric

NNT and its harm outcome counterpart, NNH, are the approaches researchers most widely use to measure risk and benefit reported in systematic reviews and evidence-based medicine [12, 13]. Also, NNT is the metric clinical practice guidelines most commonly use to address benefit-harm comparisons. NNT or NNH are the number of individuals who need to be treated over a specified period of time for one person to benefit or be harmed, respectively, and vary as the specified treatment time varies. Studies mostly present NNT and NNH separately (i.e. they are not combined on a benefit-harm comparison metric such as the ratio of NNT and NNH). For example, the Based Clinical Practice Guidelines on Antithrombotic Therapy in Atrial Fibrillation of the American College of Chest Physicians present NNTs based on a systematic review of randomized trials of oral anticoagulant therapy versus no antithrombotic therapy: “The efficacy of warfarin was consistent across studies with an overall relative risk reduction of 68 percent (95 percent confidence interval [CI], 50 to 79 percent) analyzed by intention-to-treat [18]. The absolute risk reduction implies that 31 ischemic strokes will be prevented each year for every 1,000 patients treated (or 32 patients needed to treat for 1 year to prevent one stroke, NNT = 32)”.In contrast, studies scarcely use the NNT/NNH ratio. One reason for the rare application of this benefit-harm comparison metric may be that investigators or guideline developers are reluctant to weigh benefit and harm outcomes equally on the same scale because of uncertainty about their relative clinical importance. To address this dilemma, Guyatt et al. proposed using relative value units to weight the NNT or NNH [19]. An example using the NNT/NNH ratio is a review of trials of an antidepressant drug which summarized benefit defined as response and remission of depression and harm as suicide [20].

Multicriteria decision analysis

Example where multiple benefit and harm outcomes and preferences are considered

The Analytic Hierarchy Process A is a commonly used approach for MCDA studies is. We illustrate this approach using the comparative effectiveness review of oral hypoglycemic agents for type 2 diabetes. The first step in Analytic Hierarchy Process analysis consists of defining the goal of the decision, the alternatives being considered, and the criteria that determines how well patients and clinicians can expect that alternatives will meet the goal [4, 21]. The criteria are organized into a hierarchical decision model with a desired goal of determining the best treatment of type 2 diabetes at the top; the alternatives thiazolidinediones, metformin, and sulfonylurea’s at the bottom; and the criteria in between. Operationally, we could define two criteria as being necessary for determining the best treatment: 1) its ability to maximize benefits via glucose reduction, and 2) its ability to minimize harms or medication-related adverse effects. Researchers could divide the criteria on maximizing benefits into three sub-criteria: health-related quality of life, microvascular benefit (such as improvements in incidence of neuropathy, nephropathy and diabetic retinopathy), and potential macrovascular benefit. Researchers could subdivide the criteria on minimizing risk of harm into six sub-criteria based on medication-related adverse events: congestive heart failure, fractures in women, macular edema, bladder cancer, myocardial infarction, and hypoglycemia. In the second step, researchers obtain information about how well the alternatives can be expected to fulfill the decision criteria from a systematic review. The third step consists of two parts: 1) comparing the ability of the alternative treatments to fulfill the prespecified criteria (maximizes benefit and minimizes harm) using standard Analytic Hierarchy Process pairwise comparisons, and 2) assessing the importance of these criteria to the decision goal. In the fourth step, researchers can combine the scales created in step 3 to create a summary score (the benefit-harm comparison metric) indicating how well the alternative treatments can be expected to meet the decision goal [21]. The fifth step consists of performing sensitivity analyses to explore the effects of changing the estimates or judgments used in the original analysis. The main advantage of MCDA is that it is that identifies the extent to which every criterion, judgment, and weight contributes to the benefit-harm comparison metric and that it also incorporates uncertainty. Additional visual representation of the results allow one to gain understanding and articulate the divergence between relevant stakeholders [4].

Gail/National Cancer Institute

Example where multiple benefit and harm outcomes are considered with a benefit-harm comparison metric

Some decisionmaking contexts are more complicated because there are many potential treatment outcomes as well as sources of uncertainty. A well known example of a very challenging decision is whether or not to use of tamoxifen to prevent breast cancer. Tamoxifen reduces the risk for invasive and in situ breast cancer substantially and prevents some bone fractures [2]. On the other hand, it increases the risk for endometrial cancer, stroke, and pulmonary embolism. The National Cancer Institute under the leadership of Gail developed an approach to deal with multiple outcomes [3]. Rather than simplifying the benefit-harm assessment to single outcomes, as many investigators and guideline developers do, they estimated the probability of various outcomes for women with and without tamoxifen therapy over a period of 5 years. Based on observational studies, surveillance registries or placebo arms of randomized trials, they first estimated the expected number of invasive breast cancers, in situ breast cancers, hip fractures, endometrial cancers, strokes, pulmonary embolisms, deep vein thromboses, colles’ fractures, spine fractures, and cataracts each per 10,000 women, over 5 years, in the absence of tamoxifen treatment. They estimated these numbers overall and stratified for different age and race categories. Then, based on the Breast Cancer Prevention Trial, they estimated, for each outcome, the expected number of the same outcomes but with tamoxifen treatment. Again this was per 10,000 women over 5 years, as well as overall, and it was stratified for different age and race categories. They also took competing risk from death into consideration. In order to put all outcomes on the same scale but to also consider the relative clinical importance of these outcomes, they categorized the outcomes into life-threatening, severe and other outcomes and suggested weighting them with some factor (e.g. 1 for life-threatening, 0.5 for severe and 0.0 for other outcomes). These categories and weights can be modified according to patient or treatment provider preferences. Ultimately, the results of the benefit-harm assessment are presented as the net number of events prevented or in excess per 10,000 women treated with tamoxifen over a period of 5 years. For example, for a 45-year-old woman with a uterus and a 4 percent risk of invasive breast cancer over 5 years, the net number of events prevented (weighted by their clinical importance) is 196 per 10,000 women with this profile (the expected number of prevented invasive and in situ breast cancers was 299/10,000 woman but there were 59 women/10,000 woman with harm such as endometrial cancer, stroke, pulmonary embolism, or deep vein thrombosis). The net benefit (benefit minus harm events) varied considerably and was positive for some profiles (as example above) but negative for others (e.g. black woman with age 50–59 years and a 5-year risk of invasive breast cancer of 4 percent).