Methodological standards in non-inferiority AIDS trials: moving from adherence to compliance

Study selection and methodological standards

Our aim was to consider a cohort of equivalence or non-inferiority trials published in the area of HIV/AIDS, after HAART became available. We performed a MEDLINE search using the terms equivalence OR non-inferiority AND random* AND HIV (1) and abacavir AND random* (2). 64 (1) and 136 (2) articles were identified. 5 (1) and 5 (2) were selected because they fulfilled the following requirements: randomized controlled clinical trial with 48-week minimum follow-up, initially designed as a non-inferiority or equivalence trial with a prespecified non-inferiority margin, virological primary endpoint and publication in New England Journal Medicine, JAMA, Lancet, AIDS, Clinical Infectious Diseases, Journal of Infectious Diseases and Journal of Acquired Immune Deficiency Syndrome between 2001 and 2006. Eight additional articles were identified by examining cross-references or by authors' knowledge of their existence.

We applied traditional methodological requirements for non-inferiority and equivalence trials adapted from Kirshner[7], Jones et al. [8], McAlister and Sackett [9] and Piaggio et al.[1] to eighteen [10–27] active-control trials. We also applied proposed standards in the report of non-inferiority and equivalence trials adapted from Le Henanff [28].

Statistical analysis

Intent-to-treat (ITT) or on-treatment (OT) analysis 95% confidence interval of the treatment difference were computed using the normal approximation, based on available data included in the flow chart, results section and figures. Two selected studies (ALIZE and SEAL) predefined a 90% confidence interval of the treatment difference, but their conclusions were not affected by the use of the 95% confidence interval (which was used in this paper for homogeneity). Two other selected studies (BMS-045 and CONTEXT) defined the primary endpoint as the log₁₀ reduction in HIV viral load, using a time-averaged difference method. For homogeneity with other studies, we considered the more pertinent criteria (closer to the clinical practice) of the percentage of patients with undetectable viral load (< 50 copies/ml or < 400 copies/ml) at week 48 (reported as secondary endpoint).

In case of missing data, the corresponding author of the paper was contacted. When only percentages were available with several possibilities for the numerator due to rounding, we choose on a worst case basis. If original data were censored, we used the cumulative incidence of the primary endpoint in each arm.

Significance testing in establishing non-inferiority between the two arms of a study was computed by the use of the continuity-corrected chi-square of Dunnett and Gent [29] for non-inferiority in intent-to-treat or on-treatment analysis, also on a worst case basis. Briefly, π ₁ and π ₂ represent the true proportions of patients with treatment success according to the primary outcome in a random sample of the 2 populations of patients receiving the control treatment and the new drug, respectively. In case of non-inferiority, the expected estimates of π ₁ and π ₂ are given by:

$\begin{array}{cc}{\widehat{\pi }}_1=\frac{x+y+n_2\Delta }{n_1+n_2}& {\widehat{\pi }}_2=\frac{x+y-n_1\Delta }{n_1+n_2}\end{array}$

where x and y are the observed number of success, n ₁ and n ₂ are the 2 sample sizes in the control and the experimental study groups, respectively and Δ the pre-specified margin for non-inferiority.

The continuity-corrected chi-square of Dunnett and Gent [29] (reproduced with written permission) for non-inferiority is given by:

${\chi }_c^2={(\left|x-\widehat{x}\right|-\frac{1}{2})}^2\left[\frac{1}{\widehat{x}}+\frac{1}{m-\widehat{x}}+\frac{1}{n_1-\widehat{x}}+\frac{1}{n_2-m+\widehat{x}}\right]$

where m = x + y and $\widehat{x}$ = $\widehat{\pi }$ ₁ n ₁

If Δ is the maximal acceptable difference in success rates between the 2 treatment arms and δ is the observed difference between the experimental and control arms, the equivalence hypothesis can be formulated as pair of one-sided hypothesis:

H ₀₁ : δ ≥ Δ versus H _a1 : δ < Δ with a type I error of α₁       (1)

and

H ₀₂ : δ ≥ - Δ versus H _a2 : δ > - Δ with a type I error of α₂       (2)

The type I error probability α for H ₀ rejection corresponds to H ₀₁ ∪ H ₀₂. Therefore, the P-value for equivalence is the lower chi-square value associated with max (α₁, α₂). In a non-inferiority hypothesis, only (1) is necessary. More details have been published elsewhere[30].

To avoid confusion between the P-values of superiority tests and the P-values of non-inferiority tests (both are reported in this paper), the latter have been renamed "D-values". When the normal approximation is a valid hypothesis, there is a general consistency between the two-sided 95% confidence interval approach (non-inferiority at α/2 < 2.5%) and the non-inferiority chi-square (D-value < 5%), as shown in Figure 1. D-values and P-values < 0.05 were considered statistically significant.

Efficacy of the active control and similar outcome

Rationale for the non-inferiority margin

All studies identified a pre-specified non-inferiority margin (criterion for selection). As shown in Figure 2, however, only 4/18 studies reported justification for their choice. In the CNAAB3005 study, the choice of the non-inferiority margin was based on discussion with clinical investigators and with the Food and Drug Administration. The margin of 12% was considered as the largest difference clinically acceptable. In the 903 study, the authors considered that the margin of 10% was a more stringent and conservative non-inferiority criterion. The authors of the CNA30024 commented that it was the appropriate measure for distinguishing the clinical effectiveness of 2 study treatment. Finally, the CNA30024 authors' choice relied on HIV clinicians' judgement as well as on discussion with independent reviewers. Other studies did not comment on their choice, which ranged from 10% to 15% (median: 12%). CONTEXT and BMS-045 considered a non-inferiority margin of -0.5 log₁₀ reduction in HIV viral load, without justification. Other issues regarding design are reported in Table 1.

Confidence interval and superiority testing

All but two trials reported results using the confidence interval approach. In the BEST study, the authors predefined their non-inferiority margin for sample size calculation, but the confidence interval was neither defined nor reported. In the NEFA study, although the confidence interval approach was clearly defined in the statistical analysis section of the article, none was provided in the results section. NEFA, BEST, 2NN, FTC-303, ESS40013 and SHAART studies reported non-significant superiority tests for efficacy to reinforce non-inferiority. The ALIZE and 934 studies switched from the non-inferiority to the superiority hypothesis to declare that the experimental treatment had superior efficacy in the ITT analysis set (for secondary and primary endpoints, respectively), as appropriate.

Intent-to-treat and on-treatment analysis on the primary endpoint

CNAAB3005, NEFA, SOLO, BEST, EPV20001, ALIZE, BMS-2004, SEAL and SHAART studies (Figure 2) published both ITT and OT analysis (9/18), but only the ALIZE, SOLO, EPV20001 BMS-2004 and SEAL studies found concordant results regarding non-inferiority in the two analysis. The BEST investigators provided separate conclusions for ITT and OT, as appropriate. The ALIZE-trial group conducted ITT, OT and a worst scenario analysis. In CNAAB3005 NEFA and SHAART, the conclusion was based on ITT analysis only. 2NN, FTC-303, EPV20001, ESS40013, CNA30021 and 934 studies described sufficient details to permit alternative analyses, such as OT. We have failed to compute OT analysis from the 903 and CNA30024 studies. Because of the nature of their primary outcome, CONTEXT and BMS-045 studies were not able to provide ITT and OT analysis. Both analysis were provided as secondary endpoints.

Interpretation and conclusion (non-inferiority margin versus observed upper bound of the 95% confidence interval of the difference)

BEST, SOLO, FTC-303, EPV20001, ALIZE, CNA30024, SEAL, CNA30021 and 934 conclusions' were appropriate, on the basis of available data.