Adaptive designs based on the truncated product method
© Neuhäuser and Bretz; licensee BioMed Central Ltd. 2005
Received: 12 December 2004
Accepted: 19 September 2005
Published: 19 September 2005
Adaptive designs are becoming increasingly important in clinical research. One approach subdivides the study into several (two or more) stages and combines the p-values of the different stages using Fisher's combination test.
Alternatively to Fisher's test, the recently proposed truncated product method (TPM) can be applied to combine the p-values. The TPM uses the product of only those p-values that do not exceed some fixed cut-off value. Here, these two competing analyses are compared.
When an early termination due to insufficient effects is not appropriate, such as in dose-response analyses, the probability to stop the trial early with the rejection of the null hypothesis is increased when the TPM is applied. Therefore, the expected total sample size is decreased. This decrease in the sample size is not connected with a loss in power. The TPM turns out to be less advantageous, when an early termination of the study due to insufficient effects is possible. This is due to a decrease of the probability to stop the trial early.
It is recommended to apply the TPM rather than Fisher's combination test whenever an early termination due to insufficient effects is not suitable within the adaptive design.
Randomized controlled experiments were introduced by Sir Ronald A. Fisher in the 1920s for agricultural studies and not in order to compare the effects of different treatments in humans. However, according to Palmer  the way clinical trials are conducted today is essentially unchanged from Fisher's day. In contrast to agricultural studies most clinical trials require periodic monitoring of the accumulating data, e.g. to minimize the number of experimental patients who will continue with an inferior treatment [, p. 360].
Adaptive designs with at least one interim analysis can potentially be used for periodic monitoring. All information from the first stage(s) can be used to plan the following stage(s). A number of adaptive designs have been proposed recently, for an overview see Bauer et al. . Here, we consider the adaptive procedure according to Bauer and Köhne  that uses Fisher's product test.
Let k be the number of stages (i.e., there are k - 1 interim analyses), and let p i be the one-sided p-value observed with the i-th stage's data, i = 1, ..., k. According to Fisher's product criterion [, pp. 37–39] the null hypothesis H0 can be rejected at the end of the trial if
In clinical trials boundaries for early stopping after an interim analysis may be incorporated. Obviously, in the case of p 1 ≤ c α early stopping with the rejection of H0 is possible after stage one. In general, H0 can be rejected after the j-th stage if . In addition, one may terminate the trial due to insufficient effects. A lower limit α0 can be included so that the trial is terminated without rejecting H0 if p 1 ≥ α0. According to Bauer and Köhne [, p. 1031] a value of 0.5 may be a suitable choice for α0. Bauer and Röhmel [, p. 1596] recommended α0 = 1 for establishing a dose-response relationship, that is, no early stopping without rejecting H0 at all. In this context, an early stopping due to insufficient effects is not feasible since doses in a plateau region could have been used. In that case, different doses may be used in the following stage.
Note that, in case of α0 < 1, larger boundaries for apply for early stopping with the rejection of H0. For a two-stage design, one can reject H0 after stage one if p 1 ≤ α1 for a value of α1 that lies between c α and α [4, 6]. This value can be calculated iteratively using the formula [, p. 1032]
As an alternative to Fisher's product test, Zaykin et al.  recently introduced a truncated product method for combining p-values. To be precise, instead of calculating the product of all p-values, they suggested the use of the product of only those p-values that do not exceed some fixed cut-off value τ, 0 < τ ≤ 1. The truncated product W τ is defined as
where I(.) is the indicator function. Since the p-values of the different stages are independent,
When using the truncated product method, the (1-α)-quantile of the distribution of W τ, , is the critical value for the combination test. Analogous to Fisher's combination test an can be calculated for given α0 such that the overall type I error rate is α.
Zaykin et al.  and Neuhäuser  investigated the truncated product method for combining a large number of p-values and demonstrated by simulation that it can provide high power. In this paper we investigate whether the truncated product method is also useful for the adaptive design described above. In contrast to previous applications [7, 8] we consider classical experimental questions involving only few p-values. Very recently, a rank truncated product was proposed as a further alternative . That method uses the product of the K most significant p-values where K can be chosen. Since we consider the combination of 2 to 4 p-values only, the rank truncated product does not seem to be appropriate for our aim.
We first present the comparison of the combinations with and without truncation for designs with two stages. Afterwards, designs with more than two stages are investigated. We then illustrate the method using two examples, and conclusions are given in final section.
In order to compare the adaptive procedures with and without truncation we consider the situation of two parallel groups with means μ1 and μ2. There are 100 observations per stage. These observations are subdivided into two groups and are assumed to be normally distributed with a common, but unknown variance σ2. Student's t test is performed in each of the two stages with a one-sided significance level of α = 5%.
The overall p-value, i.e. the p-value of the combination test, is defined as follows : In case the study stops after stage 1, the overall p-value equals p 1. Otherwise, the overall p-value is for Fisher's combination test and for the truncated product test.
The case α0= 0.5
First, we consider a study that is terminated early due to insufficient effects if p 1 ≥ α0 = 0.5. Without any truncation (i.e., τ = 1) we have c α = 0.0087 and α1 = 0.0233 in this case . However, when we set τ = α0 = 0.5, a smaller value for α1 but a larger boundary for is obtained. To be precise, the trial can be terminated early with the rejection of H0 if p 1 ≤ = 0.0190, and there is a significance at the end of the trial if W τ = 0.5 ≤ = 0.0095.
Power to reject H0 in a two-stage design with α0 = 0.5 (combination of t tests, one-sided, α = 0.05)
25 observations per group in stage one, 75 observations per group in stage two
τ = 1
τ = 0.5
50 observations per group and stage
τ = 1
τ = 0.5
75 observations per group in stage one, 25 observations per group in stage two
τ = 1
τ = 0.5
Boundaries c α and for two to four stages
Number of stages (k)
for τ = 0.5
α = 0.025
α = 0.05
For instance, in the case of 50 observations per group and stage and δ = 0.4 (α = 0.05) the probabilities to reject H0 after the first stage are Pr(p 1 ≤ α1) = 0.496 and Pr(p 1 ≤ ) = 0.461, respectively. The probability to stop without rejecting H0 is Pr(p 1 ≥ α0) = 0.023 irrespective of truncation. With the fixed sample size of 100 per stage the expected total sample size is 200 - 100·Pr(stop after first stage). This expected total sample size is 148 for τ = 1, but 152 in case of truncation. Hence, the slight increase in power is connected with a larger expected total sample size.
An a priori fixed sample size for stage two is uncommon within an adaptive design. Instead, a sample size reassessment can be carried out during the interim analysis . Using p 1 and the difference and variability observed in stage one, we simulated the sample size for stage two needed for an overall power of 80%. The results (not shown) indicate that, in this case, the application of the truncated product method can lead to a smaller expected total sample size.
Nevertheless, there is still a smaller probability to stop the trial after the first stage when the truncation is applied. That is a clear disadvantage in clinical development where early decisions are desirable. Therefore, despite the (small) improvement in terms of power, a truncation does not seem to be preferable within a two-stage adaptive design when α0 < 1.
The case α0= 1
As mentioned in the introduction, α0 = 1 can be a suitable choice, for example when establishing a dose-response relationship. The choice α0 = 1 leads to the same rejection boundary c α for the interim and the final analysis, respectively. Hence, there is α1 = c α and . Since c α < , the expected total sample size is decreased due to truncation even in case of a fixed sample size for stage two. For instance, in the case of 50 observations per group and stage and δ = 0.4 (α = 0.05) the probability to reject H0 after the first stage is Pr(p 1 ≤ α1) = 0.342 for τ = 1, but Pr(p 1 ≤ ) = 0.354 for τ = 0.5. The resultant expected total sample sizes are 166 and 165, respectively. Therefore, a gain in power would be of more importance in case α0 = 1.
We now present results for adaptive designs with three and four stages, respectively, and α0 = 1. Again, the behaviour of the strategies is investigated for fixed sample sizes in the separate study stages without including the option for sample size reassessment. The trial can be terminated with the rejection of H0 after the j-th stage if in case of τ = 1 or if in case of truncation. For up to four stages, Table II displays the boundaries c α and for τ = 0.5.
Simulated power to reject H0 and expected total sample sizes in three- and four-stage designs with α0 = 1 (50 observations per group and stage, combination of t tests, one-sided, α = 0.05)
τ = 1
τ = 0.5
Expected total sample
τ = 1
τ = 0.5
τ = 1
τ = 0.5
Expected total sample
τ = 1
τ = 0.5
In this section we only consider the case α = 0.025 and α0 = 1. The first example discussed in this section was presented by Bauer and Röhmel . In a two-stage dose-response study the effect of a new drug on blood pressure was investigated. Assume that the trial would have started with two medium doses. The p-value for the one-sided t test between these two doses in the interim analysis was p 1 = 0.206. Thus, the study continued with the comparison placebo vs. a higher dose, and the second stage led to p 2 = 0.0178. The product in the final analysis was p1p2 = 0.00367, the corresponding overall p-value of the non-truncated product test is 0.024. Hence, the combination test is significant even at the 0.025 level.
The second example is a hypothetical clinical study with two stages. We consider a scenario as Bauer and Köhne [, p. 1038] in their example. A clinical trial investigates a new therapy for an indication in which no efficient standard therapy is available. For the first stage five individual endpoints have been selected. The first stage's sample size is 30 each in the therapy and the control group. The changes to the baseline measurements of the five endpoints were combined into a single generalized least squares (GLS) criterion according to O'Brien , and the first stage's p-value was p 1 = 0.1758. Hence, the study continued.
The overall p-value of the final analysis based on the combination of p 1 = 0.1758 and p 2 = 0.1517 (second example) in dependence of the truncation point τ, τ = 1 corresponds to Fisher's product criterion.
p-value for TPM
The application of the truncated product method instead of Fisher's combination test within an adaptive design hardly changes the overall power. Therefore, to decide whether or not a truncation is useful one should focus on the probability to stop early and on the expected total sample size. According to these criteria, a truncation seems to be preferable in case of α0 = 1, but not for α0 < 1.
A variety of other combination functions exists , for example, the inverse normal method was proposed for adaptive designs . According to Rice  Fisher's test is "inappropriate when asking whether a set of tests, on balance, supports or refutes a common null hypothesis ... because ... Fisher's statistic is more sensitive to smaller, as compared to larger, P-values" [, p. 303–305]. In contrast, the inverse normal method is not differentially sensitive to data that support or refute a common null hypothesis. Thus, one may argue that the inverse normal method is more appropriate for an adaptive design if each stage tests the same null hypothesis. However, in the context of a dose-response study, discussed here as a motivation for α0 = 1, different doses may be tested in different stages, that is, the hypotheses tested change. The resultant question is whether at least one stage is significant, and a high sensitivity to small p-values is desirable. Consequently, Fisher's test or TPM are appropriate. An additional advantage of these two combination methods is that an early termination with rejection of the null hypothesis is possible with α0 = 1 and a full level α combination test at the end.
There is also some literature related to the efficiency of adaptive designs, and to the choice of combination functions. Wassmer , for example, compared Fisher's product criterion with an alternative adaptive design proposed by Proschan and Hunsberger  based on a conditional power function. Wassmer  concluded that "no substantial differences between the procedures were found in terms of rejection regions, power, and expected sample sizes". One of the first to investigate optimal adaptive designs for the control of conditional power were Brannath and Bauer . They constructed two-stage designs with overall and conditional power, which minimize the expected sample size for different specifications of the alternative. It transpires that there is a variety of different options to combine P-values and there is no consensus on the best method to use. In this paper we improve under special conditions Fisher's combination test using the truncated product method.
It is worthwhile to note that the truncation point τ must be specified a priori in the study protocol. Unless determined a priori, the truncated product method can be misused to alleviate an observed significance. A post-hoc choice based on the observed maximum of the individual p-values is therefore not permitted. As discussed above, τ = 0.5 may be a suitable choice. A further argument for this choice is that those p-values are excluded from the product that indicate a difference in the unanticipated direction. Note that the truncated product does not follow a χ2-distribution. Thus, a penalty results for the exclusion of large p-values. Nevertheless, this exclusion can be advantageous as demonstrated by Zaykin et al.  and above for the case of adaptive designs.
For the presentation of the power a one-sided significance level of α = 5% was chosen in this paper. However, completely analogous results can be found in case of α = 2.5%. Regarding the choice of α for one-sided tests it is referred to Neuhäuser .
In case of truncation with τ = α0 > α the power is
where f δ denotes the respective density under the alternative δ . In case of truncation with τ > α, but α0 = 1, the power is
Wassmer  presented a SAS/IML program to calculate the power for the two-stage test without truncation. Modifications of this program were used to calculate the different powers given above.
truncated product method
The authors would like to thank Roswitha Senske for technical support and a reviewer for helpful comments and suggestions.
- Palmer CR: Ethics, data-dependent designs, and the strategy of clinical trials: time to start learning-as-we-go?. Statistical Methods in Medical Research. 2002, 11: 381-402. 10.1191/0962280202sm298ra.View ArticlePubMedGoogle Scholar
- Gauch HG: Scientific method in practice. 2002, Cambridge University Press: CambridgeView ArticleGoogle Scholar
- Bauer P, Brannath W, Posch M: Flexible two-stage designs: an overview. Methods of Information in Medicine. 2001, 40: 117-121.PubMedGoogle Scholar
- Bauer P, Köhne K: Evaluation of experiments with adaptive interim analyses. Biometrics. 1994, 50: 1029-1041. Correction in Biometrics 52:380View ArticlePubMedGoogle Scholar
- Hedges LV, Olkin I: Statistical methods for meta-analysis. 1985, Academic Press: OrlandoGoogle Scholar
- Bauer P, Röhmel J: An adaptive method for establishing a dose-response relationship. Statistics in Medicine. 1995, 14: 1595-1607.View ArticlePubMedGoogle Scholar
- Zaykin DV, Zhivotovsky LA, Westfall PH, Weir BS: Truncated product method for combining P-values. Genetic Epidemiology. 2002, 22: 170-185. 10.1002/gepi.0042.View ArticlePubMedGoogle Scholar
- Neuhäuser M: Tests for genetic differentiation. Biometrical Journal. 2003, 45: 974-984. 10.1002/bimj.200390064.View ArticleGoogle Scholar
- Dudbridge F, Koeleman BPC: Rank truncated product of P-values, with application to genomewide association scans. Genetic Epidemiology. 2003, 25: 360-366. 10.1002/gepi.10264.View ArticlePubMedGoogle Scholar
- Brannath W, Posch M, Bauer P: Recursive combination tests. Journal of the American Statistical Association. 2002, 97: 236-244. 10.1198/016214502753479374.View ArticleGoogle Scholar
- Friede T, Kieser M: A comparison of methods for adaptive sample size adjustment. Statistics in Medicine. 2001, 20: 3861-3873. 10.1002/sim.972.View ArticlePubMedGoogle Scholar
- O'Brien PC: Procedures for comparing samples with multiple endpoints. Biometrics. 1984, 40: 1079-1087.View ArticlePubMedGoogle Scholar
- Loughin TM: A systematic comparison of methods for combining p-values from independent test. Computational Statistics and Data Analysis. 2004, 47: 467-485. 10.1016/j.csda.2003.11.020.View ArticleGoogle Scholar
- Lehmacher W, Wassmer G: Adaptive sample size calculations in group sequential trials. Biometrics. 1999, 55: 1286-1290. 10.1111/j.0006-341X.1999.01286.x.View ArticlePubMedGoogle Scholar
- Rice WR: A consensus combined P-value test and the family-wide significance of component tests. Biometrics. 1990, 46: 303-308.View ArticleGoogle Scholar
- Wassmer G: A comparison of two methods for adaptive interim analyses in clinical trials. Biometrics. 1998, 54: 696-705.View ArticlePubMedGoogle Scholar
- Proschan MA, Hunsberger SA: Designed extension of studies based on conditional power. Biometrics. 1995, 51: 1315-1324.View ArticlePubMedGoogle Scholar
- Brannath W, Bauer P: Optimal conditional error functions for the control of conditional power. Biometrics. 2004, 60: 715-723. 10.1111/j.0006-341X.2004.00221.x.View ArticlePubMedGoogle Scholar
- Neuhäuser M: The choice of α for one-sided tests. Drug Information Journal. 2004, 38: 57-60.Google Scholar
- Wassmer G: A technical note on the power determination for Fisher's combination test. Biometrical Journal. 1997, 39: 831-838.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/5/30/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.