This article has Open Peer Review reports available.
Correction of the significance level when attempting multiple transformations of an explanatory variable in generalized linear models
- Benoit Liquet^{1, 2, 3} and
- Jérémie Riou^{1, 2, 4}Email author
https://doi.org/10.1186/1471-2288-13-75
© Liquet and Riou; licensee BioMed Central Ltd. 2013
Received: 9 January 2013
Accepted: 17 May 2013
Published: 8 June 2013
Abstract
Background
In statistical modeling, finding the most favorable coding for an exploratory quantitative variable involves many tests. This process involves multiple testing problems and requires the correction of the significance level.
Methods
For each coding, a test on the nullity of the coefficient associated with the new coded variable is computed. The selected coding corresponds to that associated with the largest statistical test (or equivalently the smallest p _{ v a l u e }). In the context of the Generalized Linear Model, Liquet and Commenges (Stat Probability Lett,71:33–38,2005) proposed an asymptotic correction of the significance level. This procedure, based on the score test, has been developed for dichotomous and Box-Cox transformations. In this paper, we suggest the use of resampling methods to estimate the significance level for categorical transformations with more than two levels and, by definition those that involve more than one parameter in the model. The categorical transformation is a more flexible way to explore the unknown shape of the effect between an explanatory and a dependent variable.
Results
The simulations we ran in this study showed good performances of the proposed methods. These methods were illustrated using the data from a study of the relationship between cholesterol and dementia.
Conclusion
The algorithms were implemented using R, and the associated CPMCGLM R package is available on the CRAN.
Keywords
Background
In applied studies, the relationship between an explanatory and a dependent variable is routinely measured using a statistical model. For instance, in epidemiology it is quite common that a study focuses on one particular risk factor. The scientific problem is to analyze whether this risk factor has an influence on the risk of occurrence of a disease, a biological trait, or another outcome. To answer to this question, a regression model is often used in which the risk factor will be represented by a continuous X, allowing adjustment on p−1 known risk factors of the studied trait. However, the form of the effect (or the dose-effect relationship) is not known in advance, and as such, the continuous variable X is often transformed, typically into categorical variables, by grouping values into two or more categories. An example of this is seen in an The American Journal of Epidemiology (October 2009, volume 170, number 8), where four of six papers with continuous exposure used categorization, and only two kept the variable as continuous [1].
but the choice of the transformation is often subjective. The arbitrariness of the choice of cutpoints may lead to the idea of trying more than one set of values. Hence to analyze data, the statistician may have to use several transformations, and for each the statistician applies a test for “ β=0” (where β is the coefficient representing the effect of the risk factor of interest). The most favorable transformation is then chosen. The cutpoint giving the minimum p _{ v a l u e } is often termed “optimal” [2, 3]. When testing several codings of a variable, there is a problem with the multiplicity of tests performed, leading to an incorrect p _{ v a l u e } and possible overestimation of effects [4]. Generally, researchers fail to consider this problem and do not correct the significance level in relation to the number of tests performed [3], which can lead to an increase in the Type-I error [5]. The p _{ v a l u e } should thus be corrected to take into account the multiplicity of tests.
In many cases, it is now widely recognized that categorization of a continuous variable could introduce major problems to an analysis and interpretation of the associated model [1, 3]. It is important to note that the aim of this paper is not to defend this practice, but to improve a practice commonly used by epidemiologists in terms of multiple testing. Furthermore, despite known loss of power following dichotomization in the univariate case, Westfall [6] shown that dichotomizing continuous data can greatly improve the power when multiple comparisons are performed.
Many methods of correction exist, the most simple and well known being the Bonferroni rule. Several authors have improved this method to make it more powerful, however most do not take into account the correlation between the tests [7–11]. If the tests are independent, or moderately dependent, then they provide an upper bound which may be satisfactory. Efron [12] proposed a correction that account for the correlation between two consecutive tests if there is a natural order between the tests, with high correlation between adjacent tests. Liquet and Commenges [13, 14] and Hashemi and Commenges [15] proposed a more exact correction, accounting for the whole correlation matrix, for score tests obtained in logistic regression, generalized linear model and proportional hazards models.
Here, we proprose extending these studies to a categorical transformation (with m>2 categories) of the continuous variable by involving more than one parameter in the model; m−1 dummy variables are introduced in the model. The categorical transformation is a more flexible way to explore the unknown shape of the effect. In this context, we propose a method and an R program based on resampling approaches to determine the significance level for a series of several transformations (including dichotomous, Box-Cox and categorical transformations) of an explanatory variable in a Generalized Linear Model. The problem of correcting the estimation of the effect will not be examined here.
First, we revisit the example proposed by Liquet and Commenges [14] on the relationship between cholesterol and dementia [16] to provide a framework for our discussion. In section ‘Methods: Statistical context’, we present the statistical contexts relating to multiple testing; the model, the maximum test and the minimum p _{ v a l u e } procedure and finally the score tests are exposed. Section ‘Methods: Significance level correction’ presents the different methods of correction of the Type-I error. A simulation study for the different strategies of coding, and application of the model to the initial example are presented in the section ‘Results’. Concluding remarks are given in the two last sections.
Example: revisiting the PAQUID cohort example
We revisited the example presented in the article of Liquet and Commenges [13] for a coding of a binary variable in a logistic regression. This example is based on the work of Bonarek et al. [16], who studied the relationship between serum cholesterol levels and dementia. The data came from a nested case-control study of 334 elderly French subjects aged 73 and over who participated in the PAQUID cohort (37 subjects with dementia and 297 controls). The variables age, sex, level of education and wine consumption were considered as adjustment variables. The analysis focused on the influence of HDL-cholesterol(high-density lipoprotein) on the risk of dementia. Bonarek et al. [16] first considered HDL-cholesterol as a continuous variable; then, to ease clinical interpretation, they chose to transform the HDL-cholesterol into a categorical variable with four classes. Finally, as there was no significant difference between the first three quartiles, HDL-cholesterol was split into two categories with a cutpoint at the last quartile. The best p _{ v a l u e }, 0.007, was obtained in the latter analysis and was selected for interpretation. However, this p _{ v a l u e } did not take into account the numerous transformations performed to determine the best representation of the variable of interest. Legitimate questions arising from this include the following: What is the real association between dementia and HDL-cholesterol, with a correction of the Type-I error? Is it really significant? Liquet and Commenges [14] proposed correcting the p _{ v a l u e } associated with multiple transformation including dichotomous and Box-Cox transformation, however, their method cannot be used with categorical transformation.
Methods
Statistical context
Model
with $\mathbb{E}\left[{Y}_{i}\right]={\mu}_{i}={b}^{\prime}\left({\theta}_{i}\right),\mathbb{V}\mathit{\text{ar}}\left[{Y}_{i}\right]={b}^{\u2033}\left({\theta}_{i}\right)a\left(\varphi \right)$ and where a(·), b(·), and c(·) are known and differentiable functions. b(·) is three times differentiable, and its first derivative b ^{′}(·) is invertible. Parameters (θ _{ i },ϕ) belong to $\Omega \subset {\mathbb{R}}^{2}$, where θ _{ i } is the canonical parameter and ϕ is the dispersion parameter.
where ${\mathbf{0}}_{{m}_{k}-1}$ is a null vector of dimension m _{ k }−1. Under the null hypothesis ${\mathcal{\mathscr{H}}}_{0}\left(k\right)$ we have θ _{ i }(k)=γ Z _{ i }, which do not depend on k. Thus all the null hypotheses are the same, and denote it by ${\mathcal{\mathscr{H}}}_{0}$.
Maximum test and minimum P-value procedures
where P _{ m i n }=m i n(P _{1},…,P _{ K }) and p _{ m i n } is the realization of p _{ m i n }. The interest of using a procedure based on the p _{ v a l u e } is the possibility of combining statistical tests which do not follow the same distribution. In the current context, we will combine dichotomous, Box-Cox and categorical transformations with more than two levels.
Score test
We briefly present the score test used for all of the K transformations where the same null hypothesis is tested (i.e. ${\mathcal{\mathscr{H}}}_{0}$: “${\beta}_{k}={0}_{{m}_{k}-1}$” given by ${\theta}_{i}\left(k\right)=\mathit{\gamma}{Z}_{i}$ (with different alternatives)). We present the main results obtained by Liquet and Commenges [14] for the Generalized Linear Model in the context of dichotomous and Box-Cox transformations, and then consider the score test for categorical transformations.
Dichotomous and Box-cox Transformations
where $\widehat{R}$ is the vector of residuals $\widehat{{R}_{i}}={Y}_{i}-\widehat{{\mu}_{i}}$ computed under the null hypothesis, V is a diagonal matrix such that ${v}_{\mathit{\text{ii}}}=\widehat{\mathit{\text{Var}}}\left({Y}_{i}\right),H=\mathit{\text{VZ}}\left({Z}^{T}\mathit{\text{VZ}}\right){Z}^{T}$, and Z the n×p matrix with rows Z _{ i }, i=1,…,n.
The correlation between the different tests has been defined by Liquet and Commenges [14]. Asymptotically, the joint distribution of T _{1},…,T _{ K } is a multivariate normal distribution with zero mean and a certain covariance matrix. Thus Liquet and Commenges [14] propose that the p _{ v a l u e } (associated with the test T _{ m a x }) defined in (3) using numerical integration [18] be calculated. They called their method the “exact method”.
Categorical transformations
where U _{ k } and I _{ k } are respectively the score function and the Fisher information matrix under the null hypothesis [19]. To compute the p _{ v a l u e } defined in (4), it is necessary to know the joint distribution of T=(T _{1},…,T _{ K }). Some studies have defined the distribution of the multivariate χ ^{2}[20, 21]. However, even though the correlation between the different tests could be easily estimated, it has not been possible, as far as we know, to obtain the joint distribution of T=(T _{1},…,T _{ K }). To overcome this problem, we propose approximating the p _{ v a l u e } (defined in (4) by the minimum p _{ v a l u e } procedure) using a resampling method (defined in the next section) which also accounts for the correlation between the test statistics.
Significance level correction
Bonferroni method
where K is the number of tests. This method is very simple and does not require any assumption about the correlation between the different tests. It can therefore be applied directly to the different possible codings of an explanatory variable. However, this only provides an upper bound of the p _{ v a l u e }, which may be very conservative if the correlation between tests are high and the number of transformation are large.
Resampling based methods
We propose the use of resampling based methods [23, 24] with the aim of building a reference distribution for the test statistics. These procedures have the advantage of taking into account the dependence of the test statistics for evaluating the correct significance level of the minimum p _{ v a l u e } procedure (or the maximum test procedure). The principle of resampling procedures is to define new samples from the probability measure defined under ${\mathcal{\mathscr{H}}}_{0}$: “${\beta}_{k}={0}_{{m}_{k}-1}$”.
Permutation test procedure
- 1.
Apply the minimum p _{ v a l u e } procedure to the original data for the K transformations considered. We note p _{ m i n } the realization of the minimum of the p _{ v a l u e };
- 2.
As under ${\mathcal{\mathscr{H}}}_{0}$, the X _{ i } variable has no effect on the response variable, a new dataset is generated by permuting the X _{ i } variable in the initial dataset;
- 3.
Generate B new datasets ${s}_{b}^{\ast},b=1,\dots ,B$ by repeating B times the step 2;
- 4.
For each new dataset, apply the minimum p _{ v a l u e } procedure for the transfomation under consideration. We note ${p}_{\mathit{\text{min}}}^{\ast b}$ the smallest p _{ v a l u e } for each new dataset.
- 5.The p _{ v a l u e } defined in (4) is then approximated by:$\hat{{p}_{\mathit{\text{value}}}}=\frac{1}{B}\sum _{b=1}^{B}{I}_{\left\{{p}_{\mathit{\text{min}}}^{\ast b}<{p}_{\mathit{\text{min}}}\right\}},$
where I _{{·}} is an indicator function.
However, it is important to note that exchangeability need to be satisfied [25–30]. This condition is much more restrictive than it appears at first sight. In fact, Commenges [29] and Commenges and Liquet [25] showed that the permutation test approach for the score test is robust if the model has only one intercept under the null hypothesis, or if X _{ i } are independent of Z _{ i } for all i in the context of a linear model and the proportional hazards model. This issue applies in our context. Thus we investigated, the robustness of the permutation method when the exchangeability assumptions is violated.
Parametric bootstrap procedure
- 1.
Apply the minimum p _{ v a l u e } procedure to the original data for the K transformations being considered. We note p _{ m i n } the realization of the minimum of the p _{ v a l u e };
- 2.
Fit the model under the null hypothesis, using the observed data, and obtain $\widehat{\gamma}$, the maximum likelihood estimate (MLE) of γ;
- 3.Generate a new outcome ${Y}_{i}^{\ast}$ for each subject from the probability measure defined under ${\mathcal{\mathscr{H}}}_{0}$. For example, for a logistic model (where a(ϕ)=1, $b\left({\theta}_{i}\right)=\mathit{\text{log}}(1+{e}^{{\theta}_{i}})$, and ${\mu}_{i}=\mathbb{E}\left({Y}_{i}\right)={e}^{{\theta}_{i}}/(1+{e}^{{\theta}_{i}})$), we generate ${Y}_{i}^{\ast}$ according to:$P({Y}_{i}^{\ast}=1|{Z}_{i})=\frac{{e}^{\widehat{\gamma}{Z}_{i}}}{1+{e}^{\widehat{\gamma}{Z}_{i}}}.$
- 4.
Generate B new datasets ${s}_{b}^{\ast},b=1,\dots ,B$ by repeating B times the step 3;
- 5.
Apply for each new dataset, the minimum p _{ v a l u e } procedure for the transformation considered. We note ${p}_{\mathit{\text{min}}}^{\ast b}$ the smallest p _{ v a l u e } for each new dataset.
- 6.Then, the p _{ v a l u e } defined in (4) is then approximated by:$\hat{{p}_{\mathit{\text{value}}}}=\frac{1}{B}\sum _{b=1}^{B}{I}_{\left\{{p}_{\mathit{\text{min}}}^{\ast b}<{p}_{\mathit{\text{min}}}\right\}}.$
Results
Simulation study
where Z _{ i } and X _{ i } are independent and were generated according to a standard normal distribution and the vector X _{ i }(k) was a transformation of a continuous variable X _{ i }. The sample size was set to be 100. We used 1000 replications for each simulation and 1000 samples for the resampling methods.
Dichotomous transformations
Strategy for dichotomous transformations: values of the cutpoints c _{ k } according to the number of transformations ( q _{ α } represents the quantile of order α )
Number of transformations | c _{ 1 } | c _{ 2 } | c _{ 3 } | … | c _{ 9 } |
---|---|---|---|---|---|
1 | q _{1/2} | ||||
2 | q _{1/3} | q _{2/3} | |||
3 | q _{1/4} | q _{2/4} | q _{3/4} | ||
⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
9 | q _{1/10} | q _{2/10} | q _{2/10} | … | q _{9/10} |
When information on the shape of the effect of the explanatory variable was unknown we investigated the power of the methods applied above. We studied the power for a threshold effect model with a cutpoint value at the first tercile. Figure 1(b) gives the power as a function of the number of cutpoints tried. The power of the exact and resampling methods are quite similar to one another, and higher than the Bonferroni method. The difference between these methods and Bonferroni method increases with the number of cutpoints. We also observed that the power was highest at two cutpoints (two transformations). This result, was in fact, expected since we used the first and second terciles respectively as cutpoints for each dichotomous transformation. Power increased again when trying five and eight codings due to the fact that one of these codings corresponded to the first tercile. To conclude, the simulation study with dichotomous transformations showed that the resampling methods provide similar results for the Type-I error rate control and the power as those seen with the exact method.
Categorical transformations with same number of classes
Strategy for the categorical transformations in three classes: values of the cutpoints ( ${c}_{k}^{1}$ and ${c}_{k}^{2}$ ) for all transformations
Number of transformations | ${c}_{k}^{1}$ | ${c}_{k}^{2}$ |
---|---|---|
1 | q _{ 1/3 } | q _{ 2/3 } |
2 | q _{ 1/4 } | q _{ 3/4 } |
3 | q _{ 1/4 } | q _{ 1/2 } |
4 | q _{ 1/2 } | q _{ 3/4 } |
5 | q _{ 2/5 } | q _{ 4/5 } |
6 | q _{ 1/5 } | q _{ 3/5 } |
7 | q _{ 3/5 } | q _{ 4/5 } |
8 | q _{ 1/5 } | q _{ 2/5 } |
9 | q _{ 1/5 } | q _{ 4/5 } |
10 | q _{ 2/5 } | q _{ 3/5 } |
Next we considered the power of the different methods when the simulated model was specified with a categorical transformation of the continuous variable in three classes defined by cutpoints at the first and third quartile. The two resampling methods gave similar results with a higher power than the Bonferroni method (see Figure 2(b)). The power was highest for two transformations. This result was also expected because, with the strategy presented in Table 2, the transformation into three classes with cutpoints at the first and third quartile is used.
Various categorical transformations
Strategy for different categorical transformations: values of the cutpoints for all transformations
Number of transformations | ${c}_{k}^{1}$ | ${c}_{k}^{2}$ | … | ${c}_{k}^{9}$ | ${c}_{k}^{10}$ |
---|---|---|---|---|---|
1 | q _{ 1/2 } | ||||
2 | q _{ 1/3 } | q _{ 2/3 } | |||
⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
9 | q _{ 1/10 } | q _{ 2/10 } | … | q _{ 9/10 } | |
10 | q _{ 1/11 } | q _{ 2/11 } | … | q _{ 9/11 } | q _{ 10/11 } |
Robustness of resampling methods
where X _{ i }(k) is the binary coding of the X _{ i } variable with a cutpoint at the median. The coefficient β ^{∗} was computed according to η ^{2} and the variance of X _{ i }(k) variable.We tested three different binary codings with cutpoints at the first, the second and the third tercile. The strategy is used for various values of the correlation ratio (η ^{2}) from 0 to 0.6.
Example: revisiting the PAQUID cohort example
In order to find the real association between the two variables of interest in the example described at the end of Background section, we applied our newly developed approach which combined different kinds of transformations. Liquet and Commenges [14] have proposed seven dichotomous and five Box-Cox transformations. However, their method did not allow for categorical transformations. We proposed to add, to the seven dichotomous and five Box-Cox transformations for this application, four codings in three classes and four codings in four classes. The best transformation appeared to be the dichotomous transformation of HDL-cholesterol with a cutpoint at the third quartile, as already found by Bonarek et al. [16]. The Bonferroni correction gave a p _{ v a l u e } equal to 0.140, thus not significant for an α level at 0.05. The p _{ v a l u e }, which is given by both resampling based methods is 0.038. To conclude, it is important to chose a powerful method of correction, because in this context the p _{ v a l u e } with no correction given by Bonarek et al. [16] was very optimistic (0.007), and the Bonferroni correction was very conservative, yielding an incorrect conclusion. The proposed approach based on the resampling procedure gave a result which was still significant and more realistic than the uncorrected p _{ v a l u e }.
Discussion
In this paper, we have considered the problem of correction of significance level for a series of several codings of an explanatory variable in a Generalized Linear Model with several adjusting variables. The methods developed, based on resampling methods, enable us to consider categorical transformations as more flexible in order to explore the unknown shape of the effect between an explanatory and a dependent variable. The simulation studies presented above show, firstly, that the resampling method provides similar results for the Type-I error rate control and the power as those found with the exact method proposed by Liquet and Commenges [14] for dichotomous and Box-Cox transformations. Secondly, in the situation of categorical transformations, these simulations demonstrate the good performance of our proposed approaches. Finally we observed the robustness estimation of the p _{ v a l u e } by the resampling methods. These methods can be easily generalized to other models, such as the proportional hazards model, and to potentially extend the work of Hashemi and Commenges [15] in the same context.
Conclusion
To conclude, the methods developed, based on resampling, demonstrate good performances, and we have implemented different methods and different strategies of coding in an R package called CPMCGLM M (for Correction of the Pvalue after Multiple Coding in a Generalized Linear Model).
Appendix
The package CPMCGLM has been developed in R, an open source statistical software available at http://www.r-project.org. The methods presented in this paper are available in the main function CPMCGLM() for Probit, Logit, Linear, and Poisson models. Briefly, the user can specify the transformations tested: Box-Cox, dichotomous or categorical transformations. Two options are possible for defining the cutpoints of the dichotomous and the categorical transformations: the user can either specify them, or the program will automatically use the strategy based on the quantile presented in the simulation study.
Declarations
Acknowledgements
We would like to thank Luc Letenneur (ISPED, University of Bordeaux) for making the data available and the Danone Research Clinical Study Platform.
Authors’ Affiliations
References
- Bennette C, Vickers A: Against Quantiles: categorization of continuous variables in epidemiologic research, and its discontents. BMC Med Res Methodol. 2012, 12: 21-25. 10.1186/1471-2288-12-21.View ArticlePubMedPubMed CentralGoogle Scholar
- Altman D, Lausen B, Sauerbrei W, Schumacher M: Dangers of using optimal cutpoints in the evaluation of prognostic factors. J Natl Cancer Inst. 1994, 86 (11): 829-835. 10.1093/jnci/86.11.829.View ArticlePubMedGoogle Scholar
- Royston P, Altman D, Sauerbrei W: Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006, 25: 127-141. 10.1002/sim.2331.View ArticlePubMedGoogle Scholar
- Harrell FE, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996, 15 (4): 361-387. 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.View ArticlePubMedGoogle Scholar
- Miller RG: Simultaneous statistical inference. 2nd ed. 1981, New York - Heidelberg: Berlin: Springer- Verlag. XVI 299, figs. DM 44.00View ArticleGoogle Scholar
- Westfall PH: Improving power by dichotomizing (even under normality). Stat Biopharm Res. 2011, 3 (2): 353-362. 10.1198/sbr.2010.09055.View ArticleGoogle Scholar
- Simes R: An improved bonferroni procedure for multiple tests of significance. Biometrika. 1986, 73 (3): 751-754. 10.1093/biomet/73.3.751.View ArticleGoogle Scholar
- Sidak Z: Rectangular confidence regions for the means of multivariate normal distributions. J Am Stat Assoc. 1967, 62: 626-633.Google Scholar
- Holm S: A simple sequentially rejective multiple test procedure. Scand J Stat. 1979, 6: 65-70.Google Scholar
- Hommel G: A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika. 1988, 75: 383-386. 10.1093/biomet/75.2.383.View ArticleGoogle Scholar
- Hochberg Y: A sharper Bonferroni procedure for multiple test procedure. Biometrika. 1988, 75: 800-802. 10.1093/biomet/75.4.800.View ArticleGoogle Scholar
- Efron B: The length heuristic for simultaneous hypothesis tests. Biometrika. 1997, 84: 143-157. 10.1093/biomet/84.1.143.View ArticleGoogle Scholar
- Liquet B, Commenges D: Correction of the P-value after multiple coding of an explanatory variable in logistic regression. Stat Med. 2001, 20: 2815-2826. 10.1002/sim.916.View ArticlePubMedGoogle Scholar
- Liquet B, Commenges D: Computation of the p-value of the minimum of score tests in the generalized linear model, application to multiple coding. Stat Probability Lett. 2005, 71: 33-38. 10.1016/j.spl.2004.10.019.View ArticleGoogle Scholar
- Hashemi R, Commenges D: Correction of the p-value after multiple tests in a Cox proportional hazard model. Lifetime Data Anal. 2002, 8: 335-348. 10.1023/A:1020514804325.View ArticlePubMedGoogle Scholar
- Bonarek M, Barberger-Gateau P, Letenneur L, Deschamps V, Iron A, Dubroca B, Dartigues J: between cholesterol, apolipoprotein E polymorphism and dementia: a cross-sectional analysis from the PAQUID study. Neuroepidemiology. 2000, 19: 141-48. 10.1159/000026249.View ArticlePubMedGoogle Scholar
- McCullagh P, Nelder J: Genaralized Linear Models. 1989, New York: Chapman & HallView ArticleGoogle Scholar
- Genz A: Numerical computation of multivariate normal probabilities. J Comput Graphical Stat. 1992, 1: 141-149.Google Scholar
- Cox D, Hinkley D: Theoretical Statistics. 1994, London: Chapman & HallGoogle Scholar
- Royen T: Expansions for the multivariate chi-Square distribution. J Multivariate Anal. 1991, 38: 213-232. 10.1016/0047-259X(91)90041-Y.View ArticleGoogle Scholar
- Dagupsta N, Spurrier J: A class of multivariate χ2 distributions with applications to comparison with a control. Commun Stat- Theory Methods. 1997, 26: 1559-1573. 10.1080/03610929708832000.View ArticleGoogle Scholar
- Worsley K: An improved Bonferroni inequality and applications. Biometrika. 1982, 69: 297-302. 10.1093/biomet/69.2.297.View ArticleGoogle Scholar
- Westfall PH, Young S: Wiley Series in Probability and Mathematical Statistics. Applied Probability and Statistics. 1992, New York: NY Wiley, xvii, 340 pGoogle Scholar
- Yu K, Liang F, Ciampa J, Chatterjee N: Efficient p-value evaluation for resampling-based tests. Biostatistics. 2011, 12 (3): 582-593. 10.1093/biostatistics/kxq078.View ArticlePubMedPubMed CentralGoogle Scholar
- Commenges D, Liquet B: Asymptotic distribution of score statistics for spatial cluster detection with censored data. Biometrics. 2008, 64 (4): 1287-1289. 10.1111/j.1541-0420.2008.01132_1.x.View ArticlePubMedGoogle Scholar
- Romano J: On the behavior of randomization tests without a group invariance assumption. J Am Stat Assoc. 1990, 85 (411–412): 686-View ArticleGoogle Scholar
- Xu H, Hsu J: Applying the generalized partitioning principle to control the generalized familywise error rate. Biom J. 2007, 49: 52-67. 10.1002/bimj.200610307.View ArticlePubMedGoogle Scholar
- Kaizar E, Li Y, Hsu J: Permutation multiple tests of binary features do not uniformly control error rates. J Am Stat Assoc. 2011, 106 (495): 1067-1074. 10.1198/jasa.2011.tm10067.View ArticleGoogle Scholar
- Commenges D: Transformations which preserve exchangeability and application to permutation tests. J Nonparametric Stat. 2003, 15 (2): 171-185. 10.1080/1048525031000089310.View ArticleGoogle Scholar
- Westfall PH, Troendle JF: Multiple testing with minimal assumptions. Biom J. 2008, 50 (5): 745-755. 10.1002/bimj.200710456.View ArticlePubMedPubMed CentralGoogle Scholar
- Good P: Permutation Tests: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. 2000, New-York: Springer-VerlagGoogle Scholar
- Efron B, Tibshirani R: An Introduction to the Bootstrap (Chapman & Hall/CRC Monographs on Statistics & Applied Probability). 1994, London: Chapman and Hall/CRCGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/13/75/prepub
Pre-publication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.