Skip to main content

Archived Comments for: Parametric versus non-parametric statistics in the analysis of randomized trials with non-normally distributed data

Back to article

  1. Need more assurance type I error is correct

    Ken Kleinman, Harvard Medical School/Harvard Pilgrim Health Care

    22 December 2005

    The background correctly provides the following summary:

    "...techniques for statistical inference from randomized trials can only fail in one of two ways: they can inappropriately reject the null hypothesis of no difference between groups (false positive or Type I error) or inappropriately fail to reject the null (false negative or Type II error)."

    However, it goes on to claim:

    "Empirical statistical research has clearly demonstrated that the t-test does not inflate Type I (false positive) error. In a typical study, Heeren et al examined the properties of the t-test to analyze small two-group trials where data are ordinal, such as from a five point scale, and thus non-normal [3]. They found that where there was truly no difference between groups, the t-test would reject the null hypothesis close to 5% of the time."

    The single cite does not appear to support the generality of the first sentence. In particular, I question whether the proper type I error level is maintained in the actual distributions used in the article.

    We generally attach more importance to maintaining an accurate type I error and relegate power considerations to a secondary status. Thus, before using the result obtained here, I would need assurance that the type I error level is appropriately maintained in all cases as well.

    I'm frankly surprised that the author did not present this and that neither reveiwer suggested or required this analysis be performed. It would be trivial to do with the apparatus of the simulation constructed and is really essential to taking these otherwise very interesting results seriously.

    Competing interests

    I declare that I have no competing interests.

  2. Response to Dr Kleinman: Type I error

    Andrew Vickers, Memorial Sloan-Kettering Cancer Center

    19 January 2006

    I would like to thank Dr Kleinman for his comment. He is quite right with respect to his central assertion (worry about Type I error first and only consider power if Type I error rate is acceptable). I also accept his criticism that I was unclear about the type I error rates in the simulations.

    I took Kleinman's central assertion for granted and only commented when there was a concern over type I error (as when I stated that "log-transformed ANCOVA is slightly anti-conservative: when the simulation was repeated with no treatment effect, the null hypothesis was rejected for 5.23% (rather than the nominal 5%) of trials"). I should have been more explicit and stated that all simulations were repeated under the null hypothesis. Only in the case of log-transformed ANCOVA was there any evidence that the size of the test was inflated.

    Competing interests

    None declared