This article has Open Peer Review reports available.
The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional
- Morten W Fagerland^{1}Email author,
- Stian Lydersen^{2} and
- Petter Laake^{3}
https://doi.org/10.1186/1471-2288-13-91
© Fagerland et al.; licensee BioMed Central Ltd. 2013
Received: 12 March 2013
Accepted: 9 July 2013
Published: 13 July 2013
Abstract
Background
Statistical methods that use the mid-p approach are useful tools to analyze categorical data, particularly for small and moderate sample sizes. Mid-p tests strike a balance between overly conservative exact methods and asymptotic methods that frequently violate the nominal level. Here, we examine a mid-p version of the McNemar exact conditional test for the analysis of paired binomial proportions.
Methods
We compare the type I error rates and power of the mid-p test with those of the asymptotic McNemar test (with and without continuity correction), the McNemar exact conditional test, and an exact unconditional test using complete enumeration. We show how the mid-p test can be calculated using eight standard software packages, including Excel.
Results
The mid-p test performs well compared with the asymptotic, asymptotic with continuity correction, and exact conditional tests, and almost as good as the vastly more complex exact unconditional test. Even though the mid-p test does not guarantee preservation of the significance level, it did not violate the nominal level in any of the 9595 scenarios considered in this article. It was almost as powerful as the asymptotic test. The exact conditional test and the asymptotic test with continuity correction did not perform well for any of the considered scenarios.
Conclusions
The easy-to-calculate mid-p test is an excellent alternative to the complex exact unconditional test. Both can be recommended for use in any situation. We also recommend the asymptotic test if small but frequent violations of the nominal level is acceptable.
Keywords
Background
Matched-pairs data arise from study designs such as matched and crossover clinical trials, matched cohort studies, and matched case-control studies. The statistical analysis of matched-pairs studies must make allowance for the dependency in the data introduced by the matching. A simple and frequently used test for binary matched-pairs data is the McNemar test. Several versions of this test exist, including the asymptotic and exact (conditional) tests. The traditional advice is to use the asymptotic test in large samples and the exact test in small samples. The argument for using the exact test is that the asymptotic test may violate the nominal significance level for small sample sizes because the required asymptotics do not hold. One disadvantage with the exact test is conservatism: it produces unnecessary large p-values and has poor power.
Airway hyper-responsiveness (AHR) status before and after stem cell transplantation (SCT) in 21 children [1]
After SCT | ||||
---|---|---|---|---|
AHR | No AHR | Sum | ||
Before SCT | AHR | 1 | 1 | 2 |
No AHR | 7 | 12 | 19 | |
Sum | 8 | 13 | 21 |
Complete response (CR) before and after consolidation therapy [2]
After consolidation | ||||
---|---|---|---|---|
CR | No CR | Sum | ||
Before consolidation | CR | 59 | 6 | 65 |
No CR | 16 | 80 | 96 | |
Sum | 75 | 86 | 161 |
The choice between an asymptotic method and a conservative exact method—which can be summarized as a trade-off between power and preservation of the significance level—is well known from other situations involving proportions [3]. For the independent 2×2 table, a good compromise can be reached using the mid-p approach [4]. The Fisher mid-p test, which is a modification of Fisher’s exact test, combines excellent power with rare and minor violations of the significance level [5]. The modification required to transform an exact p-value to a mid-p-value is simple: the mid-p-value equals the exact p-value minus half the point probability of the observed test statistic.
The purpose of this article is to investigate whether a mid-p version of the McNemar exact conditional test can offer a similar improvement for the comparison of matched pairs as has been observed with independent proportions. A supplementary materials document (Additional file 1) shows how the mid-p test can be calculated using several standard software packages, including Excel, SAS, SPSS, and Stata.
Methods
Notation
The observed counts (and joint outcome probabilities) of a paired 2 × 2 table
Event B | ||||
---|---|---|---|---|
Success | Failure | Sum | ||
Event A | Success | n _{11}(p _{11}) | n _{12}(p _{12}) | n _{1+}(p _{1+}) |
Failure | n _{21}(p _{21}) | n _{22}(p _{22}) | n _{2+}(p _{2+}) | |
Sum | n _{+1}(p _{+1}) | n _{+2}(p _{+2}) | N (1) |
It might, however, be more realistic to assume that p _{ k l }also depends on the subject i. As denoted by Agresti 6, pp. 418–420, this is a subject-specific model. Further, this is a conditional model, since we are interested in the association within the pair, conditioned on the subject. Data from N matched pairs are then presented in N 2×2 tables, one for each pair. Collapsing over the pairs results in Table 3. Conditional independence between Y _{1} and Y _{2} is tested by the Mantel-Haenszel statistic [6, p.417]. But that test statistic is algebraically equal to the squared McNemar test statistic. In the following, we will not specify whether we test for marginal homogeneity or conditional independence.
The asymptotic McNemar test
and its asymptotic distribution is the standard normal distribution. The equivalent McNemar test statistic χ ^{2}=z ^{2}=(n _{12}−n _{21})^{2}/(n _{12}+n _{21}) is approximately chi-squared distributed with one degree of freedom under the null hypothesis. The asymptotic McNemar test is undefined when n _{12}=n _{21}=0.
The asymptotic McNemar test with continuity correction
The asymptotic McNemar test with continuity correction (CC) approximates the exact conditional test. Hence, it combines the disadvantage of an asymptotic test (significance level violations) with the disadvantage of a conditional exact test (overly conservativeness), and we do not expect it to perform well. We include it in our evaluations because it features in influential textbooks such as Altman [9] and Fleiss et al. [10]. The asymptotic McNemar test with continuity correction is undefined when n _{12}=n _{21}=0.
The McNemar exact conditional test
and the two-sided p-value equals twice the one-sided p-value. If n _{12}=(n _{12}+n _{21})/2, the p-value equals 1.0. The exact conditional test is guaranteed to have type I error rates not exceeding the nominal level.
The McNemar mid-ptest
The type I error rates of the mid-p test—as opposed to those of exact tests—are not bounded by the nominal level; however, in a wide range of designs and models, both mid-p tests and confidence intervals violate the nominal level rarely and with low degrees of infringement [11–13]. Because mid-p tests are based on exact distributions, they are sometimes called quasi-exact [14]. Additional file 1 provides details on how to calculate the McNemar mid-p test with several standard software packages.
An exact unconditional test
where $k=\text{int}({z}_{\text{obs}}^{2}+1)$, F _{ n } is the cumulative binomial distribution function with parameters (n,1/2), i _{ n }=int{h(n)}, and int is the integer function. Suissa and Shuster [15] outline a numerical algorithm to find the supremum in (10). If z _{obs}<0, the one-sided p-value is found by reversing the inequality in (8). The two-sided p-value equals twice the one-sided p-value.
Evaluation of the tests
To compare the performances of the five tests, we carried out an evaluation study of type I error rates and power. We used complete enumeration (rather than stochastic simulations) and a large set of scenarios. Each scenario is characterized by fixed values of N (the number of matched pairs), p _{1+} and p _{+1} (the probabilities of success for each event), and θ=p _{11} p _{22}/p _{12} p _{21}. θ can be interpreted as the ratio of the odds for the event Y _{2} given Y _{1}. We use θ as a convenient way to re-parameterize {p _{11},p _{12},p _{21},p _{22}} into {p _{1+},p _{+1},θ}, which includes the parameter of interest, namely the two marginal success probabilities. We used StatXact PROCs for SAS (Cytel Inc.) to calculate p-values of the exact unconditional test and Matlab R2011b (Mathworks Inc.) to calculate p-values of the four other tests and to perform the evaluation study. In cases where n _{12}=n _{21}=0, we set p=1 for the two asymptotic McNemar tests.
For the calculations of type I error rates, we used 19 values of N (10, 15, 20, …, 100), five values of θ (1.0, 2.0, 3.0, 5.0, 10.0), and 101 values of p _{1+}=p _{+1} (0.00, 0.01, 0.02, …, 1.00), a total of 9595 scenarios. The nominal significance level was 5%.
Power was calculated for N=1, 2, …, 100, θ=1.0, 2.0, 3.0, 5.0, 10.0, p _{1+}=0.1, 0.35, 0.6, and Δ=p _{+1}−p _{1+}=0.10, 0.15, 0.20, 0.25, 0.30, 0.35.
Results
Type I error rates
Evaluation of type I error rates (TIER)
mean | max | proportion | proportion | |
---|---|---|---|---|
Method | TIER | TIER | TIER> 0.05 | TIER< 0.03 |
All 9595 scenarios | ||||
McNemar asymptotic | 0.0430 | 0.0537 | 0.294 | 0.121 |
McNemar asymptotic w/CC | 0.0190 | 0.0357 | 0.000 | 0.889 |
McNemar exact | 0.0201 | 0.0367 | 0.000 | 0.880 |
McNemar mid-p | 0.0349 | 0.0495 | 0.000 | 0.260 |
Exact unconditional | 0.0373 | 0.0495 | 0.000 | 0.201 |
Subregion: 10≤N≤30(2525 scenarios) | ||||
McNemar asymptotic | 0.0352 | 0.0529 | 0.037 | 0.281 |
McNemar asymptotic w/CC | 0.0089 | 0.0237 | 0.000 | 1.000 |
McNemar exact | 0.0090 | 0.0278 | 0.000 | 1.000 |
McNemar mid-p | 0.0212 | 0.0469 | 0.000 | 0.627 |
Exact unconditional | 0.0251 | 0.0488 | 0.000 | 0.541 |
Subregion: 35≤N≤60(3030 scenarios) | ||||
McNemar asymptotic | 0.0435 | 0.0537 | 0.210 | 0.084 |
McNemar asymptotic w/CC | 0.0196 | 0.0306 | 0.000 | 0.991 |
McNemar exact | 0.0210 | 0.0306 | 0.000 | 0.989 |
McNemar mid-p | 0.0374 | 0.0474 | 0.000 | 0.176 |
Exact unconditional | 0.0408 | 0.0482 | 0.000 | 0.096 |
Subregion: 65≤N≤100(4040 scenarios) | ||||
McNemar asymptotic | 0.0476 | 0.0535 | 0.519 | 0.049 |
McNemar asymptotic w/CC | 0.0249 | 0.0357 | 0.000 | 0.743 |
McNemar exact | 0.0263 | 0.0367 | 0.000 | 0.723 |
McNemar mid-p | 0.0416 | 0.0495 | 0.000 | 0.095 |
Exact unconditional | 0.0423 | 0.0495 | 0.000 | 0.066 |
As expected, the two exact tests do not violate the nominal significance level in any of the considered scenarios. Interestingly, neither does the McNemar mid-p test.
Finally, one important comment to the interpretation of Table 4. The values of the parameters p _{1+} and p _{+1} were selected to represent the entire range of possible values and not to be a representative sample of the situations that might be encountered in practice. Scenarios with probabilities close to zero or one are thereby given more weight to the summary statistics in Table 4 than their impact in actual studies. Thus, the mean type I error rates of a typical study are likely closer to the nominal level than indicated in Table 4. The table is, however, a good illustration of the differences in performance between the five tests.
Further details of the results from the evaluation of type I error rates can be found in a supplementary materials document (Additional file 2), which contains box-plots of type I error rates from the total and various subregions of the evaluation study.
Power
The number of matched-pairs ( N ) needed to reach power of 50%, 60%, 70%, 80%, and 90%, averaged over five values of θ and three values of p _{ 1+ } , for three values of Δ=p _{ +1 } −p _{ 1+ }
Nto reach power of | |||||
---|---|---|---|---|---|
50% | 60% | 70% | 80% | 90% | |
Δ=0.15 | |||||
McNemar asymptotic | 56 | 69 | 85 | 103^{∗} | 123^{∗} |
McNemar asymptotic w/CC | 68 | 82 | 98 | 114^{∗} | 129^{∗} |
McNemar exact | 67 | 81 | 96 | 113^{∗} | 129^{∗} |
McNemar mid-p | 57 | 71 | 87 | 104^{∗} | 123^{∗} |
Exact unconditional | 57 | 71 | 88 | 106^{∗} | 126^{∗} |
Δ=0.25 | |||||
McNemar asymptotic | 23 | 28 | 34 | 42 | 54 |
McNemar asymptotic w/CC | 30 | 35 | 41 | 50 | 63 |
McNemar exact | 29 | 34 | 41 | 49 | 62 |
McNemar mid-p | 24 | 29 | 35 | 43 | 56 |
Exact unconditional | 24 | 29 | 35 | 43 | 56 |
Δ=0.35 | |||||
McNemar asymptotic | 13 | 15 | 18 | 23 | 29 |
McNemar asymptotic w/CC | 18 | 21 | 24 | 28 | 34 |
McNemar exact | 18 | 21 | 23 | 27 | 34 |
McNemar mid-p | 15 | 17 | 20 | 24 | 30 |
Exact unconditional | 15 | 17 | 20 | 24 | 29 |
The examples revisited
Discussion
The evaluation study in this article revealed several interesting observations. First, that the conservatism of the McNemar exact conditional test can be severe. A large sample size is needed to bring its type I error rates above 3% for a 5% nominal significance level. Quite often, the type I error rates of the exact conditional test were half that of the nominal level or lower. A similar conservative behavior has been observed for other exact conditional methods, for instance, Fisher’s exact test for two independent binomial proportions [5] and the Cornfield exact confidence interval for the independent odds ratio [16]. This conservatism leads to poor power and a need for unnecessary large sample sizes. We do not recommend use of the McNemar exact conditional test in any situation.
Second, the McNemar mid-p test is a considerable improvement over the exact conditional test on which it is based. It performs almost at the same level as the exact unconditional test. Whereas the exact tests are guaranteed to have type I error rates bounded by the nominal level, no such claim can be made for the mid-p test. Nevertheless, the mid-p test did not violate the nominal level in any of the 9595 scenarios considered in this evaluation. For practical use, the mid-p test is at an advantage vis-a-vis the exact unconditional test. As shown in the supplementary materials, the mid-p test is readily calculated in many commonly used software packages, including the ubiquitous Excel. The exact unconditional test, on the other hand, is computationally complex and only available in StatXact (Cytel Inc.).
Third, the asymptotic McNemar test (without CC) performs surprisingly well, even for quite small sample sizes. It often violates the nominal significance level, but not by much. The largest type I error rate of the asymptotic McNemar test we observed in this study was 5.37% with a 5% nominal level. If that degree of infringement on the nominal level is acceptable, the asymptotic McNemar test is superior to the other tests. This is notably different from comparing two independent binomial proportions, where the asymptotic chi-squared test can produce substantial violations of the type I error rate in small samples [14].
The asymptotic test with CC performs similarly to—and sometimes even more conservatively than—the exact conditional test, and we do not recommend that it is used. This was expected, and is in line with the unequivocal recommendations against using the asymptotic chi-squared test with Yates’s CC for the analysis of the independent 2×2 table [5, 13, 17].
We have only evaluated tests based on the McNemar statistic. It is also possible to construct tests using the likelihood ratio statistic; however, Lloyd [18] found no practical difference between the two statistics. We prefer the much simpler—and widely used—McNemar statistic.
Conclusions
The McNemar mid-p test is a considerably improvement on the McNemar exact conditional test. The mid-p test did not violate the nominal level in any of the 9595 scenarios considered in this article and is thus an excellent alternative to the vastly more complex exact unconditional test. The most powerful test is the McNemar asymptotic test (without CC), which we recommend if small but frequent violations of the nominal level is acceptable. We do not recommend use of the McNemar exact conditional test nor the asymptotic test with CC in any situation.
Declarations
Authors’ Affiliations
References
- Bentur L, Lapidot M, Livnat G, Hakim F, Lidroneta-Katz C, Porat I, Vilozni D, Elhasid R: Airway reactivity in children before and after stem cell transplantation. Pediatr Pulm. 2009, 44: 845-850. 10.1002/ppul.20964.View ArticleGoogle Scholar
- Cavo M, Pantani L, Petrucci MT, Patriarca F, Zamagni E, Donnarumma D, Crippa C, Boccadoro M, Perrone G, Falcone A, Nozzoli C, Zambello R, Masini L, Furlan A, Brioli A, Derudas D, Ballanti S, Dessanti ML, De Stefano V, Carella AM, Marcatti M, Nozza A, Ferrara F, Callea V, Califano C, Pezzi A, Baraldi A, Grasso M, Musto P, Palumno A: Bortezomib-Thalidomide- Dexamethasone is superior to Thalidomide-Dexamethasone as consolidation therapy after autologous hematopoietic stem cell transplantation in patients with newly diagnosed multiple myeloma. Blood. 2012, 120: 9-19. 10.1182/blood-2012-02-408898.View ArticlePubMedGoogle Scholar
- Agresti A: Dealing with discreteness: making ‘exact’ confidence intervals for proportions, differences of proportions, and odds ratios more exact. Stat Methods Med Res. 2003, 12: 3-21. 10.1191/0962280203sm311ra.View ArticlePubMedGoogle Scholar
- Lancaster HO: Significance tests in discrete distributions. J Am Stat Assoc. 1961, 56: 223-234. 10.1080/01621459.1961.10482105.View ArticleGoogle Scholar
- Lydersen S, Fagerland MW, Laake P: Recommended tests for association in 2×2 tables. Stat Med. 2009, 28: 1159-1175. 10.1002/sim.3531.View ArticlePubMedGoogle Scholar
- Agresti A: Categorical Data Analysis (3rd edn). 2013, Hoboken, NJ: Wiley,Google Scholar
- McNemar Q: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947, 12: 153-157. 10.1007/BF02295996.View ArticlePubMedGoogle Scholar
- Edwards AL: Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. Psychometrika. 1948, 13 (3): 185-187. 10.1007/BF02289261.View ArticlePubMedGoogle Scholar
- Altman DG: Practical Statistics for Medical Research. 1991, Boca Raton Fl: Chapman & Hall/CRC,Google Scholar
- Fleiss JL, Levin B, Paik MC: Statistical Methods for Rates and Proportions (3rd edn). 2003, Hoboken, NJ: Wiley,View ArticleGoogle Scholar
- Mehta CR, Walsh SJ: Comparison of exact, mid-p, and Mantel-Haenszel confidence intervals for the common odds ratio across several 2×2 contingency tables. Am Stat. 1992, 46: 146-150.Google Scholar
- Lydersen S, Laake P: Power comparison of two-sided exact tests for association in 2×2 contingency tables using standard, mid-P, and randomized test versions. Stat Med. 2003, 22: 3859-3871. 10.1002/sim.1671.View ArticlePubMedGoogle Scholar
- Hirji KF: Exact Analysis of Discrete Data. 2006, Boca Raton, FL: Chapman & Hall/CRC,Google Scholar
- Hirji KF, Tan S-J, Elashoff RM: A quasi-exact test for comparing two binomial proportions. Stat Med. 1991, 10: 1137—1153-View ArticlePubMedGoogle Scholar
- Suissa S, Shuster JJ: The 2×2 matched-pairs trial: exact unconditional design and analysis. Biometrics. 1991, 47: 361-372. 10.2307/2532131.View ArticlePubMedGoogle Scholar
- Fagerland MW: Exact and mid-P confidence intervals for the odds ratio. Stata J. 2012, 12: 505-514.Google Scholar
- Haviland MG: Yates’s correction for continuity and the analysis of 2×2 contingency tables. Stat Med. 1990, 9: 363-367. 10.1002/sim.4780090403.View ArticlePubMedGoogle Scholar
- Lloyd CJ: A new exact and more powerful unconditional test of no treatment effect from binary matched pairs. Biometrics. 2008, 64: 716-723. 10.1111/j.1541-0420.2007.00936.x.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/13/91/prepub
Pre-publication history
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.