A simple ratio-based approach for power and sample size determination for 2-group comparison using Rasch models
© Sébille et al.; licensee BioMed Central Ltd. 2014
Received: 18 December 2013
Accepted: 27 June 2014
Published: 5 July 2014
Despite the widespread use of patient-reported Outcomes (PRO) in clinical studies, their design remains a challenge. Justification of study size is hardly provided, especially when a Rasch model is planned for analysing the data in a 2-group comparison study. The classical sample size formula (CLASSIC) for comparing normally distributed endpoints between two groups has shown to be inadequate in this setting (underestimated study sizes). A correction factor (RATIO) has been proposed to reach an adequate sample size from the CLASSIC when a Rasch model is intended to be used for analysis. The objective was to explore the impact of the parameters used for study design on the RATIO and to identify the most relevant to provide a simple method for sample size determination for Rasch modelling.
A large combination of parameters used for study design was simulated using a Monte Carlo method: variance of the latent trait, group effect, sample size per group, number of items and items difficulty parameters. A linear regression model explaining the RATIO and including all the former parameters as covariates was fitted.
The most relevant parameters explaining the ratio’s variations were the number of items and the variance of the latent trait (R2 = 99.4%).
Using the classical sample size formula adjusted with the proposed RATIO can provide a straightforward and reliable formula for sample size computation for 2-group comparison of PRO data using Rasch models.
KeywordsPatient-reported outcomes Item response theory Rasch model Sample size Power
Patient-reported outcomes (PRO) are increasingly used in clinical research; they have become essential criteria that have gained major importance especially in chronically ill patients. Consequently, nowadays these outcomes are often considered as main secondary endpoints or even primary endpoints in clinical studies [1–4]. Two main types of analytic strategies are used for PRO data: so-called classical test theory (CTT) and models coming from Item Response Theory (IRT). CTT relies on the observed scores (possibly weighted sum of patients items’ responses) that are assumed to provide a good representation of a “true” score, while IRT relies on an underlying response model relating the items responses to a latent trait, interpreted as the true individual quality of life (QoL) for instance. The potential of IRT models for constructing, validating, and reducing questionnaires and for analyzing PRO data has been regularly underlined [5–7]. IRT and in particular Rasch family models  can improve on the classical approach to PRO assessment with advantages that include interval measurements, appropriate management of missing data [9–11] and of possible floor and ceiling effects, comparison of patients across different instruments . Consequently, many questionnaires are validated (or revalidated) using IRT along with CTT [13–15] allowing analysing PRO data with IRT models in clinical research.
Clinical research methodology has reached a high level of requirements through the publication of international guidelines including the CONSORT statement, the STROBE (Strengthening the Reporting of Observational Studies in epidemiology), or TREND (Transparent Reporting of Evaluations with Nonrandomized Designs), initiative for instance [16–19]. All of these published recommendations are aimed at improving the reporting of scientific investigations coming either from randomized clinical trials or observational studies and systematically include an item related to sample size justification and determination. Furthermore, good methodological standards recommend that methods used for sample size planning and for subsequent statistical analysis should be based on similar grounds. Even if guidelines have also been recently published for PRO based studies [20, 21], the reporting of such studies often lacks mentioning the justification of study size and its computation. Three main types of situations are often encountered in 2-group comparison studies: i) sample size determination is not performed whatever the intended analysis for PRO data (CTT and/or IRT), ii) tentative justification is occasionally given a posteriori for the size of studies, iii) sample size computation is made a priori but only relies on CTT (mostly using the classical formula for comparing normally distributed endpoints on expected mean scores) even if IRT models are envisaged for data analysis. In this latter case, previous studies have shown that the classical formula was inadequate for IRT models because it leads to underestimation of the required sample size . From this perspective, a method has been recently developed for power and sample size determination when designing a study using a PRO as a primary endpoint when IRT models coming from the Rasch family are intended to be used for subsequent analysis of the data . This method, named Raschpower, provides the power for a given sample size during the planning stage of a study in the framework of Rasch models. It depends on the following parameters (that are a priori assumed and fixed): the parameters related to the items of the questionnaire (items' number J and difficulties parameters δj, j = 1,…,J), the variance of the latent trait (σ2) and the mean difference between groups on the latent trait (γ). Some of these parameters are easily known a priori when planning a study (e.g. number of items) others are sometimes more difficult to reach (e.g. items difficulties, σ2, γ) and initial estimates based on the literature or pilot studies are required. Besides, whether all these parameters have the same importance regarding sample size determination for Rasch models is unknown. The aim of our paper is to explore the relative impact of these parameters on sample size computation and to identify the most relevant to be used during study design for reliable power determination for Rasch models. Our main objective is to provide a simple method for sample size determination when a Rasch model is planned for analysing PRO data in a 2-group comparison study.
The Rasch model
In the Rasch model , the responses to the items are modelled as a function of a latent variable representing the so-called ability of a patient measured by the questionnaire (e.g. QoL, anxiety, fatigue…). The latent variable is often considered as a random variable assumed to follow a normal distribution. In this model, each item is characterized by one parameter (δj for the jth item), named item difficulty because the higher its value, the lower the probability of a positive (favourable) response of the patient to this item regarding the latent trait being measured.
Let us consider that two groups of patients are compared and that a total of N patients have answered a questionnaire containing J binary items. Let Xij be a binary random variable representing the response of patient i to item j with realization xij, θi be the realization of the latent trait Θ for this patient, and γ the group effect defined as the difference between the means of the latent trait in the two groups.
where δj represents the difficulty parameter of item j and gi = 0,1 for patients in the first or second group, respectively. The latent variable Θ is usually a random variable following a normal distribution with unknown parameters μ and σ2. Marginal maximum likelihood estimation is often used for estimating the parameters of the model.
Sample size determination in the framework of the Rasch model – The Raschpower method
We assume that we want to design a clinical trial using a given dimension of a PRO (e.g. the Mental Health dimension of the SF-36) as a primary outcome in a two-group cross-sectional study. Let γ (assumed > 0) be the difference between the mean values of the latent trait (e.g. mental health) in the two groups and σ2 the common variance of the latent trait in both groups. We assume that the study involves the comparison of the two hypotheses H0: γ = 0 against the two-sided alternative H1: γ ≠ 0. If we plan to use a Rasch model that includes a group effect γ (Eq 1) to test this null hypothesis on the data that will be gathered during the study with a given power 1-βR and type I error α, determination of the required sample size can be made using an adapted formula that has been implemented in the Raschpower method . This method is based on the power of the Wald test of group effect γ for a given sample size and it is briefly described. To perform a Wald test, an estimate Γ of γ is required as well as its standard error. Since we are designing a study, some assumptions are made regarding the expected values of these parameters. More specifically, Γ is set at the assumed value for the group effect, γ, and its standard error is obtained as follows: an expected dataset of the patient’s responses is created conditionally on the planning values that are assumed for the sample size in each group, the group effect γ, the items difficulties δj, and the variance of the latent trait σ2. The probabilities and the expected frequencies of all possible response patterns for each group are computed with the statistical model that will be used for analyzing the data that will be gathered during the study: a Rasch model. The variance of the group effect is subsequently estimated using a Rasch model including a group effect with δj and σ2 fixed to their planned expected values.
where Φ is the cumulative standard normal distribution function and z 1 − α/2 the percentiles of the standard normal distribution. 1 − βR is the power of the Wald test of group effect when a Rasch model is used to detect γ at level α. In practice, γ, σ2, and the items' difficulties are unknown population parameters and initial estimates based on the literature or pilot studies are required for calculations.
Relationship between the Raschpower method and the classical formula for manifest normal variables
Where NC1 = k x NC0 (when k = 1, the sample sizes are assumed equal in both groups).
previous observations  have shown that this ratio Ra remained stable for different values of Ng and 1-βR, given γ, J and items difficulties
Hence this ratio Ra depends on the well-known classical formula and can be used to provide sample size calculations for Rasch modelling.
A simulation study has been performed in order to get more insight into the relationships between the parameters that are required when planning a study for power determination for a given sample size (γ, σ2, δj, J) and the ratio Ra. A large number of cases (106) were simulated with each case corresponding to a single parameter combination (γ, σ2, δj, J, Ng). The parameters values were randomly drawn from continuous or discrete uniform distributions, U[min-max], for: the variance of the latent trait σ2 (U[0.25-9]), the group effect γ (U[0.2xσ - 0.8xσ]), the number of items J (U[3-20]), and the sample size per group Ng assumed to be equal in both groups (U[50–500]). The items difficulty parameters δj, j = 1,…,J, were drawn from a centred normal distribution with variance σ2 and set to the percentiles of the distribution. The Raschpower method was applied on each parameter combination and provided the power 1-βR for Rasch modelling as well as the ratio Ra. Multiple linear regression was performed to assess the contribution of Ng, γ, J, and σ2 and the difficulty parameters δj, j = 1,…,J to the variation of the ratio Ra. The effects of the difficulty parameters on Ra were investigated in several ways for different values of J: i) by introducing each parameter individually δj, j = 1,…,J, ii) by introducing their mean and variance. A two-tailed P-value < 0.05 was considered significant. The variance explained by the model (R2) and the root mean square error (RMSE) were obtained and contributed to variable selection. Variables were removed if R2 and RMSE remained stable (within a 0.01 range). Post-regression diagnoses were performed to ensure that all linear regression assumptions were met (normality and homoscedasticity of residuals). Statistical analysis was performed using SAS statistical software version 9.3 (SAS Institute Inc, Cary, North Carolina).
Among the 106 parameter combinations, 15278 corresponded to the largest power for CTT and Rasch-based analysis, 100%, where the ratio cannot be computed. Hence all analyses were performed on 984722 parameter combinations.
where , for i = 1, …, 984722
Parameters estimates of the linear regression model explaining the ratio provided by the Raschpower method
NPOP = 984722
1.012 (7.0 10−5)
0.095 (1.0 10−4)
0.939 (5.0 10−4)
3.730 (7.5 10−4)
Distributions of the difference between the ratio (respectively number of subjects per group) predicted by the model and the one expected by the Raschpower method Δ R (respectively Δ N ) and according to the threshold (Thres) for Δ N
NPOP = 1996077
2.5% / Median / 97.5%
−0.049 / 0.002 / 0.043
[−1.236 ; 0.230]
−10.623 / 0.438 / 13.499
[−179.576 ; 112.064]
– Thres < ΔN < + Thres
ΔN < − Thres
ΔN > + Thres
An example of sample size determination in clinical research using the ratio – NHP data
Multiplying Ng by this ratio gives a sample size of = 197 × 1.27210 ≈ 251 patients per group that should provide the desired power of 90% for Rasch modelling of the pain dimension of the NHP questionnaire. These results were compared to those obtained with the Raschpower method using the estimated difficulty parameters from the pilot study (2.61, 2.94, 1.75, 0.46, − 0.11, 0.36, 1.28, 2.23), = 0.649, = 3.9323, and Ng = 197 per group. An 80% power (1-βR) is expected using the Raschpower method for Rasch modelling with a sample size of Ng = 197 per group (Figure 1, RP①). The proposed ratio is therefore equal to 197 / 147 = 1.34 where Nc = 147 is the number of subjects per group that provides a power of 80% using the classical formula (Figure 1, CF②). Hence, using the ratio, 197 × 1.34 ≈ 264 (NR) patients per group should provide the desired power of 90% for Rasch modelling (Figure 1, RP⑤).
Comparison of the required parameters and the results obtained using the linear regression model and the Raschpower method on the NHP data
Linear regression model
(2.61, 2.94, 1.75, 0.46, −0.11, 0.36, 1.28, 2.23)
The fact that the number of subjects given by the classical formula, based on the latent trait, has to be increased using the ratio to reach the expected power for Rasch modelling could deliver a wrong message. Indeed, it could be interpreted as if Rasch models required more subjects than CTT-based analyses would. In fact, the classical formula is directly computed from the expected difference between the latent traits in both groups and the latent trait's variance in each group, assumed to be equal. By doing so, we assume that the means and variance of the latent traits are "perfectly" known and thus do not take into account the fact that the latent trait is not an observed (manifest) variable. Hence, its estimation requires the use of a model which creates uncertainty, unlike scores that can be directly observed and measured. This uncertainty is taken into account by adjusting the sample size using the ratio to obtain an adequately sized study for Rasch modelling. Moreover, it has been underlined that the so-called effect size (difference in means over the standard deviation) on the score scale was lower than the corresponding effect size on the latent trait scale. Consequently, the sample size requested for CTT-based analysis using the effect size on the score scale is higher than its counterpart on the latent trait scale.
The proposed method can be used with confidence when J stands between 3 and 20 and especially when the variance of the latent trait is expected to be higher than 1. Otherwise (when σ2 < 1), the Raschpower method should be preferred since the ratio-based approach might under or overestimate the sample size. One of the limitations of our study is that we focused on one of the most well-known IRT model, the Rasch model. The Raschpower method has also been developed for other models that are well suited for the analysis of polytomous item responses, such as the Partial Credit Model or the Rating Scale Model (Hardouin, under revision). Moreover, the Raschpower method has recently been extended to deal with longitudinal designs  and it might be expected that this ratio would also be worthwhile in these contexts. Finally, the Raschpower method (for dichotomous and polytomous items and for cross-sectional and longitudinal designs) and the ratio-based approach (for dichotomous items) have been implemented in the free Raschpower module available at the website PRO-online http://pro-online.univ-nantes.fr.
Using the classical formula for normally distributed endpoints along with the proposed ratio only depending on the number of items and the variance of the latent trait can provide a straightforward and reliable formula for sample size computation for subsequent Rasch-based analysis of PRO data.
This study was supported by the French National Research Agency, under reference N 2010 PRSP 008 01.
- Smith EM, Pang H, Cirrincione C, Fleishman S, Paskett ED, Ahles T, Bressler LR, Fadul CE, Knox C, Le-Lindqwister N, Gilman PB, Shapiro CL, Alliance for Clinical Trials in Oncology: Effect of duloxetine on pain, function, and quality of life among patients with chemotherapy-induced painful peripheral neuropathy: a randomized clinical trial. JAMA. 2013, 309: 1359-1367.View ArticlePubMedPubMed CentralGoogle Scholar
- Lamy A, Devereaux PJ, Prabhakaran D, Taggart DP, Hu S, Paolasso E, Straka Z, Piegas LS, Akar AR, Jain AR, Noiseux N, Padmanabhan C, Bahamondes JC, Novick RJ, Vaijyanath P, Reddy SK, Tao L, Olavegogeascoechea PA, Airan B, Sulling TA, Whitlock RP, Ou Y, Pogue J, Chrolavicius S, Yusuf S, CORONARY Investigators: Effects of off-pump and on-pump coronary-artery bypass grafting at 1 year. N Engl J Med. 2013, 368: 1179-1188.View ArticlePubMedGoogle Scholar
- Cunningham MA, Swanson V, Holdsworth RJ, O'Carroll RE: Late effects of a brief psychological intervention in patients with intermittent claudication in a randomized clinical trial. Br J Surg. 2013, 100: 756-760.View ArticlePubMedGoogle Scholar
- Cartwright M, Hirani SP, Rixon L, Beynon M, Doll H, Bower P, Bardsley M, Steventon A, Knapp M, Henderson C, Rogers A, Sanders C, Fitzpatrick R, Barlow J, Newman SP, Whole Systems Demonstrator Evaluation Team: Effect of telehealth on quality of life and psychological outcomes over 12 months (Whole Systems Demonstrator telehealth questionnaire study): nested study of patient reported outcomes in a pragmatic, cluster randomised controlled trial. BMJ. 2013, 346: f653-View ArticlePubMedPubMed CentralGoogle Scholar
- Weis J, Arraras JI, Conroy T, Efficace F, Fleissner C, Görög A, Hammerlid E, Holzner B, Jones L, Lanceley A, Singer S, Wirtz M, Flechtner H, Bottomley A: Development of an EORTC quality of life phase III module measuring cancer-related fatigue (EORTC QLQ-FA13). Psychooncology. 2013, 22: 1002-1007.View ArticlePubMedGoogle Scholar
- Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, Amtmann D, Bode R, Buysse D, Choi S, Cook K, Devellis R, DeWalt D, Fries JF, Gershon R, Hahn EA, Lai JS, Pilkonis P, Revicki D, Rose M, Weinfurt K, Hays R: The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol. 2010, 63: 1179-1194.View ArticlePubMedPubMed CentralGoogle Scholar
- Langer MM, Hill CD, Thissen D, Burwinkle TM, Varni JW, DeWalt DA: Item response theory detected differential item functioning between healthy and ill children in quality-of-life measures. J Clin Epidemiol. 2008, 61: 268-276.View ArticlePubMedGoogle Scholar
- Fisher GH, Molenaar IW: Rasch Models, Foundations, Recent Developments, and Applications. 1995, New-York: Springer-VerlagGoogle Scholar
- Sébille V, Hardouin J-B, Mesbah M: Sequential analysis of latent variables using mixed-effect latent variable models: impact of non-informative and informative missing data. Stat Med. 2007, 26: 4889-4904.View ArticlePubMedGoogle Scholar
- Hardouin JB, Conroy R, Sébille V: Imputation by the mean score should be avoided when validating a patient reported outcomes questionnaire by Rasch model in presence of informative missing data. BMC Med Res Meth. 2011, 11: 105-View ArticleGoogle Scholar
- De Bock E, Hardouin JB, Blanchin M, Le Neel T, Kubis G, Bonnaud-Antignac A, Dantan E, Sébille V: Rasch-family models are more valuable than score-based approaches for analyzing longitudinal PRO with missing data. Stat Meth Med Res. in press
- Andrich D: Rating scales and rasch measurement. Expert Rev Pharmacoecon Outcomes Res. 2011, 11: 571-585.View ArticlePubMedGoogle Scholar
- Ravens-Sieberer U, Herdman M, Devine J, Otto C, Bullinger M, Rose M, Klasen F: The European KIDSCREEN approach to measure quality of life and well-being in children: development, current application, and future advances. Qual Life Res. in press
- Waller J, Ostini R, Marlow LAV, McCaffery K, Zimet G: Validation of a measure of knowledge about human papillomavirus (HPV) using item response theory and classical test theory. Prev Med. 2013, 56: 35-40.View ArticlePubMedGoogle Scholar
- Sapin C, Simeoni M-C, El Khammar M, Antoniotti S, Auquier P: Reliability and validity of the VSP-A, a health-related quality of life instrument for ill and healthy adolescents. J Adolesc Health. 2005, 36: 327-336.View ArticlePubMedGoogle Scholar
- Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996, 276: 637-639.View ArticlePubMedGoogle Scholar
- Schulz KF, Altman DG, Moher D, CONSORT Group: CONSORT: Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials. Ann Intern Med. 2010, 2010 (152): 726-732.View ArticleGoogle Scholar
- von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP: The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007, 147: 573-577.View ArticlePubMedGoogle Scholar
- Des Jarlais DC, Lyles C, Crepaz N: Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004, 94: 361-366.View ArticlePubMedPubMed CentralGoogle Scholar
- Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD, CONSORT PRO Group: Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA. 2013, 309: 814-822.View ArticlePubMedGoogle Scholar
- US Department of Health and Human Services: Guidance for Industry (Patient-reported outcome measures: use in medical product development to support labeling claims). http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf,
- Sébille V, Hardouin J-B, Le Neel T, Kubis G, Boyer F, Guillemin F, Falissard B: Methodological issues regarding power of classical test theory and IRT-based approaches for the comparison of Patient-Reported Outcome measures – A simulation study. BMC Med Res Meth. 2010, 10: 24-View ArticleGoogle Scholar
- Hardouin J-B, Amri S, Feddag M-L, Sébille V: Towards power and sample size calculations for the comparison of two groups of patients with item response theory models. Stat Med. 2012, 31: 1277-1290.View ArticlePubMedGoogle Scholar
- Julious SA: Sample sizes for clinical trials with normal data. Stat Med. 2004, 30: 1921-1986.View ArticleGoogle Scholar
- Glas CAW: The derivation of some tests for the Rasch model from the multinomial distribution. Psychometrika. 1988, 53: 525-546.View ArticleGoogle Scholar
- Blanchin M, Hardouin J-B, Le Neel T, Kubis G, Blanchard C, Miraillé E, Sébille V: Comparison of CTT and IRT based-approach for the analysis of longitudinal Patient Reported Outcome. Stat Med. 2011, 30: 825-838.PubMedGoogle Scholar
- Hardouin J-B, Audureau E, Leplège A, Coste J: Spatio-temporal Rasch analysis of Quality of life outcomes in the french general population. Measurement invariance and group comparisons. BMC Med Res Meth. 2012, 212: 182-View ArticleGoogle Scholar
- Feddag ML, Blanchin M, Hardouin JB, Sébille V: Power analysis on the time effect for the longitudinal Rasch model. J Appl Meas. 2014, in pressGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/14/87/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.