 Research article
 Open access
 Published:
A systematic approach to designing statistically powerful heteroscedastic 2 × 2 factorial studies while minimizing financial costs
BMC Medical Research Methodology volume 16, Article number: 114 (2016)
Abstract
Background
The 2 × 2 factorial design is widely used for assessing the existence of interaction and the extent of generalizability of two factors where each factor had only two levels. Accordingly, research problems associated with the main effects and interaction effects can be analyzed with the selected linear contrasts.
Methods
To correct for the potential heterogeneity of variance structure, the WelchSatterthwaite test is commonly used as an alternative to the t test for detecting the substantive significance of a linear combination of mean effects. This study concerns the optimal allocation of group sizes for the WelchSatterthwaite test in order to minimize the total cost while maintaining adequate power. The existing method suggests that the optimal ratio of sample sizes is proportional to the ratio of the population standard deviations divided by the square root of the ratio of the unit sampling costs. Instead, a systematic approach using optimization technique and screening search is presented to find the optimal solution.
Results
Numerical assessments revealed that the current allocation scheme generally does not give the optimal solution. Alternatively, the suggested approaches to power and sample size calculations give accurate and superior results under various treatment and cost configurations.
Conclusions
The proposed approach improves upon the current method in both its methodological soundness and overall performance. Supplementary algorithms are also developed to aid the usefulness and implementation of the recommended technique in planning 2 × 2 factorial designs.
Background
The factorial designs are the most common formulation for assessing the existence of interaction and the extent of generalizability of two or more factors in medical research. Notably, systematic reviews and practical guidelines have been presented in Green, Liu, and O’Sullivan [1], Kahan [2], Kent et al. [3], McAlister et al. [4], and Montgomery, Astin, and Peters [5]. The prominent advantages of factorial designs over a series of singlefactor studies can be easily seen with the 2 × 2 factorial design. Particularly, a wide range of research problems associated with interactions, main effects, and various mixtures can be examined in terms of a linear combination of mean effects. It is noteworthy that the designated linear comparison represents the substantive hypothesis of interest and reveals essential information that cannot be obtained from singlefactor studies. Comprehensive exposition and further information can be found in Kutner et al. [6] and Maxwell and Delaney [7]. In addition, useful flowcharts of twofactor studies were presented in Figure 19.11 of Kutner et al. [6] and Figure 7.2 of Maxwell and Delaney [7] for appropriate approaches to evaluating effects either in the presence or the absence of an interaction.
It follows from the independence, normality, and homogeneity of variance assumptions, that the inference for a linear combination of mean effects can be conducted with a t statistic. However, numerous studies have found that the homogeneous variances assumption is frequently untenable in many research areas. It was explicitly stressed in Golinski and Cribbie [8], Grissom [9], Keselman et al. [10], and Ruscio and Roche [11] that variances are often extremely different across treatment groups. Therefore, it is prudent to adopt proper approaches that are superior to the traditional methods under various circumstances of heterogeneous variances. For testing a hypothesis of a linear combination of group means, the approximation suggested independently by Satterthwaite [12] and Welch [13] is the most widely recommended technique to correct for variance heterogeneity (Kirk [14]; Maxwell & Delaney [7]). The procedure is sometimes referred to as the Welch–Satterthwaite test and provides a simple and robust tsolution with approximate degrees of freedom.
A research study typically requires adequate statistical power and sufficient sample size to detect scientifically credible effects. Within the context of medical trials, the power and sample size implications for subgroup analysis of treatment combination and interaction in factorial studies were noted in Beck et al. [15], Brookes et al. [16], Gonen [17], Natarajan et al. [18], and Wolbers et al. [19]. To extend the applicability of the Welch–Satterthwaite procedure in planning research designs, Shieh and Jan [20] presented two approaches to power and sample size calculations for the Welch–Satterthwaite test. The approximate method presents a particularly convenient technique for general use. Alternatively, the exact formulation is noticeably more effective in maintaining the power performance in some situations. However, the prescribed approaches for choosing sample size to provide adequate power do not consider the cost issues of different sample size choices. But the cost for treating a subject often varies with treatment groups and it is impossible for researchers to overlook budget constraints in practice. Bacchetti [21] and Bacchetti, McCulloch, and Segal [22] also emphasized that the conventional sample size procedures do not take cost into account and can produce costinefficient sample size choices. Alternatively, Allison et al. [23] advocated designing statistically powerful studies while minimizing costs.
Recently, Luh and Guo [24] studied the problem of efficient sample size allocation to reduce budget for the Welch–Satterthwaite test within the context of 2 × 2 factorial design. The optimal ratio of sample sizes suggested in Luh and Guo [24] is proportional to the ratio of the population standard deviations divided by the square root of the ratio of the unit sampling costs. Despite the presented argument and demonstration, their method is susceptible to three critical issues. First, the twostep procedure of Luh and Guo [24] involved several approximations including the use of quantiles of a standard normal distribution and a t distribution with approximate degrees of freedom. Unlike the two approaches of Shieh and Jan [20], Luh and Guo [24] did not employ the feature of a noncentral t distribution in their computation. More importantly, they did not explicitly define the nonnull distribution and the power function of the Welch–Satterthwaite test. Therefore, the resulting explication is incomplete by its absence of vital technical formulations.
Second, the optimal sample size ratios in Luh and Guo [24] were obtained with the simplified assumption that the test statistic has a normal distribution with known variances. Note that the particular formula has been adopted in Guo and Luh [25] to determine the presumably optimal sample size ratios for designing statistically powerful twosample studies while minimizing financial costs. However, it was shown in Jan and Shieh [26] that such an allocation scheme generally do not give the optimal solution. Hence, it is arguable that Luh and Guo’s [24] claim of optimal allocations requires further clarification. Third, the sample sizes need to be integer values in reality. The final sample sizes in Luh and Guo [24] are determined by rounding up the outcomes of the twostep calculations to the next largest integers. Conceivably, the use of discrete numbers induces some inexactness into the optimal evaluation. It is unlikely that a direct integer rounding process will always give the optimal result under power and cost considerations even that the optimal sample size ratios have been implemented. Instead, a systematic power calculation and cost assessment needs to be conducted to find the proper result.
In order to make a useful and wellsupported recommendation on optimal sample size allocations, this article presents an alternative approach for designing statistically powerful heteroscedastic 2 × 2 factorial studies while minimizing financial costs. A detailed account of the Welch–Satterthwaite test is presented next to document its theoretical characteristics and computational requirements. Moreover, the optimization processes of the proposed procedure with power and cost constraints are described. To provide definitive evidence, extensive empirical investigations were conducted to demonstrate the advantages of the suggested approach over the potentially defective method of Luh and Guo [24] under a variety of model configurations. Essentially, this study contributes to the literature of sample size methodology for the Welch–Satterthwaite test in two aspects. First, the suggested allocation scheme extends the design strategy of Shieh and Jan [20] to accommodate both budgetary constraints and power assessments. Second, the proposed optimization technique offers prominent improvement over Luh and Guo’s [24] method to obtain the true optimal sample sizes. For computing the attained power and optimal allocation scheme in planning research designs, the SAS computer algorithms are offered to facilitate the recommended procedure.
Methods
Consider the statistical model of a 2 × 2 heteroscedastic factorial design:
where X_{ ijk } represents the independent and normally distributed response variable with expected values μ_{ ij } and variances σ ^{2}_{ ij } , μ_{ ij } is the population mean, and σ ^{2}_{ ij } is the error variance at level i of the factor A and level j of the factor B for i and j = 1 and 2, and k = 1, …, N_{ ij }. In general, a linear combination of mean parameters is defined as
where L_{ ij } are the linear coefficients. A contrast is a special case of a linear combination in which the coefficients of the means add up to 0. Notably, the contrasts associated with the main effect A, the main effect B, and the interaction effect between A and B can be expressed as
and
respectively. An unbiased estimator \( \widehat{\uppsi} \) of the linear combination ψ is obtained by simply replacing each population mean in Equation 2 by the corresponding sample mean:
where \( {\overline{X}}_{ij}={\displaystyle \sum_{k=1}^{N_{ij}}{X}_{ijk}}/{N}_{ij} \) for i and j = 1 and 2. The hypothesis testing of H_{0}: ψ = ψ_{0} versus H_{1}: ψ ≠ ψ_{0} can be conducted with the following statistic
where ψ_{0} is a specified constant, \( {\widehat{\upomega}}^2={\displaystyle \sum_{i=1}^2{\displaystyle \sum_{j=1}^2{L}_{ij}^2{S}_{ij}^2}}/{N}_{ij} \) is the typical estimator of \( {\upomega}_{\cdot}^2=Var\left(\widehat{\uppsi}\right)={\displaystyle \sum_{i=1}^2{\displaystyle \sum_{j=1}^2{L}_{ij}^2{\sigma}_{ij}^2}}/{N}_{ij} \), and \( {S}_{ij}^2={\displaystyle \sum_{l=1}^{Nij}{\left({X}_{ijk}{\overline{X}}_{ij}\right)}^2}/\left({N}_{ij}1\right) \) is the sample variance estimator of σ ^{2}_{ ij } for i and j = 1 and 2. Under the null hypothesis H_{0}: ψ = ψ_{0}, it was demonstrated in Satterthwaite [12] and Welch [13] that the statistic T* given in Equation 5 has a convenient approximate distribution
where t(ν) is a t distribution with degrees of freedom ν and
For inferential purposes, the term of degrees of freedom ν is replaced by its counterpart \( \widehat{v} \) with direct substitution of S ^{2}_{ ij } for σ ^{2}_{ ij } in ν, where
Hence, the null distribution of T* is modified as
and the Welch–Satterthwaite procedure rejects H_{0} at the significance level α if \( \leftT*\right>{t}_{\widehat{v},a/ 2} \) where \( {t}_{\widehat{v},a/2} \) is the upper 100(α/2) percentile of the t distribution \( t\left(\widehat{v}\right) \).
Moreover, it was noted in Shieh and Jan [20] that the statistic T* has the general approximate distribution
where t(ν, δ) is a noncentral t distribution with degrees of freedom ν and noncentrality parameter
It immediately follows from the noncentral t distribution given in Equation 9 that the power function of the Welch–Satterthwaite test can be approximated by
Accordingly, Shieh and Jan [20] noted that the approximate power function π(δ) provides a useful expression because of its theoretical implications and practical applications. The numerical computation of the power level requires the evaluation of the cumulative distribution function of a noncentral t variable with respective to the quantile of a regular t distribution. Since all related functions are embedded in major statistical packages, the actual computations can be readily conducted with current computing capabilities. A SAS/IML (SAS Institute [27]) program is presented in Additional file 1 for computing π(δ) with the designated sample sizes and model configurations. More importantly, the empirical examinations presented later reveal that the approximate power function is sufficiently accurate for general purposes.
Optimal allocation scheme
The determination of an adequate and efficient allocation of sample sizes is a vital aspect in the planning stage of research studies. It is often sensible to consider the sample size issues in the presence of funding constraints. The total cost of a 2 × 2 factorial study can be represented by the overhead cost and sampling costs through the following linear cost function
where C_{ O } is the fixed overhead cost associated with the study and C_{ ij } reflects unit sampling cost of each subject in group (i, j) for i and j = 1 and 2. It is important to note that the consideration of the total number of subjects can be viewed as a special case of the cost function C_{ T }, with C_{ O } = 0 and C_{ ij } = 1 for i and j = 1 and 2. From the cost perspective, a question arises naturally in choosing the optimal sample sizes: What is the least cost for a research study to maintain its desired power level?
To develop a systematic search of the optimal solution to ensure the nominal power performance while minimizing the total cost C_{ T } defined in Equation 11, the suggested approach is conducted in two steps. With the specifications of the significance level α, the desired power level 1 – β, the null effect size ψ_{0}, and the model parameters of group means and variance components, the first step computes the preliminary sample sizes {N_{P11}, N_{P12}, N_{P21}, N_{P22}} via an optimization process by minimizing the objective cost function with the constraint that the attained power is equal to or greater than the designated level. Note that the attained power is computed with the approximate power function π(δ) of the Welch–Satterthwaite test defined in Equation 10. A closed form solution rarely exists for most situations and therefore the minimization typically requires iterative and extensive computations.
Accordingly, the NLPQN subroutine of the SAS/IML package provides an efficient approach to finding the optimal solution for cost minimization with the power function as a nonlinear constraint. It must be emphasized that the sample sizes are treated as continuous variables in the optimization process. Thus, the resulting values {N_{P11}, N_{P12}, N_{P21}, N_{P22}} are almost surely not all integers. Due to the underlying metric of integer sample sizes, in practice, the values are rounded up to the nearest integer. This simple and intuitive adjustment maintains that the corresponding power is still no less than the nominal power. But both the achieved power and total cost actually increase for the modified sample sizes {N_{M11}, N_{M12}, N_{M21}, N_{M22}} = {[N_{P11}] + 1, [N_{P12}] + 1, [N_{P21}] + 1, [N_{P22}] + 1}, where [N] denotes the integer part of N. Notably, the optimal property of the sample sizes {N_{P11}, N_{P12}, N_{P21}, N_{P22}} does not necessarily carry over to the adjusted sample sizes {N_{M11}, N_{M12}, N_{M21}, N_{M22}}. In sum, the rounding process tends to induce a suboptimal solution and the optimal set of sample sizes remains to be determined.
In the second step, a detailed comparison is conducted to find the proper result by taking into account the discrete character of sample sizes in practice. Specifically, power calculations and cost evaluations are performed for a total of 2^{4} = 16 sample size sets {N_{11}, N_{12}, N_{21}, N_{22}} with N_{ ij } = [N_{ Pij }] or [N_{ Pij }] + 1 for i and j = 1 and 2. Then the optimal allocation {N ^{*}_{11} , N ^{*}_{12} , N ^{*}_{21} , N ^{*}_{22} } is found through a screening of the sample size combinations that attain the desired power while giving the least cost. If more than one combination yields the same magnitude of least cost, the one producing the largest power is reported. Note that this finetuning procedure can be considered as a safeguard to ensure that the nearly optimal and integer sample sizes {N_{M11}, N_{M12}, N_{M21}, N_{M22}} is the true optimal solution. Unfortunately, the conducted numerical calculations revealed that the sample sizes {N_{M11}, N_{M12}, N_{M21}, N_{M22}} are rarely the correct optimal allocation. This finding justifies the suggested screening technique and notifies the deficiency of the rounding process in Luh and Guo [24]. A special purpose SAS/IML computer program is presented in Additional file 2 for performing the necessary computation.
On the other hand, Luh and Guo [24] presented a twostep procedure for obtaining the optimal result. First, using the simplified normal assumption, they showed that the optimal sample size ratio is proportional to the ratio of the population standard deviations divided by the square root of the ratio of the unit sampling costs:
i and j = 1 and 2. The initial sample sizes {N_{Z11}, N_{Z12}, N_{Z21}, N_{Z22}} are obtained with N_{ Zij } = N_{Z11·}r_{ ij }, i and j = 1 and 2, where
\( {w}^2={\displaystyle \sum_{i=1}^2{\displaystyle \sum_{j=1}^2{L}_{ij}^2{\upsigma}_{ij}^2}}/{r}_{ij}, \) and zα_{/2} and zβ are the upper 100(α/2)th and 100 · βth percentiles of the standard normal distribution, respectively. Then, to account for the approximate degrees of freedom of the Welch–Satterthwaite test, they suggested a modified process by using the sample sizes {N_{Z11}, N_{Z12}, N_{Z21}, N_{Z22}} to yield the second set of sample sizes {N_{T11}, N_{T12}, N_{T21}, N_{T22}} where N_{ Tij } = N_{T11·}r_{ ij }, i and j = 1 and 2,
and
Because the values of {N_{T11}, N_{T12}, N_{T21}, N_{T22}} are most likely fractional, a direct rounding process is applied to give the final sample sizes {N_{LG11}, N_{LG12}, N_{LG21}, N_{LG22}} where N_{ LGij } = [N_{ Tij }] + 1 for i and j = 1 and 2. In contrast to the proposed finetuning algorithm, Luh and Guo [24] overlooked the inexactness issue caused by integer sample sizes in power evaluation and cost minimization. This is one of possible causes that their method does not guarantee to produce the optimal sample sizes as shown in the subsequent numerical comparisons.
Numerical assessments
To illustrate the advantage of the proposed optimal procedure over the existing method of Luh and Guo [24], numerical appraisals were performed to assess the optimal sample size calculations of the two methods under a wide variety of model configurations. The empirical investigation consists of two studies with real and hypothetical data that correspond to the model settings in Tables 2 and 3 of Luh and Guo [24].
Study I
For the purposes of comparison, the illustrative example for the 2 × 2 factorial study of attack context and panic fear in Luh and Guo [24] is reexamined here. The data was obtained from the investigations of frequency and cost of emergency service use in Barnett and Nurmagambetov [28] and Greaves et al. [29]. To exemplify a typical research scenario most frequently encountered in the planning stage of a study, the reported findings are employed to provide planning values of the model parameters and design characteristics for future asthma study.
Specifically, the mean effects and variance components are designated as {μ_{11}, μ_{12}, μ_{21}, μ_{22}} = {1.23, 0.42, 0.13, 0.38} and {σ ^{2}_{11} , σ ^{2}_{12} , σ ^{2}_{21} , σ ^{2}_{22} } = {0.6889, 0.5184, 0.1156, 0.5929}, respectively. The contrast effect sizes associated with the main effect A (Asthma attack: recent and stable), the main effect B (Panic fear: low and high), and the interaction effect defined in Equation 3 are ψ_{ I } = 1.06, ψ_{ A } = 1.14, and ψ_{ B } = 0.56, respectively. For the definition of total cost, the fixed cost is set as C_{ O } = 0, and two sets of unit costs C_{ U } = {784.74, 267.96, 82.94, 242.44} and C_{ E } = {1, 1, 1, 1} are considered to represent varied and identical unit sampling costs for the four treatment groups. Then the proposed allocation procedure was employed to find the optimal sample sizes needed to achieve the nominal power 1 – β = 0.8 for three contrast effects and two cost structures. Throughout this empirical study, the significance level is set as α = 0.05 and the null value is ψ_{0} = 0. Overall, a total of six different sets of sample sizes were obtained.
The optimal allocation, total cost, and total sample size are summarized in Table 1. Unlike the presented procedure, the numerical outcomes reported in Table 2 of Luh and Guo [24] are based on the sample size ratios in Equation 12. For ease of illustration, the corresponding results are also presented in Table 1. In addition, the attained powers for the computed sample sizes were computed by the suggested approximate power function π(δ) given in Equation 10. Due to the approximate nature of the suggested power calculations, Monte Carlo simulation of 10,000 independent data sets was also conducted to obtain the simulated powers. Accordingly, the adequacy of the approximate power function can be evaluated by the difference between the simulated power and approximate power. In addition to the prescribed optimal sample allocation, total cot and total sample size, the approximate power, simulated power and difference are also listed in Table 1.
Study II
To further explicate the optimal behavior and profound implication of the two sample size procedures, additional numerical assessments were performed with different variability patterns and cost structures. In this study, the simulation design of Luh and Guo [24] is adopted as a convenient framework. The model configurations are {μ_{11}, μ_{12}, μ_{21}, μ_{22}} = {1, 0, 0, 1} and {σ ^{2}_{11} , σ ^{2}_{12} , σ ^{2}_{21} , σ ^{2}_{22} } = {1, 4, 9, 16}. Here, the focus is on the detection of the interaction effect ψ_{ I } = μ_{11} – μ_{12} – μ_{21} + μ_{22} = 2. For the calculation of total cost, the fixed cost is set as C_{ O } = 0, and six sets of unit costs are considered: {C_{11}, C_{12}, C_{21}, C_{22}} = {1, 1, 1, 1}, {1, 2, 3, 4}, {4, 3, 2, 1}, {1, 1, 2, 5}, {5, 2, 1, 1}, and {1, 3, 3, 1}. These combinations of unit cost patterns were chosen to represent as much as possible the extent of characteristics that are likely to be obtained in actual applications. Similar to the illustration in Study I, the main settings are assigned as the significance level α = 0.05, nominal power 1 – β = 0.8, and the null value ψ_{0} = 0. Also, Monte Carlo simulation of 10,000 independent data sets was again conducted to obtain the simulated powers. The computed sample sizes, total cost, total sample size, approximate power, simulated power, and associated error of the two competing approaches are presented in Table 2 for six different cost functions.
Results
As was pointed out above, Luh and Guo [24] did not explicitly describe the power function for the Welch–Satterthwaite test and only the simulated powers were reported in their numerical demonstration. Note that the approximate powers presented in Table 1 of both sample size procedures are computed with respect to the approximate power function π(δ) for the reported sample sizes. The appraisal and implication of power performance are valid only when the power function provides reasonably accurate results. With the target power 0.8, it is evident that all the errors between the approximate power and simulated power are contained in the interval [–0.0031, 0.0111]. Hence, the presented power function in Equation 10 and resulting powers appear to be accurate enough to validate optimization analysis. Moreover, all the approximate powers induced by the computed sample sizes are greater than the nominal level 0.8 for all 12 cases. Hence, the two contending optimal allocations provide adequate power to validate further methodological comparisons of corresponding sample size procedures.
It can be readily seen from the reported sample sizes in Table 1 that the two allocation procedures do not agree with the optimal sample size settings. The computed sample sizes of Luh and Guo’s [24] method are larger than or equal to those of the suggested technique. The last scenario associated with ψ_{ B } and C_{ E } is the only one case where the two sets of sample sizes are identical. Specifically, the sample sizes associated with the interaction effect ψ_{ I } and cost coefficient set C_{ E } are {11, 16, 13, 19} and {12, 17, 14, 19} for the proposed approach and Luh and Guo’s [24] method, respectively. In this case, the maximum difference of cell sample sizes between the two procedures is only one. This indicates that a simple rounding process may distort an optimal solution even the sample size ratios {r_{11}, r_{12}, r_{21}, r_{22}} provide a nearly optimal result under normal approximation. Moreover, for the main effect ψ_{ A } and cost coefficient set C_{ U }, the computed sample sizes for the two contending procedures are {10, 13, 12, 16} and {10, 15, 13, 17}, respectively. Hence, the cell sample sizes incur the largest difference of two units. The same situation also occurred with the setup of the main effect ψ_{ B } and cost coefficient set C_{ U }. In both scenarios, the discrepancy of more than one unit in sample size determinations reveals that the normalitybased sample size ratios r_{ ij } given in Equation 12 is also responsible for the suboptimal behavior of Luh and Guo’s [24] method. Consequently, the total cost incurred by the proposed approach is always no larger than that of the Luh and Guo [24] procedure. These numerical evidences showed that their method does not warrant optimal sample size allocation for minimizing the total cost.
With the different variability patterns and cost structures in the second empirical study, it is clear from the marginal differences between the approximate power and simulated power in Table 2 that the approximate power function π(δ) maintains a accurate solution for power calculations. Specifically, all the absolute errors are less than 0.01 for all 12 cases. Just as in the preceding study, all the approximate powers or attained powers associated with both optimal allocation methods satisfy the desired power performance 0.8 for all six cost structures. However, the computed sample sizes in Table 2 indicate that Luh and Guo’s [24] method consistently give greater total cost and larger total sample size than the suggested technique. Notably, the accuracy of their method deteriorates as the variability of the unit costs increases. For the identical unit costs {1, 1, 1, 1}, the computed sample sizes {20, 40, 60, 79} and {20, 40, 60, 80} of the two approaches are nearly the same. However, the corresponding optimal sample sizes and total costs become prominently different when the unit costs are {1, 1, 2, 5}, {5, 2, 1, 1}, and {1, 3, 3, 1}. Accordingly, the performance of the existing method of Luh and Guo [24] is sensitive to model settings and cost schemes. In view of the potentially diverse treatment and cost configurations in factorial designs, their formula does not serve as a robust procedure for general use. Consequently, the allegedly optimal sample sizes calculations of Luh and Guo [24] are actually suboptimal, and their claim of developing the most efficient allocation for heteroscedastic 2 × 2 factorial designs is incorrect.
Discussion
The 2 × 2 factorial design is widely used in different fields of research for assessing the interaction between two factors. However, violation of the homogeneity of variance assumption has been the target of criticism in applications of standard factorial ANOVA. For testing a hypothesis of a linear combination of group means, the Welch–Satterthwaite procedure emerges as a robust alternative to heteroscedasticity when distributions are normal. For the ultimate aim of selecting the optimal sample size allocation, the analytical argument and empirical performance of an optimization technique must be well examined before it can be adopted as a general methodology in practice. The large sample theory shows that in order to ensure the nominal power while minimizing total cost, an optimal ratio of sample sizes is proportional to the ratio of the population standard deviations divided by the square root of the ratio of the unit sampling costs. At first sight, Luh and Guo’s [24] sample size procedure is easy to use and seem to give practically useful results. However, it is unlikely that a direct implementation of the simple allocation formula with a simple integer rounding will give the optimal solution under cost considerations. Therefore, there is a need to provide a systematic and detailed process to calculate the final optimal sample sizes. Evidently, the proposed procedure and the current method of Luh and Guo [24] are prominently different in fundamental principles and demand varying computational efforts. Due to the complexity of the sample size optimization problem under power and cost considerations, a complete analytic examination is not feasible. Instead, numerical assessments were conducted to examine their unique feature and underlying discrepancy in order to better understand the selection of an appropriate approach for optimal sample size determination in 2 × 2 factorial studies. Detailed appraisals showed that Luh and Guo’s [24] procedure generally do not give the optimal solution. Alternatively, the described approach provides a superior solution for optimal sample size allocation.
Conclusions
To enhance the usefulness of the Welch–Satterthwaite procedure in planning research designs, this article addresses the corresponding problem of designing statistically powerful heteroscedastic 2 × 2 factorial studies while minimizing financial costs. The suggested approach outperforms the current method in both its methodological soundness and overall performance. The presented sample size optimization methodology can be useful for the advocated practice of planning research design under both power and cost considerations. Computer algorithms are also developed to facilitate the implementation of the recommended power and sample size calculations in planning 2 × 2 factorial designs.
Abbreviations
 NLPQN:

The nonlinear optimization by quasiNewton method
 SAS/IML:

The interactive matrix language software of SAS
References
Green S, Liu PY, O’Sullivan J. Factorial design considerations. J Clin Oncol. 2002;20:3424–30.
Kahan BC. Bias in randomised factorial trials. Stat Med. 2013;32:4540–9.
Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010;11:85.
McAlister FA, Straus SE, Sackett DL, Altman DG. Analysis and reporting of factorial trials: A systematic review. J Am Med Assoc. 2003;289:2545–53.
Montgomery AA, Astin MP, Peters TJ. Reporting of factorial trials of complex interventions in community settings: A systematic review. Trials. 2011;12:179.
Kutner MH, Nachtsheim CJ, Neter J, Li W. Applied linear statistical models. 5th ed. New York, NY: McGraw Hill; 2005.
Maxwell SE, Delaney HD. Designing experiments and analyzing data: A model comparison perspective. 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2004.
Golinski C, Cribbie RA. The expanding role of quantitative methodologists in advancing psychology. Can Psychol. 2009;50:83–90.
Grissom RJ. Heterogeneity of variance in clinical data. J Consult Clin Psychol. 2000;68:155–65.
Keselman HJ, Huberty CJ, Lix LM, Olejnik S, Cribbie R, Donahue B, Kowalchuk RK, Lowman LL, Petoskey MD, Keselman JC, Levin JR. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Rev Educ Res. 1998;68:350–86.
Ruscio J, Roche B. Variance heterogeneity in published psychological research: A review and a new index. Methodology. 2012;8:1–11.
Satterthwaite FE. An approximate distribution of estimate of variance components. Biom Bull. 1946;2:110–4.
Welch BL. The generalization of Students’ problem when several different population variances are involved. Biometrika. 1947;34:28–35.
Kirk RE. Experimental Design: Procedures for the behavioral sciences (4th ed.). CA: Sage.: Thousand Oaks; 2013.
Beck GJ, Berg RL, Coggins CH, Gassman JJ, Hunsicker LG, Schluchter MD, Williams GW. Design and statistical issues of the Modification of Diet in Renal Disease Trial. The Modification of Diet in Renal Disease Study Group. Control Clin Trials. 1991;12:566–86.
Brookes ST, Whitely E, Egger M, Smith GD, Mulheran PA, Peters TJ. Subgroup analyses in randomized trials: Risks of subgroupspecific analyses; power and sample size for the interaction test. J Clin Epidemiol. 2004;57:229–36.
Gonen M. Planning for subgroup analysis: a case study of treatmentmarker interaction in metastatic colorectal cancer. Control Clin Trials. 2003;24:355–63.
Natarajan R, Turnbull BW, Slate EH, Clark LC. A computer program for sample size and power calculations in the design of multiarm and factorial clinical trials with survival time endpoints. Comput Methods Prog Biomed. 1996;49:137–47.
Wolbers M, Heemskerk D, Chau TT, Yen NT, Caws M, Farrar J, Day J. Sample size requirements for separating out the effects of combination treatments: Randomised controlled trials of combination therapy vs. standard treatment compared to factorial designs for patients with tuberculous meningitis. Trials. 2011;12:26.
Shieh G, Jan SL. Power and sample size calculations for testing linear combinations of group means under variance heterogeneity with applications to meta and moderation analyses. Psicológica. 2015;36:367–90.
Bacchetti P. Current sample size conventions: Flaws, harms, and alternatives. BMC Med. 2010;8:17.
Bacchetti P, McCulloch CE, Segal MR. Simple, defensible sample sizes based on cost efficiency. Biometrics. 2008;64:577–94.
Allison DB, Allison RL, Faith MS, Paultre F, PiSunyer X. Power and money: Designing statistically powerful studies while minimizing financial costs. Psychol Methods. 1997;2:20–33.
Luh WM, Guo JH. Allocating sample sizes to reduce budget for fixedeffect 2 × 2 heterogeneous analysis of variance. J Exp Educ. 2016;84:197–211.
Guo JH, Luh WM. Optimum sample size allocation to minimize cost or maximize power for the twosample trimmed mean test. Br J Math Stat Psychol. 2009;62:283–98.
Jan SL, Shieh G. Optimal sample sizes for Welch’s test under various allocation and cost considerations. Behav Res Methods. 2011;43:1014–22.
Institute SAS. SAS/IML User’s Guide, Version 9.3. Cary, NC: SAS Institute Inc.; 2013.
Barnett SBL, Nurmagambetov TA. Costs of asthma in the United States: 2002–2007. J Allergy Clin Immunol. 2010;127:145–52.
Greaves CJ, Eiser C, Seamark D, Halpin DMG. Attack context: An important mediator of the relationship between psychological status and asthma outcomes. Thorax. 2002;57:217–21.
Acknowledgements
The authors would like to thank the editor and a reviewer for their constructive comments that led to an improved article.
Funding
This work is partially supported by Grant: MOST 1052410H033025 from the Ministry of Science and Technology of Taiwan.
Availability of data and materials
The summary statistics are available in the two articles:
Barnett SBL, Nurmagambetov TA. Costs of asthma in the United States: 2002–2007. Journal of Allergy and Clinical Immunology. 2010; 127: 145152.
Greaves CJ, Eiser C, Seamark D, Halpin DMG. (2002). Attack context: An important mediator of the relationship between psychological status and asthma outcomes. Thorax. 2002; 57: 217221.
Authors’ contributions
SLJ conceived of the study, and participated in the development of theory and helped to draft the manuscript. GS carried out the numerical computations, participated in the empirical analysis and drafted the manuscript. Both authors read and approved the final manuscript.
Authors’ information
SLJ is a professor of Applied Mathematics, Chung Yuan Christian University, Taoyuan, Taiwan 32023. GS is a professor of Management Science, National Chiao Tung University, Hsinchu, Taiwan 30010.
Competing interests
Both authors declare that they have no competing interests.
Consent for publication
Not applicable.
Ethics approval and consent to participate
Not applicable.
Author information
Authors and Affiliations
Corresponding author
Additional files
Additional file 1:
SAS IML program for computing the attained power for WelchSatterthwaite’s test. (DOC 12 kb)
Additional file 2:
SAS IML program for computing the optimal sample sizes for WelchSatterthwaite’s test to meet a designated power level with the least cost. (DOC 27 kb)
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Jan, SL., Shieh, G. A systematic approach to designing statistically powerful heteroscedastic 2 × 2 factorial studies while minimizing financial costs. BMC Med Res Methodol 16, 114 (2016). https://doi.org/10.1186/s1287401602143
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s1287401602143