Table 1 Summary of relevant literature on analysis of pnRCTs

Schweig & Pane [16]

Describe and compare models for pnRCTs with non-compliance using a simulation study.

Simulation for two levels of clustering, exact cluster sizes (m) unclear in paper, c_school = 37, c_class = 177, λ_B = 2, 8, ρ_school = 0.005, 0.05, 0.15, ρ_class = 0.0004, 0.10, 0.25, and θ = 0.087.

Clustering and non-compliance may have a substantial impact on statistical inference about intention-to-treat effects. Provide methods that may accommodate pnRCT with non-compliance, recommend using complier average causal effect estimate (CACE) and scaling by the proportion of compliers. No mention of degrees of freedom, we have assumed they used default degrees of freedom method available in R lme packages.

Flight et al. [9]

Compare models applied to four examples of pnRCTs. Compare three different methods for classifying the non-clustered control arm in pnRCTs, including: singleton clusters, one large cluster and pseudo clusters.

Examples with {m, c} = {36, 8; 24, 7; 14, 8; 5, 6}, and estimated ρ =  < 0.0001, 0.02, 0.007.

Recommend use of the heteroscedastic model, recommendations based only on re-analysis of case studies. Methods for classifying the non-clustered control arm in pnRCTs had a large impact in fully clustered mixed effects models and no measurable impact in partially nested mixed-effects models. ICCs in four examples were small.

Sterba [27]

Review of modelling developments for pnRCTs, focused on those particularly relevant to psychotherapy trials.

Recommend the inclusion of cluster variability in analysis model as it provides insight into treatment process (rather than treating it as a nuisance). Annotated Mplus commands for models

Lohr, Schochet & Sanders [19]

Report presenting a guide to design and analysis issues for pnRCTs in education research, using example trials. Discussion of degrees of freedom issue in Appendix.

Guidance document, defines pnRCT in context of education research and show methods to analyse these using SAS. Provide SAS commands for model fitting in examples.

Korendijk [18]

Compare models for pnRCTs using simulation study, investigate mis-specification for the estimation of the parameters and their standard errors.

Simulation study with m = 5, c = 10, 30, 50, 100, ρ = 0.05, 0.1, 0.2, λ_A = 1, d = 0.3.

All models perform comparably with respect to fixed effect estimates. Recommend use of partially nested mixed-effects model. Simulations were under null and ICC always greater than zero. No mention of degrees of freedom, we have we assumed default degrees of freedom used from MLwiN software, and homoscedasticity was assumed for ndividual variances between the two arms.

Sanders [17]

Compare models for pnRCTs using simulation study in terms of Type I error and power

Simulation study with {m, c} = {2, 10; 4, 4; 5, 4; 10, 2}, ρ = 0, 0.1, 0.2, 0.3, 0.4, 0.5, λ_A = 1, and ω² = 0, 0.01, 0.059, 0.138.

Type I error rate increased as ICC increased, Satterthwaite degrees of freedom had better control than Kenward-Roger degrees of freedom. Found using mixed-effects model for pnRCT when ICC is zero likely leads to never detecting intervention effects, observed Type I error rates nearly non-existent under all scenarios with ICC equal to zero. Recommend to evaluate if ICC is significantly different from zero prior to selecting analysis method. Homoscedasticity was assumed for individual variances between the two arms.

Baldwin et al. [15]

Compare analysis models for pnRCT simulation study, comparing three degrees of freedom calculations, and a pnRCT example.

Simulation for m = 5, 15, 30, c = 2, 4, 8, 16, ρ = 0, 0.05, 0.1, 0.15, 0.3, λ_B = 0.25, 1, 4, and d = 0, 0.5.

Recommend pnRCTs take account of heteroscedasticity. Satterthwaite and Kenward-Roger degrees of freedom control Type I error rate. The heteroscedastic model provides an unbiased estimate and little reduction in power compared to the homoscedastic model. Argue that using a partially nested mixed-effects model only a problem for statistical inference when the number of clusters is small. The number of clusters has greater impact on power in pnRCTs. At least eight, preferably 16 clusters, to maintain Type I error rate.

Bauer et al. [6]

Review of RCTs to ascertain the prevalence of pnRCTS in four public health and clinical research journals. Analysis models for pnRCTs extended to include pre-test measures as covariates, individual and group level covariates, and example of pnRCT

Example with clustering in one arm c = 41, m = 9, and estimated ρ = 0.02.

Out of 94 RCTs, 32% were pnRCTs, 40% iRCTs and 27% cRCT. None used methods specific to pnRCTs. Example pnRCT data could be analysed using mixed-effects models. Argue pnRCTs “often increase external validity at the expense of internal validity” (p.20).

Roberts & Roberts [7]

Examine the case of pnRCTs, heterogeneity, comparison of analysis methods for simulation study and present an example.

Simulation for m = 6, c = 8, ρ = 0, 0.1, 0.2, 0.3, λ_A = 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2 and d = 0.

Recommend pnRCTs take account of heteroscedasticity. Satterthwaite unequal variances t-test gave robust to heteroscedasticity. The heteroscedastic model gives slightly inflated test size for large ρ: suggest Satterthwaite degrees of freedom as a solution.

Lee & Thompson [28]

Describe analysis models for iRCTs with clustering and apply to two examples (using Bayesian approach)

Show that ignoring clustering may underestimate uncertainty, leading to incorrect conclusions.

Hoover [34]

Statistical tests for RCTs with clustering that differ across trial arms.

Example with clustering in both arms with m = 7 − 12, c = 3.

Provide an adjustment for the independent samples t-test for pnRCTs. Statistical impact of heterogeneity effect increases as the cluster size increases, and as heterogeneity increases. The test does not allow for the inclusion of covariates, multiple treatments, baseline measures, or non-normally distributed outcomes.