Reporting and communication of sample size calculations in adaptive clinical trials: a review of trial protocols and grant applications

Abstract

Background

An adaptive design allows modifying the design based on accumulated data while maintaining trial validity and integrity. The final sample size may be unknown when designing an adaptive trial. It is therefore important to consider what sample size is used in the planning of the study and how that is communicated to add transparency to the understanding of the trial design and facilitate robust planning. In this paper, we reviewed trial protocols and grant applications on the sample size reporting for randomised adaptive trials.

Method

We searched protocols of randomised trials with comparative objectives on ClinicalTrials.gov (01/01/2010 to 31/12/2022). Contemporary eligible grant applications accessed from UK publicly funded researchers were also included. Suitable records of adaptive designs were reviewed, and key information was extracted and descriptively analysed.

Results

We identified 439 records, and 265 trials were eligible. Of these, 164 (61.9%) and 101 (38.1%) were sponsored by industry and public sectors, respectively, with 169 (63.8%) of all trials using a group sequential design although trial adaptations used were diverse.

The maximum and minimum sample sizes were the most reported or directly inferred (n = 199, 75.1%). The sample size assuming no adaptation would be triggered was usually set as the estimated target sample size in the protocol. However, of the 152 completed trials, 15 (9.9%) and 33 (21.7%) had their sample size increased or reduced triggered by trial adaptations, respectively.

The sample size calculation process was generally well reported in most cases (n = 216, 81.5%); however, the justification for the sample size calculation parameters was missing in 116 (43.8%) trials. Less than half gave sufficient information on the study design operating characteristics (n = 119, 44.9%).

Conclusion

Although the reporting of sample sizes varied, the maximum and minimum sample sizes were usually reported. Most of the trials were planned for estimated enrolment assuming no adaptation would be triggered. This is despite the fact a third of reported trials changed their sample size. The sample size calculation was generally well reported, but the justification of sample size calculation parameters and the reporting of the statistical behaviour of the adaptive design could still be improved.

Background

The increasing demands for accelerating the development and testing of healthcare treatments have been noted across sectors [1]. Consequently, the use of adaptive designs (ADs) has had more attention due to their potential to improve efficiency when compared with conventional fixed designs [2, 3], for example, in oncology studies [4, 5]. ADs allow for controlled flexibility to modify design aspects of an ongoing trial based on interim data/results in a manner that preserves the validity and credibility of the trial results [6].

A common theme across ADs is that the trial pathway could change during execution depending on the triggered trial adaptations [7]. There are exceptional circumstances in adaptive trials when there is only one trial path, e.g., in treatment selection adaptation where a rule is to select a fixed number of leading treatments after an interim analysis such as a top-ranked promising treatment. In most cases, the sample size is a variable depending on accumulating trial data and trial path rather than a fixed number of participants known before the start of the trial [8]. For example, in group sequential design, the continuation of the trial depends on the observed data from one or more interim analyses, making the final sample size a discrete variable that can range from the earliest possible stopping point to the latest stage. Therefore, for an AD, it is worthwhile to consider how to report the sample size to reflect the uncertainty in the trial trajectory [9].

Discussion on the reporting of the sample size(s) for ADs matters [10, 11], particularly for trial budgeting and research funding applications. Simply reporting a single sample size in a trial grant application or protocol could be misleading considering the flexibility of ADs as the final sample size has a chance to deviate from the initial reported sample size.

However, despite this, trial registry records (such as on ClinicalTrials.gov) currently require a single specific number for estimated enrolment for all registered trials without considering the implications of trial adaptations.

As the uncertainty in the final trial sample size is contingent upon the types of ADs and the specific decision rules chosen, it is important to establish context-specific minimum requirements when planning sample size(s) in adaptive trials. This is essential for upholding transparency and validity in reporting sample size(s). This transparency, it could be argued, is particularly significant for research grant applications – although this needs to be weighed against restrictions on page or word length.

Existing guidance such as DELTA [12, 13] and the Adaptive designs CONSORT (CONsolidated Standards Of Reporting Trials) Extension (ACE) [6] provide recommendations on what information related to the sample size calculation could be useful when reporting the trial. However, to the best of our knowledge, the question of how to report and communicate the sample size(s) in the trial protocol or grant application for an AD remains unresolved. Given the challenges in reporting trial sample size(s) for an AD, it is beneficial to understand how previous adaptive clinical trials dealt with this problem. This study, therefore, examined previous clinical trials that used ADs to identify good practices and potential gaps for further research.

Methods

This study was conducted following the methods detailed in the established protocol [14].

Data sources

We considered trials registered on ClinicalTrials.gov between 01/01/2010 and 12/31/2022. The search also included grant applications submitted within the same period, obtained via email requests from members of the Medical Research Council (MRC)—National Institute of Health and Care Research (NIHR) Trials Methodology Research Partnership (TMRP) Adaptive Designs Working Group (ADWG) (https://www.methodologyhubs.mrc.ac.uk/research/working-groups/adaptivedesigns/) in the UK. This approach enables us to obtain both approved and rejected applications.

To be included in the methodological review, a trial was considered to have used an AD if it was designed with (pre-planned) trial adaptations using interim outcome data/results from participants in the trial. This is consistent with a consensus-driven definition of an AD [15, 16]. A trial was eligible if it:

• 1) was a phase II, III, or IV clinical trial including those with such seamless components, trials with the phase marked as “non-applicable” or missing were also eligible. This decision was made in recognition that some trials evaluate non-pharmacological interventions (e.g., surgical and behavioural therapies) that may not traditionally fit into a well-defined trial phase typical for pharmacological treatments;

• 2) was randomised with comparative efficacy or effectiveness objective(s) between treatment arms;

• 3) had an accessible trial protocol or grant application;

• 4) used an AD with at least one or more specific types of adaptations (for the definitions of types of ADs considered, refer to Appendix 1).

Identification of eligible trials

A filter function within the ClinicalTrials.gov registry was used to apply eligibility criteria 1 to 3 above to trial protocols. Following this, a keyword search was conducted on the registry to assess eligibility for criterion 4, using a comprehensive list of search terms (refer to Appendix 2). This search list was initially created based on relevant keywords identified through related literature [17, 18] and discussion with the research team and it was refined through multiple rounds of testing to improve sensitivity and specificity of the search.

Due to the limitations of online searching on ClinicalTrials.gov, which currently does not provide the functionality to search the contents of uploaded documents (such as protocols), an additional searching tool accessible via GitHub was developed by QZ using Python 3.10 (https://github.com/BiosQiang/Comprehensive-searching-on-clinicaltrial.gov).

This complementary search took place after the filtering based on criteria 1 to 3. The eligible trial information was downloaded in batches in XML format. The associated protocols were identified, downloaded and searched using the toolkit. The results of the online search were then combined with the toolkit's findings and duplicate records were removed. Finally, accessed grant applications were integrated with this combined dataset, resulting in the final database for review.

The eligibility criteria were applied continuously throughout the review process to exclude ineligible records. Figure 1 summarises the eligibility screening process.

Fig. 1

Working flow for the identification of eligible trials

Data extraction and analysis

A data extracting sheet was developed to capture key trial information including basic information on trial design; how the sample size was estimated and reported and information on trial adaptations, interim analyses and decision rules. Appendix 3 is a template of the data extraction form.

Two independent reviewers (QZ and ZY) did the data extraction. One reviewer (QZ) searched trials, screening for eligibility, reviewed and extracted required information from all eligible records. The second reviewer (ZY) reviewed 10% of randomly selected records for quality control. If disagreement between two reviewers appeared in less than 20% of all extracted variables, no further independent quality control would be needed as pre-defined in the review protocol [14]. In case of disagreement, MD reviewed the record and then discussed it with QZ and ZY to reach an agreed resolution. Further, quality control was also done through internal data queries by checking for inconsistencies of data extracted between certain variables, e.g., the classed type of AD or adaptive aspects should be consistent with specific trial adaptations. This validation process is recorded in Appendix 4.

The analysis was descriptive using summary statistics depending on the variable type, the distribution of data and the data visualisation approach. The results are reported as guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement [19], where appropriate.

Example of adaptive design and definitions for technical terms

To facilitate the subsequent discussion, a real trial example with adaptive design is provided in Table 1:

Table 1 Example of the ‘ROSE’ study

Reevaluation Of Systemic Early Neuromuscular Blockade study (ROSE, NCT02509078) was a phase III, two-arm, parallel trial testing the efficacy of cisatracurium besylate against a control group in patients with acute respiratory distress syndrome. The primary endpoint was hospital mortality within 90 days of randomization, with a planned sample size of 1,408 participants. Unlike a traditional fixed design, the trial employed a group-sequential design with two evenly spaced interim analyses and one final analysis. The two interim analyses, allowing for the possibility of early trial termination, were planned when 470 and 938 participants had been enrolled. The information time was calculated as the ratio of the 'effective sample size' at each analysis point to the final sample size.The symmetric stopping boundary was designed using the method described by Lan and DeMets [20].

For transparency and to aid interpretation, Table 2 gives some key definitions of technical terms which were used throughout this work:

Table 2 Definitions of key technical terms

Results

For trial protocols, a total of 432 trials were initially identified through the ClinicalTrials.gov registry (Fig. 2). After removing the duplicates and applying exclusion criteria, 259 eligible records were retrieved. For grant applications, an additional seven trials were obtained and six of these were eligible. Ultimately, 265 eligible trials were reviewed, and of these, 154 (58.1%) had been completed as of the analysis cut-off date (31 December 2022), meaning the last participant's final visit had occurred (including early stopping based on an adaptation), as recorded on ClinicalTrials.gov or reported by the grant application provider.

Fig. 2

Flowchart for the screening process results

In addition, since the author of the trial grant application or protocol withheld certain key information upon submission due to confidentiality considerations, the withheld information was marked as 'concealed' in the review results.

Trial characteristics

As shown in Table 3, among all the 265 eligible trials, 164 (61.9%) and 101 (38.1%) were sponsored by industry (private sector) and the public sector, respectively. Over half, 150 (56.6%), were phase III trials while 38 (14.3%) had their trial phase marked as ‘not applicable’. Over 90% of the industry-sponsored trials evaluated pharmacological treatments (drugs) or drug-involved combinations compared to 69.3% in public sector trials. The proportion of trials with time-to-event and binary primary endpoints was higher in industry and public-sponsored trials, respectively. Moreover, the most common trial design features regardless of the sponsor included superiority primary hypotheses, comparing one treatment against one comparator and use of frequentist methods for the interim and final analyses.

Table 3 Design characteristics of eligible trials

Characteristics of adaptive features or trial adaptations

A group sequential design (with options for early stopping the entire trial only) comparing two treatments was the most frequently used type of AD, which accounted for 98 (59.8%) and 71 (70.3%) of industry and public-sponsored trials, respectively (Table 4.). However, trial adaptations considered were diverse (Fig. 3), with early stopping of the entire trial (either for efficacy or futility) as the most common and planned in 238 (89.8%) trials. Multiple trial adaptations were simultaneously considered in a single trial in 61 (23.0%) trials. Sample size re-estimation (including both blinded and unblinded manner) was planned in 63 (23.8%) trials.

Table 4 Features of adaptive designs in eligible trials

Fig. 3

Prevalence of considered trial adaptations and their combinations in reviewed trials. Figure 3is an UpSet plot showing the prevalence of adaptations and their combinations across cases. It doesn’t map directly to the adaptive design types in Table 4., as the same design can involve different adaptations, and the same adaptation can apply to various designs. Trials with the details of adaptation concealed or not clearly described were excluded.

Having one interim analysis (n = 162, 61.1%) and the first interim analysis planned at the halfway point of the trial were the most common interim decision-making features (Fig. 4). Public-sponsored trials tended to have more interim analyses compared to industry-sponsored trials (Table 4.). Some public-sponsored trials planned the first interim analysis earlier, at one-third of the trial, as evident by the bimodal distribution in the timing of analysis (Fig. 4). The proportion of trials with pre-planned sample size re-estimation was higher in industry-sponsored cases (n = 48, 29.3% vs n = 17, 16.8%) and the industry-sponsored trials were more likely to perform this sample size re-estimation in an unblinded-comparative way.

Fig. 4

Distribution of the information fraction of the first interim analysis

Although most of the trials (n = 229, 86.4%) were able to clearly describe the interim decision rule(s), not all of them are replicable, e.g., 84 (61.3%) trials did not state whether the futility stopping rule would be binding or non-binding.

Sample size reporting methods and the estimated enrolment

Table 5. summarises the reporting of sample size aspects. Most sample size calculations (n = 229, 86.4%) were performed analytically and 216 (81.5%) trials described the sample size calculation methods used in detail making them reproducible; the reproducibility rate is higher in group sequential designs (147, 87.0%) and lower in multiple adaptive designs (48, 78.7%) and adaptive treatment selection designs (3, 60.0%) (Appendix 6). For the general items related to sample size that apply for both fixed and ADs, more than two-thirds of the trials regardless of sponsor did not disclose the statistical software, package, or code used for the calculation. Moreover, there was no justification for the choice of parameters used for sample size calculation in 79 (48.2%) and 37 (36.5%) industry-sponsored and public-sponsored trials, respectively.

Table 5 Reporting of sample size aspects

For items specific to ADs, the maximum, minimum, or expected sample size were metrics reported in most of reviewed trials. The maximum sample size was the most frequently reported. The minimum sample size was directly stated only in 6.0% of trials (16/265), but it was able to be directly inferred from information provided on the interim analyses plan in 220 (92.3%) trials.

Reporting the maximum and minimum sample size simultaneously was most common (Fig. 5). In contrast, only 18 (6.8%) trials reported the expected sample size, of which, 9 (50.0%) were calculated under both the null and alternative hypotheses (Table 5.).

Fig. 5

Reporting of sample size quantities (excluding 7 trials with sample size concealed)

For the enrolment of an adaptive design, the plan could be based on the minimum, maximum, or modified maximum (mMax, as defined in the methods section). Table 6 presents the metrics used by different adaptive designs to estimate enrolment. Typically, the planned metric is determined based on the sample size under the assumption that no adaptations will be triggered: for trials with early stopping as the only adaptation (e.g. group sequential design), this would be the maximum sample size; for trials with sample size re-estimation as the only adaptation, it would be the minimum sample size. When early stopping and sample size re-estimation were both planned, the target sample size was usually set assuming neither early stopping nor sample size increase would take place.

Table 6 Enrolment Estimation Across Adaptive Design Types

Reporting of the operating characteristics for the adaptive design

Table 7 summarises the details of operating characteristics of the AD that were explicitly stated. Less than half of the trials provided information on the operating characteristics of the AD. Among all the reported terms, the stagewise type I error rate and cumulative power at the interim analysis were the most frequently reported terms while relatively fewer trials had reported the probability for early stopping or taking certain trial adaptations.

Table 7 Reporting of operating characteristics

Comparison between the actual enrolment and planned enrolment size

Figure 6 illustrates how the sample size changed from the original estimation and the corresponding reasons using a Sankey plot. Additionally, the scale of the change is depicted in the accompanying density plot. Of 154 completed trials, 152 (98.7%) had information on their actual enrolment available. Of these 152 trials, 68 (44.7%) enrolled the planned sample size without significant deviation (unless modification on the sample size was specifically stated in the study results, any deviation above 5% of the original sample size was deemed significant), while 45(29.6%) had fewer participants than planned and 39 (25.7%) had more participants than planned (Fig. 6).

Fig. 6

Sankey plot for the actual enrolment status among all completed trials. *The number before and after the ‘/’ corresponds to change due to adaptive reasons and all reasons respectively. Q1 ~ Q3: The first ~ third quartile

Of the completed trials with inflated sample size, 15/39 (38.5%) trials were a result of trial adaptations, with the majority of these triggered by sample size re-estimation (n = 14) and only one trial due to the addition of a new treatment arm. The remaining 24/39 (61.5%) trials had an inflated sample size for reasons not related to trial adaptations considered (e.g., increasing the sample size to compensate for a lower event rate than expected); of which, 16 (66.7%) trials had a time-to-event primary endpoint. As a result of taking adaptations, the average ratio for sample size increasing was around 1.6.

With respect to trials that had a reduced sample size at completion, 33 /45 (73.3%) were due to pre-planned early stopping (n = 13, 28.9% and n = 20, 44.4% for futility and efficacy, respectively), while 7 trials stated that the sample size was decreased due to feasibility considerations and 1 trial was terminated early due to safety considerations. The average reduction in sample size for adaptive reason was 25%.

Examples of reporting sample size calculation for an adaptive trial

Regarding the ‘ROSE’ study (NCT02509078) introduced in Table 1, it was able to provide some information regarding the sample size and trial operating characteristics in the protocol.

In terms of the sample size estimation, nuisance parameters, targeted effect size and the corresponding justification were all provided (Table 8).

“The presumed 35% mortality rate in the control group is based on several recently published clinical trials. […] The table below shows the effect of changes in the power of the study as a function of the mortality rate on the treatment. We calculated the power under two assumptions. The first is that the absolute difference in mortality rates was 8% and the second was that the relative difference was 23% which is approximately 8% of the anticipated mortality of 35%. The second row of the table shows the current assumptions. Whether you fix the absolute or the relative difference, the power is above 80% as long as the null mortality rate is over 25%. The power goes down to 73% if the mortality rate is 20% which is below the mortality rate observed in a ARDSNet (Acute Respiratory Distress Syndrome Network) study.”

Table 8 Change in power against different effect size assumptions (recreated from ROSE protocol)

Regarding the adaptation, it states:

“There will be two interim analyses and a final analysis conducted when approximately each successive 1/3 of the patients have been enrolled. […] This trial will stop for superiority of either active or control and is designed with symmetric group sequential flexible stopping boundaries as described in Lan and DeMets (with corresponding reference), […] The information time was calculated as the ratio of the 'effective sample size' at time of each look to the final sample size.”

Regarding the enrolment, it states:

“The maximum required total sample size is 1408 subjects. […] The trial will accrue a maximum of 1408 patients.”

As shown in Table 9., the trial also provided detailed information on the decision rules and operating characteristics.

“The table presents stopping boundaries as both a required observed mortality difference and a one-sided p-value. The columns under ‘Probability of Stopping’ present the probability of stopping at each stage under the null and alternative.”

Table 9 Information on planned interim analysis and the stopping probabilities (recreated from ROSE protocol)

Although the minimum and the expected sample sizes were not explicitly stated, they can be worked out from the information provided.

Discussion

Accurately reporting the sample size can help readers understand the study design and ensures the reproducibility of methods [28]. It is also important for adequate planning of the trial budget [29], especially when an AD is used [30], since triggered trial adaptations can lead to different sample sizes.

Regarding the metrics of the sample size, the minimum and maximum values establish the range of anticipated information as well as the budgetary limitations in trial execution, which may usually make the reporting of these two metrics in trial plans reasonable. This is consistent with extant literature [31]. However, merely giving these ‘extreme’ sample size(s) might be insufficient since the minimum or maximum sample size is not always the one to be eventually achieved.

Previous studies indicated that the trial budget for an AD was usually estimated assuming the ‘worst’ situation [31] among all possible trial trajectories. This is true when early stopping is the only adaptation. Our review confirms this finding and extends it to: most trials were found to use the sample size assuming no adaptation would be triggered for planning purposes: with no sample size re-estimation, this would be equivalent to the maximum sample size; when sample size re-estimation is the only adaptation, this would correspond to the original sample size which is also the minimum.

Using the sample size assuming there will be no trial adaption in the trial plan provides a simple solution regarding the sample size to use when designing a trial, but it is not always the best choice. In this review, 48/152 (31.6%) of completed trials had their sample size changed from the original enrolment estimation for trial adaptation reasons. Considering this, reporting additional information on the sample size and planned trial adaptation could potentially reduce the allocated trial budget, if the trial is very likely to stop early [10, 32,33,34].

For an AD, other metrics, such as the most likely sample size (in terms of probability), or the expected sample size, may convey useful information to review of the grant applications and protocols and may be more meaningful on the trial portfolio level. The reporting of these metrics relies on transparent presentation of the operating characteristics of the AD, especially the probability for triggering an adaptation and when interim analysis will be performed. Gsponer et al. state that for trials with planned early stopping, the probability of stopping for success, futility and the likelihood of continuing at each interim analysis should be the main interest when giving the operating characteristics [34]. This information is of particular importance when communicating the design to funders [30] and other trial stakeholders (e.g., regulators).

Thus, it would be valuable to discuss in further studies on how to report the operating characteristics of adaptive designs and how they should be applied across different scenarios, including different statistical frameworks (frequentist and Bayesian) and complex design types such as biomarker enrichment design [35].

Our review found that the type of sponsor of the trial could impact design features and thus could possibly affect the sample size determination as well as its reporting: for example, publicly sponsored trials had their first analysis earlier than industry-sponsored trials with the time of the first analysis being bimodally distributed. One possible explanation could be that public sponsored being more sensitive to budget limitations—which may make terminating futile or unfeasible studies—in terms of ability to recruit – more desirable. It has been reported that almost all UK-based public trials from 2012 to 2016 had an internal pilot to assess feasibility in recruitment (i.e., if the study was on track to reach recruitment targets) [36]. Of these internal pilots, two-thirds were done typically after 20% of recruitment. It could be that if the internal pilot was being undertaken anyway that a study team also undertakes a formal interim analysis for futility – which could probably explain the first peak at 30% of information fraction during interim analysis, assuming prompt information of endpoint on recruitment. The gap between 20 and 30% possibly being due to aiming for more information to form an adaptive decision [37]. In this context, always allocating the budget by assuming no adaptation would take place might be questionable for certain trials (e.g., when aggressive stopping rules are used) or may not be the most effective solution from a portfolio perspective.

Finally, the different roles of researchers in clinical trial research can also influence their focus on different metrics of sample size. For public sponsors, knowing the maximum sample size helps decide the feasibility of recruitment, which is recognized as the most difficult area for determining the success of a trial [38]. For industry sponsors, knowing the minimum sample size, especially in pivotal studies, determines the least statistical information that should be gathered to support regulatory submissions for a study treatment. For both public and industry sponsors knowing the expected sample size can help allocate funding.

Conclusion

The reporting of sample size(s) for ADs demands more careful considerations than that of conventional fixed designs. Reporting the maximum, minimum and expected sample size as well as the operating characteristics in trial grant applications and protocols can provide valuable information that can be used for different purposes by diverse stakeholders.

The current review has revealed that the maximum and minimum sample sizes were commonly reported and the sample size assuming no adaptations were triggered is usually set as the target enrolment. This is despite the review showing that the actual enrolment and the target sample size did differ due to triggered trial adaptations.

The sample size calculations, details of trial adaptations and the interim decisions were generally well reported. However, this may not be generalisable to adaptive trials without accessible protocols and there is still room for improvement in justifying the parameters selected to calculate sample size and reporting the statistical behaviour of the design.

Since there are no recommendations on best practice for reporting of sample sizes for ADs in grant applications and protocols, this work has shown that further work is required to provide structured practical guidance.