Adaptive designs in clinical trials: a systematic review-part I

Abstract

Background

Adaptive designs (ADs) are intended to make clinical trials more flexible, offering efficiency and potentially cost-saving benefits. Despite a large number of statistical methods in the literature on different adaptations to trials, the characteristics, advantages and limitations of such designs remain unfamiliar to large parts of the clinical and research community. This systematic review provides an overview of the use of ADs in published clinical trials (Part I). A follow-up (Part II) will compare the application of AD in trials in adult and pediatric studies, to provide real-world examples and recommendations for the child health community.

Methods

Published studies from 2010 to April 2020 were searched in the following databases: MEDLINE (Ovid), Embase (Ovid), and International Pharmaceutical Abstracts (Ovid). Clinical trial protocols, reports, and a secondary analyses using AD were included. We excluded trial registrations and interventions other than drugs or vaccines to align with regulatory guidance. Data from the published literature on study characteristics, types of adaptations, statistical analysis, stopping boundaries, logistical challenges, operational considerations and ethical considerations were extracted and summarized herein.

Results

Out of 23,886 retrieved studies, 317 publications of adaptive trials, 267 (84.2%) trial reports, and 50 (15.8%) study protocols), were included. The most frequent disease was oncology (168/317, 53%). Most trials included only adult participants (265, 83.9%),16 trials (5.4%) were limited to only children and 28 (8.9%) were for both children and adults, 8 trials did not report the ages of the included populations. Some studies reported using more than one adaptation (there were 390 reported adaptations in 317 clinical trial reports). Most trials were early in drug development (phase I, II (276/317, 87%). Dose-finding designs were used in the highest proportion of the included trials (121/317, 38.2 %). Adaptive randomization (53/317, 16.7%), with drop-the-losers (or pick-the-winner) designs specifically reported in 29 trials (9.1%) and seamless phase 2-3 design was reported in 27 trials (8.5%). Continual reassessment methods (60/317, 18.9%) and group sequential design (47/317, 14.8%) were also reported. Approximately two-thirds of trials used frequentist statistical methods (203/309, 64%), while Bayesian methods were reported in 24% (75/309) of included trials.

Conclusion

This review provides a comprehensive report of methodological features in adaptive clinical trials reported between 2010 and 2020. Adaptation details were not uniformly reported, creating limitations in interpretation and generalizability. Nevertheless, implementation of existing reporting guidelines on ADs and the development of novel educational strategies that address the scientific, operational challenges and ethical considerations can help in the clinical trial community to decide on when and how to implement ADs in clinical trials.

Study protocol registration

https://doi.org/10.1186/s13063-018-2934-7.

Introduction

Adaptive Designs (ADs) present an alternative to convention, fixed trial designs. ADs are a type of clinical trial design in which carefully planned changes may occur during the study. Specifically, ADs allow for prospectively planned modification(s) to one or more aspects of the study based on accumulating data from the trial [1]. Previous work has suggested that AD makes trials more flexible, and increases the ability to answer research questions by providing the option to shorten trial duration if answers are learned early, or lengthen the trial to ensure the question is meaningfully answered [2, 3]. These potential benefits can be achieved while preserving the integrity and validity of the trial [2] through careful planning before the trial begins (pre-specification) and with proper adjustment for the possible alterations during the trial [2, 4, 5]. All clinical trials should have stopping rules for safety, but ADs often include stopping rules to terminate arms that are not working (futility) which can prevent exposing more people to ineffective or suboptimal treatments and also provide an ethical justification for increasing the application of ADs [1]. ADs have been strongly recommended by the Food and Drug Administration (FDA) [2].

A defining feature of AD trial is that changes are pre-planned (written into the study protocol) and have pre-defined rules, which allow these modifications to roll out during the trial without additional approvals, such as changes to sample size or the number of treatment arms or the allocation ratio of patients to different treatment arms [2], as well as early termination of the trials if an intervention is not safe, or not effective. Additionally, ADs can often provide information about the effectiveness, futility, and safety of interventions earlier than fixed designs, due to an increased number of interim analyses. Earlier identification of ineffective therapies, can cut down on the overall participant burden and cost of a trial [6] and limit exposure to unsafe interventions [7].

Despite several potential advantages over conventional trial designs, the suitability of ADs depends largely on the clinical question being addressed [2]. There are challenges to designing and operating AD clinical trials, including an increased amount of time to design, logistic challenges to preserve trial integrity, ethical considerations and the need to prospectively consider the statistical complexity that each adaptation will bring about.

Although there are many different kinds of adaptations and ADs that have been previously described. Although AD trials share some of the common features (for example that changes are preplanned), the specific trial objectives and research question will determine the acceptability and complexity of adaptation(s) implemented. For instance, adaptive dose-finding trials seek to identify the effective target doses for each patient type by minimizing the dispersion of date around the right dose across different patients [4]. Adaptive randomization designs are a type of AD where pre-specified modifications in treatment allocations vary with the accrued response data (response-adaptive randomization) [5]. These modifications shift the randomization ratio in favour of more participants being allocated to the trial arm with the most promising risk/benefit ratio [6].

Another feature that can be applied in AD trials is group sequential design. These trials incorporate specific stopping rules that specify when to stop trials early for safety, futility (drop-the-loser), or efficacy (pick-the-winner). Seamless designs combine initial safety/efficacy data gathering (phase II) and confirmatory phases (Phase III) into one trial protocol for further investigations in the subsequent trial stages [9]. Seamless designs offer an efficient way to reduce sample sizes for dose optimization and accelerate the development of targeted agents using shared trial infrastructure[10]. A practical advantage of the seamless adaptive trial is that it does not need two separate clinical trial applications, approvals or set-up procedures at study centres, thereby reducing the time taken to evaluate a new intervention[11]. In adaptive enrichment designs, the trial inclusion criteria are modified such that trial participants who have a higher likelihood to benefit from the intervention are increased or “enriched”. However, this may lead to a high uncertainty about the treatment effects in populations who are not “enriched” and may induce statistical bias; adaptive sample size re-estimation designs have been developed address potential biases from adaptive enrichment designs [12]. Sample size re-estimation may be done either in a blinded or unblinded manner based on the criteria of treatment effect-size, conditional power and/or reproducibility probability [2].

Given the limited available resources to conduct clinical trials, and the large number of clinical decisions that are made without population specific evidence, there is an urgent need for trial designs that present efficient and feasible alternatives while meeting ethical, regulatory, and methodological standards [2, 3, 5]. This systematic review seeks to capture the methodological features, study characteristics and reported barriers in real-world examples of trials using ADs. This will help familiarize clinical trialists, ethics boards, regulators and other interested parties with examples of trials that incorporate AD.

Methods

Protocol registration

The protocol for this review was published [13] in October 2018 https://doi.org/10.1186/s13063-018-2934-7

Search strategy

We performed a systematic review, reporting according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA) [8]. Initially, a health science librarian (CJN) conducted a comprehensive systematic search of the literature on April 10, 2020. The initial search was designed in Ovid Medline and peer-reviewed following PRESS guidelines as outlined in the study protocol [13]. The original strategy was limited to full-text manuscripts on drugs and vaccines in English, French- and Dutch-languages and published between 2010 and 2017. We found a small number (n=16) of randomized controlled trials (RCTs) that specifically reported an “adaptive design” as a keyword. After testing our pre-defined search strategy, we realized that the AD methods are often not specifically reported as such in the keywords, title and abstract of papers. To overcome this, the results of this search were peer-reviewed by a second librarian and the search was further refined. To our knowledge we present a novel systematic review protocol for adaptive trials. This search strategy included was modified from our initial published protocol [13].

The final search included manuscripts published up to April 2020, conducted in the following databases: Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations and Daily <1946 to April 20, 2020>; Embase <1974 to 2020 Week 16>; and International Pharmaceutical Abstracts <1970 to April 2020>. The initial protocol indicated that two additional databases – CENTRAL (Wiley) and MathSciNet – would be used for this review, however, given the large volume of results, these databases were excluded. In addition, the original intent was to limit results to English-, French- and Dutch-language published between 2010 and 2017. The final search was limited to human studies, published in English, French, or Dutch from 2010 to April 2020. Readers are invited to visit the University of Manitoba institutional repository [14] for the complete search histories.

Selection procedures

Four reviewers (MA, AR, AP, DP) independently reviewed the search results for studies to be included. We screened the titles and abstracts of the 23,886 citations independently, in duplicate. Published protocols, reports, and a secondary analyses that reported using an adaptive method were selected. We included designs with one or more modifications of the trial such as the sample size, the number of treatments, or the allocation ratio to different arms. All diseases and populations (children and adults) were included. Interventions were limited to drugs and vaccines because there are separate regulatory guidance and ethical considerations for device and behavioural interventions in Canada and around the world. Comparator (control groups) were not restricted and we did not include any trial registrations (e.g., ClinicalTrials.gov) as there was not enough data provided on registrations to characterize methodological features.

Data extraction and syntheses

Four reviewers (MBE, AR, AP, DP) extracted data using standardized data extraction forms in REDCap [15] following the appropriate training and approvals by the University of Manitoba. Discrepancies or disagreements were resolved by rechecking the data, discussion, and consensus between all authors, with adjudication by an arbiter (MBE or LEK) in the event of non-consensus. Data abstracted from included trial details including the key and basic study characteristics of the design, population (age group, disease, location of recruitment), statistical analysis plan, nature of the interventions and control groups, primary and secondary outcomes, planned sample size and sample size methodology, intended analysis, type of adaptation(s), the rationale for adaptation where available, reported challenges, and study limitations that were categorized as regulatory, ethical, logistical, statistical, or other. The strength of the included body of evidence was not evaluated, as the purpose of this study is primarily descriptive.

Results

Search results

The literature search resulted in 23,886 citations, 2639 removed during deduplication, and 19,984 citations were excluded as they did not meet our inclusion criteria. If there was uncertaintly regarding the application of adaptive designs, full text articles were retrieved. There were 1263 full text articles that underwent a detailed evaluation for inclusion, and a total of 317 trial report were included. See Fig. 1 for the PRISMA study flow diagram [16].

Fig. 1 

Study Flow diagram, results of literature search

Description of the included studies

The included 317 publications, were mostly 267/317 (84.2%) clinical trial reports, and there were 50/317 (15.8%) study protocols. There were 390 adaptations reported in 317 studies. Several included trials reported publishing additional information including “appendices and online supplements” 186/317 (58.7%) or “statistical analysis plan” 18/317 (5.7%), and another publication including study protocols 62/317 (19.6%). A large number of these reports were open-access publications (187/317, 59.9%). Most were registered with a clinical trial registry (247/317, 77.9%). Characteristics of included studies are found in Table 1.

Table 1 Characteristics of the included studies (N=317)

Most studies were conducted in North America 140/317 (44.2%), followed by Europe 56 (17.7%), Asia 25 (7.9%), Australia 20 (6.3%), with the remaining trials conducted in Africa 19 (6.0%) and South America 15 (4.8%). Location was not reported in 6 trials (1.8%). The majority (n=193/317 studies, 61.1%) of included trials were multicentre. Most of the included studies had a relatively short treatment duration of fewer than 30 days (n=132/317, 41.64%), 30 to 90 days (n=53/317, 16.7%) and 90 to 365 days in 109 studies (34.4%). Intervention duration was not reported in 36 studies (11%). The mean duration of follow-up ranges from a single dose one day study to up to 798 days, up to a maximum of 1512 days, with follow-up time ranging from a single day up to 2400 days.

Tables 2 and 3 include a summary of data reported in the adaptive design clinical trials for further description regarding the adaptation designs used. The majority of trials 265 of 317 (83.9%) were limited to adults (over 18 years of age), 16 trials were limited to only children (under 18 years of age), and 28 enrolled both children and adults, 8 trials did not report the age of the participants. The age range across all included trial included from neonates (0-28 days) to geriatric patients (80 years). 169 studies were performed in cancer patients, 17 studies in the therapeutic area of virology, 8 in vascular hematology, 7 studies in diabetes, and 6 in cardiology. Almost all (315/317 = 99.4%) investigated drugs or biologics, three investigated vaccines, and 22 (6.9%) investigated both drugs/biologics and vaccines in combination with radiotherapy or surgery. Of the 317 included studies, 171/317 (53.9%) used parallel groups, and 124 /317 were single-arm (39.1%) studies.

Table 2 Summary of the adaptation’s designs* described in included trials (N=317)

Table 3 Summary of data of adaptive design considerations (N=317)

Most trials were in early phase of drug development and were classified as phase I or phase II. Approximately half of the trials (152/317, 47.1%) were phase II trials, mostly in the therapeutic area of oncology, and were aimed at establishing efficacy and choosing doses for the phase III of the trials. 125 studies (38.7%) were phase I trials of new drugs, aimed to assess the safety of treatment across a range of available doses in order to identify the maximum tolerable dose. Only 59/317 (18.6%) of the trials were in phase III, and 10 trials in phase IV (3.1%).

Summary of adaptative designs used

The most common type of adaptation utilized in these studies was the adaptive dose-finding methodology (121/317, 38.2 %). Some studies utilized more than one adaptation type of adaptive design. Table 2 outlines the adaptations outlined in included studies. Most adaptive dose-finding studies (121/317, 38.2 %), were reported in oncology during the phase I and II stages of the included trials and were evaluating the maximum tolerated dose of the study treatment. Studies using this methodology had relatively small sample sizes studies, ranging from 19 to 889 participants. Continual reassessment methods were commonly used (60/317, 18.9%), aimed to find the maximum tolerated dose from different doses followed by the Bayesian logistic regression approach (6/317,1.9%), and the 3+3 design (4/317, 1.3%).

The second most described method was adaptive randomization design (53/317, 16.7%) which included modification of randomization schedules or treatment, covariate-adaptive randomization, and response-adaptive randomization, in which the participants are increasingly randomized to receive a trial arm which indicates a more promising risk/benefit ratio.

Other designs which were less commonly reported were group sequential multi-stage design (47/317, 14.8%), which aimed to stop the trials early for safety, futility, or efficacy. Similarly, drop-the-losers (pick-the-winner) multi-arm/ multistage designs were used in 29 trials (9.1%) allowing adding additional arms and were mostly used in phase II. These AD designs combine safety and interim treatment selection and confirmatory phases into one trial for further investigations in the subsequent stages.

Adaptive seamless phase II-III designs, which combine a learning stage and a confirmatory stage, was reported in 27 trials (8.5%). Sample size re-estimation design was described in 14 trials (4.4%) where the target sample size was modified based on the observed data in the interim to achieve the desired power. Table 2 outlines the types of adaptions that were reported in included trials.

Unplanned changes to the trial design

Unplanned changes to the trial design were reported in 22 trials (22/317, 7%) where protocol modifications were made after the recruitment of the first participant. The most common unplanned modification included adding additional recruitment sites. Poor accrual rate and extended funding were the main reasons for the adding new participating sites. Other commonly reported unplanned modification included increase in the sample size, and a change the eligibility criteria or endpoints. Only 6% of studies included other unplanned modifications such as the changes in the statistical design, length of stay of participants, time point of interim analysis, the inclusion of additional doses and modifications to interventions. This modification included either an alternative substitute due to unavailability of the intervention, or a change in the route of administration owing to adverse events. Refer to Additional file 1 for further details on unplanned changes.

Statistical methods

Approximately 203/317 (64%) studies used frequentist statistical methods and 75/317 (23.7%) used Bayesian statistical methods. Descriptive statistics were used in 17/317 studies (5.4%), which include measures of central tendency such as mean, median, standard deviation, percentage, and correlation. Table 4 includes a detailed summary of the statistical analysis used in the adaptive design clinical trials.

Table 4 Summary of the statistical analysis used in the adaptive design clinical trials

Stopping boundaries

Various stopping boundaries were reported that allowed for stopping a trial prematurely due to safety, futility/efficacy or both based on the results of the interim analysis. Additional files 2 illustrates examples of clinical trials with adaptive designs with further details and complete descriptions of what is reported in these trials. Of the 317 AD studies included in this systematic review, 131 (131/317, 41%) included stopping boundaries to prematurely terminate the clinical trials for futility, efficacy, superiority, non-inferiority, and/or safety. Most of these rules were based on frequentist criteria such as alpha spending functions, O'Brien-Fleming boundary predictive probability, conditional power, critical value criteria (P values and Z boundaries, confidence interval (CI) or standard error (SE), suboptimal response to therapy as compared to another drug placebo and safety). Most of the studies were part of drug development programs for neoplasms utilizing outcomes including progression-free survival, overall survival, response to therapy (both complete and partial response) and drug safety. Futility stopping boundaries, when an intervention was determined to not likely be effective, were more common than superiority/non-inferiority, futility, efficacy and safety-stopping boundaries.

Logistical challenges

While there are certainly additional trial planning challenges specific to AD, the most common logistical challenges reported in the extant literature within the period of our review were similar to those reported using non-adaptive designs. These include slow participant recruitment, financial issues, difficulty in the identification of outcomes, and severe side effects due to the medication under investigation [3, 17,18,19]. Compared to non-adaptive designs, the use of ADs adds logistical challenges to ensuring appropriate trial conduct and trial integrity. These challenges include, but are not limited to termination of the medication production, drug supply for multi-arm studies, low infection rates for biological infectious agents, inadequate clinical research and regional infrastructure, and lack of prior clinical trials experience (based on the International Conference on Harmonisation-Good Clinical Practice (ICH-GCP) guidelines), the stringent regulatory standards of western agencies, community resistance, difficulty reaching remote field sites, or inability of the participants to travel long distances, decrease in the quality of life of the participants and vaccine storage issues. Refer to Additional File 3, Summary of the logistical challenges reported in the included trials, for further details regarding these challenges.

Discussion

Adaptive designs are complex, and this systematic review highlights the methodological features that have been reported in pediatric and adult trials. We have reviewed the literature on trial designs and described how adaptive design methods are used from 2010 to 2020. The most common forms of adaptive design were dose-finding and adaptive randomization designs. Oncology was the most common clinical area observed to use adaptive designs. Frequentist statistical methods were more commonly used than Bayesian methods, and the most common barrier reported to using Bayesian analysis was insufficient knowledge. The logistical and operational concerns reported in this review are mostly not unique to adaptive designs [20,21,22]. Recruitment challenges reported in conventional and AD trials highlight an urgent need to better understand the dynamics around investigator/trial team and trial-specific factors that can influence participant engagement in the design and operations of clinical trials. Better communication strategies to help convey the different adaptive methods to members of the clinical research community are needed. This should entail engaging in an iterative design built in close communication and collaboration among clinical experts, patient advocates, regulators, pharmaceutical companies, funders and biostatisticians [23].

Many included studies lacked important information on the type of adaptations, including the rationale with respect to the research question. In addition, there were limitations with reporting specifically on how and when data was analyzed (at what stage of the trial). Often in our included studies, it was not clear when the interim analysis was performed and how the sample size re-estimation and adjustment were done. Thus, it was not possible for us to determine if the adapted inclusion criteria were justified, and or if they might have introduced biases into the study. Moreover, it was often unclear who had access to interim results and how adaptive decisions were made. None of the trials reported clearly how they adjusted and accounted for biases introduced by the adaptive study design. This lack of methodological transparency could potentially jeopardize the integrity and uptake of adaptive trials. Thankfully, in 2020 reporting guidelines for reporting adaptive trials were published which should hopefully improve interpretability in future reviews. [24] This review serves as a snapshot of the reporting gaps, and should be re-evaluated in a future study.

Globally, regulators have called for an increase in the use of modern and efficient trial designs. Clinical trials that use ADs are an attractive option because they can potentially increase trial efficiency. However, there is still a lack of knowledge and acceptance among researchers about ADs [19]. Concerns about the appropriate use of ADs in trials revolve around a lack of knowledge and expertise among researchers, a lack of infrastructure support for the planning of AD control trials, complicated statistical analysis methods that are not widely understood, the lack of suitable software to aid both the design and conduct of trials [2], and concerns on how the funders and regulator’s view ADs. All these factors limit the use of ADs in clinical research, thereby denying trialists and patients the appropriation of the range of benefits that AD clinical trials offer. The acceptability of AD, incurring both the potential benefits and additional challenges, depends on the research question[25, 26]. For instance, group sequential designs that repeatedly assess clinical outcomes over multiple interim analysis is effective in trial cost minimization and patient risk reduction. However, the design is poor at gathering information on long-term treatment effects, effects on secondary end points, and may even produce less precise estimates [27]. Similarly, response adaptive randomization designs that produces unequal sample size across treatment and control arms can impact the statistical efficiency of the trial, likely increasing bias in favor of the arm that over-pools patients after a favorable treatment response [25]. Seamless Phase II/III designs can help minimize delays associated with protocol development, and logistical issues, however, it often restricts the flexibility of modifying the confirmatory phase after trial commencement [28].

An understandable critique of ADs is that the adaptive methodological and/or statistical process can introduce operational biases that may be difficult to predict and control, which may inadvertently make the target population to be shifted with regards to location and scale [17]. Although the use of pre-planned statistical analysis plans including extensive simulation and statistical tools such as the Bonferroni-Holm test procedure (based on conditional error rates of individual treatment–control comparisons) can help control and limit these operational biases and the attendant type 1 errors [29, 30], they may raise scientific concerns resulting from unfamiliarity during grant submissions, review and approval. An increase in logistical constraints in safeguarding the trial conduct and integrity as well as the need for specific analytical methods/simulations (that may not be readily accessible as many trial team lack dedicated support from statisticians with expertise in AD) are additional challenges associated with ADs in practice [1].

There are few published reviews of specific ADs in the literature [2,3,4, 14, 18, 31]. These papers present focused discussions on an overview of ADs, provide a select description of case studies to illustrate their use, or focus on the statistical considerations, statistical solutions, and simulations of ADs. Our review provides the first comprehensive summary of the use of ADs in adult and pediatric trials and provides a snapshot of AD trial reporting. We hope that it will enhance understanding of and familiarity with adaptive methods of clinical trials for the research community as well as increase the confidence of trialists who choose to incorporate them. In a follow-up to this paper (Part II), we will contrast the application of adaptive designs in trials that enrolled adults with those of pediatric populations, with the goal of informing viable educational strategies that can foster trial efficiency and familiarity with AD amongst the clinical research community.

Limitations of this review

Our search was limited to 2010 to April 2020, based on the FDA guidance [1] on AD in clinical trials that was first issued in 2010, and evaluates its uptake before the adaptive design reporting standards (CONSORT extension) were published in 2020. It is worth mentioning that ADs were used frequently in clinical trials during the COVID-19 pandemic, due to their operational efficiency. However, given the timing of our literature search, COVID-19-related trials are not captured in this review. We also did not attempt to identify trials that are not published or evaluate non-drug/vaccine interventions, which may limit the application of our findings on AD clinical trials to those contexts. Given our focus on trial reports, planned studies that may have encountered barriers to implementing an AD and could not begin at all (e.g., planned trials unable to overcome ethical or regulatory hurdles for the proposed adaptations for approval) would not have been reflected in the published literature. An additional limitation is that publication word counts could have limited how the manuscripts described the rationale for the adaptation-specific challenges.

Implications for practice

The results of this review suggest that there is a need for rigorous, immersive training in adaptive trial designs for health care providers, researchers, the public and other interested parties who design and implement clinical trials. Proper guidance on planning ADs trials with simulations of possible adaptation scenarios for risk-benefit assessments, supported with clinical input and statistical analysis is necessary to optimize outcomes. In the face of the implementation challenges associated with ADs, developing a framework on how to operationalize them is an important means of overcoming these challenges [17].

Implications for research

To increase the capacity for adaptive trials, a qualitative study to obtain practical feedback from regulators, research ethics board members, biostatisticians, clinicians, and scientists, as well as representatives from patient groups and the public, on challenges in applying AD in trials to inform recommendations on best practices, is warranted. Given the ethical requirement for monitoring participant safety, an external adaptation committee should be established to continually monitor modifications based on interim data. Guidelines for creating and communicating with adaptation committees are warranted. To increase uptake of AD trials, enforced application of reporting tools and methodological transparency will ensure regulators, trialists and the public can clearly understand the published literature as well as how adaptations are applied and how biases are eliminated (or minimized).

Conclusions

Clinical trials are needed that can efficiently inform optimal treatment practices. This review provides an overview of the methodological features of adaptive designs that have been reported in clinical trials. Clarity about methodological, operational and ethical features may help increase familiarity and ultimately uptake of trials that incorporate ADs.