A systematic review of randomisation method use in RCTs and association of trial design characteristics with method selection

Abstract

Background

When conducting a randomised controlled trial, there exist many different methods to allocate participants, and a vast array of evidence-based opinions on which methods are the most effective at doing this, leading to differing use of these methods. There is also evidence that study characteristics affect the performance of these methods, but it is unknown whether the study design affects researchers’ decision when choosing a method.

Methods

We conducted a review of papers published in five journals in 2019 to assess which randomisation methods are most commonly being used, as well as identifying which aspects of study design, if any, are associated with the choice of randomisation method. Randomisation methodology use was compared with a similar review conducted in 2014.

Results

The most used randomisation method in this review is block stratification used in 162/330 trials. A combination of simple, randomisation, block randomisation, stratification and minimisation make up 318/330 trials, with only a small number of more novel methods being used, although this number has increased marginally since 2014. More complex methods such as stratification and minimisation seem to be used in larger multicentre studies.

Conclusions

Within this review, most methods used can be classified using a combination of simple, block stratification and minimisation, suggesting that there is not much if any increase in the uptake of newer more novel methods. There seems to be a noticeable polarisation of method use, with an increase in the use of simple methods, but an increase in the complexity of more complex methods, with greater numbers of variables included in the analysis, and a greater number of strata.

Background

Randomisation is an important feature in clinical trials; it allows valid estimation of standard errors, is vital to conceal allocations and maintain this concealment in blinded trials and allowing researchers to eliminate important sources of bias. For example, selection bias is introduced when recruiters assign participants to certain treatments leading to group imbalances. Chronological bias, caused by imbalances in the timing of the assignment of treatments over the recruitment period, can also lead to group differences. Randomisation, implemented correctly, mitigates the risk that the study findings could be influenced by these and other unknown sources of bias. Simple randomisation creates entirely unpredictable random sequences, whereas methods such as block and stratified randomisation are designed to generate more balanced groups with respect to group size and population characteristics, potentially at the cost of more predictable sequences.

Each method may perform differently depending on the design of the trial and there is a lack of consensus on which methods are most appropriate in different situations. For example, simple randomisation can be highly effective in large trials but is not advised if a trials target sample size is less than 200 participants to avoid potential imbalances in treatment group size, which can affect the power of subsequent analysis [1]. Minimisation is considered by some as “the platinum standard” of randomisation methods for achieving balanced groups, whilst others strongly discourage the use of dynamic methods for this purpose [2, 3], regarding less complex methods such as simple or block randomisation as more effective [4].

It has been suggested that choice of allocation method should be dependent on the study context, objectives, and resources [5]. It was also stated that simpler methods are preferred, and that the size of the study was a key factor in selecting a randomisation method. Previous reviews have described the association between randomisation methodology and sample size [4], and changes in reporting before and after CONSORT guidance updates [6].

The aim of this study is to describe current randomisation methods use, investigate associations between choice of method and trial population characteristics, and the extent to which method selection has changed over time.

Methods

This review looks at the randomisation methods used in trials reported in five high impact journals. The BMJ, NEJM, JAMA and The Lancet were identified to allow comparisons with previous similar reviews [4, 6]. The NIHR HTA library was added as an additional journal to widen the scope of this review.

Papers from the four clinical journals were identified from the database PubMed. This search was defined as papers where the title and/or abstract contained the words “RCT”, “randomised” or “randomized”, published from January 2019 up to the end of December 2019.

The NIHR HTA library was searched to identify journal reports published from January 2019 up to the end of December 2019 (and January 2014 to the end of December 2014) categorised as primary research. For each article, we verified that the study had not already been published in one of the other four journals. This was also done for the 2014 extraction, comparing if these fit the search criteria originally used. A full breakdown of this search criteria, along with reasons for exclusions and final numbers is given in Fig. 1.

Fig. 1

Article search results including identified papers, reasons for exclusions and final numbers of papers included

Each paper was reviewed to collect information on the randomisation method used and trial design features such as sample size, number of arms and centres, blinding methods used and details of variables used in the randomisation (see Additional file 2: Appendix Table 1 for full details of the data extraction). If this information could not be found in the paper this was recorded, and then other relevant sources were searched; the protocol, statistical analysis plan or other related publications. If the randomisation method could not be retrieved the author was contacted to obtain this information. Data was only recorded as unknown if it could not be verified in any of these sources.

Table 1 Summary table showing the main study characteristics of 2019 papers split by randomisation method

Validation was performed at two stages. A random sample of 20 papers deemed ineligible for review were checked to confirm agreement on the inclusion criteria. Then, a random sample of 24 papers were independently extracted by another reviewer. Separate definitions were given for disagreements and discrepancies (see Additional file 3: Appendix Table 2 for full definition). Discrepancies covered any discordant items, and all discrepancies found were resolved by consensus. We decided that if the number of disagreements was greater than 5%, then a further sample would be examined.

Table 2 Comparison of inclusion of randomisation variables in subsequent analysis for the different randomisation methods

Trial characteristics and choice of randomisation method were compared with the 2014 Ciolino et al. review [6]. The use of stratification or minimisation methods to balance allocation, the type of variables used in balancing, and inclusion of such variables in analysis were compared with a 2010 review by Kahan et al. [7] In addition, where randomisation balanced for recruiting site or centre, the method of any subsequent adjustment in the analysis was also explored.

Results

For the random sample of 20 excluded papers, there was 100% agreement between reviewers. For the full data extraction review, 10 discrepancies were found across 24 items in 26 papers, meaning the proportion of discrepancies fell below our prespecified 5% threshold at 1.1%.

Our search returned 633 articles of which 414 (65%) were eligible for inclusion. Main reasons for exclusions were that a paper consisted of a meta-analysis or that it was a comment or correction. For the NIHR HTA library, as the only search criteria was primary research in 2019, many of the papers reviewed contained qualitative research (see Fig. 1 for a breakdown of reasons for exclusions).

The most commonly used randomisation method in 2019 in individually randomised trials was block stratified randomisation in 162/330 trials (47%). Almost two thirds 228/330 used some form of stratification within their randomisation. All but 12 of the randomisation methods used can be classified as using simple, block, stratification minimisation or a combination of these.

Within this review, 348 of the trials were individually randomised, and 37 were cluster randomised. For the cluster randomised trials, 25 of these trials can be classified using the above methods (again with the most common being stratification) and additional methods used included step wedged randomisation, matched pairs of clusters and allocation of clusters to minimise imbalance.

One trial began using block stratification and during recruitment detected imbalances in groups and so switched their allocation method to minimisation. In this case we have classified this randomisation as block stratification as this was the intended randomisation method at the design phase.

Of the 12 randomisation methods that are classified as other, 3 of these used Bayesian methods, 4 defined their methods as dynamic or adaptive, 1 used a biased coin design, 1 used a big stick design, 1 defined their method as using cohorts and 2 did not properly specify the methods and there was no access to the protocol (and no response from the authors).

Table 1 shows a breakdown of trial population characteristics by randomisation method. We can see that sample size and number of centres relate to randomisation method selection. Interestingly, the number of centres is much larger for studies that used stratification. After filtering out trials where centre was included in the randomisation and those that were not, the median number of centres is still much larger in stratified studies, but not those that included centre as a stratification variable (see Additional file 4: Appendix Table 3).

Table 3 The characteristics of each variable used to randomise split by covariate inclusion in the analysis

From Table 2, comparing the trials with no covariates in the analysis to those with at least one covariate in the analysis, these trials are much more likely to be stratified by centre, suggesting that generally if a trial is stratified by centre, centre is not subsequently included in the analysis.

Table 3 provides a breakdown by the individual characteristics of the variable. Comparing covariates that were not included in the analysis to those that were, covariates not included in the analysis are less likely to be binary than those included in the analysis and are more likely to be non-ordinal categorical. This may be due to the fact that centre is often not included within the analysis, supported by the fact that covariates not included in the analysis generally had a much greater number of categories.

In Table 4, comparing stratification variables to minimisation variables we see a similar pattern, with a higher percentage of stratification variables being non-ordinal categorical and having more than five categories. To confirm this, we split these tables out to look specifically at centre and those variables that were prognostic. This effect can still be seen even when looking at prognostic variables only suggesting there is another reason for this effect (Table not shown).

Table 4 The characteristics of each variable used to randomise split by randomisation method (stratification or minimisation)

In order to look more closely at the difference between variables included in stratified randomisation and minimisation, we compared sample size with the number of randomisation variables included (shown in Fig. 2), and then sample size with the number of strata (in this case defined to be the product of categories for these variables (Shown in Fig. 3)). Both Figs. 2 and 3 suggest that stratification is more common in higher sample sizes, but that minimisation is used more frequently for more complex designs, when there is a greater number of randomisation variables and strata.

Fig. 2

Plot of sample size vs the number of randomisation variables included. Note: y axis is presented as log base 10

Fig. 3

Plot of sample size vs the number of strata across all randomisation variables. Note: Both axis are presented as log base 10

This suggests that while a greater percentage of stratification variables had greater than five categories than minimisation variables, trials that used minimisation included a greater number of randomisation variables and so overall had a greater number of strata.

Table 5 presents a comparison of our results with those published by Ciolino et al. Here the original results from Ciolino are presented, alongside these results combined with the HTA journal and then our findings from 2019. Comparisons between 2014 and 2019 show two notable trends. In some ways trials seem to be more simplistic, with an increase in the number of trials that do not include a covariate within the randomisation, also seen in the increase in the use of simple randomisation. Nevertheless, of those trials that do include a covariate in the analysis, there is an increase in the number of covariates being used in the randomisation.

Table 5 A comparison of current practice with a review conducted in 2014

Finally, a comparison of our findings alongside the review from Kahan et al., is shown in Table 6. This shows an overall increase in the use of prognostic factors in the randomisation process, and more importantly, an increase in adjustment for balancing factors in subsequent analysis. Exploring the relationship between the number of centres, sample size, and subsequent adjustment for centre in the final analysis, we found that overall, random effects were more commonly used to adjust for centre and fixed effects were generally used in studies with less than 15 sites (shown in Additional file 1: Appendix Fig. 1).

Table 6 A comparison with a previous review of the inclusion of randomisation factors in the analysis

Discussion

This review gives an insight into the randomisation methods most commonly used and shows that there have been very few changes in this methodology since 2014, with a method containing an element of stratification being favoured. Response adaptive randomisation methods seem to be less commonly used, but this may be due to the types of trials being conducted, as these methods are only appropriate in very specific circumstances which may be less likely to be published in the journals we have reviewed.

We expected to see trials with larger sample sizes using less complex methods, as achieving balance is more likely to happen without intervention in larger sample sizes; however we observed the opposite to be true. It is possible that researchers may have been reluctant to use more complex methods with smaller sample sizes, as a large number of strata with a small sample size can lead to overstratification [8]; however this is perhaps surprising as these methods were devised to create more balanced groups with smaller sample sizes.

European Medicines Agency (EMA) guidance on covariate adjustment was issued in 2015 [9], so was not available during the 2014 review conducted by Ciolino but was available during our review. The EMA discuss the benefits of stratification for improving estimation efficiency. It also states that all stratification variables should be included within subsequent analysis unless they were chosen only for administrative reasons, and in this case justification for non-inclusion should be given. Their views on dynamic methods are more cautious as they state the possibility of issues with type 1 error control when using these methods.

When discussing inclusion on centre in an analysis, the EMA recommends all variables should be included in the analysis but recognises there are many issues leading to unreliable estimates, and states that in the case of non-inclusion of centre this should be explained, and sensitivity analysis should be used to demonstrate there is no effect of leaving it out of the model.

ICH E9 recommends that inclusions of centre should be dependent on the effect of centre, and discusses situations in which inclusions of centre in subsequent analysis may not be practical, such as those with a large number of smaller centres that recruit a limited number of participants [10].

We noted that trials which had stratified by centre tended not to include centre in subsequent analysis. This may allude to a more practical issue where models including centre can be a lot more unstable if there are a large number of centres and some centres with small recruitment numbers [11].

Looking at trends in methods use over time, we were surprised to see that fewer studies included a covariate in the analysis, but those that did were including a larger number of covariates than previously seen in 2014. This may suggest a polarisation of opinions, with some researchers sticking to less complex methods without inclusion of covariates, whilst those that do include covariates assume a benefit in including larger numbers of them. Additionally, generally we can see those studies including less than 5 variables in their randomisation are much more likely to use stratification, where above this minimisation becomes more common, which is consistent with existing guidance [12].

Our findings agree with McPherson’s [5], that the preferred method of randomisation is block stratified randomisation. Overall, the only design characteristics that seem to factor into researchers’ decisions on which randomisation method they select is related to the sample size and the number of centres, creating a need for more in-depth research into how these decisions are actually made.

Comparing our findings to those from Kahan et al., there is a large increase in trials adjusting for balancing factors in subsequent analysis, suggesting a large influence of this research in practice. When including centre in the analysis, this may be included in multiple different ways, as discussed by Senn [13]. In our review, more studies adjusted for site in their analysis after including site in the randomisation than Kahan found in 2010, although almost half of the studies did not adequately specify how site was included (as a random or fixed effect). There does however seemed to be a correlation between the number of centres a trial has and whether a random or fixed effect was used.

This study benefits from being a large and comprehensive review. The four clinical journals were selected as they were used in many other reviews, allowing this review to update previously published research but also as they are considered high impact journals, and since each journal is committed to the CONSORT statement [14], it was felt they would provide more complete data.

The generalisability of results may be considered a limitation of this review as it only focuses on five high impact journals. The addition of the NIHR HTA library was intended to widen the scope of journals from the other 4 clinical journals, but we do still acknowledge that the papers reviewed will still have limited generalisability to all clinical trials.

Another limitation of this review is that results may be considered slightly out of date. However, due to the COVID-19 pandemic, we felt that publications after this year would not be representative of usual practice, hence we felt that 2019 gave the most up to date representative review of the literature.

The findings of this review demonstrate that there is some evidence of trial design characteristics contributing to a researcher’s choice of randomisation method. There has not been much evolution in randomisation methodology, with simple, block, stratified and minimisation making up 95% of individually randomised trials in this review, suggesting little uptake in any newer emerging randomisation methods. The one noticeable change in practice is the polarisation of method use, which adds evidence to the fact the differing opinions on randomisation method affects methods use in clinical trials. Future research will focus on how study features factor into a researcher’s decision when choosing a randomisation method, and other factors that may affect their choice.