An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom Clinical Research Network database

The National Institute of Health Research Evaluation, Trials and Studies Coordinating Centre (NETSCC) defines a pilot trial for a randomised controlled trial (RCT) as ‘a version of the main study…run in miniature to test whether the components of the study can all work together’ and a feasibility study for an RCT as ‘research done before a main study to answer the question “Can this study be done?”.[1] However, whilst some authors, including Arain et al. [2] recommend these definitions, in truth there is no consensus. Stallard [3] reports a reason for this as being in part, due to the wide variety of purposes for which pilot trials are undertaken.

Thabane et al. [4] give a number of reasons as to why pilot trials may be conducted. They state that conducting a pilot trial before a main study can increase the likelihood that the main study will be a success, and may potentially help to avoid ‘doomed’ main trials. They also state that in many cases, pilot trials are performed in order to generate data for sample size calculations in the main study.

Prescott and Soeken [5] meanwhile, suggest five pilot trial aims based on a review of then-current nursing research text books including: a feasibility assessment; adequacy of instrumentation and answering methodological questions.

To address the aims of a pilot trial a sample size justification is required. Hertzog [6] highlights that there is little published guidance on for a pilot trial sample size. However, when applying for funding for a pilot trial, a review panels would expect a justification for the planned sample size. This justification could be based on a number of methods:

Setting an appropriate sample size for any study is important. If a study is too large it may be judged to be unethical as participants may be unnecessarily exposed to risks and burdens [11]. There is the additional issue that setting the sample size too high may lead to a preventable failure to reach the recruitment target [12]. While Julious [9] highlights that a sample size that is too small will have an imprecisely estimated variance, which could impact on the design of a future definitive study.

This paper aims to build on the work of Lancaster et al. [12] who reviewed pilot trials published from 2000 to 2001 in seven major journals and Arain et al. [2] who revisited the same seven journals from 2007 to 2008 to see if there had been any change in how pilot trials were reported.

Arain et al. [2] concluded that pilot trials are poorly reported and that the authors are often not explicit as to the purpose of their pilot trial. They also found that sample size calculations were only performed and reported in 35% of the trials and that those identified using the key word ‘pilot’ were more likely to have a pre-study sample size calculation.

Using data from the United Kingdom Clinical Research Network (UKCRN) Database we extend the work of Lancaster et al. [12] and Arain et al. [2] by investigating the sample size of pilot and feasibility trials for RCTs currently running in the United Kingdom (UK). The aim was to investigate on-going sample sizes for pilot/ feasibility trials in the UK. Although as discussed, there are definitions of pilot and feasibility available, we recognise that in reality the terms are often used interchangeably. However, Arain et al. [2] found that there were some differences between the designs of studies labelled pilot and feasibility. Therefore, in this investigation we will distinguish between pilot and feasibility trials in the analysis. We will further look at whether the sample sizes chosen varies between the two study types (pilot or feasibility), as defined by the principal investigator in their UKCRN Database entry.

The paper will also investigate if the sample size chosen for the trial is influenced by factors such as how the trial is funded or the type of endpoint.

The three research aims of the paper are:

1 To describe the sample sizes set for trials labelled pilot versus feasibility

2 To describe the sample sizes set for trials with a dichotomous compared to a continuous endpoint

3 To describe the sample sizes set in trials funded by industry, public bodies or charities.

Building on the work of Lancaster et al. [12] and Arain et al. [2] the trials analysed in this paper were trials currently running in the United Kingdom on the date the search was conducted, giving us a wide range of information regarding target sample sizes. All the trials that met the inclusion criteria stated a target sample size for their trial within their research summary. Although it is not a requirement in none of the summaries was there a justification given for the target sample size given.

Moore et al. [16] highlighted that it is not unusual for study proposal reviewers to come across a statement such as “No sample size justification is needed because of the pilot nature of the proposed study”, but they state that pilot trials are not exempt from needing a clear rationale for the number of patients to be included. However, Arain et al. [2] discovered that only a small proportion of published pilot trials report pre-study sample size calculations as most journal editors state that it is not mandatory criterion for publication.

An investigation of the expected benefits, risks and costs of the study is required to justify a target sample size [16]. However, it is important to remember that a target sample size for a pilot or feasibility study is only a preliminary figure and has a great degree of uncertainty. For example, the researchers may find that more participants drop out than first presumed. We have shown that target sample sizes vary for preliminary trials. Considering the median sample sizes for pilot and feasibility trials our data shows that on average feasibility studies are larger than pilot trials: although there is wide variability in the sample sizes across all types of trial. The median sample size per arm across all the types of study was 30.

With regards to target sample size according to funder, a study of registered drug trials by Bourgeois et al. [17], across a wide variety of types of trial, found that those funded by industry were more likely to have a larger sample size than those funded by government sources. However, our analysis indicated that publicly funded pilot trials were larger than industry funded pilot trials.

Campbell et al. [18] describe sample size calculations for studies that have dichotomous, ordered categorical and continuous endpoints. They state that approximately 30% fewer patients are required for a study with a continuous endpoint – in our research we found that for a dichotomous endpoint compared to a continuous the median sample size was 20% bigger.

Looking at the differences in sample size according to type of primary endpoint and funder we found that there is a larger difference in sample size between trials with a dichotomous endpoint compared to a continuous endpoint for publicly funded trials compared to industry funded trials.

It would be beneficial to follow-up the pilot and feasibility trials discussed in this paper to see how many go on to be published – to see if there is a difference between those published and not published. Another possible extension would be to investigate the different sample sizes of trials dependent on whether the primary endpoint of the trial is based on efficacy or feasibility.

The limitations of this study include the fact that only one trial registry was used to collect the data meaning that it is possible that eligible trials that were not registered with the UKCRN are missing from the analysis. If these trials differ in some way from the trials listed on the UKCRN then this could affect the conclusions made. The database used only trials being carried out in the UK, which could also affect the generalisability of the results. The search was only carried out by one reviewer and was not repeated to check for accuracy. In addition, only two search terms were used; pilot and feasibility therefore, some trials labelled for example, exploratory or preliminary could have been missed during data extraction. However, these search terms were used to maintain consistency with previous research [2, 12].

All trials should have a sample size justification. Not all trials however need to have a sample size calculation. For feasibility and pilot trials, while a sample size justification is important, a formal calculation may not be appropriate. In our study we found that the median pilot study sample sizes for two arm trials were 36 and 30 per arm respectfully for dichotomous and continuous endpoints.