Questions asked and answered in pilot and feasibility randomized controlled trials

In the last decade a number of authors have reviewed the justification for describing a trial as a pilot or feasibility study in terms of its content and the questions it addresses [1–5]. Lancaster et al [2] list as legitimate objectives of a pilot study: sample size calculation; providing a dummy run of trial procedures/the protocol; testing data collection forms or questionnaires; testing how randomization procedures work; determining recruitment and consent rates; examining the acceptability of the intervention; and selection of the most appropriate primary outcome measure. Thabane et al [5] categorize reasons to conduct a pilot study into four groups: assessing the feasibility of processes that are key to the success of the main study; assessing time and resource problems; potential human and data management problems; and scientific issues including the assessment of treatment safety, dose, response effects and variance of the effect. They also include a checklist of items to include in reports of pilot studies. Arain et al [4] recommend the NIHR Evaluation, Trials and Studies Coordinating Centre definitions [6] which describe a pilot study as a miniature version of a main study run to test whether components of the main study work together, while feasibility studies are pieces of research done before a main study to answer the question "Can this study be done?". According to these definitions both pilot and feasibility studies play a preliminary role in the design stage of a subsequent larger trial, and do not themselves address efficacy.

Arain et al [4] comment that researchers applying for funding for trials inadequately powered to address clinically meaningful hypotheses may adopt the designation of a pilot study in the hope of a more favourable review. When studies are prepared for publication authors may similarly believe that labeling a small trial as a pilot increases its chance of acceptance. Although Arain et al found that the editors of five high ranking medical journals did not encourage publication of pilot studies because of their perceived lack of rigour, it is possible that other journals are more accommodating. In an editorial in the journal Circulation, Loscalzo [7] proposed a binary classification: trials designated a priori as pilots, and those redefined a posteriori. During a five year period 41 pilot trials were published in the journal. Many had been designated as pilots at the request of the editorial office to alert readers to uncertainty in the generalizability of their results and their preliminary and exploratory nature. Such a policy is likely to result in trials primarily addressing efficacy being described as pilots, contrary to the NIHR and other recent definitions.

In comparison to the well established pathways of development for pharmaceuticals, prior to 2000 there was little specific guidance on the development of procedures involved in trials of non-pharmacological interventions, possibly because of their complex and heterogeneous nature. The MRC guidelines for the evaluation of complex interventions published in 2000 [8] and revised in 2009 [9] emphasize the importance of testing procedures before planning an evaluation, and also the circular nature of development, feasibility and piloting, evaluation, implementation, and further development. There is no specific guidance for non-pharmacological interventions that do not meet the MRC definition of complexity. In this paper we review a random sample of 50 papers reporting randomized controlled trials (RCTs) published in journals covered by the MEDLINE and EMBASE databases where authors described their trial as a 'pilot' or addressing 'feasibility' in the title. We review papers published between 2000 and 2009, the ten years following the publication of the first MRC guidelines on complex interventions. We examine stated objectives, results and conclusions drawn, and in particular whether these relate to methodological issues, efficacy or safety/toxicity: comparisons are drawn between papers reporting drug and non-drug trials.

We searched the EMBASE and MEDLINE databases on 29^th July 2010 to identify papers reporting parallel group trials with one or both of the words 'Pilot' and 'Feasibility' in the title. To be included, papers had to be published between 2000 and 2009, written in English, studying humans and indexed as an RCT. Using computer generated random numbers we selected 50 of those identified for full review [10–59]. The sample size was chosen taking into account resources available and the detailed review required, it allows percentages between 10% and 17.5% to be estimated with a 95% confidence interval (CI) of ± 10%. The search was repeated on the 21^st February 2011 to gain a more complete estimate of the number of relevant papers in 2009: two papers were selected from those identified on this date to replace papers found to be ineligible at a late stage.

A form was designed to document characteristics of the selected papers. It was tested by all three authors on three randomly selected papers which were not part of the main review sample. Minor modifications were made after the first two papers of the main sample had been reviewed. We defined drug trials as those involving the administration of a discrete chemical entity, substance, or biological agent by mouth or other route, for example by injection. Questions on the form related to: any blinding and in particular double blinding; the numbers of active and placebo/control arms; whether multiple centres were involved (sometimes deduced from the number of institutional review boards mentioned); the target and actual sample sizes; and any justification given for the target sample size.

Research objectives stated in the Abstract and Introduction sections were coded as relating to methodological issues, efficacy, or safety/toxicity, as were statements summarizing results and conclusions chosen for inclusion in the Abstract. Objectives had to be explicitly stated, it was not enough for the reviewer to deduce what the objectives might have been from results presented, or conclusions drawn. Efficacy conclusions in the Abstract were rated as indicating that the experimental intervention had not been shown to have benefit or had no benefit, that it showed promise, or that it showed actual benefit. The accuracy of conclusions drawn by the authors was not verified. An example of an efficacy conclusion in the Abstract rated as indicating the intervention had not been shown to have benefit or had no benefit is

"With the numbers studied, we failed to find a significant difference between the two groups; thus we have no evidence of a benefit from botulinum toxin injection in the treatment of chronic tennis elbow" [34];

one rated as indicating the intervention to have promise is

"Therefore, the colonic colplasty seems to be an attractive pouch design because of its feasibility, simplicity, and effectiveness." [31];

and one rated as indicating the intervention to have actual benefit is

"Sleep educational programs for secondary students are recommended to improve information about sleep." [23].

Numerical results presented in the Results sections were classified as relating to methodological issues, efficacy, or safety/toxicity. We looked for counts of trial participants experiencing methodological problems or side effects, or numerical summaries of statistical findings: text statements that a procedure was feasible for example were not enough to qualify as a result. Methodological results were recorded separately for: recruitment, retention, compliance/adherence to intervention, blinding procedures, acceptability of the intervention to participants, other aspects of the intervention, outcome assessment, logistics of the randomization procedure, acceptability of trial procedures, or the logistics of multi-centre procedures. Selection of these topics was in part based on the issues that Lancaster et al [2] list as constituting pilot studies. We checked the Methods sections to see whether methodological results were reported there, sometimes they formed part of a CONSORT flowchart [60] for example. Depth of coverage was coded as none, brief, detailed or tabulated/graphical presentation. Significance tests and CIs presented for efficacy outcomes were examined to see whether they indicated between or within group significant differences, but no attempt was made to judge which findings were the primary evaluation of efficacy.

In the Discussion sections we again rated coverage of methodological issues, efficacy, and safety/toxicity as none, minimal, substantial, or the major focus of the section. An example of discussion concerning methodological issues rated as minimal was

"The results are promising but not conclusive because of the low numbers of patients studied, and we recommend that a sufficiently powered study should be performed." [29];

and the following example was rated as substantial

"In order to show an assumed clinically relevant difference of 2 kg, with an 80% power and a type-I error of 5%, 300 patients would be needed (150 in each treatment group). If a subsequent study were to be planned, it would be advisable to use the mean change in grip strength as a primary variable because the variability for this parameter was lowest in the present study and it came close to identifying a significant difference between groups (p = 0.196). In addition, grip strength is a quantifiable measurement of effect, unlike the more subjective measurement of pain." [34].

The NIHR definitions indicate that pilot and feasibility studies should be preliminary research prior to a main study: we were therefore interested in whether authors stated they were conducting a further trial (or were scheduled to start one in the near future). If this was mentioned it was usually in the Discussion section. Since the above two quotes were the only mention of future trials in each paper we did not consider either set of authors to be actually conducting a future trial. Comments in the Discussion concerning lack of power or small sample size were noted.

Finally we recorded whether Conclusions sections contained statements concerning methodological issues, efficacy, or safety/toxicity. The Conclusion section could be a specifically labelled section, a paragraph of the Discussion clearly listing conclusions, or presented as a box: where there was no such section missing was coded not the absence of a relevant conclusion.

The 50 papers were assessed by MS and difficulties arising were discussed with RMP and MW. Blyth-Still-Casella 95% CIs for single percentages and exact CIs for Rate Ratios (RR) were obtained in StatXact [61]. Ordinal ratings were compared between groups in Mann-Whitney U tests, and percentages in exact Pearson's chi-squared tests.

For the main part the pilot and feasibility trials in our review did not primarily address methodological issues. Although lessons learnt about planning trials were discussed in the majority of papers it was often to an extent we rated as minimal, and in only 6 (12%) of papers was it stated that the authors were actually conducting a subsequent trial or about to start one. This frequency is not dissimilar to the 9% of pilot studies reviewed by Lancaster et al [2], subsequently found to have been followed by a larger study [4]. Even though authors themselves may not proceed to a larger RCT it is possible that others reading the paper will. Our impression was that many of the trials fell into the latter of Loscalzo's [9] two classes: namely those designated as pilots a posteriori possibly after failing to demonstrate the hoped for effects or because of inadequate sample size. In all but one of the 11 papers including a power calculation, sample size was determined to achieve power in testing efficacy. In other papers it was impossible to be sure what a priori objectives were from the published paper alone, but since methodological issues were discussed to greater depth in the non-drug trials, they are more likely to fall into Loscalzo's class of a priori pilot trials. Even amongst the non-drug trials there was generally an emphasis on efficacy.

We interpreted efficacy as the examination of change in an outcome variable not clearly related to safety/toxicity. The final group of scientific objectives for pilot studies listed by Thabane et al [5] includes obtaining estimates of the treatment effect and its variance. Thabane et al also discuss the distinction between pilot and proof-of-concept studies which they define as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive. Arnold et al [3] include the assessment of mechanisms, possibly using surrogate measures, as a legitimate objective of pilot trials, to establish proof-of-principal and potential efficacy. Many of the papers in our review may lie on the margins between pilot and proof-of-concept investigations and their emphasis on efficacy should perhaps be interpreted in this light, however none were described as proof-of- either concept or principal studies in their title [10 - 59], though one [42] was described as a phase II pilot study.

Thabane et al [5] recommend that explicit criteria indicating that a subsequent trial is feasible should be stated: they describe the criteria set out in advance for proceeding from the pilot to the main Prophylaxis of Thromboembolism in Critical Care Trial (PROTECT) [62], also reviewed by Arnold et al [3]. In contrast, Gardener et al [63] describe a case study where unanticipated problems arising during a pilot lead to abandoning a subsequent RCT even though pre-stated objectives indicated the methodology to be feasible. They identify the availability of funding and the contemporary health service environment as issues likely to impact on the decision to proceed. We believe that studies evaluating the feasibility of trial procedures are essentially exploratory in nature. Researchers should examine carefully the success of procedures and react to unanticipated problems to get the best possible design for their next trial. Aspects of the design aren't decided in isolation, predicted recruitment under a set of eligibility criteria may be adequate if there is a change in outcome variable for example. If extensive changes are made it may be advisable to retest the feasibility of the protocol.

Others have searched for pilot/feasibility trials adopting different criteria leading to different populations of papers surveyed. Lancaster et al [2] could find no guidance on how to search MEDLINE for pilot/feasibility trials, and restricted their search to papers in six top ranking medical journals with the words 'pilot' or 'feasibility' in the title, abstract or keywords: of the 115 hits retrieved 25 (22%) were not suitable for a variety of reasons. Arain et al [4] repeated Lancaster et al's search procedure seven years later with a rate of unsuitable papers (30%) similar to ours of 33%. Arnold et al [3] initially searched MEDLINE for pilot trials in critical care medicine, but then canvassed known clinical investigators because of the poor indexing of pilot trials. They describe five pilot trials fulfilling the requirements of either addressing methodological issues relating to the feasibility of subsequent trials, or assessing mechanisms of intervention. Reviews based on published papers do not address internal pilots which continue into the main phase and are unlikely to be reported separately, though where a decision is taken not to continue the pilot phase could be written up. We chose to include only full papers in our study: different issues may arise amongst pilot or feasibility trials that are published as letters or brief reports. Sampling from all journals covered by MEDLINE and EMBASE and restricted to trials incorporating random allocation, the characteristics of the pilot/feasibility trials we found is not unexpected. The majority bear little resemblance to the recent definitions proposed for pilot/feasibility studies.

Our main findings are that RCTs described by their authors as pilots or addressing feasibility most commonly focus on efficacy, in just over a half (56%) were issues that might inform the planning of a subsequent trial addressed in reasonable depth. In addition to efficacy pilot drug trials also addressed safety, while pilot non-drug trials were more likely to additionally address methodological issues. While the median sample size was quite small at 34, there were three trials recruiting over 100 participants, demonstrating that sample size very much depends on circumstances even in the context of pilot/feasibility trials.