- Research
- Open access
- Published:
Scoping review and characteristics of publicly available checklists for assessing clinical trial feasibility
BMC Medical Research Methodology volume 22, Article number: 142 (2022)
Abstract
Background
Whether there is sufficient capacity and capability for the successful conduct and delivery of a clinical trial should be assessed by several stakeholders according to transparent and evidence-based criteria during trial planning. For this openly shared, user-tested, and validated tools are necessary. Therefore, we systematically examined the public availability and content of checklists which assess the study-level feasibility in the planning phase of clinical trials.
Methods
In our scoping review we systematically searched Medline, EMBASE, and Google (last search, June 2021). We included all publicly available checklists or tools that assessed study level feasibility of clinical trials, examined their content, and checked whether they were user-tested or validated in any form. Data was analysed and synthesised using conventional content analysis.
Results
A total of 10 publicly available checklists from five countries were identified. The checklists included 48 distinct items that were classified according to the following seven different domains of clinical trial feasibility: regulation, review and oversight; participant recruitment; space, material and equipment; financial resources; trial team resources; trial management; and pilot or feasibility studies. None of the available checklists appeared to be user-tested or validated.
Conclusions
Although a number of publicly available checklists to assess the feasibility of clinical trials exist, their reliability and usefulness remain unclear. Openly shared, user-tested, and validated feasibility assessment tools for a better planning of clinical trials are lacking.
Background
Evidence-based health care relies on high quality clinical research. Randomized controlled trials (RCTs) are the method of choice to assess preventive and therapeutic interventions and are a cornerstone in the final phase of drug development and in comparative effectiveness research [1, 2]. Conducting high quality clinical trials, however, is challenging [1, 2]; requiring specialized capacities and capabilities in the areas of the clinical conduct of studies, adherence to ICH E6 Good Clinical Practice guidelines, regulatory aspects, data management, financial regulations, protection of human beings, and project management [3, 4]. Feasibility assessment during trial planning is an evaluation whether there is sufficient capacity and capability for the successful conduct and delivery of a clinical trial [5]. This trial planning process aims to ensure that the design is practical within the intended setting, the resources required for delivery are available, and recruitment targets are realistic. It is distinctly different from a feasibility study. A feasibility study “asks whether something can be done, should we proceed with it, and if so, how” [6] and it can be among other considerations part of the feasibility assessment of a trial.
If the feasibility of a clinical trial is not established before commencement of the trial, there is the risk of poor performance, insufficient recruitment, and an unacceptable high number of protocol violations [3]. Previous research indicated that one out of four RCTs are not completed as planned, primarily because of poor recruitment [7, 8], which could likely be avoided with appropriate planning [9]. The same has been found with regards to non-randomized clinical trials [10]. Poor assessment of feasibility is a known factor which adversely affects efficient trial conduct [11]. Proper feasibility assessment during the planning process can help avoid premature discontinuations of clinical trials, which constitute a considerable waste of research resources. The assessment could be done by using checklists or tools that could tell the user whether a trial is likely to be successful or needs further adjustments at the planning stage. Whereas checklists exist with the aim to improve the reporting quality of pilot- and feasibility trials [6, 12], it is currently not known, whether there are validated and publicly available checklists that could be used to assess the study level feasibility of clinical trials during their planning. Therefore, we aimed to systematically identify available checklists with a focus on whole study and not site level feasibility assessment and to examine their contents.
Methods
A scoping review was conducted to identify feasibility checklists. Scoping reviews are used to map the existing literature, and are considered particularly suitable for complex or heterogeneous areas of research [13]. This study is reported adhering to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) extension for Scoping Reviews [14]. No written protocol exists for this scoping review.
Eligibility criteria
Publicly available checklists that can be used to assess study level feasibility of a clinical trial, independent of the planning stage (e.g. protocol) or source (any country, university, funding agency, clinical trial organisation), were eligible to be included in the study. We aimed to focus on study level feasibility checklists and therefore excluded checklists focusing only on site level or program level feasibility assessment (see Supplementary file and Supplementary excel file for details) [15].
Information sources, search, and selection
We systematically searched Medline and Embase via Ovid from inception to June 2021 without any language restrictions. In addition, BS and VG independently searched the internet via Google and the homepages of relevant research stakeholder organisations with pre-specified word combinations; initially in October 2019 and updated in June 2021. The internet was searched until BS and VG felt that saturation was reached (i.e. the last 20 hits did not reveal any new relevant information). The search strategy is presented in the Supplementary information file.
Based on the eligibility criteria, VG along with either BS, AG, ATH, AS, TF, CMP, or MB screened all titles and abstracts of references found through the literature search for potentially eligible publications. Full texts of potentially eligible checklists were then independently screened by VG and either BS, ATH, AG, or MB. In case of disagreement, consensus was reached by discussion.
Data extraction
Of the included checklists, VG extracted information on source, country name, institution type, intended users, any description of user testing or validation, whether the checklist is provided with any instructions on how it has to be applied or how the result should be interpreted. All extractions were cross-checked by BS, ATH, or AG.
Analysis
Using the included checklist documents, VG performed conventional content analysis [16], focusing on items or themes common across checklists as well as those unique to individual policies. VG read and coded all checklists, with initial themes being identified inductively using a process of open coding (i.e., no specific preconceived codes were identified or used; rather, codes emerged directly from the data). A coding framework was developed by a progressive process of classifying, comparing and refining text passages to create categories. The final coding framework was checked by the other co-authors to ensure consistency and validity. The results of the analysis were summarized in tables. One presenting characteristics of the included checklists and the other presenting the identified items that may determine study level feasibility. In terms of critical appraisal of individual checklists, we checked whether they were user-tested or validated. The raw data can be found in a Supplementary excel file.
Results
Selection of sources
The literature search identified 6221 references of which one was included in the final analysis, and the internet search identified 35 potential checklists of which nine were included (Fig. 1). Thus, in total we identified 10 publicly available checklists to determine study level feasibility of clinical trials.
Characteristics of included checklists
Out of the 10 included checklists, 5 were issued by universities [5, 17,18,19,20], 3 by health care facilities [21,22,23], and 2 by national research organisations [24, 25] (Table 1). The checklists were from a total of 5 different countries, nine checklists were published in English and one in German. Intended users of the checklists were funders and investigators [24], health care facilities and investigators [5, 25], funders [17], investigator only [18, 19], investigator and department chair or designee [20] or investigator and research facilitator [21]. In one case, a university department required investigators to submit the filled checklist to their institutional review board before approval of a new trial [20].
Only four checklists indicated the information required for a feasibility assessment: trial protocol only [20, 23], informed consent form, and other facility specific forms [22], or grant proposal [24]. None of the included checklists provided information on whether they were user tested or validated. With the exception of one checklist [23], they did not provide any instructions on how to fill out the checklist or how to interpret the results of the assessment. In addition, the underlying evidence-base for the development of the identified checklists (e.g., whether the checklist items were based on expert consensus) was missing for all checklists.
Feasibility assessment items
A total of 48 distinct items in relation to assessing the feasibility of a clinical trial were identified (Table 2). These items were further categorised into 7 domains: regulation, review and oversight; participant recruitment; space, material and equipment; financial resources; trial team resources; trial management; and pilot or feasibility studies.
The number of distinct items according to our classification ranged in checklists from 7 to 28 with a median of 21 items (Table 2). There were four checklists that included about half of all identified 48 items and the other six checklists less than that. The domain with the highest number of identified distinct items was participant recruitment (14 items), followed by trial team resources (11 items). All other domains contained nine or less items.
The most frequently mentioned items (mentioned by 8/10 checklists) across all domains were “The target population is available “ (domain: participant recruitment), "Access to professional support and required facilities is available" and “Equipment is appropriate and sufficient “ (domain: Space, equipment and material), "Current standard practice at trial site(s) is compatible with trial protocol" (domain: Trial management), and “Adequate staffing is identified and available within the trial period” (domain: Trial team resources). Also frequent (mentioned by 7/10 checklist) were the items: “The clinical trial is compliant with local regulations”, “Competing trials are known”, “Study drug and comparator are available” and “Adequate staffing is identified and available within the trial period” (Table 2).
Discussion
This scoping review found ten checklists issued by universities, national research organizations, or health care facilities that are publicly available to assess study level feasibility of clinical trials. We identified 48 distinct items for trial feasibility assessment. The most frequently mentioned individual items were “The target population is available “, "Access to professional support and required facilities is available", “Equipment is appropriate and sufficient “, "Current standard practice at trial site(s) is compatible with trial protocol" and “Adequate staffing is identified and available within the trial period”. The number of items differed considerably across feasibility checklists. Only four of the ten checklists contained about half of the identified 48 items, the other five checklists less than that. For only four of the identified checklists the documentary basis (e.g. trial protocol) for the assessment was specified, and for none of the checklists the choice of items was justified or the way of compiling items explained. None of the available checklists appeared to be user-tested or validated. Thus, the validity, practicability of available trial feasibility checklists, and whether or not the implementation of such checklists indeed leads to more successful trial conduct appears uncertain. No single checklist is likely to cover all the items required to assess feasibility for every trial and is reliant on the user completing the checklist as intended [26]. Furthermore, checking for feasibility during trial planning has to be seen in the context of a comprehensive framework of clinical research that covers all stages of a clinical trial, i.e. concept, planning and feasibility, conduct, analysis and interpretation, and reporting and knowledge translation [27]. Trial success may also depend on these other phases. Thus, equivalent tools are conceivable for the other phases, too. For example, the implementation of a risk- based monitoring tool during trial conduct.
Comparison to other literature
Although there is substantial literature on feasibility studies, reporting guidelines, and since 2015 even an online journal fully dedicated to pilot and feasibility studies exists [6, 12, 28], the actual assessment whether there is sufficient capacity and capability for the successful conduct and delivery of a clinical trial seems to be a neglected topic in the literature. We only found a single article of a publicly available feasibility checklist with our systematic literature search [5]. There are viewpoints, commentaries, or perspectives articles discussing different aspects of clinical trial feasibility without providing a practical checklist or describing scientific work for a systematic tool development [3, 11, 15]. The here mentioned key factors for trial feasibility assessment largely overlap with the domains from our content analysis. Butryn et al., for instance, considered optimal resource allocation, operational efficiency, financial viability, and enrolment success as essential components for trial feasibility; and the success of each component is best achieved through close collaboration between the principal investigator, the research team, information technology specialists, and ancillary departments (e.g. radiology) [3]. As a reaction to another prematurely discontinued RCT due to poor recruitment, an editorial by Maas raised the overdue question about criteria for pre-study feasibility assessment and suggested that clinical trial registries such as clinicaltrials.gov should consider requiring information about trial feasibility assessments [29]. Given the high prevalence of premature trial discontinuations due to recruitment or organisational problems [7, 8], and the associated huge amount of wasted resources, it is surprising that the clinical trial community has not yet adequately responded to the obvious need for more effective trial feasibility assessment.
Limitations
Our scoping review has the following limitations: First, we might have missed available trial feasibility checklists despite our comprehensive search strategy including an internet search in addition to a literature search of two large electronic databases [30]. We chose this approach, because we assumed that we had to rely on websites and online publications of research institutions. Inherent risks of searching the internet are selection bias (bubble effect) and the issue of limited reproducibility due to the non-transparent and non-consistent search algorithm by Google.com [31]. Second, we focused only on publicly available checklists. Searching for unpublished checklists or tools would have required a different approach (e.g. survey of clinical trial stakeholders). However, in our opinion this is a minor limitation as we aimed to provide an overview of publicly available tools that can also be accessed by any stakeholder. Third, we could not assess the quality of the identified checklists since we did not find any information on how they were developed. A detailed description of advantages and disadvantages of the different checklists would require comprehensive user testing, ideally using a sample of RCTs that are currently in the planning phase. Fourth, the provided overview of suggested feasibility assessment items is not a recommendation for how an ideal checklist should look like (e.g. not all items might be relevant to trial success or some items may be missing) and is not ready to implement – it is rather a first step for systematic and transparent tool development (see Future directions).
These limitations, however, are hardly relevant for our conclusion that user-tested and validated clinical trial feasibility assessment checklists or tools are lacking. We think that our search allowed us to identify the available checklists that an investigator would find who probably conducts less-extensive searches of the internet or literature databases.
Future directions
Our overview of suggested items for trial feasibility assessment may be used as a starting point for the systematic and transparent development of a reliable, valid, and user-friendly feasibility assessment tool involving relevant stakeholders such as trial investigators, trial support organizations, research ethics committees, and funding agencies. A large international group of stakeholders could first examine whether there are any missing items, more or less important items (grading the importance of items) and bring forward feasibility checklists or tools that are not publicly available. A resultant item list could then undergo a consensus process across stakeholders using the Delphi technique to determine which items need to be considered in an effective trial feasibility checklist and how assessment results should be applied. Subsequently, empirical user testing and validation work is important. A similar tool development process has recently been successfully completed for an assessment of subgroup effect credibility [32]. Finally, evidence needed to be generated (e.g. a cohort of trials either randomised to using a feasibility checklist or not) in order to investigate whether the implementation of a feasibility checklist indeed leads to more successful trial conduct (e.g. measured by enrolment success). Furthermore, empirical research needs to establish how trial success is associated with individual items that appear relevant for study level feasibility. It might well be that some of these items are gatekeepers and, thus, more important than others for trial success (e.g. whether or not a pilot trial was conducted).
Conclusions
This scoping review identified ten publicly available checklists to assess the feasibility of RCTs. None of the available checklists appeared to be user-tested or validated. We extracted 48 distinct items for trial feasibility assessment that can be grouped into seven categories. Our results only describe the currently available checklists and suggested assessment items, and we make no recommendations at this stage of the project on how to assess the feasibility of a clinical trial. Instead, we provide the evidence-base for the transparent development of an improved checklist or tool for trial feasibility assessment, including user testing and validation and encourage relevant stakeholders, e.g. university hospitals, research ethics committees, funding agencies to get involved.
Availability of data and materials
Details on the information search and the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist can be found in a Supplementary information file. The raw data can be found in a Supplementary excel file.
References
Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet. 2001;357(9253):373–80.
Duley L, et al. Specific barriers to the conduct of randomized trials. Clin Trials. 2008;5(1):40–8.
Butryn T, et al. Keys to success in clinical trials: A practical review. International Journal of Academic Medicine. 2016;2(2):203–16.
The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) Database. INTEGRATED ADDENDUM TO ICH E6(R1): GUIDELINE FOR GOOD CLINICAL PRACTICE E6(R2). 2016. https://database.ich.org/sites/default/files/E6_R2_Addendum.pdf. Accessed Aug 2021.
Bertram W, et al. Optimising recruitment into trials using an internal pilot. Trials. 2019;20(1):207.
Eldridge SM, et al. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS ONE. 2016;11(3):e0150205.
Bernardez-Pereira S, et al. Prevalence, characteristics, and predictors of early termination of cardiovascular clinical trials due to low recruitment: insights from the ClinicalTrials.gov registry. Am Heart J. 2014;168(2):213–9 e1.
Kasenda B, et al. Prevalence, characteristics, and publication of discontinued randomized trials. JAMA. 2014;311(10):1045–51.
Briel M, et al. A systematic review of discontinued trials suggested that most reasons for recruitment failure were preventable. J Clin Epidemiol. 2016;80:8–15.
Carlisle B, et al. Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials. Clinical Trials. 2015;12(1):77–83.
Duley L, et al. What are the main inefficiencies in trial conduct: a survey of UKCRC registered clinical trials units in the UK. Trials. 2018;19:15.
Thabane L, et al. Methods and processes for development of a CONSORT extension for reporting pilot randomized controlled trials. Pilot Feasibility Stud. 2016;2:25.
Arksey H, O’Malley L. Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology: Theory and Practice. 2005;8(1):19–32.
Tricco AC, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.
Rajadhyaksha V. Conducting feasibilities in clinical trials: an investment to ensure a good study. Perspect Clin Res. 2010;1(3):106–9.
Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.
University of Calgary Clinical Trials. Clinical Study Feasibility Check List. https://ucalgary.ca/research/files/research/clinical-study-feasibility-checklist.docx. Accessed June 2021.
The University of North Carolina, Office of Clinical Trials. Feasibility Assessment. https://research.unc.edu/wp-content/uploads/sites/61/2020/04/Feasibilitiy-Assessment-V03-27-2020.doc . Accessed June 2021.
University Hospital Basel, Department of Clinical Research. Machbarkeitsabschätzung. https://studienwegweiser.dkfbasel.ch/print/196. Accessed June 2021.
University of Wisconsin-Madison, Institute for Clinical and Translational Research. Reasearch feasibility attestation form. https://ictr.wisc.edu/protocol-feasibility/ . Accessed June 2021.
Kings College Hospital National Health Service foundation trust United Kingdom. Feasibility Checklist. https://www.kch.nhs.uk/Doc/mi%20-%20132.4%20-%20feasibility%20proforma.docx. Accessed June 2021.
Health First. Clinical Research Feasibility Assessment Form A. https://hf.org/internal_access/clinical_research/11_hrmc_clin_res_ops_feasibility_assessment_forma_all_v3.doc. Accessed June 2021.
University of Florida Health Cancer Center Clinical Research Office. UFHCC CRO Feasibility Group Questionnaire. https://cancer.ufl.edu/files/2018/11/Feasibility-Group-Charter-FINAL-08Nov2018_Ver2.0.pdf. Accessed June 2021.
Association of Medical Research Charities (AMRC) and NIHR Medicines for Children Research Network (MCRN), National Institute for Health Research. Points to consider when assessing the feasibility of research. http://www.ct-toolkit.ac.uk/routemap/feasibility-and-investigator-selection/downloads/AMRC-MCRN-Feasibility-Guide-May2011.pdf. Accessed June 2021.
The Clinical Research Centre Ministry of Health. Checklist for Study Feasibility Assessment. https://www.crc.gov.my/wp-content/uploads/documents/intranet/SOP/pre_study/SOP_R_ISR_1_02/WS-ISR-1-02-01%20Checklist%20for%20Study%20Feasibility%20Assessment.docx. Accessed June 2021.
Clay-Williams R, Colligan L. Back to basics: checklists in aviation and healthcare. BMJ Qual Saf. 2015;24(7):428–31.
von Niederhausern B, et al. Academic response to improving value and reducing waste: A comprehensive framework for INcreasing QUality In patient-oriented academic clinical REsearch (INQUIRE). PLoS Med. 2018;15(6):e1002580.
Lancaster GA. Pilot and feasibility studies come of age! Pilot Feasibility Stud. 2015;1(1):1.
Maas AI. Clinical trials: do we need criteria for pre-study feasibility assessment? Acta Neurochir (Wien). 2016;158(11):2045–6.
Haddaway NR, et al. The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. Plos One. 2015;10(9):e0138237.
Curkovic M, Kosec A. Bubble effect: including internet search engines in systematic reviews introduces selection bias and impedes scientific reproducibility. BMC Med Res Methodol. 2018;18(1):130.
Schandelmaier S, et al. Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses. CMAJ. 2020;192(32):E901–6.
Acknowledgements
Not applicable.
Funding
Benjamin Speich is supported by a return grant from the Swiss National Science Foundation (P300PB_177933). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
Viktoria Gloy: Conceptualization, Data curation, Formal analysis, Methodology, Writing—original draft. Benjamin Speich: Data curation, analysis, Writing- review and editing. Alexandra Griessbach: Data curation, Writing- review and editing. Ala Taji Heravi: Data curation, Writing- review and editing. Alexandra Schulz: Data curation, Writing- review and editing. Thomas Fabbro: Data curation, Writing- review and editing. Christiane Pauli Magnus: Data curation, Writing- review and editing. Stuart McLennan: Data curation, Analysis, Writing- review and editing. Wendy Bertram: Data curation, Analysis, Writing- review and editing. Matthias Briel: Conceptualization, Data curation, Analysis, Writing- review and editing. The author(s) read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
All authors read and approved the final manuscript and gave their consent for publication.
Competing interests
All authors have declared that no competing interests exist.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Gloy, V., Speich, B., Griessbach, A. et al. Scoping review and characteristics of publicly available checklists for assessing clinical trial feasibility. BMC Med Res Methodol 22, 142 (2022). https://doi.org/10.1186/s12874-022-01617-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12874-022-01617-6