Skip to main content

Analysis of the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic treatments

Abstract

Background

Pragmatic clinical trials (PCTs) are designed to reflect how an investigational treatment would be applied in clinical practice. As such, unlike their explanatory counterparts, they measure therapeutic effectiveness and are capable of generating high-quality real-world evidence. However, the conduct of PCTs remains extremely rare. The scarcity of such studies has contributed to the emergence of the efficacy-effectiveness gap and has led to calls for launching more of them, including in the field of oncology. This analysis aimed to identify self-labelled pragmatic trials of antineoplastic interventions and to evaluate whether their use of this label was justified.

Methods

We searched PubMed® and Embase® for publications corresponding with studies that investigated antitumor therapies and that were tagged as pragmatic in their titles, abstracts and/or index terms. Subsequently, we consulted all available source documents for the included trials and extracted relevant information from them. The data collected were then used to appraise the degree of pragmatism displayed by the PCTs with the help of the validated PRECIS-2 tool.

Results

The literature search returned 803 unique records, of which 46 were retained upon conclusion of the screening process. This ultimately resulted in the identification of 42 distinct trials that carried the ‘pragmatic’ label. These studies examined eight different categories of neoplasms and were mostly randomized, open-label, multicentric, single-country trials sponsored by non-commercial parties. On a scale of one (very explanatory) to five (very pragmatic), the median PCT had a PRECIS-2 score per domain of 3.13 (interquartile range: 2.57–3.53). The most and least pragmatic studies in the sample had a score of 4.44 and 1.57, respectively. Only a minority of trials were described in sufficient detail to allow them to be graded across all domains of the PRECIS-2 instrument. Many of the studies examined also had features that arguably precluded them from being pragmatic altogether, such as being monocentric or placebo-controlled in nature.

Conclusion

PCTs of antineoplastic treatments are generally no more pragmatic than they are explanatory.

Peer Review reports

Background

Randomized controlled trials (RCTs) are considered the bedrock of evidence-based medicine [1]. Their reliance on randomization as a core methodological principle renders them invaluable for assessing the benefits and risks of experimental therapeutic approaches, it has been argued [2]. Both individually and collectively, these studies generate data that shape clinical guidelines in many fields, including oncology. Although the first cancer RCTs were initiated by academic institutions, the growing complexity of the legal and ethical framework which governs the conduct of clinical trials has made it challenging to set up such studies with limited resources, resulting in the gradual expansion of the pharmaceutical industry’s role in developing antitumor medicines [3]. Demonstrating this shift towards the for-profit development of anticancer agents, a recent analysis showed that approximately 90% of RCTs which investigated systemic treatments against common cancers and whose results were published in major scientific journals in the past decade had been either fully or partially funded by companies [4].

The increasingly commercial nature of clinical research in oncology has been accompanied by a rise in the use of surrogate endpoints [4], which for the most part lack validation as predictors of overall survival or quality of life [5,6,7]. Moreover, trials undertaken by manufacturers to support the regulatory approval and market entry of antineoplastic drugs often feature comparator arms deviating from the standard of care [8,9,10] and recruit patients on the basis of stringent eligibility criteria that reduce the external validity of their findings [11,12,13]. Despite their inherent limitations, studies that incorporate elements like these into their design offer important insights into the efficacy of investigational cancer therapies, i.e. their effects measured under ideal circumstances, when they are applied in highly controlled and artificial environments [14,15,16]. Such trials, which are known as explanatory RCTs [17], address the question of whether an intervention can work, regardless of any external factors that could influence the treatment outcome [14, 18]. They are of particular interest to industry sponsors since they can present a product in the best possible light, and because regulators demand them as part of the requirements that need to be satisfied in order to obtain a marketing authorization [19].

Conversely, so-called pragmatic RCTs [17] investigate whether an intervention will actually work, taking into account additional aspects that could have an impact on the therapeutic response [14, 18]. As such, they provide clarity on the effectiveness of anticancer medicines, i.e. their effects assessed under real-life conditions, when they are employed in the clinic [14,15,16]. These studies are characterized by their patient-centered outcome measures, their clinically relevant comparator arms, their real-world settings, their routine follow-up schedules, and their inclusive recruitment of participants, bringing about heterogeneous samples [18, 20, 21]. Pragmatic RCTs produce valuable insights for patients and clinicians by filling existing evidence gaps and contributing to the realization of a learning health system [22]. Moreover, such trials are of great value to payers and healthcare policymakers as they can directly inform health technology assessment (HTA) and reimbursement-related decision-making [23, 24]. Nevertheless, studies of this nature are infrequently performed, judging by the number of self-labelled pragmatic trials reported in the literature [25,26,27]. According to one estimate [26], they represented fewer than 1% of all RCTs carried out between 1990 and 2010. Pharmaceutical companies are reluctant to conduct them due to their associated business risks, their omission from the list of studies that are standardly imposed on manufacturers, and the paucity of regulatory guidance on how they should be run, among other reasons [19, 20, 28]. Consequently, pragmatic RCTs have so far mainly been undertaken independently from commercial interests [28], using public funding when available [23, 29, 30].

The dominance of explanatory RCTs has contributed to the emergence of what has come to be known as the efficacy-effectiveness gap, i.e. the phenomenon where significant disparities may be seen between a treatment’s efficacy as recorded in pre-approval trials and its effectiveness as observed in clinical practice [31]. In the field of oncology, where the discrepancy between the survival rates of study participants and real-life patients receiving the same therapies can be large, this gap manifests itself particularly clearly [32,33,34,35,36]. Regulators are cognizant of its existence, but they have also dismissed calls for requiring marketing authorization applicants to submit data from pragmatic trials as part of their product dossiers, fearing that this would introduce more uncertainty into the evaluation of the application and lead to the rejection of potentially useful agents [37]. Nonetheless, such studies, being considered sources of high-quality real-world evidence [38, 39], are accepted by regulatory authorities if they allow sponsors to deliver on their post-approval commitments [39, 40]. Bridging the efficacy-effectiveness gap may necessitate an overhaul of the current medicines development paradigm and the creation of new models of partnership between the various stakeholders involved in the process of getting a drug from the bench to the bedside [41,42,43,44].

The distinction between pragmatic and explanatory RCTs is not always apparent: many trials exhibit characteristics of both [18]. In reality, clinical studies exist on a spectrum, ranging from fully pragmatic to fully explanatory, with most trials positioning themselves somewhere in between the two ends [18]. To determine an RCT’s place on this continuum, an instrument was developed that enables researchers to evaluate the degree of pragmatism displayed by the study, namely the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) tool [18]. This tool, which was revised after its original publication and is referred to as PRECIS-2 in its current version [21], makes its users appraise the trial in question across nine distinct domains (Table 1), each of which is rated on a scale from 1 (i.e. very explanatory) to 5 (i.e. very pragmatic). The cumulative score awarded then reflects the extent to which the study integrates facets of pragmatism into its design. PRECIS-2, which unlike its predecessor has been validated [45], is in principle intended to be applied prospectively by investigators, when they are writing a trial protocol, so that they are aware of the impact of the study’s setup on the generalizability of the results they will eventually acquire [18, 21]. However, the instrument can also be employed retrospectively, to grade pragmatic RCTs which have already been completed, in the context of a systematic review [46].

Table 1 Overview of the nine domains of the PRECIS-2 instrument, their descriptions, and the corresponding questions that users of the tool need to ask themselves when evaluating the degree of pragmatism exhibited by a particular trial [21, 47]

It has been noted that many self-described pragmatic trials of pharmacological interventions are placebo-controlled and/or double-blinded studies, sometimes performed in a single institution or hospital, examining substances that have not yet been licensed [25]. Given that these features inherently limit the real-life applicability of a trial’s findings, this observation suggests that the ‘pragmatic’ label is often misused, likely because claims of pragmatism can attract attention from readers [25, 48, 49]. Moreover, RCTs that call themselves pragmatic do not commonly explain why they deserve to be designated as such, and the use of the PRECIS-2 tool for this purpose remains extremely rare [50]. Some authors have advocated the conduct of pragmatic cancer trials to address the efficacy-effectiveness gap in oncology [41, 42, 44, 51,52,53,54], but it remains unclear what the current landscape of these studies looks like. In this article, we identify clinical trials from the literature that investigated antineoplastic treatments and that were tagged as pragmatic at least once. Subsequently, we assess whether they were justifiably labelled as such by applying the PRECIS-2 instrument.

Methods

Literature screening and study selection

We searched PubMed® and Embase® for publications which outlined the design and/or the results of clinical trials of antineoplastic interventions and whose titles, abstracts or index terms made reference to the pragmatic nature of the studies in question. To this end, a search strategy was developed which contained synonyms and variations of the terms ‘pragmatic clinical trial’ (hereinafter abbreviated as PCT) and ‘neoplasm’, including the corresponding MeSH® and Emtree® descriptors (Additional File 1). The list of PubMed® and Embase® entries that were retrieved on 3 February 2021 using this strategy was imported into EndNote® for removal of duplicates. After all duplicates were eliminated, the shortened list was uploaded to Rayyan®, a website which was designed to facilitate the process of reviewing scientific literature in a systematic manner. The titles and abstracts of the items catalogued in Rayyan® were screened by a team of five researchers (RS, KD, KK, JL, JV), with each individual item being examined by three different persons (half of the items by RS, KD and JV, and the other half by RS, KK and JL). Based on various inclusion and exclusion criteria that were defined a priori (Table 2), the researchers selected the publications that they thought should be scrutinized further, without being able to see each other’s selections. Once they had gone through the items in Rayyan® that were assigned to them, the researchers discussed their choices amongst each other (in two groups of three) and any disagreements were resolved through consensus. This procedure was repeated for the full text screening, ultimately leading to the identification of a number of distinct trials carrying the ‘pragmatic’ label.

Table 2 Overview of the inclusion and exclusion criteria applied during the literature screening and study selection process

Data extraction and PRECIS-2 scoring

Next, an in-depth analysis of the identified PCTs was undertaken by consulting publicly accessible sources of information about these studies, such as journal articles, conference posters and abstracts, records from ClinicalTrials.gov and other trial registries, study protocols, and dedicated trial websites, if available. These source materials were again reviewed by five researchers (RS, KD, KK, JL, JV), following the same method that was employed during the literature screening phases. The researchers collected specific data on the studies (Table 3) by filling in a standardized data extraction form. Simultaneously, they applied the PRECIS-2 tool according to the instructions provided by its creators [21, 46, 47] and graded the trials across the nine domains that this instrument encompasses, documenting the motivation behind each score given by quoting relevant excerpts from the source materials. If a domain could not be appraised due to a lack of information found, it was left blank and the trial-level score per domain was determined by adjusting the denominator to reflect the number of assessable domains (e.g. if two domains could not be evaluated and the remaining seven together received a score of 21, the trial-level score per domain would be 21/7 = 3). The researchers were blinded to each other’s data extraction forms and PRECIS-2 scores until they had all completed their review of the source materials. Eventually, they met to compare their forms and scores and to settle any conflicting interpretations by mutual agreement.

Table 3 Overview of the characteristics that were recorded for each trial included in the analysis, grouped by category

Data analysis and reporting

Once the separate data extraction forms had been merged into one and the PRECIS-2 scores had been agreed upon by the researchers, the data gathered were analyzed descriptively in Excel® and inferentially using IBM® SPSS® Statistics 28.0. For nominal variables (e.g. type of neoplasm, primary endpoint, comparator treatment, etc.), proportions were calculated. Ordinal (e.g. sample-level PRECIS-2 scores per domain), discrete (e.g. number of participants, study duration, number of assessable PRECIS-2 domains, etc.) and continuous (e.g. journal impact factor, trial-level PRECIS-2 scores per domain) variables were characterized by their median values, with the associated interquartile ranges (IQRs) being represented by their lower and upper boundaries. Mann–Whitney U tests were performed to verify whether trials with particular features displayed a PRECIS-2 score per domain that was significantly different from that of the studies that did not have those features. To probe the strength of the correlation between the sample size of the trials and their PRECIS-2 score per domain, Kendall’s tau-b coefficient was computed. The significance threshold was set at 0.05.

Results

Literature screening and study selection

The search strategy returned 1,075 results in total, of which 473 were obtained from PubMed® and 602 from Embase® (Fig. 1). After 272 duplicates were filtered out in EndNote®, the titles, abstracts and index terms of the remaining 803 publications were screened in Rayyan® for adherence to the prespecified eligibility criteria. 64 items were withheld for the full text screening, which eventually led to the identification of 42 distinct trials that had been tagged as pragmatic at least once, either by the authors themselves or by the PubMed® and Embase® reviewers that manually assign relevant MeSH® and Emtree® descriptors to new entries in these two repositories.

Fig. 1
figure 1

Flowchart of the literature screening and study selection process

Characteristics of included trials

The characteristics of the included trials are summarized in Table 4. The completed data extraction form outlining the information collected for each individual trial is provided in Additional File 2.

Table 4 Overview of the characteristics of the 42 trials included in the analysis

General aspects

The 42 identified PCTs encompassed a wide variety of different neoplasms, which could be categorized into eight distinct groups (Fig. 2A): gastrointestinal neoplasms (9/42, 21.4%), breast neoplasms (9/42, 21.4%), hematological neoplasms (6/42, 14.3%), gynecological neoplasms (4/42, 9.5%), genitourinary neoplasms (3/42, 7.1%), lung neoplasms (3/42, 7.1%), skin neoplasms (3/42, 7.1%), and neoplasms of the head and neck (2/42, 4.8%). Of the three studies that did not focus on a specific tumor type, two dealt with spinal metastases of various primary origins and one was a basket trial. The bulk (23/42, 54.8%) of the PCTs in our sample had been launched in the past decade (Fig. 2B). The oldest study was started in 1982, whereas the most recent one began recruiting participants in 2018. Seven trials were ongoing at the time of writing, and completed PCTs had run for a median of 5.5 years (IQR: 3–8). When results were available for a study (39/42, 92.9%), they had generally been published in high-impact journals (median impact factor of 9.162, with IQR of 4.853–41.316). For most of the PCTs (23/42, 54.8%), the study protocol was publicly accessible, usually as an appendix to a publication detailing the trial’s findings or as a separate article.

Fig. 2
figure 2

Breakdown of the included studies by (A) the types of neoplasms they focused on (with the area of each rectangle/tumor type being proportional to its share of the sample), B their decades of initiation, (C) the types of interventions they investigated, and (D) the extent to which the industry was involved in their conduct. The percentages shown may not add up to 100% due to rounding (A), or because the categories displayed are not mutually exclusive (C)

Design elements

Half (21/42, 50.0%) of the trials included in the analysis covered particular phases of the clinical development paradigm, with phase II (9/21, 42.9%) and phase III (7/21, 33.3%) studies being more prevalent than those of any other phases (I, I/II, II/III, or IV; 5/21, 23.8%). While most (31/42, 73.8%) of the examined PCTs employed randomization and could therefore be considered RCTs, non-randomized and single-arm trials together accounted for more than a quarter (11/42, 26.2%) of the studies scrutinized. The use of masking was exceptional: only two trials (4.8%) had implemented blinding schemes into their design, and in both cases neither the participants nor the investigators were at any point in the data collection process aware of which intervention(s) they had been allocated to receive or administer. All phase III PCTs (7/7, 100%) were randomized, open-label studies.

The sample mainly (30/42, 71.4%) consisted of trials that evaluated two treatment strategies against each other. For a majority (24/42, 57.1%) of the analyzed PCTs, at least one of the study interventions could be classified as pharmacological in nature (e.g. chemotherapy, targeted therapy, hormonal therapy; Fig. 2C). Many trials also investigated therapeutic approaches that comprised surgical (13/42, 31.0%) or radiotherapeutic (16/42, 38.1%) procedures. Over a third (15/42, 35.7%) of the studies explored combinations of different treatment modalities (e.g. intraoperative radiotherapy). If comparator arms were present in the PCTs (35/42, 83.3%), they typically reflected what constituted the standard of care or the most commonly used choice of therapy at the time the trials were initiated (28/35, 80.0%). Placebo controls were exclusively featured in the two blinded studies.

With regard to the outcome measures recorded within the trials, a majority (26/42, 61.9%) of the PCTs had been statistically designed to generate data on one or more surrogate endpoints (e.g. progression-free survival, disease-free survival, response rate, etc.), including all (9/9, 100%) of the phase II and a significant proportion (5/7, 71.4%) of the phase III studies. Overall survival was infrequently (6/42, 14.3%) adopted as a primary endpoint, but often (26/42, 61.9%) served as a secondary one instead. In most (23/42, 54.8%) trials, an assessment of patients’ quality of life was carried out, albeit rarely (1/23, 4.3%) with the intent of delivering on the study’s main objective. Some PCTs (5/42, 11.9%) had been set up to document other types of clinically meaningful outcomes, such as changes in the mobility status or pain response of their participants.

Organizational facets

The studies’ sample sizes, whether actual (for completed trials) or planned (for ongoing trials), varied significantly, ranging from 4 to 3,581. The median phase II PCT had 68 participants (IQR: 51–121), whereas the median phase III one had 501 (IQR: 231.5–748). Collectively, the trials that were reviewed in this analysis had a median of 161 patients (IQR: 87.5–538.5) taking part in them, ignoring two prematurely terminated PCTs from which no meaningful conclusions could be drawn, and assuming studies that were still running reached their accrual targets. Approximately three-quarters (31/42, 73.8%) of the trials had been conducted across multiple hospitals or institutions and could thus be described as multicentric. However, only a minority (13/42, 31.0%) of the PCTs had been rolled out in more than a single country. The three countries which saw the most studies being undertaken within their borders were the United Kingdom (16/42, 38.1%), the United States (8/42, 19.0%) and the Netherlands (7/42, 16.7%).

The legal sponsorship of the trials was almost always (39/42, 92.9%) assumed by an entity that did not stand to gain monetarily from their outcomes, such as a university hospital, a research consortium, a learned society, or a not-for-profit organization (Fig. 2D). University College London alone was the sponsor of six (14.3%) of the studies in our sample. The few (3/42, 7.1%) PCTs that were sponsored by companies examined interventions of a commercial nature, namely a type of gene therapy, a topical cream, and a chemotherapeutic agent. Nevertheless, many (12/39, 30.8%) of the trials that were not performed at the industry’s behest still received some degree of financial or material support from the manufacturers of the products they investigated (e.g. educational grants, free drug supplies, etc.).

PRECIS-2 scores

The completed data extraction form listing the PRECIS-2 scores given to each individual trial is provided in Additional File 2.

For the median PCT, 8 domains of the PRECIS-2 tool could be assessed (IQR: 7–9), resulting in a trial-level score per domain of 3.13 after rounding (IQR: 2.57–3.53). The scores awarded did not depend on whether or not the studies employed randomization (P = 0.059), took place in multiple countries (P = 0.276), or received support from the industry (P = 0.066). Moreover, PCTs for which a protocol was available were not graded significantly higher than those for which no such document could be retrieved (P = 0.294). Multicentric trials on the other hand garnered greater scores than their monocentric counterparts (P = 0.010).

A positive correlation of moderate strength was observed between the studies’ sample sizes (planned or actual) and their PRECIS-2 scores (τb = 0.234, P = 0.031). PCTs that were labelled as pragmatic in the index terms of their corresponding publications exclusively (n = 9) were rated as more explanatory overall (P = 0.023). The most pragmatic study included in the analysis was the UKHAN1 trial [55] (PRECIS-2 score per domain of 4.44, Fig. 3A), while the most explanatory one was the RASHEC RCT [56] (PRECIS-2 score per domain of 1.57, Fig. 3B).

Fig. 3
figure 3

PRECIS-2 wheels of (A) the highest-scoring, (B) the lowest-scoring, and (C) the median trial. The concentric circles represent the different possible scores that can be awarded to each domain of the PRECIS-2 tool, with the innermost one being equivalent to 1 and the outermost one corresponding with 5. The area of the polygons enclosed by the green lines connecting the ‘score dots’ is proportional to the degree of pragmatism exhibited by the studies in question. Note that for (B), the ‘recruitment’ and ‘flexibility (adherence)’ domains could not be graded, and the matching dots are therefore missing

At the sample level, the PRECIS-2 domains that accumulated the highest scores were ‘primary analysis’ (median of 4.5, with IQR of 4–5), ‘recruitment’ (median of 4, with IQR of 3–5) and ‘setting’ (median of 4, with IQR of 3–5), whereas the ones that accrued the lowest were ‘organization’ (median of 2, with IQR of 1–3) and ‘flexibility (adherence)’ (median of 2, with IQR of 2–3). A detailed breakdown of the median PRECIS-2 scores by domain is presented in Table 5 and displayed in Fig. 3C. In most cases (24/42, 57.1%), one or more domains of the PRECIS-2 instrument could not be appraised for a given PCT due to a lack of information provided in the source documents that were consulted for each study. Even when a protocol was accessible (23/42, 54.8%), the appraisal of certain domains was oftentimes (7/23, 30.4%) impossible for this reason. ‘Flexibility (adherence)’ (20/42, 47.6%) and ‘recruitment’ (21/42, 50.0%) were the least frequently assessable domains altogether, with the former sometimes being inapplicable (8/42, 19.0%). Just four domains were scorable across all trials, namely ‘eligibility’, ‘flexibility (delivery)’, ‘primary outcome’ and ‘setting’.

Table 5 Overview of the sample-level PRECIS-2 scores per domain. The last column of the table displays the number of times the individual domains could not be graded, either because they were not applicable to the studies in question (e.g. flexibility in terms of adherence cannot be assessed for trials of surgical techniques, as these are single-application interventions), or due to a lack of relevant information provided in the source documents that were consulted for each PCT

Discussion

To our knowledge, this is the first study that has analyzed the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic interventions. We found that such trials were heterogeneous in terms of the methodological and organizational features they displayed, and that they were overall only slightly more pragmatic than explanatory in their setup, as evidenced by their moderate PRECIS-2 scores. Additionally, we observed that in most cases, the PCTs could not be graded across all domains of the PRECIS-2 tool because their design and procedures had been inadequately reported in their corresponding publications.

Our analysis shows that the number of pragmatic-labelled studies initiated in the field of oncology has roughly doubled every decade since the 1980s. This rising trend suggests that the use of PCTs to investigate the effectiveness of antitumor therapies has become more popular over time. However, it is debatable whether this actually means that sponsors of trials examining antineoplastic interventions are increasingly embracing the concept of pragmatism, as it could also signify that the ‘pragmatic’ descriptor is being misapplied more frequently. In fact, the PRECIS-2 scores that were obtained seem to support the latter interpretation, and prior literature points in that direction as well [25, 50]. Nevertheless, the expanding role of real-world evidence in the development of anticancer treatments [57, 58] will likely spur the future conduct of PCTs, since these studies are considered to be some of the most robust sources of such evidence [38, 39].

Some authors [59,60,61,62,63,64,65,66,67,68,69,70,71,72] have employed the PRECIS-2 instrument as a critical appraisal tool to investigate the extent to which the trials that they included in their systematic or scoping literature reviews could be classified as PCTs. Others [73,74,75,76,77,78,79,80,81,82,83,84,85,86] have used it in a similar manner to us, with the aim of verifying whether studies that were tagged as pragmatic or that were running as part of a publicly funded PCT program deserved to carry the ‘pragmatic’ label. However, few publications [69, 71, 72] in which PRECIS-2 assessments were undertaken looked at trials of antineoplastic interventions, and only one reported the outcomes of an analysis that was comparable to ours in terms of its sample size and methodological depth. Sorigue and Kuittinen [72] evaluated the position of studies underpinning the therapeutic recommendations listed in the 2020 European Society for Medical Oncology clinical guidelines for follicular lymphoma on the pragmatic-explanatory continuum. In total, they incorporated 28 distinct trials into their review, more than half of which could only be scrutinized based on articles outlining their results due to the inaccessibility of their protocols. The median trial received a PRECIS-2 score per domain of 3.5, which is slightly higher than in our analysis. This is remarkable, given that the studies we examined had been explicitly labelled as pragmatic, whereas the ones appraised by Sorigue and Kuittinen [72] had not. Moreover, the sample-level PRECIS-2 scores per domain differed significantly across the two analyses: while ‘organization’ was for instance the joint-most explanatory domain for our trials, it was among the most pragmatic for those reviewed by Sorigue and Kuittinen [72]. These observations further illustrate how the ‘pragmatic’ label is often used in a misleading manner, offering no assurance that the study in question is more pragmatic than any given trial not carrying this tag. Nevertheless, it should be stressed that a direct comparison between our findings and those of Sorigue and Kuittinen [72] may not be fully appropriate: unlike us, Sorigue and Kuittinen [72] did not consider the ‘flexibility (adherence)’ and ‘recruitment’ domains of the PRECIS-2 instrument at all in their analysis, since they could not find any meaningful information on the basis of which these domains could be graded. Consequently, differences in scoring may be the result of divergence in the way the PRECIS-2 instrument was applied.

Concerning the methodological design of the PCTs in our analysis, two studies employed the so-called trials-within-cohorts (TwiCs) approach [87], namely VERTICAL [88] and RECTAL-BOOST [89]. These studies are therefore examples of cohort multiple randomized controlled trials (cmRCTs) [87, 90, 91]. In such trials, patients are followed prospectively as part of a large observational cohort and a randomly selected subset among them is given the choice to undergo the investigational intervention [87, 90, 91]. Participants who are randomized to remain within the cohort receive usual care and make up the control group of the study. Since these patients are not informed of the fact that a trial is being conducted and that treatment allocation has taken place, the recruitment process can be significantly more efficient and less contrived for cmRCTs than for regular RCTs [90,91,92]. Moreover, because their cohorts are intrinsically embedded into clinical practice, cmRCTs are standardly performed in a real-world setting [87, 90, 91]. As a result, cmRCTs tend to be very pragmatic with respect to the ‘recruitment’ and ‘setting’ domains of the PRECIS-2 tool. However, as TwiCs schemes are relatively new, investigators may not yet be very familiar with them. Consequently, additional training could be required, lowering the score that can be given for the ‘organization’ domain. Besides cmRCTs, our sample also contained a basket trial [93], namely TAPUR [94]. Here as well, the innovative nature and relative complexity of biomarker-driven studies [93] limit the degree to which they can be pragmatic in terms of their organization. Furthermore, the stringent criteria that such trials apply to determine which drugs should be administered to which patients [93] inherently restrict the level of flexibility that is allowed with regard to the delivery of the targeted agents in question. Nevertheless, basket trials can still be justifiably labelled as pragmatic, as long as they implement enough elements of pragmatism into their design.

Notably, over a quarter of the PCTs that were examined in this analysis could not be regarded as RCTs, either because they were single-arm trials or because their participants were not randomly assigned to their study arms. In principle, a lack of randomization does not necessarily exclude a trial from being pragmatic: in their seminal paper introducing the concept of pragmatism [17], Schwartz and Lellouch never specifically refer to randomized trials, only to trials in general. However, the authors of the PRECIS-2 tool explicitly state that their instrument is intended to be applied to RCTs [21], even though none of its domains directly pertain to the method used for treatment allocation. In fact, it can be argued that randomization is a highly artificial procedure, since in a real-life environment, patients and clinicians can actively choose which interventions they will undergo or administer. Nevertheless, RCT designs are methodologically indispensable for producing robust and actionable evidence which is free from confounding-related biases [2, 22]. Hence, it may be reasonable to consider randomization separately from the PRECIS-2 assessment, for example as part of a critical appraisal exercise based on the Cochrane risk-of-bias tool RoB 2 [95]. By randomizing at the site level rather than the individual participant level, RCTs can preserve their statistical strengths and at the same time emulate real-world practice more closely. This technique, which is referred to as cluster randomization [96], is commonly employed in PCTs [97] but was not featured in any of the studies in our sample.

Despite reviewing all of the available source documents for the included studies, we were unable to appraise most of the PCTs in our sample across every domain of the PRECIS-2 instrument. Other authors [63, 65, 68, 70, 72, 76, 79,80,81] have likewise struggled to conduct exhaustive PRECIS-2 evaluations within the context of a systematic or scoping literature review. The fields corresponding with the ‘flexibility (adherence)’ and ‘recruitment’ domains on the tool’s accompanying appraisal sheet seem to be especially difficult to complete, even when considering that the former domain is not universally applicable. The inability to fully grade a given trial due to missing information is a known limitation of retrospective PRECIS-2 assessments [98]. Examining the protocol in addition to publications describing the results of the study does not always solve the problem: in our analysis, almost one-third of the trials for which this document could be accessed had at least one outstanding PRECIS-2 domain. Echoing the proposals made by Dal-Ré et al. [25], we believe that PCT sponsors and investigators should (a) systematically and prospectively evaluate the degree of pragmatism exhibited by their studies using the PRECIS-2 tool, and (b) publish the scores given for each domain along with the underlying motivations so that journal editors, reviewers and readers can independently validate them.

None of the studies in our sample that were started after 2009 (i.e. the year in which the first version of the PRECIS instrument was released [18]) substantiated their claim of being pragmatic by reporting their PRECIS or PRECIS-2 scores. Whenever justification for the use of the ‘pragmatic’ label was provided, it mostly addressed just one or two domains of these tools, usually the ones relating to the nature of the eligibility criteria that were applied to select participants for the trial, the setting in which the study took place, or the level of flexibility that was permitted regarding the administration of the investigational intervention. Janiaud et al. [50] have made similar observations: in their review of 73 pragmatic-labelled RCTs whose results were published in 2016, they found that 45% offered no explanation for why they carried that label, and that only one trial publication included a PRECIS assessment. The studies that tried to justify their ‘pragmatic’ tag mainly did so by citing the same type of arguments that we came across in our analysis. There seems to be a general lack of awareness among PCT sponsors and investigators of the PRECIS(-2) instrument’s existence and of the fact that pragmatism is a multifaceted concept, encompassing all aspects of a trial’s design and organization. Additional efforts may be needed to educate these stakeholders on the hallmarks of PCTs. In light of the important role such studies can play in the decision-making of HTA agencies, payers, clinicians and patients, it is imperative that the ‘pragmatic’ descriptor is employed correctly in the literature.

Some authors [25, 98] have argued that certain design elements can preclude a trial from being pragmatic altogether. More specifically, it has been asserted that placebo-controlled, blinded, single-center and/or pre-approval studies can never be considered PCTs, since such characteristics would inherently render them explanatory. In our analysis, we encountered many trials that displayed at least one of these features. However, instead of immediately excluding the possibility that they could be pragmatic, we still subjected them to a full PRECIS-2 evaluation. That way, if a study scored low for one or two domains, it could nonetheless accrue a relatively high total score by exhibiting a greater degree of pragmatism across the other domains of the PRECIS-2 instrument. This approach is consistent with the recommendations formulated by the tool’s developers [46]. Nevertheless, if the core aim of a PCT is indeed to emulate clinical practice, we would agree that having a monocentric setup and making use of masking would prevent that aim from being achievable. Regardless of how the concept of pragmatism is interpreted, it is remarkable that a large number of pragmatic-labelled trials of antitumor treatments show such overt signs of actually being considerably explanatory.

Strengths of the present study include its wide scope, encompassing all types of neoplasms and interventions, its comprehensive review of virtually every document available on the included trials, and its methodological rigor stemming from its blinded and comparative approaches with respect to data extraction and PRECIS-2 scoring. However, this analysis also suffers from a number of limitations. Firstly, our literature search only allowed us to find studies that had been tagged as pragmatic in the titles, abstracts or index terms of their corresponding publications. This implies that we were unable to identify PCTs for which no results or design characteristics had been reported yet or which had not explicitly been labelled as pragmatic. Nevertheless, the former would likely not have been assessable using the PRECIS-2 instrument given the scarcity of accessible information on such trials, and the latter were not considered for inclusion as they would not have contributed to answering the main research question. To compose a detailed overview of the PCT landscape in oncology, a different search strategy would need to be used, such as the one described by Taljaard et al. [99]. Secondly, several domains of the PRECIS-2 tool were challenging to appraise without in-depth knowledge on the way in which usual care is organized for patients with specific tumors. Since none of the researchers involved in the PRECIS-2 assessment had any experience working as a clinician, some facets of clinical practice may have been unknown to them. To familiarize themselves with such aspects, they consulted additional source materials (e.g. journal articles, chapters from medical textbooks, websites targeted towards patients) where necessary. Lastly, our sample size was relatively small, which could have negatively affected the power of the statistical tests that were performed. As a result, we may have mistakenly failed to reject their null hypotheses on multiple occasions.

Conclusions

Pragmatic-labelled trials of antineoplastic interventions with published results and/or designs varied in their characteristics, but were mainly randomized, open-label, multicentric, single-country studies that were sponsored by non-commercial parties and that compared pharmacological therapies against standard-of-care treatments in terms of their potential to improve outcomes serving as surrogate markers for the survival and/or quality of life of patients with specific tumors. Despite having been tagged as pragmatic, the trials generally did not exhibit a high degree of pragmatism, judging by the relatively low PRECIS-2 scores they accrued. Some of the studies also displayed features that were arguably irreconcilable with their claims of being pragmatic, like having a monocentric setup or a placebo comparator arm. However, most of the trials were not described in enough detail in their corresponding publications to enable them to be appraised across all domains of the PRECIS-2 tool. Sponsors of future PCTs undertaken in the field of oncology and beyond should systematically justify their use of the ‘pragmatic’ label by prospectively evaluating the position of their studies on the pragmatic-explanatory continuum with the help of the PRECIS-2 instrument or its forthcoming successor, and such an evaluation should be independently reviewed and validated.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

cmRCT:

Cohort multiple randomized controlled trial

HTA:

Health technology assessment

IQR:

Interquartile range

PCT:

Pragmatic clinical trial

PRECIS:

PRagmatic Explanatory Continuum Indicator Summary

RCT:

Randomized controlled trial

TwiCs:

Trials-within-cohorts

References

  1. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128:305–10. https://doi.org/10.1097/prs.0b013e318219c171.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Collins R, Bowman L, Landray M, Peto R. The magic of randomization versus the myth of real-world evidence. N Engl J Med. 2020;382:674–8. https://doi.org/10.1056/nejmsb1901642.

    Article  PubMed  Google Scholar 

  3. Broes S, Saesen R, Lacombe D, Huys I. Past, current, and future cancer clinical research collaborations: the case of the European Organisation for Research and Treatment of Cancer. Clin Transl Sci. 2021;14:47–53. https://doi.org/10.1111/cts.12863.

    Article  PubMed  Google Scholar 

  4. Del Paggio JC, Berry JS, Hopman WM, Eisenhauer EA, Prasad V, Gyawali B, et al. Evolution of the randomized clinical trial in the era of precision oncology. JAMA Oncol. 2021;7:728–34. https://doi.org/10.1001/jamaoncol.2021.0379.

    Article  PubMed  Google Scholar 

  5. Kim C, Prasad V. Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration’s Approval of Oncology Drugs. Mayo Clin Proc. 2016;91:713–25. https://doi.org/10.1016/j.mayocp.2016.02.012.

    Article  Google Scholar 

  6. Prasad V, Kim C, Burotto M, Vandross A. The strength of association between surrogate end points and survival in oncology. JAMA Intern Med. 2015;175:1389–98. https://doi.org/10.1001/jamainternmed.2015.2829.

    Article  PubMed  Google Scholar 

  7. Gyawali B, Hey SP, Kesselheim AS. Evaluating the evidence behind the surrogate measures included in the FDA’s table of surrogate endpoints as supporting approval of cancer drugs. EClinicalMedicine. 2020;21:100332. https://doi.org/10.1016/j.eclinm.2020.100332.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Hilal T, Sonbol MB, Prasad V. Analysis of control arm quality in randomized clinical trials leading to anticancer drug approval by the US Food and Drug Administration. JAMA Oncol. 2019;5:887–92. https://doi.org/10.1001/jamaoncol.2019.0167.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Hilal T, Gonzalez-Velez M, Prasad V. Limitations in clinical trials leading to anticancer drug approvals by the US food and drug administration. JAMA Intern Med. 2020;180:1108–15. https://doi.org/10.1001/jamainternmed.2020.2250.

    Article  CAS  PubMed  Google Scholar 

  10. Naci H, Davis C, Savović J, Higgins JPT, Sterne JAC, Gyawali B, et al. Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014–16: cross sectional analysis. BMJ. 2019;366:l5221. https://doi.org/10.1136/bmj.l5221.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16:495. https://doi.org/10.1186/s13063-015-1023-4.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Jin S, Pazdur R, Sridhara R. Re-evaluating eligibility criteria for oncology clinical trials: analysis of investigational new drug applications in 2015. J Clin Oncol. 2017;35:3745–52. https://doi.org/10.1200/jco.2017.73.4186.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Unger JM, Hershman DL, Osarogiagbon RU, Gothwal A, Anand S, Dasari A, et al. Representativeness of Black Patients in Cancer Clinical Trials Sponsored by the National Cancer Institute Compared With Pharmaceutical Companies. JNCI Cancer Spectr. 2020;4:034. https://doi.org/10.1093/jncics/pkaa034.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217–24 https://doi.org/10.31887/dcns.2011.13.2/npatsopoulos.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Sedgwick P. Explanatory trials versus pragmatic trials. BMJ. 2014;349:g6694. https://doi.org/10.1136/bmj.g6694.

    Article  PubMed  Google Scholar 

  16. Roland M, Torgerson DJ. Understanding controlled trials: What are pragmatic trials? BMJ. 1998;316:285. https://doi.org/10.1136/bmj.316.7127.285.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48. https://doi.org/10.1016/0021-9681(67)90041-0.

    Article  CAS  PubMed  Google Scholar 

  18. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–75. https://doi.org/10.1016/j.jclinepi.2008.12.011.

    Article  PubMed  Google Scholar 

  19. Zwarenstein M, Treweek S. What kind of randomized trials do we need? J Clin Epidemiol. 2009;62:461–3. https://doi.org/10.1016/j.jclinepi.2009.01.011.

    Article  PubMed  Google Scholar 

  20. Tunis SR, Stryer DB, Clancy CM. Practical Clinical Trials. JAMA. 2003;290:1624–32. https://doi.org/10.1001/jama.290.12.1624.

    Article  CAS  PubMed  Google Scholar 

  21. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. https://doi.org/10.1136/bmj.h2147.

    Article  PubMed  Google Scholar 

  22. Simon GE, Platt R, Hernandez AF. Evidence from Pragmatic Trials during Routine Care — Slouching toward a Learning Health System. N Engl J Med. 2020;382:1488–91. https://doi.org/10.1056/nejmp1915448.

    Article  PubMed  Google Scholar 

  23. Neyt M, Christiaens T, Demotes J, Walley T, Hulstaert F. Publicly funded practice-oriented clinical trials: of importance for healthcare payers. J Comp Eff Res. 2016;5:551–60. https://doi.org/10.2217/cer-2016-0018.

    Article  PubMed  Google Scholar 

  24. Ratner J, Mullins D, Buesching DP, Cantrell RA. Pragmatic Clinical Trials: US Payers’ Views on Their Value. Am J Manag Care. 2013;19:e158-65 https://www.ajmc.com/view/pragmatic-clinical-trials-us-payers-views-on-their-value.

    PubMed  Google Scholar 

  25. Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018;16:49. https://doi.org/10.1186/s12916-018-1038-2.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials. 2012;9:436–46. https://doi.org/10.1177/1740774512450097.

    Article  PubMed  Google Scholar 

  27. Vallvé C. A critical review of the pragmatic clinical trial. Med Clin (Barc). 2003;121:384–8. https://doi.org/10.1016/s0025-7753(03)73957-8.

    Article  PubMed  Google Scholar 

  28. Buesching DP, Luce BR, Berger ML. The role of private industry in pragmatic comparative effectiveness trials. J Comp Eff Res. 2012;1:147–56. https://doi.org/10.2217/cer.12.9.

    Article  PubMed  Google Scholar 

  29. Weinfurt KP, Hernandez AF, Coronado GD, DeBar LL, Dember LM, Green BB, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol. 2017;17:144. https://doi.org/10.1186/s12874-017-0420-7.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, et al. A guide to research partnerships for pragmatic clinical trials. BMJ. 2014;349:g6826. https://doi.org/10.1136/bmj.g6826.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Nordon C, Karcher H, Groenwold RHH, Ankarfeldt MZ, Pichler F, Chevrou-Severac H, et al. The “Efficacy-Effectiveness Gap”: Historical Background and Current Conceptualization. Value Health. 2016;19:75–81. https://doi.org/10.1016/j.jval.2015.09.2938.

    Article  PubMed  Google Scholar 

  32. Phillips CM, Parmar A, Guo H, Schwartz D, Isaranuwatchai W, Beca J, et al. Assessing the efficacy-effectiveness gap for cancer therapies: A comparison of overall survival and toxicity between clinical trial and population-based, real-world data for contemporary parenteral cancer therapeutics. Cancer. 2020;126:1717–26. https://doi.org/10.1002/cncr.32697.

    Article  PubMed  Google Scholar 

  33. Cramer-Van Der Welle CM, Peters BJM, Schramel FMNH, Klungel OH, Groen HJM, Van De Garde EMW. Systematic evaluation of the efficacy–effectiveness gap of systemic treatments in metastatic nonsmall cell lung cancer. Eur Respir J. 2018;52:1801100 https://doi.org/10.1183/13993003.01100-2018.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Boyle JM, Hegarty G, Frampton C, Harvey-Jones E, Dodkins J, Beyer K, et al. Real-world outcomes associated with new cancer medicines approved by the Food and Drug Administration and European Medicines Agency: A retrospective cohort study. Eur J Cancer. 2021;155:136–44. https://doi.org/10.1016/j.ejca.2021.07.001.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Green AK, Curry M, Trivedi N, Bach PB, Mailankody S. Assessment of Outcomes Associated With the Use of Newly Approved Oncology Drugs in Medicare Beneficiaries. JAMA Netw Open. 2021;4:e210030. https://doi.org/10.1001/jamanetworkopen.2021.0030.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Soni PD, Hartman HE, Dess RT, Abugharib A, Allen SG, Feng FY, et al. Comparison of population-based observational studies with randomized trials in oncology. J Clin Oncol. 2019;37:1209–16. https://doi.org/10.1200/jco.18.01074.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Eichler HG, Abadie E, Breckenridge A, Flamion B, Gustafsson LL, Leufkens H, et al. Bridging the efficacy–effectiveness gap: a regulator’s perspective on addressing variability of drug response. Nat Rev Drug Discov. 2011;10:495–506. https://doi.org/10.1038/nrd3501.

    Article  CAS  PubMed  Google Scholar 

  38. Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel GJMW, Grobbee DE. Pragmatic trials and real world evidence: Paper 1. Introduction J Clin Epidemiol. 2017;88:7–13. https://doi.org/10.1016/j.jclinepi.2016.12.023.

    Article  PubMed  Google Scholar 

  39. Food and Drug Administration (FDA). Framework for FDA’s Real-World Evidence Program. 2018. https://www.fda.gov/media/120060/download. Accessed 7 Jun 2022.

    Google Scholar 

  40. European Medicines Agency (EMA). Scientific guidance on post-authorisation efficacy studies. 2016. https://www.ema.europa.eu/documents/scientific-guideline/scientific-guidance-post-authorisation-efficacy-studies-first-version_en.pdf. Accessed 7 Jun 2022.

    Google Scholar 

  41. Kempf E, Bogaerts J, Lacombe D, Liu L. “Mind the gap” between the development of therapeutic innovations and the clinical practice in oncology: A proposal of the European Organisation for Research and Treatment of Cancer (EORTC) to optimise cancer clinical research. Eur J Cancer. 2017;86:143–9. https://doi.org/10.1016/j.ejca.2017.08.028.

    Article  PubMed  Google Scholar 

  42. Lacombe D, Bogaerts J, Tombal B, Maignen F, Osipienko L, Sullivan R, et al. Late translational research: putting forward a new model for developing new anti-cancer treatments that addresses the needs of patients and society. Mol Oncol. 2019;13:558–66. https://doi.org/10.1002/1878-0261.12431.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Lacombe D, O’Morain C, Casadei B, Hill K, Mateus E, Lories R, et al. Moving forward from drug-centred to patient-centred research: A white paper initiated by EORTC and developed together with the BioMed Alliance members. Eur Respir J. 2019;53:1801870. https://doi.org/10.1183/13993003.01870-2018.

    Article  PubMed  Google Scholar 

  44. Lacombe D, Quaglio G, Lejeune S, Saesen R, Rübig P. Establishing treatment optimisation as part of personalised medicine development. Eur J Cancer. 2019;113:96–7. https://doi.org/10.1016/j.ejca.2019.03.014.

    Article  PubMed  Google Scholar 

  45. Loudon K, Zwarenstein M, Sullivan FM, Donnan PT, Gágyor I, Hobbelen HJSM, et al. The PRECIS-2 tool has good interrater reliability and modest discriminant validity. J Clin Epidemiol. 2017;88:113–21. https://doi.org/10.1016/j.jclinepi.2017.06.001.

    Article  PubMed  Google Scholar 

  46. Zwarenstein M, Thorpe K, Treweek S, Loudon K. PRECIS-2 for retrospective assessment of RCTs in systematic reviews. J Clin Epidemiol. 2020;126:202–6. https://doi.org/10.1016/j.jclinepi.2020.06.023.

    Article  PubMed  Google Scholar 

  47. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe K, Zwarenstein M. PRECIS-2 Toolkit. 2015. https://www.precis-2.org/help/documentation/toolkitdownload. Accessed 7 Jun 2022.

    Google Scholar 

  48. Dal-Ré R, Mentz RJ, Rosendaal FR. Thoughtful selection and use of scientific terms in clinical research: The case of “pragmatic” trials. J Investig Med. 2021;69:1056–8. https://doi.org/10.1136/jim-2021-001789.

    Article  PubMed  Google Scholar 

  49. Dal-Ré R. The misleading use of the term pragmatic in pre-licensing medicine trials. Eur J Clin Pharmacol. 2019;75:1033–5. https://doi.org/10.1007/s00228-019-02672-7.

    Article  PubMed  Google Scholar 

  50. Janiaud P, Dal-Ré R, Ioannidis JP. Assessment of Pragmatism in Recently Published Randomized Clinical Trials. JAMA Intern Med. 2018;178:1278–80. https://doi.org/10.1001/jamainternmed.2018.3321.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Mullins CD, Montgomery R, Tunis S. Uncertainty in Assessing Value of Oncology Treatments. Oncologist. 2010;15:58–64. https://doi.org/10.1634/theoncologist.2010-s1-58.

    Article  PubMed  Google Scholar 

  52. Banerjee R, Prasad V. Pragmatic trials with prespecified subgroups: what oncologists can learn from COVID-19. Nat Rev Clin Oncol. 2020;18:7–8. https://doi.org/10.1038/s41571-020-00448-y.

    Article  PubMed Central  Google Scholar 

  53. Koehler M, Donnelly ET, Kalanovic D, Dagher R, Rothenberg ML. Pragmatic randomized clinical trials: a proposal to enhance evaluation of new cancer therapies with early signs of exceptional activity. Ann Oncol. 2016;27:1342–8. https://doi.org/10.1093/annonc/mdw143.

    Article  CAS  PubMed  Google Scholar 

  54. Nipp RD, Yao NA, Lowenstein LM, Buckner JC, Parker IR, Gajra A, et al. Pragmatic study designs for older adults with cancer: Report from the U13 conference. J Geriatr Oncol. 2016;7:234–41. https://doi.org/10.1016/j.jgo.2016.02.005.

    Article  PubMed  Google Scholar 

  55. Tobias JS, Monson K, Gupta N, MacDougall H, Glaholm J, Hutchison I, et al. Chemoradiotherapy for locally advanced head and neck cancer: 10-year follow-up of the UK Head and Neck (UKHAN1) trial. Lancet Oncol. 2010;11:66–74. https://doi.org/10.1016/s1470-2045(09)70306-7.

    Article  PubMed  Google Scholar 

  56. Salehi S, Åvall-Lundqvist E, Legerstam B, Carlson JW, Falconer H. Robot-assisted laparoscopy versus laparotomy for infrarenal paraaortic lymphadenectomy in women with high-risk endometrial cancer: A randomised controlled trial. Eur J Cancer. 2017;79:81–9. https://doi.org/10.1016/j.ejca.2017.03.038.

    Article  PubMed  Google Scholar 

  57. Skovlund E, Leufkens HGM, Smyth JF. The use of real-world data in cancer drug development. Eur J Cancer. 2018;101:69–76. https://doi.org/10.1016/j.ejca.2018.06.036.

    Article  CAS  PubMed  Google Scholar 

  58. Greshock J, Lewi M, Hartog B, Tendler C. Harnessing Real-World Evidence for the Development of Novel Cancer Therapies. Trends Cancer. 2020;6:907–9. https://doi.org/10.1016/j.trecan.2020.08.006.

    Article  PubMed  Google Scholar 

  59. Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review Transl Behav Med. 2017;7:751–72. https://doi.org/10.1007/s13142-017-0502-4.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Lu L, Zhou L, Dong J, Xiang Y, Wen Z, Lu L, et al. The application of PRECIS-2 ratings in randomized controlled trials of Chinese herbal medicine. Oncotarget. 2017;8:107002–10. https://doi.org/10.18632/oncotarget.22204.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Le Tourneau N, Germann A, Thompson R, Ford N, Schwartz S, Beres L, et al. Evaluation of HIV treatment outcomes with reduced frequency of clinical encounters and antiretroviral treatment refills: A systematic review and meta-analysis. PLoS Med. 2022;19:e1003959. https://doi.org/10.1371/journal.pmed.1003959.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Titz M, Schwameis M, Kienbacher C, Buchtele N, Roth D, Duma A, et al. Generalisability of randomised trials evaluating perioperative β-blocker therapy in noncardiac surgery. Br J Anaesth. 2020;125:926–34. https://doi.org/10.1016/j.bja.2020.08.006.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Fitzpatrick C, Gillies C, Seidu S, Kar D, Ioannidou E, Davies MJ, et al. Effect of pragmatic versus explanatory interventions on medication adherence in people with cardiometabolic conditions: a systematic review and meta-analysis. BMJ Open. 2020;10:e036575. https://doi.org/10.1136/bmjopen-2019-036575.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Robinson NB, Fremes S, Hameed I, Rahouma M, Weidenmann V, Demetres M, et al. Characteristics of randomized clinical trials in surgery From 2008 to 2020: a systematic review. JAMA Netw Open. 2021;4:e2114494. https://doi.org/10.1001/jamanetworkopen.2021.14494.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Gastaldon C, Mosler F, Toner S, Tedeschi F, Bird VJ, Barbui C, et al. Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review PLoS One. 2019;14:e0222891. https://doi.org/10.1371/journal.pone.0222891.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Devos F, Foissac F, Bouazza N, Ancel PY, Tréluyer JM, Chappuy H. Study characteristics impacted the pragmatism of randomized controlled trial published in nursing: a meta-epidemiological study. J Clin Epidemiol. 2019;116:18–25. https://doi.org/10.1016/j.jclinepi.2019.07.017.

    Article  PubMed  Google Scholar 

  67. Sepehrvand N, Alemayehu W, Das D, Gupta AK, Gouda P, Ghimire A, et al. Trends in the Explanatory or Pragmatic Nature of Cardiovascular Clinical Trials Over 2 Decades. JAMA Cardiol. 2019;4:1122–8. https://doi.org/10.1001/jamacardio.2019.3604.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Choi MY, Barnabe C, Barber CE, Bykerk V, Pope JE, Hazlewood GS. Pragmaticism of Randomized Controlled Trials of Biologic Treatment With Methotrexate in Rheumatoid Arthritis: A Systematic Review. Arthritis Care Res. 2019;71:620–8. https://doi.org/10.1002/acr.23620.

    Article  Google Scholar 

  69. Xu C, Zhang Y, Chen L, Li WF, Tang SQ, Tang LL, et al. Association between outcome disparities and pragmatic features related to clinical trial and real-world settings in nasopharyngeal carcinoma: A population-based retrospective cohort study, 2006–2016. Radiother Oncol. 2020;151:306–13. https://doi.org/10.1016/j.radonc.2020.08.019.

    Article  PubMed  Google Scholar 

  70. Ettori-Ajasse I, Tatin E, Forbes G, Eldridge S, Dibao-Dina C. How pragmatic are the randomised trials used in recommendations for control of glycosylated haemoglobin levels in type 2 diabetic patients in general practice: An application of the PRECIS II tool. Trials. 2020;21:281. https://doi.org/10.1186/s13063-020-4215-5.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Mittal N, George A, Leanos L, Kent P. Children’s Oncology Group (COG) clinical trials as exemplars of pragmatism in clinical research. J Clin Oncol. 2017;35:e18188. https://doi.org/10.1200/jco.2017.35.15_suppl.e18188.

    Article  Google Scholar 

  72. Sorigue M, Kuittinen O. Robustness and pragmatism of the evidence supporting the European Society for Medical Oncology guidelines for the diagnosis, treatment, and follow-up of follicular lymphoma. Expert Rev Hematol. 2021;14:655–68. https://doi.org/10.1080/17474086.2021.1943351.

    Article  CAS  PubMed  Google Scholar 

  73. Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:23. https://doi.org/10.1186/s13063-016-1158-y.

    Article  Google Scholar 

  74. Varas-Doval R, Saéz-Benito L, Gastelurrutia MA, Benrimoj SI, Garcia-Cardenas V, Martinez-Martínez F. Systematic review of pragmatic randomised control trials assessing the effectiveness of professional pharmacy services in community pharmacies. BMC Health Serv Res. 2021;21:156. https://doi.org/10.1186/s12913-021-06150-8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Lipman PD, Loudon K, Dluzak L, Moloney R, Messner D, Stoney CM. Framing the conversation: Use of PRECIS-2 ratings to advance understanding of pragmatic trial design domains. Trials. 2017;18:532. https://doi.org/10.1186/s13063-017-2267-y.

    Article  PubMed  PubMed Central  Google Scholar 

  76. Dal-Ré R. Articles provided insufficient information to conduct an appropriate retrospective assessment of the pragmatic/explanatory features of medicine trials with the PRECIS-2 tool. Eur J Clin Pharmacol. 2020;76:1093–102. https://doi.org/10.1007/s00228-020-02901-4.

    Article  PubMed  Google Scholar 

  77. Forbes G, Loudon K, Treweek S, Taylor SJC, Eldridge S. Understanding the applicability of results from primary care trials: lessons learned from applying PRECIS-2. J Clin Epidemiol. 2017;90:119–26. https://doi.org/10.1016/j.jclinepi.2017.06.007.

    Article  PubMed  Google Scholar 

  78. Dal-Ré R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2018;24:258–61. https://doi.org/10.1111/jep.12796.

    Article  PubMed  Google Scholar 

  79. Chen G, Ma Q. Development of A Precis-2 Scoring Grid: How Pragmatic are Pragmatic Trials of Surgical and Pharmaceutical Interventions? Value Health. 2015;18:A729–30. https://doi.org/10.1016/j.jval.2015.09.2781.

    Article  Google Scholar 

  80. Hohenschurz-Schmidt D, Kleykamp BA, Draper-Rodi J, Vollert J, Chan J, Ferguson M, et al. Pragmatic trials of pain therapies: A systematic review of methods. Pain. 2022;163:21–46. https://doi.org/10.1097/j.pain.0000000000002317.

    Article  PubMed  Google Scholar 

  81. Chan KW, Lee PW, Leung CP sha, Law YK, Gao L, Chan GC wang, et al. PRAgmatic Clinical Trial Design of Integrative MediCinE (PRACTICE): A Focus Group Series and Systematic Review on Trials of Diabetes and Kidney Disease. Front Med. 2021;8:668913. https://doi.org/10.3389/fmed.2021.668913.

  82. Palmer JA, Mor V, Volandes AE, McCreedy E, Loomer L, Carter P, et al. A dynamic application of PRECIS-2 to evaluate implementation in a pragmatic, cluster randomized clinical trial in two nursing home systems. Trials. 2018;19:453. https://doi.org/10.1186/s13063-018-2817-y.

    Article  PubMed  PubMed Central  Google Scholar 

  83. Brunette CA, Miller SJ, Majahalme N, Hau C, MacMullen L, Advani S, et al. Pragmatic Trials in Genomic Medicine: The Integrating Pharmacogenetics In Clinical Care (I-PICC) Study. Clin Transl Sci. 2020;13:381–90. https://doi.org/10.1111/cts.12723.

    Article  PubMed  Google Scholar 

  84. Carlson LE, Oberoi DV, Qureshi M, Subnis U. Integrative Oncology Trials in the Real World: Assessing the Pragmatism of an Ongoing Integrative Oncology Trial of Mindfulness and T’ai Chi/Qigong. J Altern Complement Med. 2018;24:926–32. https://doi.org/10.1089/acm.2018.0208.

    Article  PubMed  Google Scholar 

  85. Darker C, Loudon K, O’Connell N, Castello S, Burke E, Vance J, et al. An application of PRECIS-2 to evaluate trial design in a pilot cluster randomised controlled trial of a community-based smoking cessation intervention for women living in disadvantaged areas of Ireland. Pilot Feasibility Stud. 2022;8:19. https://doi.org/10.1186/s40814-022-00969-6.

    Article  PubMed  PubMed Central  Google Scholar 

  86. Jamali M, Frey MA. Are Findings From a Pragmatic Dry Needling Trial Always Applicable in the Real World? J Orthop Sport Phys Ther. 2021;51:470–1. https://doi.org/10.2519/jospt.2021.0202.

    Article  Google Scholar 

  87. TwiCs network. Trials within Cohorts – Facilitating Multiple Randomised Controlled Trials. https://www.twics.global. Accessed 7 Jun 2022.

  88. Pielkenrood BJ, van der Velden JM, van der Linden YM, Bartels MMT, Kasperts N, Verhoeff JJC, et al. Pain Response After Stereotactic Body Radiation Therapy Versus Conventional Radiation Therapy in Patients With Bone Metastases-A Phase 2 Randomized Controlled Trial Within a Prospective Cohort. Int J Radiat Oncol Biol Phys. 2021;110:358–67. https://doi.org/10.1016/j.ijrobp.2020.11.060.

    Article  PubMed  Google Scholar 

  89. Couwenberg AM, Burbach JPM, Berbee M, Lacle MM, Arensman R, Raicu MG, et al. Efficacy of Dose-Escalated Chemoradiation on Complete Tumor Response in Patients with Locally Advanced Rectal Cancer (RECTAL-BOOST): A Phase 2 Randomized Controlled Trial. Int J Radiat Oncol Biol Phys. 2020;108:1008–18. https://doi.org/10.1016/j.ijrobp.2020.06.013.

    Article  PubMed  Google Scholar 

  90. Relton C, Torgerson D, O’Cathain A, Nicholl J. Rethinking pragmatic randomised controlled trials: introducing the “cohort multiple randomised controlled trial” design. BMJ. 2010;340:963–7. https://doi.org/10.1136/bmj.c1066.

    Article  Google Scholar 

  91. van der Velden JM, Verkooijen HM, Ayoung-Afat D, Burbach JPM, van Vulpen M, Relton C, et al. The cohort multiple randomized controlled trial design: a valid and efficient alternative to pragmatic trials? Int J Epidemiol. 2017;46:96–102. https://doi.org/10.1093/ije/dyw050.

    Article  PubMed  Google Scholar 

  92. Couwenberg AM, Burbach JPM, May AM, Berbee M, Intven MPW, Verkooijen HM. The trials within cohorts design facilitated efficient patient enrollment and generalizability in oncology setting. J Clin Epidemiol. 2020;120:33–9. https://doi.org/10.1016/j.jclinepi.2019.12.015.

    Article  PubMed  Google Scholar 

  93. Tao JJ, Schram AM, Hyman DM. Basket Studies: Redefining Clinical Trials in the Era of Genome-Driven Oncology. Annu Rev Med. 2018;69:319–31. https://doi.org/10.1146/annurev-med-062016-050343.

    Article  CAS  PubMed  Google Scholar 

  94. Mangat PK, Halabi S, Bruinooge SS, Garrett-Mayer E, Alva A, Janeway KA, et al. Rationale and Design of the Targeted Agent and Profiling Utilization Registry Study. JCO Precis Oncol. 2018;2:1–14. https://doi.org/10.1200/po.18.00122.

    Article  Google Scholar 

  95. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. https://doi.org/10.1136/bmj.l4898.

    Article  PubMed  Google Scholar 

  96. Lorenz E, Köpke S, Pfaff H, Blettner M. Cluster-randomized studies Dtsch Arztebl Int. 2018;115:163–8. https://doi.org/10.3238/arztebl.2018.0163.

  97. Ford I, Norrie J. Pragmatic Trials. N Engl J Med. 2016;375:454–63. https://doi.org/10.1056/nejmra1510059.

    Article  PubMed  Google Scholar 

  98. Dal-Ré R, de Boer A, James SK. The design can limit PRECIS-2 retrospective assessment of the clinical trial explanatory/pragmatic features. J Clin Epidemiol. 2020;126:193–201. https://doi.org/10.1016/j.jclinepi.2020.03.027.

    Article  PubMed  Google Scholar 

  99. Taljaard M, McDonald S, Nicholls SG, Carroll K, Hey SP, Grimshaw JM, et al. A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity. J Clin Epidemiol. 2020;124:75–84. https://doi.org/10.1016/j.jclinepi.2020.05.003.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Robbe Saesen's work as a Fellow at EORTC Headquarters was supported by a grant from the EORTC Cancer Research Fund (ECRF). This publication was supported by a donation from Kom op tegen Kanker (Stand up to Cancer), the Flemish cancer society from Belgium. The funding sources had no role in the design of the study, the analysis and interpretation of the data, the writing of the manuscript or the decision to submit it for publication.

Author information

Authors and Affiliations

Authors

Contributions

RS, DL and IH were responsible for the conception and the design of the study. RS, KD, KK, JL and JV performed the literature review, extracted information from the source documents identified and carried out the PRECIS-2 assessment. RS analyzed the data and prepared the manuscript. All authors reviewed the manuscript and provided comments. RS revised the manuscript based on those comments. All authors approved the submitted version of the manuscript.

Corresponding author

Correspondence to Robbe Saesen.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saesen, R., Depreytere, K., Krupianskaya, K. et al. Analysis of the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic treatments. BMC Med Res Methodol 23, 148 (2023). https://doi.org/10.1186/s12874-023-01975-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-023-01975-9

Keywords

  • Pragmatic trials
  • Explanatory trials
  • Randomized controlled trials
  • Oncology
  • Cancer
  • Neoplasms
  • Treatments
  • Efficacy-effectiveness gap
  • Real-world evidence
  • PRECIS-2