High-quality evidence regarding the risks and benefits of treatments is lacking across a multitude of clinical specialties [1–7], and the clinical research enterprise is not focused on generating the kind of information that drives national guidelines for clinical care. Similarly, traditional clinical trials are often unable to provide high quality evidence in a timely or cost-effective manner [8, 9]. High quality evidence is typically generated by conducting randomized controlled trials (RCTs) using costly stand-alone, non-reusable systems, which may be separate from clinical care, designed with rigorous inclusion and exclusion criteria, and conducted under ideal conditions [10]. These design features are a double-edged sword: they help ensure that findings reflect true variation as a result of an intervention, but have been criticized as being so specific as to not be generalizable to broader populations and settings. Many are calling for a change, including the National Academies of Medicine (formerly the Institute of Medicine), to a system in which data for research are gathered during routine clinical care to enable continual learning, i.e., a “learning healthcare system” [11].

Pragmatic clinical trials (PCTs) are trials that use data collected in the electronic health record (EHR) as part of routine care, or are “embedded” in routine care, and are a foundational component of such a system. By their nature, PCTs are designed to show real-word effectiveness in broad, generalizable patient groups (as opposed to the more restricted protocols and populations found in exploratory randomized trials) [12]. These PCTs have the potential to significantly decrease the evidence gap and inform real-world practice with digital health data collected at the point of care. They involve critical partnerships between health care systems and academic investigators to embed clinically meaningful research questions into the infrastructure of the health system to generate real-world generalizable results in an efficient manner.

Although there have been numerous publications describing nuanced challenges and solutions encountered in the conduct of specific Demonstration Projects, this is the first to widen the focus and describe the NIH Collaboratory more generally. We build on the knowledge created by the Cores to review important generalizable lessons learned over the first four years since the Collaboratory was established. We also discuss the remaining opportunities and challenges for the Collaboratory and its forthcoming projects, and for the medical community more broadly regarding PCTs to power a learning health system.

To provide expertise to the NIH Collaboratory Demonstration Projects, the following working groups, or Cores were created to explore key elements of PCTs: electronic health records; phenotypes, data standards and data quality; patient-reported outcomes; health care systems interactions; ethics & regulatory; biostatistics and study design; and stakeholder engagement. The Collaboratory Coordinating Center  at the Duke Clinical Research Institute (with colleagues at the Johns Hopkins Berman Institute of Bioethics, Harvard Pilgrim Health Care Institute, Group Health Research Institute/Kaiser Permanente Washington Health Research Institute, Center for Medical Technology Policy, and Medical College of Wisconsin) serves 3 roles: (1) to provide support to the demonstration projects to help ensure success; (2) to cull experiences and create generalizable solutions regarding challenges in designing, conducting, and disseminating results from PCTs; and (3) to disseminate lessons learned, tools, and other resources to the broader community, including the creation of a Knowledge Repository and the Living Textbook of Pragmatic Clinical Trials: Rethinking Clinical Trials .

There have been two funding opportunities so far, one in 2012 and one in 2014, and we expect another funding opportunity in 2017 to support 10 additional PCTs. In 2012, seven demonstration projects were funded (RFA-RM-12-002, Table 1), and in 2014, three additional demonstration projects were added to the Collaboratory to focus on interventions that improve the health of patients with multiple chronic conditions (RFA-RM-13-012). In recognition of the challenge of conducting trials in ongoing clinical care settings and other uncertainties associated with their design, a two-phase, cooperative agreement approach was used for their funding. In the one-year planning phase, the demonstration project investigators refined their study protocols in cooperation with other experts within the Collaboratory network and Core groups, piloted aspects of the proposed designs, and developed revised proposals for their projects that were submitted to NIH. Projects deemed acceptable were funded for the four-year execution phase. This two-phase approach was beneficial in that it (1) drew on the broader intellectual community of the Collaboratory to contribute to the planning of each project, (2) highlighted the questions and uncertainties that arise during the planning of PCTs within healthcare systems, and (3) increased the likelihood that the projects would be successful in answering their scientific question.

Because the Collaboratory pragmatic trial interventions are tested in the context of routine medical care, data are collected from the EHR and claims, and the studies involve large sample sizes, the engagement of healthcare delivery organizations, providers, researchers, and representatives from the NIH (12 NIH institutes and centers are involved) has been critical. The Collaboratory’s success depends entirely on the willingness of investigators to share their challenges in real time with the rest of the network, including the NIH staff. This differs from many research enterprises, in which investigators often feel motivated to present an image of perfection to everyone outside of their project and especially to the sponsors. The Collaboratory projects were selected with the understanding that they would generate challenges that the research community must learn to overcome. The transformation of challenges into lessons learned that are then shared with the broader community is a direct result of the good faith, generosity, and candor of the Collaboratory demonstration project investigators and their health system partners.

The information regarding lessons learned was gleaned from a number of sources. The Collaboratory held yearly Steering Committee meetings (Bethesda, MD, 2012–2107) in which all the Principal Investigators (PIs) gather to share their experience and knowledge through interviews and presentations. Members of the Coordinating Center interview each of the PIs at these meetings to gather lessons learned, and each PI presents lessons learned at a general session followed by an opportunity for discussion. Leaders of each of the Cores also hold monthly calls with the PIs and their teams to discuss problems and explore solutions, and these lessons are gathered as a part of minutes from the call. In addition, all the PIs and some members of their teams are authors on this paper.

In a traditional randomized, controlled trial, it is possible to exert significant control over what happens in the different arms of the trial. Because the demonstration projects are tackling issues that are major public health concerns, such as opioid prescribing and hospital-based infections, competing initiatives are likely to be launched at some of the study sites. Conversely, providers in the experimental arm might resist the intervention due to burden. Providers in the “usual care” arm might voluntarily adopt the trial intervention, especially if there is growing evidence that the intervention is associated with better outcomes. There were a number of opioid reduction related initiatives rolled out during the implementation of PPACT, thereby changing the nature of the underlying usual care. In the words of Lynn DeBar, PI of the PPACT trial: “Our control is not controlled. We can’t control it, and that’s a big challenge [48].” For another example, because observational data suggest that longer hemodialysis sessions are beneficial, dialysis units in the TiME trial, including some of those randomized to usual care, have increased session durations for their patients [41]. Staff turnover might result in interventions being implemented in different ways over time and across sites. As an example, in the PROVEN trial, some participating nursing homes experienced multiple changes in the advance care planning champion in the first year of the study, requiring retraining and interruptions in patients and family members being offered the videos. It is not surprising, therefore, that facilities with more champion turnover had lower rates of showing the video, particularly to long-stay residents. For another example, in the PPACT trial competing local initiatives requiring highly skilled nurses or behavioral specialists resulted in some instability in the intervention staffing and, consequently, some variability in disciplines represented on the core interdisciplinary intervention team across performance sites over time. These and other factors make it necessary to carefully monitor the activities in all of the sites involved in the study, so that at the very least it will be possible to describe the nature of these changes that might affect the difference between the arms.

Reliance on the EHR for data to support research investigations is challenging for many reasons. First, even when sites are a part of a single corporate entity, local coding varies, and cross-site data standardization is essential. Because no two electronic health record (EHR) systems are alike, the solution requires engagement of local data experts. Tools, such as the ones developed for SPOT do not transfer from one site to another, and local adaptation is necessary. Each health system may have a distinct process or pathway for implementation of an intervention, and a multi-site trial will need to adapt to those. In the case of the SPOT trial, even though all of the sites use Epic, they have different preferred methods for implementing standardized assessment tools, and the investigators needed to adapt to those different preferred pathways.

Additionally, because the EHR is optimized for clinical practice and sometimes billing, integrating study-related data elements into the EHR has implications for clinical workflow and compliance. A seemingly small change to the work flow to accommodate a research project likely could have major implications for the IT system. The investigators in LIRE, a study that adds epidemiological benchmarks to lumbar spine imaging reports, found that even minor modifications to reports met with initial resistance since these required allocations of scarce IT resources. Overcoming this sort of hurdle required close coordination with site PIs and local leadership buy-in. Some cases offer a different scenario: for PROVEN, the two corporate partners use a different EHR but with the help of devoted efforts from corporate leadership, were able to add a new record into their EHR system to permit staff to document each time the video was offered and subsequently shown to the patient or family members.

The NIH Collaboratory is an important part of the overall effort to improve the national capacity to generate evidence to inform healthcare decisions by patients, providers, and payers. Even before the demonstration projects have completed their work, much has been learned. Planning activities are substantial and should be supported as part of a phased approach, such as the planning and implementation phases used by NIH for the Collaboratory. New challenges frequently arise during execution; change is the only constant, and because of this, early and ongoing engagement with all stakeholders is critical. As the new demonstration projects begin, we hope to learn from these lessons and generate others, as there remain opportunities for additional learning in order to make intelligent investments in PCTs. There is also the need to train a new clinical trials workforce, to train reviewers for PCTs, and align interests for all stakeholders to contribute to a national evidence generation system.