Skip to main content

Advertisement

Readiness assessment for pragmatic trials (RAPT): a model to assess the readiness of an intervention for testing in a pragmatic trial

Article metrics

Abstract

Background

Pragmatic randomized, controlled trials (PCTs) test the effectiveness of interventions implemented in routine clinical practice. Because PCT findings are generalizable, this approach is gaining momentum among interventionists and funding agencies seeking to accelerate the testing and adoption of evidence-based strategies to improve care and outcomes. Particular attention is being paid to non-pharmacological interventions, which are often complex and may be difficult to uniformly implement across multiple sites. While many such non-pharmacological interventions have proven efficacious in small trials, most have not been widely adopted. PCTs could accelerate effectiveness testing and adoption, yet there are no established criteria to identify interventions ready for testing in a PCT.

Methods

We convened 30 interventionists and healthcare leaders to identify criteria to assess the readiness of non-pharmacological interventions for PCTs. Based on this discussion, we created a model with multiple domains, qualitative scoring guidelines for each domain, and a graphical summary of readiness assessments. All workshop participants had an opportunity to review and comment on the resulting model; three piloted it with their own interventions. Several other experts also provided input.

Results

The Readiness Assessment for Pragmatic Trials (RAPT) model enables interventionists to assess an intervention’s readiness for PCTs. RAPT includes nine domains: implementation protocol, evidence, risk, feasibility, measurement, cost, acceptability, alignment, and impact. Domains reflect a range of considerations regarding the feasibility of successfully employing PCT methods and the prospect of an intervention’s widespread adoption, if proven effective. Individuals evaluating an intervention are asked to qualitatively assess each domain from low to high readiness. In this report, we provide assessment guidelines and examples of scored interventions.

Conclusions

RAPT is the first model to help interventionists and funders assess the extent to which interventions are ready for PCTs. Scoring efficacious interventions using RAPT can inform research team discussions regarding whether or not to advance an intervention to effectiveness testing using a PCT and how to design that PCT.

Background

Pragmatic randomized, controlled trials (PCTs), which test effectiveness under real-world conditions [1], may help to address the need to accelerate the adoption of evidence-based interventions. Because they can be undertaken relatively quickly and result in highly-generalizable findings, PCTs are gaining favor among interventionists and funding agencies. Since 2012, the National Institutes of Health (NIH) Common Fund has been strengthening national capacity to conduct PCTs in healthcare systems through its Health Care System Collaboratory, which captures and publishes best practices from more than a dozen NIH-funded PCTs [2]. More recently, the National Institute on Aging (NIA) and other stakeholders have sought to stimulate PCTs focused specifically on non-pharmacological dementia interventions, [3] recognizing that few efficacious dementia interventions have been replicated.

While a widely-adopted framework, the Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) [4] helps trialists assess the pragmatism of their study design when undertaking PCTs, there are no established criteria to assess efficacious interventions for their readiness for PCTs—and moving forward with a PCT when an intervention is not sufficiently “ready” can have serious consequences, ranging from wasted time and money to false conclusions. To address the need to formally assess interventions’ readiness for PCTs, we convened experts to recommend criteria that researchers should consider in decision-making and trial design. This paper presents the resulting Readiness Assessment for Pragmatic Trials (RAPT) model.

Methods

As an official activity following a federally-funded national research summit on dementia care [5] the National Institute on Aging sponsored a one-day expert workshop in December 2017. Part of the agenda centered on generating recommendations regarding criteria to assess the readiness of interventions for PCTs.

The workshop and its results, including infrastructure recommendations, are detailed elsewhere [6, 7]. Three of the authors served as chairs (RB, SM, and VM) and two presented and attended (EJ and EM). Participants (N = 30; listed in the Acknowledgements) included researchers with expertise relevant to the conduct of PCTs, such as data, regulatory, and ethical issues; interventionists; and healthcare leaders with experience translating evidence-based interventions into practice. Senior NIA staff (N = 6) also attended.

After the workshop, we summarized participants’ recommended criteria for determining the characteristics of interventions ready for PCTs [6, 7], and used recommended criteria to define domains to assess interventions for PCT readiness. Inspired by the PRECIS framework, we developed initial guidelines for assessing each domain and a graphical summary to plot domain assessments. We emailed the resulting model to the 30 workshop participants and asked them to review the content and to pilot test an assessment, if possible, with their own intervention.

Finally, we presented an updated draft to approximately 30 additional experts, including interventionists, who attended a Brown and Hebrew SeniorLife seminar in a series focused on the design and execution of pragmatic and cluster-randomized trials. The final model reflects discussion from this seminar.

Results

We made minor iterative revisions to RAPT based on feedback first from NIA workshop participants and later from Brown and Hebrew SeniorLife seminar attendees. Of the 30 workshop participants, seven provided thorough comments and three pilot tested it with their interventions. Most suggestions involved minor wordsmithing changes, with the exception of shifting the assessment from a numeric to a qualitative scale; we made this final change following the group discussion at the Brown and Hebrew SeniorLife seminar. The resulting model includes nine domains that reflect a range of considerations regarding the feasibility of successfully employing PCT methods to test a non-pharmacologic intervention and the prospect of an intervention’s widespread adoption, if proven effective. The model asks individuals assessing an intervention to score each domain on a spectrum from low to high and enables them to summarize results graphically (Fig. 1). Table 1 provides scoring guidance to consider.

Fig. 1
figure1

The blank Readiness Assessment for Pragmatic Trials (RAPT) graphical summary “wheel”

Table 1 Readiness Assessment for Pragmatic Trials (RAPT) domains and scoring guidance

Implementation protocol

Is the intervention protocol sufficiently detailed to be replicated? In this context, protocol refers to an operations manual or implementation guide, not the Institutional Review Board submission. To be ready for a PCT, an intervention should have a well-documented protocol. Ideally, an intervention’s outcomes should be measurable and captured using existing data sources or systems, so that the intervention – if proven effective – can be readily adopted on a broad scale. Although it may be possible to make minor revisions to existing systems, such as electronic health records, major revisions and any data collection that would burden staff will affect widespread adoption.

Evidence

To what extent does the evidence base support the intervention’s efficacy? Ideally, there should be an evidence base establishing efficacy using rigorous methods, such as randomized, controlled trials (RCTs). Efficacy studies may also have demonstrated which intervention components, alone or in combination, are associated with improvement. An intervention with mixed findings or less robust evidence may not be sufficiently efficacious for a PCT. Furthermore, overwhelming evidence for efficacy doesn’t necessarily translate to overwhelming evidence for effectiveness. While an intervention may work in a controlled setting, it is vital to test the intervention in a real-world context (such as a PCT).

Risk

Is it known how safe the intervention is? PCTs often involve providers implementing interventions as a new standard of care with all patients. An intervention should therefore be of minimal risk, with careful consideration given to potential adverse events and unintended consequences. This is particularly true when an intervention targets people who are vulnerable, such as those with ADRD or residing in nursing homes and other residential care settings.

Feasibility

To what extent can the intervention be implemented under existing conditions? An intervention should be feasible for providers and healthcare systems to implement as part of routine clinical care, with existing staff, infrastructure, and other resources, such as reimbursement models. If resources are absent or insufficient, this could affect implementation both in the PCT and when the intervention is broadly disseminated.

Measurement

To what extent can outcomes be captured during the conduct of the PCT? Ideally, an intervention’s outcomes should be measurable and captured using existing data sources or systems, to ensure that the PCT can be implemented and evaluated without resource-intensive primary data collection. Existing data systems may also be useful for ongoing audit and feedback to drive the intervention’s adoption and spread. Although it may be possible to make minor revisions to existing systems, such as electronic health records, major revisions and any data collection that would burden staff will affect widespread adoption.

Cost

How likely is the intervention to be economically viable? The benefits of an intervention should outweigh its costs, so that there is a realistic business case for providers’ implementation or reimbursement. Without making the business case, providers are unlikely to widely adopt an intervention, even if it demonstrates effectiveness in a PCT.

Acceptability

How willing are providers likely to be to adopt the intervention? Leadership and frontline staff should believe that an intervention is feasible in their unique environment and workflow and addresses a need. While providers’ perceptions of acceptability may be driven, in part, by evidence demonstrating an intervention’s impact, some baseline understanding of their willingness to adopt the intervention during effectiveness testing will help when determining how likely the intervention is to be implemented fully (with fidelity) or once proven effective.

Alignment

To what extent does the intervention align with external stakeholders’ priorities? While acceptability focuses on providers, alignment focuses on external stakeholders: policymakers, payors, patients and their families, advocates, and others. An intervention should address a current or anticipated priority for most or all external stakeholders. For example, it may be a topic included in state or national policy, payment, or programs. Without alignment, the intervention is unlikely to be widely adopted.

Impact

How useful will the results be? As with alignment, this domain relates to the likelihood of the intervention being widely adopted; it differs in that it focuses on the results of the PCT vs. the need for the intervention itself. Providers and stakeholders (policymakers, payors, advocates, and others) should believe that the outcomes will be useful to inform clinical care and policy. An intervention is unlikely to be widely adopted if results are not likely to be perceived as useful.

Scored examples

Figure 2 presents graphical summaries of RAPT assessments for the three non-pharmacological dementia interventions that serve as examples: A) the MUSIC & MEMORY individualized music program for nursing home residents with dementia, B) Advance Care Planning Specialist Program, and C) Reserve for Delirium Superimposed on Dementia.

Fig. 2
figure2

Examples of three scored interventions

MUSIC & MEMORY (A) involves providing individualized music to persons with dementia using personalized music devices [8] and became popular among nursing homes in part because of a widely-viewed documentary, Alive Inside, showcasing the intervention [9]. It scores highest on domains related to alignment and acceptability, because it addresses national priorities related to improving dementia care in nursing homes and is already widely adopted by providers familiar with anecdotal results and seeking strategies to improve care for residents with ADRD. It scores lowest on domains related to the intervention’s evidence and implementation protocol, because it has not been prospectively evaluated using rigorous methods and existing training materials, while numerous, stop shy of a detailed protocol. Several of the authors (RB, EM, and VM) are currently undertaking a PCT of MUSIC & MEMORY that included an initial phase designed to address these weaknesses, focused on developing a protocol and pilot testing evaluation strategies [10].

The Advance Care Planning (ACP) Specialist Program (B) is a nursing home intervention that involves providing standardized staff education and strengthening facilities’ procedures, to enable systematic advance care planning facilitation by existing staff [11, 12]. The interventionists are currently implementing a PCT applying the intervention to nursing home residents with ADRD [13]. Because it aims to strengthen a process that is already occurring as part of standard practice, the ACP Specialist Program is low risk and scores high on that domain. It also scores high for alignment, because nursing home staff widely recognize advance care planning as valuable, but challenging, and welcome assistance improving the process. In contrast, medium scores for several domains, such as feasibility and measurement, reflect the fact that the intervention requires nursing home leaders to review and revise existing policies, explicitly commit staff time (which may require shifting commitments), and modify electronic documentation to capture advance care planning.

Reserve for Delirium Superimposed on Dementia (DSD) (C) is an intervention that involves using cognitive stimulation to improve outcomes among persons with dementia receiving post-acute care. While an RCT comparing Reserve for DSD to usual care found no differences in the duration and severity of delirium, it did find better executive function (an important cognitive domain affected by delirium) and a shorter post-acute care length of stay in the intervention group vs. a control group [14]. The intervention scores highest for alignment; most stakeholders believe delirium is a significant clinical problem for persons with dementia because of its effects on health and cost outcomes. It also scores high for protocol, because the RCT involved developing an interventional manual, producing a video on implementing the intervention, and publishing the protocol [15]. The lowest score pertains to feasibility, since the RCT protocol involved using research staff to implement the intervention.

Discussion

RAPT is the first model to help interventionists and funding agencies, among others, assess the extent to which interventions are ready for effectiveness testing in routine clinical practice using PCT methods. Assessing an intervention using RAPT may inform discussion and decisions about whether to proceed with a PCT and, in conjunction with PRECIS, how best to design that PCT, including any preliminary work necessary to complete prior to proceeding.

This model arose from discussion at a NIA-funded workshop that focused on non-pharmacological dementia interventions. A federally-funded research summit in 2017 concluded that improving care for persons with dementia is an urgent public health challenge requiring high-quality evidence, [5] yet few efficacious studies have been consistently replicated [16]. The NIA workshop, which followed the summit, therefore focused on interventions and PCTs targeting this population. Nonetheless, we believe RAPT is broadly applicable to interventions across all clinical fields: none of the nine domains are specific to dementia and many of those who helped to vet the model were interventionists whose research does not center on dementia.

An intervention’s RAPT assessment can be plotted on a wheel, allowing interventionists to graphically summarize all nine domains and to quickly identify any domains requiring particular consideration. Some RAPT domains may be unknown prior to a PCT (e.g., risk or evidence) and will subsequently receive a low assessment; however, this alone should not prevent researchers from moving forward with a PCT. We do not propose any specific weighting or any thresholds for “passing” the overall assessment and likewise do not recommend directly comparing interventions’ assessments to one another. Rather, we suggest that each intervention’s readiness for a PCT be considered based on its individual strengths and weaknesses, with results used to inform the research team’s discussion about whether or not to advance an intervention to a PCT and—together with PRECIS—how to design that PCT. In other words, a low assessment may not preclude conducting a PCT, but could give insight into considerations that improve the PCT’s design and conduct.

To increase reliability, we also recommend having multiple people independently score an intervention and then compare results to resolve any discrepancies and inform discussion. While RAPT includes detailed scoring guidance, some domains are more subjective than others. For example, it is relatively straightforward to ascertain whether or not an intervention has a detailed implementation protocol, such as an operations manual (protocol domain). However, assessing providers’ beliefs about whether an intervention is feasible and needed (acceptability domain) may prove difficult for interventionists, unless the research team includes clinicians or others intimately familiar with the day-to-day realities of the targeted healthcare setting. Identifying ways to incorporate providers’ thoughts may be necessary; just as important as incorporating the view of patients and families into the initial development of interventions.

In developing RAPT, we applied it to our own intervention, Music & Memory, post-hoc. This allowed us to recognize weaknesses—particularly around feasibility—that would have been helpful to discuss in detail when designing the large PCT that we are now conducting in nursing homes across the country. Thankfully, our trial did include a pilot phase that enabled us to identify and address critical issues before proceeding with our full-scale PCT. The pilot was intended to include transforming training materials into a detailed implementation protocol, addressing an area in which the intervention did score low in our example. It ultimately also addressed issues we didn’t adequately foresee, particularly around the feasibility of implementing a seemingly “simple” intervention in low-resource nursing homes that lack quality improvement infrastructure, face frequent leadership turnover, and include numerous competing priorities. We incorporated pilot findings into the implementation protocol and trial design.

Conclusions

RAPT was developed based on expert recommendations and is the first model to help interventionists and funding agencies determine the extent to which interventions are ready for PCTs. Evaluating efficacious interventions using RAPT can inform research team discussion regarding whether or not to advance an intervention to effectiveness testing using a PCT and how do design that PCT.

Availability of data and materials

Not applicable.

Abbreviations

ACP:

Advance Care Planning

ADRD:

Alzheimer’s disease and related dementias

DSD:

Delirium Superimposed on Dementia

NIA:

National Institute on Aging

PCT:

Pragmatic randomized, controlled trial

RAPT:

Readiness Assessment for Pragmatic Trials

RCT:

Randomized, controlled trial

References

  1. 1.

    Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217–4.

  2. 2.

    National Institutes of Health Collaboratory. Rethinking clinical trials: a living textbook of pragmatic clinical trials. http://rethinkingclinicaltrials.org/. Accessed 1 Aug 2018.

  3. 3.

    Department Of Health and Human Services, National Institute on Aging (NIA). NIA AD/ADRD health care systems research collaboratory (U54 clinical trial required). https://grants.nih.gov/grants/guide/rfa-files/RFA-AG-19-009.html. Accessed 1 Aug 2018.

  4. 4.

    Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–75.

  5. 5.

    U.S. Department of Health & Human Services, ASPE., Office of the Assistant Secretary for Planning and Evaluation. National research summit on care, services, and supports for persons with dementia and their caregivers: report to the national advisory council on alzheimer’s research, care, and services. https://aspe.hhs.gov/pdf-report/national-research-summit-care-services-and-supports-persons-dementia-and-their-caregivers-final-summit-report. Accessed 1 Aug 2018.

  6. 6.

    U.S. Department of Health & Human Services, ASPE, Office of the Assistant Secretary for Planning and Evaluation. State of the science for pragmatic trials of non-drug interventions to improve outcomes among persons with dementia and their caregivers [Online]. https://aspe.hhs.gov/pdf-report/state-science-pragmatic-trials-non-pharmacological-interventions-improve-outcomes-among-persons-dementia-and-their-caregivers. Accessed 26 Mar 2018.

  7. 7.

    Baier RR, Mitchell SL, Jutkowitz E, Mor V. Identifying and supporting nonpharmacological dementia interventions ready for pragmatic trials: results from an expert workshop. J Am Med Dir Assoc. 2018;19:560–2.

  8. 8.

    Music & Memory. Music & Memory. https://musicandmemory.org/. Accessed 1 Sep 2018.

  9. 9.

    Rossato-Bennett, M. Alive Inside. http://www.aliveinside.us/. Accessed 1 Sep 2018.

  10. 10.

    Mor V. METRICAL - music and Memory: a pragmatic trial for nursing home residents with Alzheimer's disease. 2017, National Institute on Aging: Brown University. https://projectreporter.nih.gov/project_info_details.cfm?aid=9422034&icde=41259844. Accessed 17 Sep 2018.

  11. 11.

    Unroe KT, Nazir A, Holtz LR, Maurer H, Miller E, Hickman SE, La Mantia MA, Bennett M, Arling G, Sachs GA. The optimizing patient transfers, impacting medical quality, andImproving symptoms:transforming institutional care approach: preliminary data from the implementation of a Centers for Medicare and Medicaid Services nursing facility demonstration project. J Am Geriatr Soc. 2015;63:165–9.

  12. 12.

    Hickman SE, Unroe KT, Ersek MT, Buente B, Nazir A, Sachs GA. An interim analysis of an advance care planning intervention in the nursing home setting. J Am Geriatr Soc. 2016;64:2385–92.

  13. 13.

    Hickman SE, Unroe KT. A nursing home pragmatic clinical trial of approaches (aligning patient preferences: a role offering Alzheimer's patient, caregivers, and healthcare providers education and support. 2017, National Institute on Aging: Indiana University-Purdue University at Indianapolis. https://projectreporter.nih.gov/project_info_description.cfm?aid=9423503&icde=41259762&ddparam=&ddvalue=&ddsub=&cr=1&csb=default&cs=ASC&pball=. Accessed 17 Sep 2018.

  14. 14.

    Kolanowski A, Fick D, Litaker M, Mulhall P, Clare L, Hill N, Mogle J, Boustani M, Gill D, Yevchak-Sillner A. Effect of cognitively stimulating activities on symptom Management of Delirium Superimposed on dementia: a randomized controlled trial. J Am Geriatr Soc. 2016;64:2424–32.

  15. 15.

    Kolanowski AM, Hill N, Clare L, Marx P. Practical protocol for implementing cognitive stimulation in persons with delirium superimposed on dementia. Nonpharmacol Ther Dement. 2012;2:101–10.

  16. 16.

    Jutkowitz E, Brasure M, Fuchs E, Shippee T, Kane RA, Fink HA, Butler M, Sylvanus T, Kane RL. Care-delivery interventions to manage agitation and aggression in dementia nursing home and assisted living residents: a systematic review and meta-analysis. J Am Geriatr Soc. 2016;64:477–88.

Download references

Acknowledgements

We thank the 30 participants who attended the December 2017 expert workshop, “State of the science for pragmatic trials of non-drug interventions to improve outcomes among persons with dementia and their caregivers,” whose recommended criteria for determining the characteristics of dementia interventions ready for PCTs inspired us to create the RAPT model:David Bass, PhD, Benjamin Rose Institute on Aging; Abraham Brody, PhD, NYU Rory Meyers College of Nursing; David Dore, PharmD, PhD, Optum Analytics – Life Sciences and Brown University School of Public Health; Gary Epstein-Lubow, MD, Hebrew SeniorLife and Brown University School of Public Health; Richard Fortinsky, PhD, UConn Health Center on Aging; Nicole Fowler, PhD, MHSA, Indiana University School of Medicine; Elizabeth Galik, PhD, CRNP, FAAN, FAANP, University of Maryland School of Nursing; Joseph Gaugler, PhD, University of Minnesota School of Nursing; David Gifford, MD, MPH, American Health Care Association/National Center for Assisted Living; Laura Gitlin, PhD, Johns Hopkins School of Nursing; Corita Grudzen, MD, MHSH, FACEP, NYU Lagone Health; Lisa Gwyther, MWS, LCSW, Duke Medical Center; Laura Hanson, MD, MPH, University of North Carolina School of Medicine; Susan Hickman, PhD, Indiana University School of Nursing; Lee Jennings, MD, MSHS, University of Oklahoma Health Sciences Center; Ann Kolanowski, PhD, RN, FGSA, FAAN, Penn State University; Mark Kunik, MD, MPH, US Department of Veterans Affairs; Julie Lima, PhD, Brown University School of Public Health; Cheryl Phillips, MD, SNP Alliance; Greg Sachs, MD, Indiana University School of Medicine; Quincy Miles Samus, PhD, John Hopkins Medicine; David Schulke, PhD, Health Quality Strategies, LLC; Richard Schulz, University of Pittsburgh; Robyn Stone, DrPh, LeadingAge; Helena Temkin-Greener, PhD, MS, University of Rochester Medical Center; Joan Teno, MD, MS, University of Oregon; Thomas Travison, PhD, Hebrew SeniorLife; Kathleen Unroe, MD, MHA, Indiana University School of Medicine; Kathleen Welsh-Bohmer, PhD, Duke University School of Medicine; and Ann Wyatt, MSW, CaringKind.

Three pilot tested the first version of the scoring criteria and graphical summary with their interventions. We give particular thanks to Susan Hickman, PhD, Indiana University School of Nursing, Kathleen Unroe, MD, MHA, Indiana University School of Medicine, and Ann Kolanowski, PhD, RN, FGSA, FAAN, Pennsylvania State University for their provision of the Advance Care Planning Specialist Program (SH and KU) and Reserve for DSM (AK) scored examples and their permission to include them here.

Finally, we thank Rebekah Gardner, MD, FACP, Healthcentric Advisors, for naming the model.

Funding

This work was supported by supplemental funding provided by the NIA (P01-AG-027296), to sponsor the December 2017 expert workshop. NIA staff did not have any role in creating the RAPT model or in writing this manuscript.

Author information

RRB, SM, and VM chaired the NIA meeting that resulted in the criteria included in this model; EJ and EM presented and attended. Following the meeting, RRB conceptualized the model; RRB elicited workshop participants’ input and drafted the paper; RRB and EM wrote the scored example. EJ, EM, SM, and VM reviewed and revised drafts of the paper. All authors read and approved the final manuscript.

Correspondence to Rosa R. Baier.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Pragmatic clinical trial
  • Pragmatic trial
  • Effectiveness
  • Translational research
  • Implementation science
  • Model
  • Framework