In all three guidelines, elements of the RARE-Bestpractice Working Group’s pilot framework were successfully used. The specific elements, and our experience with their use, are described below.
The national Hemophilia Foundation-McMaster guideline on care models in hemophilia
In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: indirect evidence; qualitative research; and expert-based evidence.
The NHF-McMaster hemophilia guideline project started with a formal process of priority setting, question generation and prioritization, and identification of patient-important outcomes. Persons with hemophilia (PWH) and their families as well as health care providers were extensively involved in these initial steps. A modified Delphi process was used to finalize guideline questions and outcomes [22, 23].
Due to the paucity of evidence on models of care in hemophilia, the methods group conducted parallel systematic searches for meta-analyses from other chronic diseases to generate indirect evidence [26]. .Ultimately, searches in congestive heart failure, chronic obstructive pulmonary disease, asthma, and diabetes were performed. These diseases shared some features with hemophilia: chronicity, high resource use, involvement over the life span (for asthma and diabetes), and delivery of care via multidisciplinary integrated models [26]. Before the Guideline Panel met (and before they saw the meta-analyzed data for prioritized outcomes), they were asked to judge whether the reviews were “sufficiently direct” that they could inform the care of persons with hemophilia. Based on their summated responses, assessments of indirectness were then incorporated into the evidence profiles. In turn, use of evidence judged to be sufficiently direct provided valuable information on patient preferences, harms, costs and health equity.
There was also scant published information on stakeholder (including patient) experiences and perspectives around health and psychosocial outcomes of importance, acceptability of different models of care, impact of different models of care on health inequities, and feasibility of implementing different models of care. An experienced qualitative researcher on the guideline development team designed and executed a pilot qualitative study to explore these areas [24]. At the outset, stratified purposeful sampling was used to recruit individuals with the following perspectives: people with hemophilia (PWH), parents of PWH aged 18 and under, healthcare providers (hematologists, nurses, physiotherapists, social workers), insurance company representatives, and policy makers. A variety of traditional and social media approaches were used for recruitment. Snowball sampling (where key informants known to members of the research team inform stakeholders about the study) was also used as the study progressed. Participants engaged in semi-structured telephone interviews lasting 25–60 min. Techniques of thematic content analysis were used to identify major and minor themes in the data. The open-ended nature of the interview questions allowed participants to respond according to their own experiences, opinions, and beliefs. Clear themes emerged. Ultimately, this study helped the Guideline Panel learn about care models in 26 hemophilia treatment centers across the United States, and provided insightful information on outcomes of importance to stakeholders, access challenges, impact of care models on health equity, and challenges around acceptability, feasibility, and implementation.
The NHF-McMaster hemophilia guideline adapted surveys developed for a guideline project in Saudi Arabia to systematically capture expert-based clinical observations from the Guideline Panel, as well as a selection of other individuals identified as experts in the field [22, 23, 33]. Respondents were asked, not for their opinions, but for objective information that they could attest to, supported by unpublished data and/or their own observations of the effects of different models of care and care providers on important outcomes. Responses were transparent; the experts were asked to “sign off” on their responses, which were recorded in their entirety. This information was collated and presented to the Guideline Panel.
Patient registries were not used in the NHF-McMaster hemophilia guideline, as the Panel had access to large longitudinal and cross sectional observational studies. However, the guideline ultimately called for Hemophilia Treatment Centers in the United States to reaffirm their commitment to collecting and sharing high quality patient-level data; this enhanced data collection capacity should strengthen the evidence base for future guidelines.
The McMaster RARE BestPractices guideline on diagnosis and Management of Catastrophic Antiphospholipid Syndrome (CAPS)
In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: patient registry data; indirect evidence; and expert-based evidence.
Similar to the NHF-McMaster hemophilia guideline, the initial development stages of the CAPS guideline revealed low certainty to no direct evidence for important outcomes identified by the Panel. Therefore, in addition to the systematic reviews conducted for each guideline question, further methods of data retrieval and evidence creation were used.
The CAPS methods group and Guideline Panel identified the CAPS Registry as a potential source of data to inform the guideline. The CAPS Registry was initiated in 2000 by the European Forum on Antiphospholipid Antibodies to capture diagnosed cases worldwide [28]. Currently, clinical, laboratory, and therapeutic data from over 500 patients are recorded in the registry. The CAPS Registry is the most comprehensive source of patient information in CAPS, with data from both newly diagnosed patients and published case reports. Through on-site, real-time access to the CAPS Registry during evidence generation and panel meeting stages, the Panel was provided with an up to date overview of all indexed CAPS patients. Data from the CAPS Registry (sex, age at time of diagnosis, precipitating factors) informed the Panel on the natural history and impact of the disease. The proportion of deaths associated with each treatment was presented to potentially capture their effects. The CAPS Registry was particularly useful in answering a treatment question regarding the use of rituximab. During the Panel meeting, a search for the mortality of all registry patients who had received rituximab was performed, and compared to the mortality of a contemporaneous cohort of patients not treated with rituximab. A crude odds ratio representing patient mortality with rituximab was presented to the Panel; although rated as very low certainty evidence, it allowed the Panel to consider the potential effects of the drug when previously no information would have been available beyond Panel Member experience.
In contrast to the NHF-McMaster hemophilia guideline, the CAPS methods group did not conduct novel research on patient values and preferences, acceptability, feasibility, or equity. Indirect evidence from other conditions was however used to inform the Panel’s seven outcomes of interest: mortality, permanent organ dysfunction, permanent neurologic deficit, complete recovery, thromboembolic event, and amputation. Recognizing that individuals with CAPS often suffer limb loss and neurologic deficits, the Panel considered patients with diabetes who had limb amputations, and patients with stroke who had neurologic deficits. Health utilities of stroke, acute thromboembolism, and major intracranial bleeds were also considered. Finally, indirect evidence from patients with acute arterial and venous thromboembolism and their bleeding complications (resulting from the use of aspirin, warfarin, and low molecular weight heparins) were considered.
Expert-based evidence was systematically captured from CAPS Panel Members to provide clinical observations for questions around therapy and diagnosis. The collected expert-based evidence was used to inform two key areas. First, the responses around each treatment option provided data on their relative usage in practice. For example, systematically collected observations revealed that only one Panel expert had experience using rituximab. This reflected potential uncertainty around the use of rituximab in wider clinical practice for the Panel to consider. Second, systematically collected observations from Panel Members were discussed and compared to data from systematically reviewed published studies. Panel Members, both spontaneously during the panel meeting and when prompted during an exploratory interview at the end of the guideline development process, stated this was valuable in adopting a standardized and objective way of describing their own practice and experience. Many admitted that their perception of the relevance and direction of the evidence changed when they looked at their experience in a semi-quantitative way. A template of the standardized expert observation form used in the hemophilia and CAPS guidelines is in Appendix A.
The rare-Bestpractices guideline on sickle cell disease (SCD)
In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: qualitative research; and expert-based evidence.
The SCD guideline process started with identifying a source guideline that could be used for the Adolopment process, including to identify subject areas of interest, prioritize questions, and identify relevant studies [33]. After a scoping exercise, the National Heart, Lung and Blood Institute (NHLBI) guidelines on Evidence Based Management of Sickle Cell Disease were selected as the source guideline [39]. The guideline adolopment approach then went through a formal process to prioritize the subject areas tackled by the NIH guideline, prioritize questions for selected areas, and identify patient-important outcomes. For these steps, structured surveys were sent to all Panel Members. Although the Panel prioritized a number of patient important outcomes, due to scarcity of data, the team ultimately focused on the outcomes considered in the NHLBI guidelines.
A methods group, independent from the Guideline Panel, conducted the literature search in three sequential steps for each of the prioritized questions: (1) screening of the NIH guidelines reference list; (2) electronic database search for relevant systematic reviews; and (3) electronic database search for relevant primary studies. Systematic searches were also conducted for values attached to the outcomes of interest, preferences for the interventions under consideration, and resource use associated with the intervention options. The Panel was asked to review the list of identified studies and suggest if any were missed; however no studies reporting direct evidence were identified by the experts beyond what had been captured through the first two search steps. The SCD methods group did not conduct systematic searches for indirect evidence in related diseases.
Panel Members were asked to complete online surveys looking at values and preferences, and resource use related to the interventions of interest. Panelists were also asked to suggest additional published and unpublished information of interest relating to benefits and harms, values and preferences, resource use, acceptability, feasibility, and equity. These additional pieces of information were provided in the form of unstructured communications from Panel Members to the methodology team, and were discussed during the face-to-face Panel Meeting. At their in-person meeting, Panelists were asked to complete a survey soliciting feedback about the guideline development process.
Ethics approval
The qualitative study conducted as part of the NHF-McMaster hemophilia guideline was submitted to and approved by the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada. Written informed consent was obtained from all participants in the qualitative study. Ethics approval for collection of expert based evidence in all the described guidelines was not deemed necessary, as it involved surveys of guideline panel members, whose ethical treatment and consent to participate in guideline related activities was considered implicit. All other work described in this study involved systematic reviews of anonymized and published data, and thus did not require ethics submission and approval as per the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada.