Skip to main content
Download PDF

Strategies for eliciting and synthesizing evidence for guidelines in rare diseases

BMC Medical Research Methodology volume 19, Article number: 67 (2019) Cite this article

Abstract

Background

Rare diseases are a global public health priority. Though each disease is rare, when taken together the thousands of known rare diseases cause significant morbidity and mortality, impact quality of life, and confer a social and economic burden on families and communities. These conditions are, by their nature, encountered very infrequently by individual clinicians, who may feel unprepared to address their diagnosis and treatment. Clinical practice guidelines are necessary to support clinical and policy decisions. However, creating guidelines for rare diseases presents specific challenges, including a paucity of high certainty evidence to inform panel recommendations.

Methods

This paper draws from the authors’ experience in the development of clinical practice guidelines for three rare diseases: hemophilia, sickle cell disease, and catastrophic antiphospholipid syndrome.

Results

We have summarized a number of strategies for eliciting and synthesizing evidence that are compatible with the rigorous, internationally accepted standards for guideline development set out by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. These strategies include: use of pre-existing and ad hoc qualitative research, use of systematic observation forms, use of registry data, and thoughtful use of indirect evidence. Their use in three real guideline development efforts, as well as their theoretical underpinnings, are discussed. Avenues for future research to improve clinical practice guideline creation for rare diseases – and any disease affected by a relative lack of evidence - are also identified.

Conclusions

Rigorous clinical practice guidelines are needed to improve the care of the millions of people worldwide who suffer from rare diseases. Innovative evidence elicitation and synthesis methods will benefit not only the rare disease community, but also individuals with common diseases who have rare presentations, suffer rare complications, or require nascent therapies. Further refinement and improved uptake of these innovative methods should lead to higher quality clinical practice guidelines in rare diseases.

Peer Review reports

Background

Rare diseases: a Global Health priority

The U.S. National Institutes of Health (NIH) Office of Rare Diseases Research defines a disease as rare if it affects less than 1 in 1500 people, while the European Union defines rare diseases as those affecting less than 1 in 2000 people [1, 2]. Rare diseases frequently cause significant morbidity and mortality, gravely affect quality of life, and confer a social and economic burden on families and communities. Though any one rare disease will affect a small number of people, when taken together, rare diseases affect over 25 million American citizens and 30 million European Union citizens. Moreover, a disease defined as rare in the general population may be endemic to specific populations, conferring disproportionate impact on these communities. Rare diseases challenge traditional health economy perspectives [3]. Many rare diseases have no specific therapies. Those that can be treated with specific orphan drugs are characterized by extremely high costs for the individual, though these amount to a fraction of the total healthcare costs directed towards more common diseases [4, 5]. These considerations, paired with the paucity of available evidence and the inadequacy of accepted metrics to measure quality of life in patients affected by chronic rare diseases (known as the disability paradox), make it challenging to apply standard comparative effectiveness strategies to rare diseases [3, 6].

Barriers to evidence generation in rare diseases

Most rare diseases are not a focus for policymakers, funders, and researchers in many countries. Pharmaceutical companies may invest significant financial resources in bringing orphan drugs to the market, but often struggle to recover costs through sales [7]. Without Orphan Drug Regulations and cross-sector sponsorship initiatives, there are few economic incentives to develop treatments for rare diseases [8, 9]. And as the number of individuals affected by a rare disease is, by definition, small, a critical mass of people who can advocate for research and development may not exist.

Performing properly sized studies of these diseases, which minimize bias and confounding, is difficult when the number of affected individuals (and potential study participants) for any one disease is low [10]. Adequately powered studies are rarely done because of difficulties in participant recruitment; challenges also include unclear or difficult-to-confirm diagnostic criteria, and lack of reliable patient registries that can support study planning and recruitment [11, 12]. Use of a placebo may not be an option in many rare conditions due to the severe course of the untreated disease, or because patients are unwilling to enroll in studies where they may not receive a badly-needed active treatment option. In order to compensate for low enrollment and/or lack of a comparator, several alternative trial designs (e.g., parallel RCT-cohort trial, sequential design, risk based allocation, hierarchical designs, placebo phase trials, Bayesian designs) have been proposed [13, 14]. However, these innovative types of trials are technically more difficult to design and run than conventional RCTs [15].

Even if a researcher successfully designs and executes a trial, dissemination of results can be challenging. Journal editors may be hesitant to publish studies with a narrow audience, on the assumption that it will have little impact. And much like in common diseases, if a rare disease trial’s findings are negative, publication becomes even more difficult.

When published studies are available, they are often heterogeneous. Patients may have widely varying baseline characteristics, interventions and comparators may be used differently, and outcomes and measurement techniques may differ. Patient reported outcomes, such as disease-specific quality of life instruments, may not exist, may not be reported at all, or may be reported with instruments that are neither properly validated nor straightforward to interpret; investigators often rely on an array of surrogate outcomes instead. Heterogeneity in studies can also make aggregating data challenging. The end result of all of these challenges is a slim and inconsistent body of evidence, much of which is low certainty and does not address critically important outcomes.

Barriers to guideline development for rare diseases

The barriers to guideline development for rare diseases are considerable, spanning from evidence appraisal and synthesis, to issuing recommendations for care, to knowledge translation. Ideally, clinical practice guidelines summarize evidence for patient-important outcomes, and use clear criteria to generate recommendations [16, 17]. In many methodological approaches, high certainty evidence often leads to strong recommendations, whereas low certainty evidence usually leads to conditional, or weaker recommendations. Guideline developers may be unable to generate strong recommendations for diagnosis and/or treatment strategies in rare diseases, due to the lack of high certainty evidence. Guideline users may find these conditional recommendations, which have little evidentiary base, unhelpful or insufficiently directive for frontline clinical care. The ultimate consequence could be suboptimal clinical decisions for individuals with rare diseases. These barriers have previously been described by the RARE-Bestpractices Working Group, a 4 year project (January 2013–December 2016) funded by the European Commission under the FP7 Cooperation Work Programme: Health-2012. The project’s focus was sharing best practices and methodologic knowledge in the clinical management of rare diseases.

Our aim was to use a dedicated framework for guideline development in the field of rare diseases, as proposed by the RARE-Bestpractices Working Group [18]. This framework is intended as an extension of existing guideline development strategies, but has not previously been tested in real rare disease guidelines, Our goal was to determine if: a) the novel strategies for eliciting and synthesizing evidence can be feasibly implemented in guideline development for rare diseases; and b) if the framework helps overcome all or some of the barriers to guideline development described above [18].

Methods

We piloted the suggested framework in three specific guideline development processes undertaken in three different rare conditions. All three guidelines were coauthored and developed by authors of this report. All three guidelines also explicitly aimed to achieve a double goal: 1) produce a guideline to be used in clinical practice; and 2) implement our framework to overcome expected barriers in guideline issuing in rare diseases. The individuals most focused on the latter goal where the guideline methodologists, whilst the clinical panel members acted as end users trying out and experiencing the impact of our framework.

The first guideline focused on hemophilia, an X-linked bleeding disorder caused by the hereditary lack of a coagulation factor. Blood clotting is severely impaired in hemophilia, resulting in serious bleeding with minimal provocation. Hemophilia A (deficiency of coagulation factor VIII) and hemophilia B (deficiency of coagulation factor IX) are both rare; hemophilia A affects fewer than 1 in 10,000 people and hemophilia B affects fewer than 1 in 50,000 people. In 2016, the National Hemophilia Foundation in the United States partnered with McMaster University in Canada to publish a guideline that explored the optimal model of care delivery, and the optimal composition of the care team, in this rare disease. The guideline used the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) system and the GIN-McMaster guideline development checklist, and adhered to the principles set out by the Institute of Medicine in their 2011 document, “Guidelines We Can Trust.” [19,20,21] The guideline and its associated studies have been published in the Journal “Haemophilia” and are included in the National Guideline Clearinghouse [22,23,24,25,26].

The second guideline focused on catastrophic antiphospholipid syndrome (CAPS), also known as Asherson’s syndrome. CAPS is a clinical syndrome characterized by the rapid onset of multifocal thrombosis associated with multi-organ failure in patients meeting the serological criteria for antiphospholipid syndrome [27]. The mortality rate for CAPS is about 50%. CAPS is extremely rare, with approximately 500 individuals identified in the published literature since the disorder was first described in 1992 [28, 29]. In 2015, the European RARE-Bestpractices project group partnered with McMaster University to produce guidelines that explored therapeutic and diagnostic questions in this rare disease. The guidelines adhered to GRADE methodology and the GIN-McMaster Guideline Development Checklist [20, 21]. The guideline has been submitted for publication in a peer reviewed journal.

The third guideline focused on sickle cell disease (SCD), an inherited condition in which a mutated form of hemoglobin distorts red blood cells in low oxygen conditions. These crescent shaped “sickle” cells can obstruct the flow of blood in blood vessels, so that oxygen cannot reach nearby tissues. SCD results in anemia, acute painful crises, chronic pain, strokes, and damage to vital organs. The overall prevalence of SCD per 10,000 live births is approximately 30 in African Americans, 0.1 in Caucasian Americans, and 0.3 in Hispanic Americans [30]. In 2016, the European RARE-Bestpractices project (funded by the European Union Commission) organized a group that partnered with the GRADE Centers from American University of Beirut and McMaster Univerisity to produce guidelines exploring diagnostic and therapeutic questions in sickle cell disease [31]. This project’s methods were based on the GRADE and GRADE Adolopment approaches [32, 33]. The project resulted in a manuscript reporting the guideline development process and the resulting recommendations. This manuscript is in the process of journal submission.

All three guidelines were developed using an overarching approach proposed by the GRADE working group. GRADE is a methodological approach that helps guideline developers present summaries of evidence in a structured and standardized way, make transparent judgments about the quality of evidence, and then systematically move towards developing recommendations [34]. GRADE has been adopted by over 100 organizations worldwide, including the World Health Organization [35, 36]. It encompasses not only guideline development for questions about treatment, but also for diagnosis [37, 38]. Most of the applications of GRADE to date have been outside the field of rare diseases, and it is unproven if GRADE can be efficiently applied to rare diseases. For this study, we utilized the RARE-Bestpractice Working Group’s pilot framework for applying GRADE to rare disease guidelines, applying elements of it to all three guidelines and assessing its feasibility and practicability [18]. These rare disease specific elements were as follows:

  1. 1.

    Qualitative research methods can be used to generate evidence on patient values and preferences, equity, acceptability, feasibility, and implementability. These factors are essential to inform Guideline Panel recommendations when using the GRADE system. They are also critical in the guideline implementation phase as they support health policy decision making. However in both rare and common diseases, published evidence on these factors can be difficult to find; it may be necessary to generate it to support guideline development.

  1. 2.

    Expert-based evidence can be systematically solicited through structured observation forms. Despite the paucity of published evidence, there is often a great deal of knowledge about rare diseases – usually from health care providers who cumulatively have many years of real world experience treating patients. Individually, they may not have enough data to publish a peer reviewed study. Furthermore, due to publication bias, data describing ineffective therapies may remain unpublished. However, when data are collected using a structured approach, this “expert-based evidence” becomes invaluable to inform recommendations for clinical practice.

  1. 3.

    Patient registries can be used to supplement the published literature, particularly for questions where scant data were available. Registry data can be used to identify or explore relevant patient subgroups. Registries can also give a snapshot of the impact and natural history of a disease, and the variety and effects of treatments that are used. Registries have the benefit of prospectively capturing patient information; a robust registry, comprehensive in the depth of captured data and inclusive of all known patients, can be equivalent to a high-quality observational study.

  1. 4.

    Finally, indirect evidence can be used. The concept of directness (or indirectness) of evidence is well-established in GRADE, and is used to assess whether the body of evidence used to inform guideline recommendations is directly applicable to the health questions of interest. Sometimes, guideline developers may not be able to find published evidence that directly addresses their clinical questions. In such cases, they can turn to evidence that differs to some degree in terms of population, intervention, comparator, and/or outcome of interest. Indirect evidence becomes particularly valuable in rare diseases. For example, indirect evidence may include extrapolation of data from a population affected by a more common disease that shares some features with the rare disease. (For example, data from epilepsy informing guidelines on tuberous sclerosis.) When the search for applicable/relevant indirect evidence is performed, a specific process to minimize bias can be adopted.

Results

In all three guidelines, elements of the RARE-Bestpractice Working Group’s pilot framework were successfully used. The specific elements, and our experience with their use, are described below.

The national Hemophilia Foundation-McMaster guideline on care models in hemophilia

In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: indirect evidence; qualitative research; and expert-based evidence.

The NHF-McMaster hemophilia guideline project started with a formal process of priority setting, question generation and prioritization, and identification of patient-important outcomes. Persons with hemophilia (PWH) and their families as well as health care providers were extensively involved in these initial steps. A modified Delphi process was used to finalize guideline questions and outcomes [22, 23].

Due to the paucity of evidence on models of care in hemophilia, the methods group conducted parallel systematic searches for meta-analyses from other chronic diseases to generate indirect evidence [26]. .Ultimately, searches in congestive heart failure, chronic obstructive pulmonary disease, asthma, and diabetes were performed. These diseases shared some features with hemophilia: chronicity, high resource use, involvement over the life span (for asthma and diabetes), and delivery of care via multidisciplinary integrated models [26]. Before the Guideline Panel met (and before they saw the meta-analyzed data for prioritized outcomes), they were asked to judge whether the reviews were “sufficiently direct” that they could inform the care of persons with hemophilia. Based on their summated responses, assessments of indirectness were then incorporated into the evidence profiles. In turn, use of evidence judged to be sufficiently direct provided valuable information on patient preferences, harms, costs and health equity.

There was also scant published information on stakeholder (including patient) experiences and perspectives around health and psychosocial outcomes of importance, acceptability of different models of care, impact of different models of care on health inequities, and feasibility of implementing different models of care. An experienced qualitative researcher on the guideline development team designed and executed a pilot qualitative study to explore these areas [24]. At the outset, stratified purposeful sampling was used to recruit individuals with the following perspectives: people with hemophilia (PWH), parents of PWH aged 18 and under, healthcare providers (hematologists, nurses, physiotherapists, social workers), insurance company representatives, and policy makers. A variety of traditional and social media approaches were used for recruitment. Snowball sampling (where key informants known to members of the research team inform stakeholders about the study) was also used as the study progressed. Participants engaged in semi-structured telephone interviews lasting 25–60 min. Techniques of thematic content analysis were used to identify major and minor themes in the data. The open-ended nature of the interview questions allowed participants to respond according to their own experiences, opinions, and beliefs. Clear themes emerged. Ultimately, this study helped the Guideline Panel learn about care models in 26 hemophilia treatment centers across the United States, and provided insightful information on outcomes of importance to stakeholders, access challenges, impact of care models on health equity, and challenges around acceptability, feasibility, and implementation.

The NHF-McMaster hemophilia guideline adapted surveys developed for a guideline project in Saudi Arabia to systematically capture expert-based clinical observations from the Guideline Panel, as well as a selection of other individuals identified as experts in the field [22, 23, 33]. Respondents were asked, not for their opinions, but for objective information that they could attest to, supported by unpublished data and/or their own observations of the effects of different models of care and care providers on important outcomes. Responses were transparent; the experts were asked to “sign off” on their responses, which were recorded in their entirety. This information was collated and presented to the Guideline Panel.

Patient registries were not used in the NHF-McMaster hemophilia guideline, as the Panel had access to large longitudinal and cross sectional observational studies. However, the guideline ultimately called for Hemophilia Treatment Centers in the United States to reaffirm their commitment to collecting and sharing high quality patient-level data; this enhanced data collection capacity should strengthen the evidence base for future guidelines.

The McMaster RARE BestPractices guideline on diagnosis and Management of Catastrophic Antiphospholipid Syndrome (CAPS)

In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: patient registry data; indirect evidence; and expert-based evidence.

Similar to the NHF-McMaster hemophilia guideline, the initial development stages of the CAPS guideline revealed low certainty to no direct evidence for important outcomes identified by the Panel. Therefore, in addition to the systematic reviews conducted for each guideline question, further methods of data retrieval and evidence creation were used.

The CAPS methods group and Guideline Panel identified the CAPS Registry as a potential source of data to inform the guideline. The CAPS Registry was initiated in 2000 by the European Forum on Antiphospholipid Antibodies to capture diagnosed cases worldwide [28]. Currently, clinical, laboratory, and therapeutic data from over 500 patients are recorded in the registry. The CAPS Registry is the most comprehensive source of patient information in CAPS, with data from both newly diagnosed patients and published case reports. Through on-site, real-time access to the CAPS Registry during evidence generation and panel meeting stages, the Panel was provided with an up to date overview of all indexed CAPS patients. Data from the CAPS Registry (sex, age at time of diagnosis, precipitating factors) informed the Panel on the natural history and impact of the disease. The proportion of deaths associated with each treatment was presented to potentially capture their effects. The CAPS Registry was particularly useful in answering a treatment question regarding the use of rituximab. During the Panel meeting, a search for the mortality of all registry patients who had received rituximab was performed, and compared to the mortality of a contemporaneous cohort of patients not treated with rituximab. A crude odds ratio representing patient mortality with rituximab was presented to the Panel; although rated as very low certainty evidence, it allowed the Panel to consider the potential effects of the drug when previously no information would have been available beyond Panel Member experience.

In contrast to the NHF-McMaster hemophilia guideline, the CAPS methods group did not conduct novel research on patient values and preferences, acceptability, feasibility, or equity. Indirect evidence from other conditions was however used to inform the Panel’s seven outcomes of interest: mortality, permanent organ dysfunction, permanent neurologic deficit, complete recovery, thromboembolic event, and amputation. Recognizing that individuals with CAPS often suffer limb loss and neurologic deficits, the Panel considered patients with diabetes who had limb amputations, and patients with stroke who had neurologic deficits. Health utilities of stroke, acute thromboembolism, and major intracranial bleeds were also considered. Finally, indirect evidence from patients with acute arterial and venous thromboembolism and their bleeding complications (resulting from the use of aspirin, warfarin, and low molecular weight heparins) were considered.

Expert-based evidence was systematically captured from CAPS Panel Members to provide clinical observations for questions around therapy and diagnosis. The collected expert-based evidence was used to inform two key areas. First, the responses around each treatment option provided data on their relative usage in practice. For example, systematically collected observations revealed that only one Panel expert had experience using rituximab. This reflected potential uncertainty around the use of rituximab in wider clinical practice for the Panel to consider. Second, systematically collected observations from Panel Members were discussed and compared to data from systematically reviewed published studies. Panel Members, both spontaneously during the panel meeting and when prompted during an exploratory interview at the end of the guideline development process, stated this was valuable in adopting a standardized and objective way of describing their own practice and experience. Many admitted that their perception of the relevance and direction of the evidence changed when they looked at their experience in a semi-quantitative way. A template of the standardized expert observation form used in the hemophilia and CAPS guidelines is in Appendix A.

The rare-Bestpractices guideline on sickle cell disease (SCD)

In this guideline, the following elements of the RARE-Bestpractice Working Group’s framework were piloted: qualitative research; and expert-based evidence.

The SCD guideline process started with identifying a source guideline that could be used for the Adolopment process, including to identify subject areas of interest, prioritize questions, and identify relevant studies [33]. After a scoping exercise, the National Heart, Lung and Blood Institute (NHLBI) guidelines on Evidence Based Management of Sickle Cell Disease were selected as the source guideline [39]. The guideline adolopment approach then went through a formal process to prioritize the subject areas tackled by the NIH guideline, prioritize questions for selected areas, and identify patient-important outcomes. For these steps, structured surveys were sent to all Panel Members. Although the Panel prioritized a number of patient important outcomes, due to scarcity of data, the team ultimately focused on the outcomes considered in the NHLBI guidelines.

A methods group, independent from the Guideline Panel, conducted the literature search in three sequential steps for each of the prioritized questions: (1) screening of the NIH guidelines reference list; (2) electronic database search for relevant systematic reviews; and (3) electronic database search for relevant primary studies. Systematic searches were also conducted for values attached to the outcomes of interest, preferences for the interventions under consideration, and resource use associated with the intervention options. The Panel was asked to review the list of identified studies and suggest if any were missed; however no studies reporting direct evidence were identified by the experts beyond what had been captured through the first two search steps. The SCD methods group did not conduct systematic searches for indirect evidence in related diseases.

Panel Members were asked to complete online surveys looking at values and preferences, and resource use related to the interventions of interest. Panelists were also asked to suggest additional published and unpublished information of interest relating to benefits and harms, values and preferences, resource use, acceptability, feasibility, and equity. These additional pieces of information were provided in the form of unstructured communications from Panel Members to the methodology team, and were discussed during the face-to-face Panel Meeting. At their in-person meeting, Panelists were asked to complete a survey soliciting feedback about the guideline development process.

Ethics approval

The qualitative study conducted as part of the NHF-McMaster hemophilia guideline was submitted to and approved by the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada. Written informed consent was obtained from all participants in the qualitative study. Ethics approval for collection of expert based evidence in all the described guidelines was not deemed necessary, as it involved surveys of guideline panel members, whose ethical treatment and consent to participate in guideline related activities was considered implicit. All other work described in this study involved systematic reviews of anonymized and published data, and thus did not require ethics submission and approval as per the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada.

Discussion

Rare diseases pose a real challenge to patients, providers, payers and policy makers. Many of these conditions have no satisfactory therapeutic options, while others have very expensive therapeutic options with relatively little evidence to support them [2, 40]. Health care practitioners and policy makers often have little experience with rare conditions. Thus, clinical practice guidelines which summarize the available evidence and provide recommendations developed by experts in a structured and transparent manner are potentially helpful in supporting decision making at various levels.

Our study – in which we applied a framework for evidence synthesis, allowing for the flexible and transparent use of indirect, expert-based, and ad hoc newly generated evidence – successfully addressed some of the “evidence gaps” encountered when developing three target rare disease guidelines. These “gaps” are common to guideline development for many rare diseases. Guidelines in common diseases frequently rely on comparative evidence of treatment effects (generated when two or more interventions, or an intervention and a placebo, are compared). This reliance on comparative studies is often not practical for rare disease guidelines. Indeed, there is often only one treatment for any given rare disease, or there is only very low certainty evidence for the effect of different treatments.

A major strength of our study is that it shows how, in the absence of disease-specific comparative studies, valuable evidence can still be retrieved and successfully used. Systematic reviews of evidence from similar, more common diseases can provide valuable comparative information to Guideline Panels. Qualitative research has proven to be a viable data source for values and preferences, acceptability, feasibility, and equity [41]. These important aspects are underreported in both rare and common diseases. Our study suggests that patient registries are a feasible source of information for guideline developers as well. Registries can capture long term longitudinal data that are difficult to capture in conventional experimental study designs due to ethical, regulatory and financial constraints. Registries may be of particular use in rare diseases, many of which are chronic, and affect individuals over the life span. Registries are considered key instruments for developing rare disease clinical research, enhancing patient care and health planning, and improving social, economic and quality-of-life outcomes. Indeed, it is usually the case that no single institution, and in many cases no single country, has sufficient data on a given rare disease that can be applied broadly to clinical and translational research. However, the fragmentation and heterogeneity of existing registries, which are often the result of spontaneous initiatives, limit the general applicability of their observations [42]. The use of registries in guideline development is subject to other limitations. Ideally, registries must be designed to collect information on patient-important outcomes. Registries must also be set up to capture data that are amenable to pooling in order to achieve a sufficient sample size for key outcomes. This requires significant foresight as the registry is launched, ongoing efforts to maintain recruitment, and assurance that the technology to securely share data is maintained. In the CAPS guideline, many analyses could not be performed because of missing variables that would have controlled for confounding. The guideline development group could therefore provide only crude estimates and unadjusted analyses. Another concern with using registry data is duplication of data from published studies, since populations with rare disease populations are small. For example, data from published studies of CAPS patients were likely incorporated into the CAPS Registry, as part of the registry recruitment strategy is periodic canvassing of the literature to identify and recruit cases. In these circumstances, it may be important to inform the Panel that analyses from the registry could potentially be based on duplicate data already presented in the studies retrieved from the systematic review. Furthermore, use of registries almost inevitably requires involvement of the creators of the registry, which can lead to important intellectual conflicts of interest.

Another strength of our study is its confirmation that unpublished non-experimental data (including observational data and qualitative evidence) can be elicited and collected in such a way to be usable by Guideline Panels. Systematically collected evidence from field experts can provide this information; further, “going to the source” may overcome dissemination bias, albeit at the risk of introducing selection bias (which must be carefully monitored).

Either of the two approaches indicated above can overcome the tendency of guideline developers to focus only on randomized controlled trials and large observational studies to inform their work, which may not be possible outside the field of common diseases. As users demand high-quality guidelines for rare diseases, it will be important to incorporate non-experimental and non-comparative data collected in a systematic and transparent fashion.

There were clear limitations to our work, and to the RARE-Bestpractice Working Group’s framework. Qualitative research did provide valuable information in the NHF-McMaster guideline. However, in general, published qualitative data can be difficult to find in rare diseases – either because these studies are not conducted, or because they are not published. Guideline developers can conduct their own qualitative studies (as the NHF-McMaster group did), but this takes time and resources. Qualitative research may have complemented the limited quantitative evidence base in the CAPS guideline. In particular, it may have provided insight into patient perspectives, which are not reported in the literature. However, identifying patients to interview was a major limitation to conducting a qualitative study of CAPS; the number of patients diagnosed with this rare condition every year is low, the presentation is precipitous and often fulminant, and the mortality rate is close to 50%. In such circumstances, patient organizations may be able to assist with identifying patient perspectives. Partnering with a patient organization ultimately helped the CAPS guideline development team recruit a patient representative for the Guideline Panel. When published qualitative studies are available, there is still little consensus in how to rate the quality of evidence. GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research), a GRADE project group, has proposed that an assessment of confidence for individual review findings from qualitative evidence syntheses be based on four components: methodological limitations of the incorporated studies, relevance of the studies to the review question, coherence of the review finding, and adequacy of data supporting the review finding [43]. This is an active area of study, but is currently a major constraint on the utility of qualitative evidence in guideline development.

Finally, our study identified limitations to the use of expert-based evidence. Guideline developers have been advised to help Panel Members identify the evidence underlying their opinions, and judge the quality of that evidence [44]. However, expert-based evidence can be challenging to incorporate into guidelines. We maintain that it should be collected transparently and systematically, and subjected to the same level of Guideline Panel appraisal as other evidence. This evidence can be subject to bias if it tilts from objectively collected data into subjectively developed opinion. Experts should be encouraged to provide statements in advance, after systematically having reviewed cases they have managed, to provide the most objective information. This would provide Panels with unique data gained from years of clinical experience. Experts should also be required to sign off on their statements, which may increase its trustworthiness.

Further research into how novel methods of evidence creation would support the guideline update process is also needed. For example, if a guideline for a rare disease relies heavily on evidence from a more common disease with a rapidly evolving knowledge base, the rare disease guideline may require more frequent updating through reviews of indirect evidence. Guideline developers must also be able to efficiently respond to changes in the knowledge base that may impact guideline recommendations, but can only be detected by ongoing qualitative research (for example, shifts in patient values over time). Finally, it is clear that non-experimental and non-comparative forms of evidence can play a valuable role in guideline creation. Several groups, including AID (Appraising and Including Different Knowledge in Guideline Development) and GRADE-CERQual are actively exploring this concept.

Conclusion

After theoretically exploring the barriers to issuing guidelines on rare diseases, we proposed an operational approach consistent with GRADE [18]. We have now applied this approach in three specific rare conditions [22]. Our work confirms that priority setting, question generation and prioritization, and outcome identification and ranking, all pursued with extensive involvement of patients and experts are possible in the development of rare disease guidelines, and the process can parallel that in more common diseases. Furthermore, we have come to a number of conclusions: We have concluded that intentional and systematic gathering of (unpublished) observational evidence from experts is a more efficient and transparent alternative to informal “around the table discussion” by the Guideline Panel. We have also concluded that it is feasible to use indirect evidence from other diseases and access a patient registry of the target disease to complement published low certainty evidence. Finally, we have established the feasibility of qualitative research as an important way to glean information about values and preferences, acceptability, feasibility, and health equity – all key factors to consider when making practice recommendations.

All three rare disease guidelines in this study maintained a clear link between quality of evidence and the strength of the ultimate recommendations, as specified by the GRADE Working Group framework. Rigorous clinical practice guidelines are needed to improve the care of the millions of people worldwide who suffer from rare diseases. We recommend that systematic processes must be used to create evidence-based practice guidelines that are useful to patients, clinicians, researchers, industry, and policy makers in the rare disease community. However, these processes must also be dynamic and responsive to the unique challenge presented by rare diseases: a dearth of high certainty evidence. Innovative evidence elicitation and synthesis methods will benefit not only the rare disease community, but also individuals with common diseases who have rare presentations, suffer rare complications, or require nascent therapies. Further refinement and improved uptake of these innovative methods should lead to higher quality clinical practice guidelines in rare diseases.

Abbreviations

AID:

Appraising and Including Different Knowledge in Guideline Development

CAPS:

Catastrophic antiphospholipid syndrome

CERQual:

Confidence in the Evidence from Reviews of Qualitative research

GRADE:

Grading of Recommendations, Assessment, Development and Evaluation

NHLBI:

National Heart, Lung and Blood Institute

PWH:

Persons with hemophilia

SCD:

Sickle cell disease

References

  1. 1.

    Adams E, Deutsche J, Okoroh E, Owens-Mcalister S, Majumdar S, Ullman M, Damiano ML, Recht M, Casto L, Droze K, et al. An inventory of healthy weight practices in federally funded haemophilia treatment centres in the United States. Haemophilia : the official journal of the World Federation of Hemophilia. 2014;20(5):639–43.

    CAS  Article  Google Scholar 

  2. 2.

    Ahmad Kia Daliri A, Mamikhani J. Cost-effectiveness of prophylaxis against on-demand treatment in boys with severe haemophilia in Iran. Value Health. 2009;12(3):A133.

    Article  Google Scholar 

  3. 3.

    Abrahamyan L, Willan AR, Beyene J, McLimont M, Blanchette V, Feldman BM. Using value-of-information methods when the disease is rare and the treatment is expensive--the example of hemophilia a. J Gen Intern Med. 2014;29(Suppl 3):S767–73.

    Article  Google Scholar 

  4. 4.

    Swenson SM, Chamberlain LJ, Sanders LM, Sundaram V, Wise PH. Outpatient pharmacy expenditures for children with serious chronic illness in California, 2010-2012. Jama. 2015;314(4):405–7.

    CAS  Article  Google Scholar 

  5. 5.

    Zhou ZY, Koerper MA, Johnson KA, Riske B, Baker JR, Ullman M, Curtis RG, Poon JL, Lou M, Nichol MB. Burden of illness: direct and indirect costs among persons with hemophilia a in the United States. J Med Econ. 2015;18(6):457–65.

    Article  Google Scholar 

  6. 6.

    Berger KC, Feldman BM, Wasserman J, Schramm W, Blanchette V, Fischer K. Securing reimbursement for patient centered haemophilia care: major collaborative efforts are needed. Haematologica. 2016;101(3):266–8.

    Article  Google Scholar 

  7. 7.

    Haffner ME. Adopting orphan drugs--two dozen years of treating rare diseases. N Engl J Med. 2006;354(5):445–7.

    CAS  Article  Google Scholar 

  8. 8.

    Fiorentino R, Liu G, Pariser AR, Mulberg AE. Cross-sector sponsorship of research in eosinophilic esophagitis: a collaborative model for rational drug development in rare diseases. J Allergy Clin Immunol. 2012;130(3):613–6.

    Article  Google Scholar 

  9. 9.

    Dunoyer M. Accelerating access to treatments for rare diseases. Nat Rev Drug Discov. 2011;10(7):475–6.

    CAS  Article  Google Scholar 

  10. 10.

    Lilford RJ, Thornton JG, Braunholtz D. Clinical trials and rare diseases: a way out of a conundrum. BMJ (Clinical research ed). 1995;311(7020):1621–5.

    CAS  Article  Google Scholar 

  11. 11.

    Luisetti M, Campo I, Scabini R, Zorzetto M, Kadija Z, Mariani F, Ferrarotti I. The problems of clinical trials and registries in rare diseases. Respir Med. 2010;104(Suppl 1):S42–4.

    Article  Google Scholar 

  12. 12.

    Groft SC, de la Paz MP. Rare diseases - avoiding misperceptions and establishing realities: the need for reliable epidemiological data. Adv Exp Med Biol. 2010;686:3–14.

    Article  Google Scholar 

  13. 13.

    Peyvandi F, Farrugia A, Iorio A, Key NS, Srivastava A. Joint WFH‐ISTH session: issues in clinical trial design. Haemophilia. 2014;20:137–44.

  14. 14.

    Iorio A, Marcucci M. Clinical trials and haemophilia: does the Bayesian approach make the ideal and desirable good friends? Haemophilia : the official journal of the World Federation of Hemophilia. 2009;15(4):900–3.

    CAS  Article  Google Scholar 

  15. 15.

    Gupta S, Faughnan ME, Tomlinson GA, Bayoumi AM. A framework for applying unfamiliar trial designs in studies of rare diseases. J Clin Epidemiol. 2011;64(10):1085–94.

    Article  Google Scholar 

  16. 16.

    Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, Nasser M, Meerpohl J, Post PN, Kunz R, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66(7):719–25.

    Article  Google Scholar 

  17. 17.

    Andrews JC, Schunemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, Rind D, Montori VM, Brito JP, Norris S, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol. 2013;66(7):726–35.

    Article  Google Scholar 

  18. 18.

    Pai M, Iorio A, Meerpohl J, Taruscio D, Laricchiuta P, Mincarone P, Morciano C, Leo CG, Sabina S, Akl E, et al. Developing methodology for the creation of clinical practice guidelines for rare diseases: a report from RARE-Bestpractices. Rare Diseases. 2015;3(1):e1058463.

    Article  Google Scholar 

  19. 19.

    Institute of Medicine. Clinical practice guidelines we can trust. Washington DC: National Academy of Sciences; 2011.

    Google Scholar 

  20. 20.

    Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW, Phillips B, Lelgemann M, Lethaby A, Bousquet J, et al. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy. 2009;64(5):669–77.

    CAS  Article  Google Scholar 

  21. 21.

    Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M, Brignardello-Petersen R, Laisaar KT, Kowalski S, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne. 2014;186(3):E123–42.

    Article  Google Scholar 

  22. 22.

    Pai M, Key NS, Skinner M, Curtis R, Feinstein M, Kessler C, Lane SJ, Makris M, Riker E, Santesso N, et al. NHF-McMaster guideline on care models for Haemophilia management. Haemophilia : the official journal of the World Federation of Hemophilia. 2016;22(Suppl 3):6–16.

    Article  Google Scholar 

  23. 23.

    Pai M, Santesso N, Yeung CH, Lane SJ, Schunemann HJ, Iorio A. Methodology for the development of the NHF-McMaster guideline on care models for Haemophilia management. Haemophilia : the official journal of the World Federation of Hemophilia. 2016;22(Suppl 3):17–22.

    Article  Google Scholar 

  24. 24.

    Lane SJ, Sholapur NS, Yeung CH, Iorio A, Heddle NM, Sholzberg M, Pai M. Understanding stakeholder important outcomes and perceptions of equity, acceptability and feasibility of a care model for haemophilia management in the US: a qualitative study. Haemophilia : the official journal of the World Federation of Hemophilia. 2016;22(Suppl 3):23–30.

    Article  Google Scholar 

  25. 25.

    Yeung CH, Santesso N, Pai M, Kessler C, Key NS, Makris M, Navarro-Ruan T, Soucie JM, Schunemann HJ, Iorio A. Care models in the management of haemophilia: a systematic review. Haemophilia : the official journal of the World Federation of Hemophilia. 2016;22(Suppl 3):31–40.

    Article  Google Scholar 

  26. 26.

    Yeung CH, Santesso N, Zeraatkar D, Wang A, Pai M, Sholzberg M, Schunemann HJ, Iorio A. Integrated multidisciplinary care for the management of chronic conditions in adults: an overview of reviews and an example of using indirect evidence to inform clinical practice recommendations in the field of rare diseases. Haemophilia : the official journal of the World Federation of Hemophilia. 2016;22(Suppl 3):41–50.

    Article  Google Scholar 

  27. 27.

    Asherson RA. The catastrophic antiphospholipid syndrome. J Rheumatol. 1992;19(4):508–12.

    CAS  PubMed  Google Scholar 

  28. 28.

    Cervera R. CAPS registry. Lupus. 2012;21(7):755–7.

    CAS  Article  Google Scholar 

  29. 29.

    de Groot PG, Derksen RH. Pathophysiology of the antiphospholipid syndrome. J Thromb Haemost : JTH. 2005;3(8):1854–60.

    Article  Google Scholar 

  30. 30.

    Strahan JE, Canfield MA, Drummond-Borg LM, Neill SU. Ethnic and gender patterns for the five congenital disorders in Texas from 1992 through 1998. Tex Med. 2002;98(9):80–6.

    PubMed  Google Scholar 

  31. 31.

    Taruscio D, Morciano C, Laricchiuta P, Mincarone P, Palazzo F, Leo CG, Sabina S, Guarino R, Auld J, Sejersen T, et al. RARE-Bestpractices: a platform for sharing best practices for the management of rare diseases. Orphanet J Rare Dis. 2014;9(Suppl 1):O14.

    Article  Google Scholar 

  32. 32.

    Tugwell P, Knottnerus JA. Adolopment - a new term added to the clinical epidemiology lexicon. J Clin Epidemiol. 2017;81:1–2.

    Article  Google Scholar 

  33. 33.

    Schunemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, Brignardello-Petersen R, Neumann I, Falavigna M, Alhazzani W, et al. GRADE evidence to decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.

    Article  Google Scholar 

  34. 34.

    Terracciano L, Brozek J, Compalati E, Schunemann H. GRADE system: new paradigm. Curr Opin Allergy Clin Immunol. 2010;10(4):377–83.

    Article  Google Scholar 

  35. 35.

    Oxman A. DECIDE: Developing and Evaluating Communication Strategies to Support Informed Decisions and Practices Based on Evidence. In: European Commission under the Seventh Framework Programme; 2011-2015.

    Google Scholar 

  36. 36.

    Almonte T, Decker K, Seroski W, Walker I, Webert K, Bos C, Strike K, Waterhouse L, Goldsmith R, Chan AKC. The importance of haemophilia treatment Centre administrators in patient care. Haemophilia : the official journal of the World Federation of Hemophilia. 2012;18(5):e366–7.

    CAS  Article  Google Scholar 

  37. 37.

    Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, McGinn T, Hayden J, Williams K, Shea B, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ (Clinical research ed). 2015;350:h870.

    Google Scholar 

  38. 38.

    Spencer FA, Iorio A, You J, Murad MH, Schunemann HJ, Vandvik PO, Crowther MA, Pottie K, Lang ES, Meerpohl JJ, et al. Uncertainties in baseline risk estimates and confidence in treatment effects. BMJ (Clinical research ed). 2012;345:e7401.

    Google Scholar 

  39. 39.

    National Heart, Lung, and Blood Institute. Evidence-based management of sickle cell disease: expert panel report, 2014. Washington, DC: National Institutes of Health; 2014.

  40. 40.

    Rode J. Rare diseases: understanding this public health priority. Paris: EURORDIS; 2005.

  41. 41.

    MacLean S, Mulla S, Akl EA, Jankowski M, Vandvik PO, Ebrahim S, McLeod S, Bhatnagar N, Guyatt GH. Patient values and preferences in decision making for antithrombotic therapy: a systematic review: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141(2 Suppl):e1S–23S.

    CAS  Article  Google Scholar 

  42. 42.

    Taruscio D, Gainotti S, Mollo E, Vittozzi L, Bianchi F, Ensini M, Posada M. The current situation and needs of rare disease registries in Europe. Public health genomics. 2013;16(6):288–98.

    CAS  Article  Google Scholar 

  43. 43.

    Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.

    Article  Google Scholar 

  44. 44.

    Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

No funding has been received in support of the preparation of this manuscript. The research projects leading to these results have received funding from National Hemophilia Foundation (hemophilia guideline), as well as the European Union Seventh Framework Programme (FP7/2007–2013) under Grant Agreement n° 305690 (RARE-Bestpractices project, www.rarebestpractices.eu). The opinions presented here reflect only the authors’ views. Neither the National Hemophilia Foundation nor the European Union are liable for any use that may be made of the information contained herein.

Availability of data and materials

The data that support the findings of this study are available from the authors of the three named clinical practice guidelines but restrictions apply to the availability of data from the CAPS and SCD guidelines; these data are currently being used to produce manuscripts for publication, and so are not publicly available. Data are however available from the authors of this manuscript upon reasonable request and with permission of the authors of the three named clinical practice guidelines.

Author information

Affiliations

  1. McMaster University, Hamilton, Canada

    Menaka Pai, Cindy H. T. Yeung, Elie A. Akl, Christopher Hillis, Kimberly Legault, Nancy Santesso, Madeleine Verhovsek, Holger J. Schünemann & Alfonso Iorio

  2. Hamilton Regional Laboratory Medicine Program, Hamilton, Canada

    Menaka Pai & Madeleine Verhovsek

  3. American University of Beirut GRADE Center, Beirut, Lebanon

    Elie A. Akl & Andrea Darzi

  4. Cochrane Germany, Medical Center, University of Freiburg, Freiburg, Germany

    Joerg J. Meerpohl

  5. National Center for Rare Diseases, Istituto Superiore di Sanità, Rome, Italy

    Domenica Taruscio

  6. Cochrane Canada, Hamilton, Canada

    Nancy Santesso & Holger J. Schünemann

  7. Hamilton General Hospital, Room 1-270A, 237 Barton Street East, Hamilton, ON, L8L 2X2, Canada

    Menaka Pai

Authors
  1. Menaka Pai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cindy H. T. Yeung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elie A. Akl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrea Darzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christopher Hillis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kimberly Legault
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Joerg J. Meerpohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Nancy Santesso
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Domenica Taruscio
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Madeleine Verhovsek
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Holger J. Schünemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Alfonso Iorio
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MP, CY, MV and AI were responsible for the conception, design, and drafting of this manuscript. All authors (MP, CY, EA, AD, CH, KL, JM, NS, DT, MV, HS, AI) made substantial contributions to acquisition, analysis, and interpretation of data. All authors (MP, CY, EA, AD, CH, KL, JM, NS, DT, MV, HS, AI) were involved in revising and giving final approval of the submitted manuscript. All authors (MP, CY, EA, AD, CH, KL, JM, NS, DT, MV, HS, AI) have agreed both to be personally accountable for their own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Menaka Pai.

Ethics declarations

Authors’ information

Not applicable.

Ethics approval and consent to participate

The qualitative study conducted as part of the NHF-McMaster hemophilia guideline was submitted to and approved by the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada. Written informed consent was obtained from all participants in the qualitative study. Ethics approval for the remainder of the described work (surveys of guideline panel members, whose ethical treatment and consent to participate in guideline related activities was considered implicit, as well as systematic reviews of anonymized and published data) was not sought, as it was determined not to require ethics submission and approval as per the Hamilton Integrated Research Ethics Board in Hamilton, Ontario, Canada.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pai, M., Yeung, C.H.T., Akl, E.A. et al. Strategies for eliciting and synthesizing evidence for guidelines in rare diseases. BMC Med Res Methodol 19, 67 (2019). https://doi.org/10.1186/s12874-019-0713-0

Download citation

Keywords

  • Clinical practice guideline
  • Evidence-based medicine
  • Health care quality, access, and evaluation
  • Rare diseases

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us