- Research article
- Open Access
An exploration of how developers use qualitative evidence: content analysis and critical appraisal of guidelines
BMC Medical Research Methodology volume 20, Article number: 160 (2020)
Clinical practice guidelines have become increasingly widely used to guide quality improvement of clinical practice. Qualitative research may be a useful way to improve the quality and implementation of guidelines. The methodology for qualitative evidence used in guidelines development is worthy of further research.
A comprehensive search was made of WHO, NICE, SIGN, NGC, RNAO, PubMed, Embase, Web of Science, CNKI, Wanfang, CBM, and VIP from January 1, 2011 to February 25, 2020. Guidelines which met IOM criteria and were focused on clinical questions using qualitative research or qualitative evidence, were included. Four authors extracted significant information and entered this onto data extraction forms. The Appraisal of Guidelines for Research and Evaluation (AGREE II) tool was used to evaluate the guidelines’ quality. The data were analyzed using SPSS version 17.0 and R version 3.3.2.
Sixty four guidelines were identified. The overall quality of the guidelines was high (almost over 60%). Domain 1 (Scope and Purpose) was ranked the highest with a median score of 83% (IQ 78–83). Domain 2 (Stakeholder involvement) and Domain 5 (Applicability) were ranked the lowest with median scores of 67% (IQ 67–78) and 67% (IQ 63–73) respectively. 20% guidelines used qualitative research to identify clinical questions. 86% guidelines used qualitative evidence to support recommendations (mainly based on primary studies, a few on qualitative evidence synthesis). 19% guidelines applied qualitative evidence when considering facilitators and barriers to recommendations’ implementation. 52% guideline developers evaluated the quality of the primary qualitative research study using the CASP tool or NICE checklist for qualitative studies. No guidelines evaluated the quality of qualitative evidence synthesis to formulate recommendations. 17% guidelines presented the level of qualitative research using the grade criteria of evidence and recommendation in different forms such as I, III, IV, very low. 28% guidelines described the grades of the recommendations supported by qualitative and quantitative evidence. No guidelines described the grade of recommendations only supported by qualitative evidence.
The majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to implementation of recommendations’. However, more attention needs to be paid to the methodology. For example, no experts proficient in qualitative research were involved in guideline development groups, no assessment of the quality of qualitative evidence synthesis was included and there was lack of details reported on the level of qualitative evidence or grade of recommendations.
Qualitative research can be defined as research that involves “the collection, analysis and interpretation of data that are not easily reduced to numbers; these data relate to the social world and the concepts and behaviors of people within it” . Data from qualitative research can address certain types of significant questions that may not be answered by quantitative research methods, such as “how” and “why”a given intervention engenders its effects. Qualitative research is now widely used for a variety of purposes in the field of healthcare, for example, the identification of patients’ concerns, the manner in which people select and use healthcare services, and the circumstances under which healthcare interventions play a role in practice [2, 3].
Taking the merits of qualitative research into account, it has attracted the attention of guideline developers and is gradually becoming accepted to inform guideline recommendations, for example WHO (World Health Organization) has affirmed in its handbook for guideline development that qualitative evidence should be considered and used in the process of guideline development and the WHO Guidelines Review Committee (GRC) internet site also provides additional guidance on when and how to use qualitative research data to inform WHO guidelines . Many professional scholars and researchers have also used qualitative research or evidence to conduct projects on the development and implementation of guidelines such as addressing questions about the values and preferences of relevant stakeholders (e.g., patients, caregivers, and the public), the acceptability and feasibility of the interventions and the influence of the interventions on equity and human rights [4,5,6,7,8,9]. This provides opportunities for qualitative research methodologists to be involved in the process of developing guideline recommendations [10, 11] and exploring facilitators of and barriers to the guideline’s implementation .
As Lewin & Glenton said, qualitative research may be entering a new era of being used in the process of guideline development, and it is beneficial for decision making . Our aim was to further understanding of the way qualitative evidence has been used in the process of the existing guideline development process, for example, whether qualitative evidence was retrieved or how many recommendations are supported by qualitative evidence. To achieve this we conducted a systematic search, a rigorous quality evaluation of guidelines, and comprehensive information extraction related to qualitative evidence in guidelines. We also performed content analysis for the purpose of providing clear views on the roles and functions of qualitative evidence in the process of guideline development.
The systematic review was performed according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines .
Criteria for guideline selection
We included guidelines focused on improving healthcare that met the following criteria: 1) the guidelines were primarily published in Chinese or English from January 1, 2011 to February 25, 2020. In 2011, IOM (Institute of Medicine) claimed that for a CPG to be trustworthy it needs to “be developed via a transparent process by a group of multidisciplinary experts (including patient representatives), screened for minimal potential bias and conflicts of interest, and supported by a systematic review of the evidence” . This, which is the first statement of criteria for clinical practice guidelines, plays an important role in guideline development, so we chose it as the start date for retrieval; 2) the guidelines met the above mentioned IOM criteria; 3) the guidelines mainly focused on clinical questions, such as diagnosis, treatment or care for certain diseases or patients symptoms, to provide suggestions for healthcare staff or community health services; 4) qualitative research or qualitative evidence was used in the process of guidelines development; 5) if the guidelines were updated, only the most recent version of the guidelines were included. The guidelines were excluded, if they had the following characteristics: 1) the same guidelines had been repeatedly published in multiple journals; 2) the full texts of guidelines were not available.
Search strategy for guidelines
Relevant representative guidelines repositories, such as WHO, NICE (the National Institute for Health and Care Excellence), SIGN (Scottish Intercollegiate Guidelines Network), NGC (National Guideline Clearinghouse), RNAO (Registered Nurses’ Association of Ontario), and other databases, including three English databases (PubMed, Embase, Web of Science), four Chinese databases (China National Knowledge Infrastructure, CNKI; Wanfang Data; Chinese BioMedical Literature Database, CBM; and VIP Database for Chinese Technical Periodicals, VIP), were systematically searched from January 1, 2011 to February 25, 2020. The search strategy used MeSH terms, Title/Abstract and text words. Taking PubMed as an example, the retrieval strategy is shown in Fig. 1.
Guidelines selection and data extraction
Three (C.L.,Y.X.S and J.Z) authors experienced in literature retrieval independently selected eligible guidelines. Three reviewers (D.D.L.,Y.C and C.F) extracted significant information from the guidelines and completed data extraction forms by means of reading the text content of the guideline, references and the online relevant attachments. The detailed process of data extraction is presented in Additional file 1. The forms included: (1) the basic characteristics of included guidelines (such as title, publication/update date, and developer); (2) how qualitative research or evidence was used in the process of the guidelines development (were experts proficient in qualitative research invited to be involved in guideline development group, was qualitative research used to identify clinical questions, was qualitative evidence retrieved; was this used to support recommendations; and was this applied when considering facilitators and barriers to recommendations’ implementation); (3) details of the methodology for qualitative research or evidence used in the development process of guidelines (such as qualitative research quality assessment tool, the quality of the primary qualitative research study used to formulate recommendations and the grade of recommendations supported by qualitative evidence).
We hypothesized that the development of guidelines using qualitative research or evidence would be relevant to these items in the forms. The hypothesis was based on related methodological literature, COnsolidated criteria for REporting Qualitative research (COREQ) checklists  and discussion between all authors with methodologists in evidence-based guidelines development who were willing to engage in dialogue with us. Another researcher (Y.H.J) examined the data extraction forms to make sure no errors had occurred.
Appraisal of included guidelines
Two researchers (Y.YW and D.H) independently evaluated the quality of the guidelines by using the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool, which consists of 23 items under 6 domains involving scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence . Each item was rated from 1 to 7 points with 1 point for “strongly disagree” and 7 points for “strongly agree”. We summarized the domain scores individually and scaled the total of that domain, calculated by the following formula: (obtained score - minimal possible score)/(maximal possible score - minimal possible score) × 100% .
Descriptive statistics were computed for the scores for each AGREE domain. Data for each AGREE II domain were provided as medians and interquartile ranges (IQRs). Intraclass correlation coefficients (ICCs) were calculated to evaluate the agreement between two reviewers for each domain [18, 19]. When the ICC value was less than 0.4, the consistency between raters was poor; if the ICC range was from 0.4 ~ 0.75, the consistency between raters was moderate; and a value of ICC over 0.75 the consistency was high . The data were analyzed using SPSS version 17.0 (SPSS Inc. Chicago, IL, USA) and R version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria) for Windows.
Guideline identification and selection
The searches identified 10,245 discrete records, of which 449 were selected for a full-text review. Sixty-four guidelines were eventually included [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84]. The flow diagram for the guidelines is shown in Fig. 2.
Characteristics of included guidelines
As Table 1 shows, the sixty-four guidelines concentrated on different topics such as cancers, chronic pain and smoking, and were developed by NICE, SIGN, RNAO, WHO or other professional organizations. The majority of guideline developers used GRADE (the Grading of Recommendations Assessment, Development and Evaluation) criteria for grading of evidence and recommendations. When formulating recommendations, they considered the quality of evidence, the risk-benefit analysis of some interventions, supporting resources and stakeholders’ values and preferences. The number of recommendations ranged from 2 to 262. The largest number of recommendations supported only by qualitative evidence in each included guideline was 8 . The largest number of recommendations supported by both qualitative and quantitative evidence in each included guideline was 23 . The majority of recommendations were supported by qualitative evidence based on primary studies, a few on systematic reviews).
Quality appraisal of the guidelines
The ICC values for all six domains were over 0.75, which indicated high consistency in the assessment results between the two raters.
As Table 2 and Fig. 3 show. The final domain scores ranged between 0% (domain 6 of 6 guidelines) [75, 77, 78, 81, 82, 84] and 96% (domain 6 of 11 guidelines) [21, 22, 25,26,27, 29,30,31,32,33,34]. When comparing the total domain scores, Domain 1 (Scope and Purpose) was ranked the highest with a median score of 83% (IQ 78–83). Domain 2 (Stakeholder involvement) and Domain 5 (Applicability) were ranked the lowest with median scores of 67% (IQ 67–78) and 67% (IQ 63–73) respectively. The median scores of Domains 3, 4, 6 (Rigour of development, Clarity of presentation, Editorial independence) were 71% (IQ 69–74), 72% (IQ 58–78) and 79% (IQ 75–83) respectively.
The process of the guidelines development using qualitative research or evidence
As Fig. 4 shows, no guideline developers invited experts proficient in qualitative research to be involved in guideline development groups. 20% guidelines (13/64) used qualitative research to identify clinical questions [68, 71, 73,74,75, 77,78,79,80,81,82,83,84]. 83% (53/64) guidelines retrieved qualitative evidence [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70, 75, 77, 81]. 86% (55/64) guidelines used qualitative evidence to support recommendations [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70, 72, 75,76,77, 81]. And 19% (12/64) guidelines applied qualitative evidence when considering facilitators and barriers to recommendations’ implementation [55, 56, 60, 62,63,64,65,66,67,68,69,70].
The methodology for evidence used in the guidelines development
As Table 3 shows, one guideline used qualitative research based on grounded theory, phenomenology . 52% (27/52) guideline developers evaluated the quality of the primary qualitative research study using the CASP (the Critical Appraisal Skills Programme) tool or NICE checklist for qualitative studies [35, 38, 46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70]. No guidelines evaluated (0/18) the quality of qualitative evidence synthesis used to formulate recommendations. 17% (11/64) guidelines presented the level of qualitative research using the grade criteria of evidence and recommendation in different forms such as I, III, IV, very low [35,36,37,38,39,40, 42, 44, 73, 77, 81]. They were based on JBI, GRADE or adapted from SIGN or Pati D. A framework [35,36,37,38,39,40,41,42,43,44,45, 85,86,87] respectively. 28% guidelines (15/54) described the grades of the recommendations supported by qualitative and quantitative evidence in different ways such as “strong”, “good”, “B”, “C” or “D” and “weak” [21, 22, 24, 25, 27, 28, 30,31,32,33,34, 73, 76, 77, 81], which also complied with JBI, GRADE or adapted from SIGN and (or) Pati D. A framework respectively. But no guidelines (0/10) described the grade of recommendations supported only by qualitative evidence.
Our review shows that the majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to recommendations’ implementation. However, the methodology still needs more attention, as there were, no experts proficient in qualitative research involved in guideline development group, no assessment of the quality of qualitative evidence synthesis and a lack of detailed reporting the level of qualitative evidence and its grade of recommendations’.
The summary findings of this review
The majority of the included guidelines introduced the overall aim of the guideline, the specific health questions, and the target population in tabulated form, bold, or using separate paragraphs. They described the gathering and synthesis of the evidence, gave details of updating and dealt with the language, structure, and format of the guideline recommendations.. However, the guidelines still had some noticeable shortcomings. For instance, a few guidelines did not describe the methods of formulating recommendations [74, 76, 82]; a few did not clearly introduce the different options for management of the conditions or health issues [76, 82]; a minority of guidelines did not give details of conflict of interest statements [75, 77, 78, 81, 82, 84]. In addition, although the majority of the guidelines stated that the guideline development group consisted of all relevant professional experts, and clearly defined the guidelines’ target users, a number of developers did not consider values and preferences of the target population [71, 78, 83, 84] or lacked adequate information on how they gained patients, doctors or other stakeholders’ views. And also the majority of the guidelines did not describe facilitators and barriers to their application in detail.
The methodological quality of qualitative evidence affects interpretation of its results. Unfortunately, while the majority of guidelines developers used qualitative evidence synthesis to formulate recommendations, they did not appraise confidence in each individual review, which resulted in some difficulties in explaining relevant themes or theories formulated in different articles. In addition, only three of the grade systems used, referred to single qualitative studies or synthesis of qualitative research as a level of the grade criteria of evidence and recommendation [35,36,37,38,39,40,41,42,43,44,45, 85,86,87]. The majority of guideline developers did not concentrate on the important influence of qualitative evidence on the grade criteria of evidence and recommendation.
Comparison of findings with prior research
When comparing our findings with similar relevant articles, lack of statements about conflict of interest, details on how to gain patients, doctors or other stakeholders’ views, consideration of facilitators and barriers to guidelines’ implementation are also common issues e.g. oncology CPGs , inflammatory bowel disease guidelines , nursing CPGs , guidelines for management of cholangiocarcinoma . Our review firstly identified whether qualitative research or evidence had been used to obtain stakeholders’ values and preferences, and in identifying facilitators and barriers to guidelines’ implementation in the process of guidelines development. Other researchers also used qualitative research to explore practice gaps based on existing guidelines: Feyissa et al. used a semi-structured interview to assess contextual barriers and facilitators to the implementation of a guideline developed to reduce HIV-related stigma and discrimination (SAD) in the Ethiopian healthcare setting ; Lind et al. interviewed local politicians, chief medical officers and health professionals at acute care hospitals to investigate perceptions regarding guidelines for palliative care and identify obstacles and opportunities for their implementation in acute care hospitals .
In Addition, qualitative research is increasingly being recognised as having an important role to play in addressing questions relating to interventions or system complexity, and guideline development processes. As with our topic, other researchers have also focused on the methodology of involving qualitative research in the development process of guidelines. Flemming et al. provided guidance for the choice of qualitative evidence synthesis methods in the context of guideline development for complex interventions by using a best fit framework synthesis to address interactions between components of complex interventions; interactions of interventions with context and multiple (health and non-health) outcomes; using meta-ethnography to deal with sociocultural acceptability of an intervention . In addition, Moore et al. also put forward designs and methods for the applicability of quantitative and qualitative evidence in guidelines including complexity-related questions of interest in the guideline, types of synthesis used in the guideline, mixed-method review design and integration mechanisms, observations, concerns and considerations .
Implications for guideline developers
The development of guidelines is a complex undertaking which needs a significant focus on its methodology. Based on our findings, we put forward some proposals for guideline developers, which may be helpful to improve their guideline’s quality. Firstly, guidelines developers can record and report details about how they reach agreement on recommendations and how they deal with possible disagreement when formulating recommendations and present different options for the same CQs with information on population characteristics or clinical situations for each option. Secondly, they can also develop a series of methods to avoid potential COI before the initiation of the guideline development project. Guideline developers may also obtain the target population’ views by interviewing stakeholders or extracting some relevant themes from existing qualitative data on the topic of interest. Finally, guideline developers should formally consider how to evaluate and grade single qualitative studies or synthesis of qualitative research into the grade system for guideline development prior to start-up of the guideline development project, and identify which factors influence the grade classification with the help of experts proficient in qualitative research. They should also select appropriate tools to appraise the quality of qualitative evidence such as CASP tool, NICE checklist for primary studies, GRADE-CERQual (Grading of Recommendations Assessment, Development and Evaluation-Confidence in the Evidence from Reviews of Qualitative research) for qualitative evidence synthesis, which is an approach for assessing how much confidence to place in findings from qualitative evidence syntheses in terms of four components (methodological limitations, coherence, adequacy of data, relevance) [13, 96].
Limitations and strengths
Our study has some potential limitations. Firstly, although we selected eligible guidelines by means of reading their text content, references and the online relevant attachments, we used a quick search strategy on the guideline development. We also used the filter capability when using Endnote to manage literature from databases. But because of the size of the task there may be selection bias because of unavailable guidelines published in government documents, books or other guideline publication platforms. Additionally, we did not specify how many guidelines were recommended, recommended with modifications, and not recommended, because AGREE II protocol states that no overall score is calculated to determine if a CPG is recommended or not recommended and the main focus of this article was the methodology for qualitative research or qualitative evidence used in guidelines development . Nonetheless, there may be several advantages. Firstly, a systematic literature search was performed for screening eligible guidelines. Secondly, we discussed the potential effect of qualitative research or evidence on the AGREE II appraisal, and then put forward some suggestions on how to use qualitative research or evidence to improve the quality of future guidelines. Thirdly, this is the first attempt to systematically analyze the role of qualitative research or evidence in guidelines development based on published guidelines.
Suggestions for ongoing research
Qualitative research or qualitative evidence will be extensively used in the guideline development process in the future. There are three interesting topics needing further research. Firstly, when available data exists, this can be explored to provide more reliable conclusions related to the potential association between AGREE appraisal and the identification, incorporation and reporting of qualitative research by means of statistical methods such as non-parametric tests. Secondly, it will be interesting to compare the use of qualitative and quantitative data when formulating recommendations in guidelines, perhaps by matching guidelines on similar topics or key questions, and comparing those which did and didn’t use use qualitative evidence. Thirdly, exploring how qualitative research may be used to obtain the information related to conflict of interest will also be useful to inform guideline transparency. These topics are worthy of future exploration.
The majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to recommendations’ implementation. However, more attention needs to be given to the methodology, for instance, no experts proficient in qualitative research have been involved in guideline development group, there has been no assessment of the quality of qualitative evidence synthesis, and there is a lack of detail when reporting on the level of qualitative evidence and its grade recommendations’.
Availability of data and materials
All data generated or analyzed are included in this published article.
World Health Organization
Guidelines Review Committee
Scottish Intercollegiate Guidelines Network
National Guideline Clearinghouse
The National Institute for Health and Care Excellence
Registered Nurses’ Association of Ontario
China National Knowledge Infrastructure
Chinese BioMedical Literature Database
VIP Database for Chinese Technical Periodicals
- AGREE II:
The Appraisal of Guidelines for Research and Evaluation
Intraclass Correlation Coefficient
The Grading of Recommendations Assessment, Development and Evaluation
Intensive Care Unit
The Critical Appraisal Skills Programme
The Joanna Briggs Institute
Grading of Recommendations Assessment, Development and Evaluation-Confidence in the Evidence from Reviews of Qualitative research
Murphy E, Dingwall R, Greatbatch D, et al. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998;2:iii.
Payne JB, Dance KV, Farone M, et al. Patient and caregiver perceptions of lymphoma care and research opportunities: A qualitative study. Cancer. 2019;125(22):4096–104.
Moses C, Flegg K, Dimaras H. Patient knowledge, experiences and preferences regarding retinoblastoma and research: A qualitative study. Health Expect. 2020;10.1111/hex.13043. https://doi.org/10.1111/hex.13043. [Published online ahead of print, 2020 Feb 29].
WHO. Handbook for Guideline Development (2nd ed). Available at: https://www.who.int/publications/guidelines/guidelines_review_committee/en/.
Tan TP, Stokes T, Shaw EJ. Use of qualitative research as evidence in the clinical guideline program of the National Institute for Health and Clinical Excellence. Int J Evid Based Healthc. 2009;7(3):169–72.
NICE. Developing NICE Guidelines the Manual. 2017. Available at: https://www.nice.org.uk/process/pmg20/chapter/introduction-andoverview.
Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the Optimize MNH guidance process. Implement Sci. 2016;11:98.
SIGN. A guideline developer’s handbook. 2015. Available at: http://www.sign.ac.uk/sign-50.html.
Carroll C. Qualitative evidence synthesis to improve implementation of clinical guidelines. BMJ. 2017;356:j80.
Kelson M, Akl EA, Bastian H, et al. Integrating values and consumer involvement in guidelines with the patient at the center: article 8 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):262–8.
Armstrong MJ, Mullins CD, Gronseth GS, et al. Impact of patient involvement on clinical practice guideline development: a parallel group study. Implement Sci. 2018;13(1):55.
Mudge S, Hart A, Murugan S, et al. What influences the implementation of the New Zealand stroke guidelines for physiotherapists and occupational therapists? Disabil Rehabil. 2017;39(5):511–8.
Lewin S, Glenton C. Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(1):126.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement, the PRISMA GROUP. PLoS Med. 2009;6(6):e1000097 https://doi.org/10.1371/journal/pmed.1000097.
Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011. Available at: http://www.nap.edu/catalog/13058.html.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.
AGREE Next Steps Consortium. The AGREE II Instrument [Electronic version]. 2017. Available at: http://www.agreecollaboration.org.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Kramer MS, Feinstein AR. Clinical biostatistics: LIV. The biostatistics of concordance. Clin Pharmacol Ther. 1981;29(1):111–23.
Shrout PE, Fleiss JL. Intraclass correlaions: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8.
SIGN. Management of epithelial ovarian cancer. 2018. Available at: https://www.sign.ac.uk/sign-135-management-of-epithelial-ovarian-cancer.
SIGN. Diagnosis and management of epilepsy in adults. 2018. Available at: https://www.sign.ac.uk/sign-143-diagnosis-and-management-of-epilepsy-in-adults.
SIGN. Management of stable angina. 2018. Available at: https://www.sign.ac.uk/sign-151-stable-angina.
SIGN. Cardiac rehabilitation. 2017. Available at: https://www.sign.ac.uk/sign-150-cardiac-rehabilitation.
SIGN. Assessment, diagnosis and interventions for autism spectrum disorders. 2016. Available at: https://www.sign.ac.uk/sign-145-assessment,-diagnosis-and-interventions-for-autism-spectrum-disorders.
SIGN. Management of chronic heart failure. 2016. Available at: https://www.sign.ac.uk/sign-147-management-of-chronic-heart-failure.
SIGN. Acute coronary syndrome. 2016. Available at: https://www.sign.ac.uk/sign-148-acute-coronary-syndrome.
SIGN. British guideline on the management of asthma. 2016. Available at: https://www.sign.ac.uk/sign-153-british-guideline-on-the-management-of-asthma.
SIGN. Glaucoma referral and safe discharge. 2015. Available at: https://www.sign.ac.uk/sign-144-glaucoma-referral-and-safe-discharge.
SIGN. Brain injury rehabilitation in adults. 2013. Available at: https://www.sign.ac.uk/sign-130-brain-injury-rehabilitation-in-adults.
SIGN. Management of hepatitis C. 2013. Available at: https://www.sign.ac.uk/sign-133-management-of-hepatitis-c.
SIGN. Management of chronic pain. 2013. Available at: https://www.sign.ac.uk/sign-136-management-of-chronic-pain.
SIGN. Management of adult testicular germ cell tumours. 2011. Available at: https://www.sign.ac.uk/sign-124-management-of-adult-testicular-germ-cell-tumours.
SIGN. Diagnosis and management of colorectal cancer. 2011. Available at: https://www.sign.ac.uk/sign-126-diagnosis-and-management-of-colorectal-cancer.
RNAO. Implementing supervised injection services. 2018. Available at: https://rnao.ca/bpg/guidelines/implementing-supervised-injection-services.
RNAO. Promoting and supporting the initiation, exclusivity, and continuation of breastfeeding for newborns, infants, and young children. 2018. Available at: https://rnao.ca/bpg/guidelines/breastfeeding-promoting-and-supporting-initiation-exclusivity-and-continuation-breast.
RNAO. Adult asthma care: Promoting control of asthma (Second Edition). 2017. Available at: https://rnao.ca/bpg/guidelines/adult-asthma-care.
RNAO. Crisis intervention for adults using a trauma-informed approach: Initial four weeks of management (Third Edition). 2017. Available at: https://rnao.ca/bpg/guidelines/crisis-intervention.
RNAO. Delirium, dementia, and depression in older adults: Assessment and care (Second Edition). 2016. Available at: https://rnao.ca/bpg/guidelines/assessment-and-care-older-adults-delirium-dementia-and-depression.
RNAO. Person- and family-centred care. 2015. Available at: https://rnao.ca/bpg/guidelines/person-and-family-centred-care.
RNAO. Care transitions. 2014. Available at: https://rnao.ca/bpg/guidelines/care-transitions.
RNAO. Preventing and addressing abuse and neglect of older adults: Person-centred, collaborative, system-wide approaches. 2014. Available at: https://rnao.ca/bpg/guidelines/abuse-and-neglect-older-adults.
RNAO. Primary prevention of childhood obesity (Second Edition). 2014. Available at: https://rnao.ca/bpg/guidelines/primary-prevention-childhood-obesity.
RNAO. Assessment and management of foot ulcers for people with diabetes (Second Edition). 2013. Available at: https://rnao.ca/bpg/guidelines/assessment-and-management-foot-ulcers-people-diabetes-second-edition.
RNAO. Promoting safety: Alternative approaches to the use of restraints. 2012. Available at: https://rnao.ca/bpg/guidelines/promoting-safety-alternative-approaches-use-restraints.
NICE. Depression in children and young people: identification and management. 2019. Available at: https://www.nice.org.uk/guidance/ng134.
NICE. Pancreatic cancer in adults: diagnosis and management. 2018. Available at: https://www.nice.org.uk/guidance/ng85.
NICE. Antimicrobial stewardship: changing risk related behaviours in the general population. 2017. Available at: https://www.nice.org.uk/guidance/ng63.pdf.
NICE. Eating disorders: recognition and treatment. 2017. Available at: https://www.nice.org.uk/guidance/ng69.
NICE. Healthcare-associated infections: prevention and control in primary and community care. 2017. Available at: https://www.nice.org.uk/guidance/cg139.
NICE. Hip fracture: management. 2017. Available at: https://www.nice.org.uk/guidance/cg124.
NICE. Immunisations: reducing differences in uptake in under 19s. 2017. Available at: https://www.nice.org.uk/guidance/ph21.
NICE. Intermediate care including reablement. 2017. Available at: https://www.nice.org.uk/guidance/ng74.
NICE. Suspected cancer: recognition and referral. 2017. Available at: https://www.nice.org.uk/guidance/ng12.
NICE. Coexisting severe mental illness and substance misuse: community health and social care services. 2016. Available at: https://www.nice.org.uk/guidance/ng58.
NICE. Oral health for adults in care homes. 2016. Available at: https://www.nice.org.uk/guidance/ng48.
NICE. Skin cancer prevention. 2016. Available at: https://www.nice.org.uk/guidance/ph32.
NICE. Maternal and child nutrition. 2014. Available at: https://www.nice.org.uk/guidance/ph11.
NICE. Needle and syringe programmes. 2014. Available at: https://www.nice.org.uk/guidance/ph52.
NICE. Physical activity: brief advice for adults in primary care. 2013. Available at: https://www.nice.org.uk/guidance/ph44.
NICE. Service user experience in adult mental health: improving the experience of care for people using adult NHS mental health services. 2011. Available at: https://www.nice.org.uk/guidance/cg136.
NICE. Type 2 diabetes prevention: population and community-level interventions. 2011. Available at: https://www.nice.org.uk/guidance/ph35.
WHO. WHO recommendations: intrapartum care for a positive childbirth experience. 2018. Available at: https://www.who.int/reproductivehealth/publications/intrapartum-care-guidelines/en.
WHO. Guidelines for managing advanced HIV disease and rapid initiation of antiretroviral therapy. 2017. Available at: https://www.who.int/hiv/pub/guidelines/advanced-HIV-disease/en.
WHO. Protecting, promoting and supporting breastfeeding in facilities providing maternity and newborn services. 2017. Available at: http://www.who.int/nutrition/publications/guidelines/breastfeeding-facilities-maternity-newborn/en.
WHO. Guidelines on HIV self-testing and partner notification: supplement to consolidated guidelines on HIV testing services. 2016. Available at: https://www.who.int/hiv/pub/vct/hiv-self-testing-guidelines/en.
WHO. WHO recommendations on antenatal care for a positive pregnancy experience. 2016. Available at: https://www.who.int/reproductivehealth/publications/maternal_perinatal_health/anc-positive-pregnancy-experience/en.
WHO. Health worker roles in providing safe abortion care and post-abortion contraception. 2015. Available at: https://www.who.int/reproductivehealth/publications/unsafe_abortion/abortion-task-shifting/en/-33k.
WHO. WHO recommendations on health promotion interventions for maternal and newborn health. 2015. Available at: https://www.who.int/maternal_child_adolescent/documents/health-promotion-interventions/en/-70k.
WHO. WHO recommendations Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting. 2012. Available at: https://www.who.int/reproductivehealth/publications/maternal_perinatal_health/978924504843/en.
National Center for Cardiovascular Diseases (NCCD), Cardiovascular Committee of Beijing Nursing Association (CCBNA), Research Center of Nursing Theory and Practice Chinese Academy of Medical Sciences & Peking Union Medical College (RC-NTPC-AMSPUMC), Evidence-based Medical Center of Lanzhou University (EBMCLU). Construction of Nursing Practice Guideline for Emergency Percutaneous Coronary Intervention. Chin J Nurs. 2019;54:36–41.
Committee of Breast Cancer Society (CBCS), Committee Specialist of Breast Surgeon (CSBS). Expert consensus on breast tumor plastic surgery and breast reconstruction (2018 Edition). China Oncol. 2018;28(6):439–80.
Maternity Hospital of Fudan University (MHFU), School of Nursing Fudan University (SNFU), Shanghai Evidence-based Nursing Center (SEBNC). Gestational diabetes mellitus clinical nursing practice guideline. 2018. Available at: http://nursing.ebn.fudan.edu.cn/WebContent_list.aspx?BID=263&SID=279.
Pediatric Hospital of Fudan University (PHFU), School of Nursing Fudan University (SNFU), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU), Shanghai Evidence-based Nursing Center (SEBNC). Evidence-based guidelines for breastfeeding of hospitalized newborns. 2017. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2018/7/201807042226014857_1.pdf.
Tian L, Li HL, Tao M, et al. Construction of clinical nursing guideline on cancer related fatigue in adults. Nurs Res China. 2017;31:1564–8.
Guangdong Medical Association Accelerated Rehabilitation Surgeons Branch (GD-MAARSB). Clinical application of anaesthesia in accelerated rehabilitation surgery of colorectal surgery in lingnan Expert consensus on operation specification (2016 Edition). J Dig Oncol (Electronic Edition). 2016;8(4):209–19.
Shanghai Public Health Clinical Center (SPHCC), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). HIV/AIDS nursing clinical practice guidelines. 2016. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/3/201903112057079741_1.pdf.
National Center for Cardiovascular Diseases (NCCD), Heart Failure Committee of Chinese Medical Association (HFCCMA), Beijing Nursing Society (BNS). Nursing practice guideline of acute heart failure. Chin Nurs Manage. 2016;16:1179–88.
JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU), Shanghai Evidence-based Nursing Center (SEBNC), Pediatric Hospital of Fudan University (PHFU). Clinical nursing practice guideline for enteral nutrition for infants with congenital heart disease. 2016. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/1/201901242006519494_1.pdf.
East China Hospital of Fudan University (ECHFU), School of Nursing Fudan University (SNFU). Clinical Practice guideline for nasogastric tube feeding among adult patients. 2015. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/2/201902281629196565_1.pdf.
Fudan University Shanghai Cancer Center (FUSCC), School of Nursing Fudan University (SNFU), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). Clinical practice guidelines of peripherally inserted central catheter (PICC) catheterization. 2014. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/7/201907150909333584_1.pdf.
Zhou YF, Qin W, Liu RY, et al. The construction of an evidence-based practice guideline on prevention and management of medication errors in hospitalized adult patients. J Nurs Sci. 2014;29:1–4.
Tang HT. Development of clinical practice guideline for oral care on critically ill patients with endotracheal intubation. Shanghai: School of Nursing Fudan University; 2013. p. 1–100.
JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). Clinical practice guideline on inpatient fall prevention. 2011. Available at: http://nursing.ebn.fudan.edu.cn/xzzx/UploadFiles/file/201510/1612274173499.pdf.
JBI. Levels of evidence FAME. Available at: http://joannabriggs.org/Levels%20of%20Evidence%20%20FAME.
JBI. Grades of Recommendation. Available at: http://joannabriggs.org/Grades%20of%20Recommendation.
The GRADE Working Group. What is GRADE? [EB/OL]. Available at: http://www.gradeworkinggroup.org.
Saleh RR, Majeed H, Tibau A, et al. Undisclosed financial conflicts of interest among authors of American Society of Clinical Oncology clinical practice guidelines. Cancer. 2019;125(22):4069–75.
Grindal AW, Rishad K, Scaffidi MA, et al. Financial Conflicts of Interest in Inflammatory Bowel Disease Guidelines. Inflamm Bowel Dis. 2019;25(4):642–5.
Wang Y, Yu S, Wang L, et al. The methodology for developing nursing clinical practice guidelines over recent decades in China: A critical appraisal using AGREE II. J Nurs Manag. 2020;28(4):976–97.
Paschalis G, Alan A, Roberts Keith J, et al. Appraisal of the current guidelines for management of cholangiocarcinoma-using the Appraisal of Guidelines Research and Evaluation II (AGREE II) Instrument. Hepatobiliary Surg Nutr. 2020;9:126–35.
Angela B, Adrienne Y, Alison M, et al. EXploring practice gaps to improve PERIoperativE Nutrition CarE (EXPERIENCE Study): a qualitative analysis of barriers to implementation of evidence-based practice guidelines. Eur J Clin Nutr. 2019;73(1):94–101.
Lind S, Wallin L, Brytting T, et al. Implementation of national palliative care guidelines in Swedish acute care hospitals: a qualitative content analysis of stakeholders' perceptions. Health Policy. 2017;121(11):1194–201.
Flemming KA, Booth A, Garside R, et al. Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Glob Health. 2019;4(Suppl 1):e000882.
Moore G, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.
Lewin S, Glenton C, Munthe-Kaas H, et al. Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.
Ethics approval and consent to participate
Consent for publication
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Wang, Y., Liang, D., Lu, C. et al. An exploration of how developers use qualitative evidence: content analysis and critical appraisal of guidelines. BMC Med Res Methodol 20, 160 (2020). https://doi.org/10.1186/s12874-020-01041-8
- Qualitative research
- Guideline development
- AGREE II