Skip to main content
Download PDF

An exploration of how developers use qualitative evidence: content analysis and critical appraisal of guidelines

BMC Medical Research Methodology volume 20, Article number: 160 (2020) Cite this article

Abstract

Background

Clinical practice guidelines have become increasingly widely used to guide quality improvement of clinical practice. Qualitative research may be a useful way to improve the quality and implementation of guidelines. The methodology for qualitative evidence used in guidelines development is worthy of further research.

Methods

A comprehensive search was made of WHO, NICE, SIGN, NGC, RNAO, PubMed, Embase, Web of Science, CNKI, Wanfang, CBM, and VIP from January 1, 2011 to February 25, 2020. Guidelines which met IOM criteria and were focused on clinical questions using qualitative research or qualitative evidence, were included. Four authors extracted significant information and entered this onto data extraction forms. The Appraisal of Guidelines for Research and Evaluation (AGREE II) tool was used to evaluate the guidelines’ quality. The data were analyzed using SPSS version 17.0 and R version 3.3.2.

Results

Sixty four guidelines were identified. The overall quality of the guidelines was high (almost over 60%). Domain 1 (Scope and Purpose) was ranked the highest with a median score of 83% (IQ 78–83). Domain 2 (Stakeholder involvement) and Domain 5 (Applicability) were ranked the lowest with median scores of 67% (IQ 67–78) and 67% (IQ 63–73) respectively. 20% guidelines used qualitative research to identify clinical questions. 86% guidelines used qualitative evidence to support recommendations (mainly based on primary studies, a few on qualitative evidence synthesis). 19% guidelines applied qualitative evidence when considering facilitators and barriers to recommendations’ implementation. 52% guideline developers evaluated the quality of the primary qualitative research study using the CASP tool or NICE checklist for qualitative studies. No guidelines evaluated the quality of qualitative evidence synthesis to formulate recommendations. 17% guidelines presented the level of qualitative research using the grade criteria of evidence and recommendation in different forms such as I, III, IV, very low. 28% guidelines described the grades of the recommendations supported by qualitative and quantitative evidence. No guidelines described the grade of recommendations only supported by qualitative evidence.

Conclusions

The majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to implementation of recommendations’. However, more attention needs to be paid to the methodology. For example, no experts proficient in qualitative research were involved in guideline development groups, no assessment of the quality of qualitative evidence synthesis was included and there was lack of details reported on the level of qualitative evidence or grade of recommendations.

Peer Review reports

Background

Qualitative research can be defined as research that involves “the collection, analysis and interpretation of data that are not easily reduced to numbers; these data relate to the social world and the concepts and behaviors of people within it” [1]. Data from qualitative research can address certain types of significant questions that may not be answered by quantitative research methods, such as “how” and “why”a given intervention engenders its effects. Qualitative research is now widely used for a variety of purposes in the field of healthcare, for example, the identification of patients’ concerns, the manner in which people select and use healthcare services, and the circumstances under which healthcare interventions play a role in practice [2, 3].

Taking the merits of qualitative research into account, it has attracted the attention of guideline developers and is gradually becoming accepted to inform guideline recommendations, for example WHO (World Health Organization) has affirmed in its handbook for guideline development that qualitative evidence should be considered and used in the process of guideline development and the WHO Guidelines Review Committee (GRC) internet site also provides additional guidance on when and how to use qualitative research data to inform WHO guidelines [4]. Many professional scholars and researchers have also used qualitative research or evidence to conduct projects on the development and implementation of guidelines such as addressing questions about the values and preferences of relevant stakeholders (e.g., patients, caregivers, and the public), the acceptability and feasibility of the interventions and the influence of the interventions on equity and human rights [4,5,6,7,8,9]. This provides opportunities for qualitative research methodologists to be involved in the process of developing guideline recommendations [10, 11] and exploring facilitators of and barriers to the guideline’s implementation [12].

As Lewin & Glenton said, qualitative research may be entering a new era of being used in the process of guideline development, and it is beneficial for decision making [13]. Our aim was to further understanding of the way qualitative evidence has been used in the process of the existing guideline development process, for example, whether qualitative evidence was retrieved or how many recommendations are supported by qualitative evidence. To achieve this we conducted a systematic search, a rigorous quality evaluation of guidelines, and comprehensive information extraction related to qualitative evidence in guidelines. We also performed content analysis for the purpose of providing clear views on the roles and functions of qualitative evidence in the process of guideline development.

Methods

The systematic review was performed according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines [14].

Criteria for guideline selection

We included guidelines focused on improving healthcare that met the following criteria: 1) the guidelines were primarily published in Chinese or English from January 1, 2011 to February 25, 2020. In 2011, IOM (Institute of Medicine) claimed that for a CPG to be trustworthy it needs to “be developed via a transparent process by a group of multidisciplinary experts (including patient representatives), screened for minimal potential bias and conflicts of interest, and supported by a systematic review of the evidence” [15]. This, which is the first statement of criteria for clinical practice guidelines, plays an important role in guideline development, so we chose it as the start date for retrieval; 2) the guidelines met the above mentioned IOM criteria; 3) the guidelines mainly focused on clinical questions, such as diagnosis, treatment or care for certain diseases or patients symptoms, to provide suggestions for healthcare staff or community health services; 4) qualitative research or qualitative evidence was used in the process of guidelines development; 5) if the guidelines were updated, only the most recent version of the guidelines were included. The guidelines were excluded, if they had the following characteristics: 1) the same guidelines had been repeatedly published in multiple journals; 2) the full texts of guidelines were not available.

Search strategy for guidelines

Relevant representative guidelines repositories, such as WHO, NICE (the National Institute for Health and Care Excellence), SIGN (Scottish Intercollegiate Guidelines Network), NGC (National Guideline Clearinghouse), RNAO (Registered Nurses’ Association of Ontario), and other databases, including three English databases (PubMed, Embase, Web of Science), four Chinese databases (China National Knowledge Infrastructure, CNKI; Wanfang Data; Chinese BioMedical Literature Database, CBM; and VIP Database for Chinese Technical Periodicals, VIP), were systematically searched from January 1, 2011 to February 25, 2020. The search strategy used MeSH terms, Title/Abstract and text words. Taking PubMed as an example, the retrieval strategy is shown in Fig. 1.

Fig. 1
figure 1

Search strategy on PubMed

Full size image

Guidelines selection and data extraction

Three (C.L.,Y.X.S and J.Z) authors experienced in literature retrieval independently selected eligible guidelines. Three reviewers (D.D.L.,Y.C and C.F) extracted significant information from the guidelines and completed data extraction forms by means of reading the text content of the guideline, references and the online relevant attachments. The detailed process of data extraction is presented in Additional file 1. The forms included: (1) the basic characteristics of included guidelines (such as title, publication/update date, and developer); (2) how qualitative research or evidence was used in the process of the guidelines development (were experts proficient in qualitative research invited to be involved in guideline development group, was qualitative research used to identify clinical questions, was qualitative evidence retrieved; was this used to support recommendations; and was this applied when considering facilitators and barriers to recommendations’ implementation); (3) details of the methodology for qualitative research or evidence used in the development process of guidelines (such as qualitative research quality assessment tool, the quality of the primary qualitative research study used to formulate recommendations and the grade of recommendations supported by qualitative evidence).

We hypothesized that the development of guidelines using qualitative research or evidence would be relevant to these items in the forms. The hypothesis was based on related methodological literature, COnsolidated criteria for REporting Qualitative research (COREQ) checklists [16] and discussion between all authors with methodologists in evidence-based guidelines development who were willing to engage in dialogue with us. Another researcher (Y.H.J) examined the data extraction forms to make sure no errors had occurred.

Appraisal of included guidelines

Two researchers (Y.YW and D.H) independently evaluated the quality of the guidelines by using the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool, which consists of 23 items under 6 domains involving scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence [17]. Each item was rated from 1 to 7 points with 1 point for “strongly disagree” and 7 points for “strongly agree”. We summarized the domain scores individually and scaled the total of that domain, calculated by the following formula: (obtained score - minimal possible score)/(maximal possible score - minimal possible score) × 100% [17].

Statistical analyses

Descriptive statistics were computed for the scores for each AGREE domain. Data for each AGREE II domain were provided as medians and interquartile ranges (IQRs). Intraclass correlation coefficients (ICCs) were calculated to evaluate the agreement between two reviewers for each domain [18, 19]. When the ICC value was less than 0.4, the consistency between raters was poor; if the ICC range was from 0.4 ~ 0.75, the consistency between raters was moderate; and a value of ICC over 0.75 the consistency was high [20]. The data were analyzed using SPSS version 17.0 (SPSS Inc. Chicago, IL, USA) and R version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria) for Windows.

Results

Guideline identification and selection

The searches identified 10,245 discrete records, of which 449 were selected for a full-text review. Sixty-four guidelines were eventually included [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84]. The flow diagram for the guidelines is shown in Fig. 2.

Fig. 2
figure 2

Flow diagram of guidelines identification and selection

Full size image

Characteristics of included guidelines

As Table 1 shows, the sixty-four guidelines concentrated on different topics such as cancers, chronic pain and smoking, and were developed by NICE, SIGN, RNAO, WHO or other professional organizations. The majority of guideline developers used GRADE (the Grading of Recommendations Assessment, Development and Evaluation) criteria for grading of evidence and recommendations. When formulating recommendations, they considered the quality of evidence, the risk-benefit analysis of some interventions, supporting resources and stakeholders’ values and preferences. The number of recommendations ranged from 2 to 262. The largest number of recommendations supported only by qualitative evidence in each included guideline was 8 [68]. The largest number of recommendations supported by both qualitative and quantitative evidence in each included guideline was 23 [70]. The majority of recommendations were supported by qualitative evidence based on primary studies, a few on systematic reviews).

Table 1 The basic characteristics of guidelines included
Full size table

Quality appraisal of the guidelines

The ICC values for all six domains were over 0.75, which indicated high consistency in the assessment results between the two raters.

As Table 2 and Fig. 3 show. The final domain scores ranged between 0% (domain 6 of 6 guidelines) [75, 77, 78, 81, 82, 84] and 96% (domain 6 of 11 guidelines) [21, 22, 25,26,27, 29,30,31,32,33,34]. When comparing the total domain scores, Domain 1 (Scope and Purpose) was ranked the highest with a median score of 83% (IQ 78–83). Domain 2 (Stakeholder involvement) and Domain 5 (Applicability) were ranked the lowest with median scores of 67% (IQ 67–78) and 67% (IQ 63–73) respectively. The median scores of Domains 3, 4, 6 (Rigour of development, Clarity of presentation, Editorial independence) were 71% (IQ 69–74), 72% (IQ 58–78) and 79% (IQ 75–83) respectively.

Table 2 Analysis of the included N-CPGs according to AGREE II (%)
Full size table
Fig. 3
figure 3

The summary of scaled domain score over all included guidelines

Full size image

The process of the guidelines development using qualitative research or evidence

As Fig. 4 shows, no guideline developers invited experts proficient in qualitative research to be involved in guideline development groups. 20% guidelines (13/64) used qualitative research to identify clinical questions [68, 71, 73,74,75, 77,78,79,80,81,82,83,84]. 83% (53/64) guidelines retrieved qualitative evidence [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70, 75, 77, 81]. 86% (55/64) guidelines used qualitative evidence to support recommendations [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70, 72, 75,76,77, 81]. And 19% (12/64) guidelines applied qualitative evidence when considering facilitators and barriers to recommendations’ implementation [55, 56, 60, 62,63,64,65,66,67,68,69,70].

Fig. 4
figure 4

The process of the guidelines development using qualitative research or evidence. a Experts proficient in qualitative research to involve in guideline development group. b Using qualitative research to identify clinical questions. c Retrieving qualitative evidence. d Using qualitative evidence to support recommendations. e Applying qualitative evidence when considering facilitators and barriers of recommendations' implementation

Full size image

The methodology for evidence used in the guidelines development

As Table 3 shows, one guideline used qualitative research based on grounded theory, phenomenology [55]. 52% (27/52) guideline developers evaluated the quality of the primary qualitative research study using the CASP (the Critical Appraisal Skills Programme) tool or NICE checklist for qualitative studies [35, 38, 46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70]. No guidelines evaluated (0/18) the quality of qualitative evidence synthesis used to formulate recommendations. 17% (11/64) guidelines presented the level of qualitative research using the grade criteria of evidence and recommendation in different forms such as I, III, IV, very low [35,36,37,38,39,40, 42, 44, 73, 77, 81]. They were based on JBI, GRADE or adapted from SIGN or Pati D. A framework [35,36,37,38,39,40,41,42,43,44,45, 85,86,87] respectively. 28% guidelines (15/54) described the grades of the recommendations supported by qualitative and quantitative evidence in different ways such as “strong”, “good”, “B”, “C” or “D” and “weak” [21, 22, 24, 25, 27, 28, 30,31,32,33,34, 73, 76, 77, 81], which also complied with JBI, GRADE or adapted from SIGN and (or) Pati D. A framework respectively. But no guidelines (0/10) described the grade of recommendations supported only by qualitative evidence.

Table 3 The methodology for qualitative research or evidence in the process of included guidelines development
Full size table

Discussion

Our review shows that the majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to recommendations’ implementation. However, the methodology still needs more attention, as there were, no experts proficient in qualitative research involved in guideline development group, no assessment of the quality of qualitative evidence synthesis and a lack of detailed reporting the level of qualitative evidence and its grade of recommendations’.

The summary findings of this review

The majority of the included guidelines introduced the overall aim of the guideline, the specific health questions, and the target population in tabulated form, bold, or using separate paragraphs. They described the gathering and synthesis of the evidence, gave details of updating and dealt with the language, structure, and format of the guideline recommendations.. However, the guidelines still had some noticeable shortcomings. For instance, a few guidelines did not describe the methods of formulating recommendations [74, 76, 82]; a few did not clearly introduce the different options for management of the conditions or health issues [76, 82]; a minority of guidelines did not give details of conflict of interest statements [75, 77, 78, 81, 82, 84]. In addition, although the majority of the guidelines stated that the guideline development group consisted of all relevant professional experts, and clearly defined the guidelines’ target users, a number of developers did not consider values and preferences of the target population [71, 78, 83, 84] or lacked adequate information on how they gained patients, doctors or other stakeholders’ views. And also the majority of the guidelines did not describe facilitators and barriers to their application in detail.

The methodological quality of qualitative evidence affects interpretation of its results. Unfortunately, while the majority of guidelines developers used qualitative evidence synthesis to formulate recommendations, they did not appraise confidence in each individual review, which resulted in some difficulties in explaining relevant themes or theories formulated in different articles. In addition, only three of the grade systems used, referred to single qualitative studies or synthesis of qualitative research as a level of the grade criteria of evidence and recommendation [35,36,37,38,39,40,41,42,43,44,45, 85,86,87]. The majority of guideline developers did not concentrate on the important influence of qualitative evidence on the grade criteria of evidence and recommendation.

Comparison of findings with prior research

When comparing our findings with similar relevant articles, lack of statements about conflict of interest, details on how to gain patients, doctors or other stakeholders’ views, consideration of facilitators and barriers to guidelines’ implementation are also common issues e.g. oncology CPGs [88], inflammatory bowel disease guidelines [89], nursing CPGs [90], guidelines for management of cholangiocarcinoma [91]. Our review firstly identified whether qualitative research or evidence had been used to obtain stakeholders’ values and preferences, and in identifying facilitators and barriers to guidelines’ implementation in the process of guidelines development. Other researchers also used qualitative research to explore practice gaps based on existing guidelines: Feyissa et al. used a semi-structured interview to assess contextual barriers and facilitators to the implementation of a guideline developed to reduce HIV-related stigma and discrimination (SAD) in the Ethiopian healthcare setting [92]; Lind et al. interviewed local politicians, chief medical officers and health professionals at acute care hospitals to investigate perceptions regarding guidelines for palliative care and identify obstacles and opportunities for their implementation in acute care hospitals [93].

In Addition, qualitative research is increasingly being recognised as having an important role to play in addressing questions relating to interventions or system complexity, and guideline development processes. As with our topic, other researchers have also focused on the methodology of involving qualitative research in the development process of guidelines. Flemming et al. provided guidance for the choice of qualitative evidence synthesis methods in the context of guideline development for complex interventions by using a best fit framework synthesis to address interactions between components of complex interventions; interactions of interventions with context and multiple (health and non-health) outcomes; using meta-ethnography to deal with sociocultural acceptability of an intervention [94]. In addition, Moore et al. also put forward designs and methods for the applicability of quantitative and qualitative evidence in guidelines including complexity-related questions of interest in the guideline, types of synthesis used in the guideline, mixed-method review design and integration mechanisms, observations, concerns and considerations [95].

Implications for guideline developers

The development of guidelines is a complex undertaking which needs a significant focus on its methodology. Based on our findings, we put forward some proposals for guideline developers, which may be helpful to improve their guideline’s quality. Firstly, guidelines developers can record and report details about how they reach agreement on recommendations and how they deal with possible disagreement when formulating recommendations and present different options for the same CQs with information on population characteristics or clinical situations for each option. Secondly, they can also develop a series of methods to avoid potential COI before the initiation of the guideline development project. Guideline developers may also obtain the target population’ views by interviewing stakeholders or extracting some relevant themes from existing qualitative data on the topic of interest. Finally, guideline developers should formally consider how to evaluate and grade single qualitative studies or synthesis of qualitative research into the grade system for guideline development prior to start-up of the guideline development project, and identify which factors influence the grade classification with the help of experts proficient in qualitative research. They should also select appropriate tools to appraise the quality of qualitative evidence such as CASP tool, NICE checklist for primary studies, GRADE-CERQual (Grading of Recommendations Assessment, Development and Evaluation-Confidence in the Evidence from Reviews of Qualitative research) for qualitative evidence synthesis, which is an approach for assessing how much confidence to place in findings from qualitative evidence syntheses in terms of four components (methodological limitations, coherence, adequacy of data, relevance) [13, 96].

Limitations and strengths

Our study has some potential limitations. Firstly, although we selected eligible guidelines by means of reading their text content, references and the online relevant attachments, we used a quick search strategy on the guideline development. We also used the filter capability when using Endnote to manage literature from databases. But because of the size of the task there may be selection bias because of unavailable guidelines published in government documents, books or other guideline publication platforms. Additionally, we did not specify how many guidelines were recommended, recommended with modifications, and not recommended, because AGREE II protocol states that no overall score is calculated to determine if a CPG is recommended or not recommended and the main focus of this article was the methodology for qualitative research or qualitative evidence used in guidelines development [17]. Nonetheless, there may be several advantages. Firstly, a systematic literature search was performed for screening eligible guidelines. Secondly, we discussed the potential effect of qualitative research or evidence on the AGREE II appraisal, and then put forward some suggestions on how to use qualitative research or evidence to improve the quality of future guidelines. Thirdly, this is the first attempt to systematically analyze the role of qualitative research or evidence in guidelines development based on published guidelines.

Suggestions for ongoing research

Qualitative research or qualitative evidence will be extensively used in the guideline development process in the future. There are three interesting topics needing further research. Firstly, when available data exists, this can be explored to provide more reliable conclusions related to the potential association between AGREE appraisal and the identification, incorporation and reporting of qualitative research by means of statistical methods such as non-parametric tests. Secondly, it will be interesting to compare the use of qualitative and quantitative data when formulating recommendations in guidelines, perhaps by matching guidelines on similar topics or key questions, and comparing those which did and didn’t use use qualitative evidence. Thirdly, exploring how qualitative research may be used to obtain the information related to conflict of interest will also be useful to inform guideline transparency. These topics are worthy of future exploration.

Conclusion

The majority of the included guidelines were high-quality. Qualitative evidence was mainly used to identify clinical questions, support recommendations, and consider facilitators and barriers to recommendations’ implementation. However, more attention needs to be given to the methodology, for instance, no experts proficient in qualitative research have been involved in guideline development group, there has been no assessment of the quality of qualitative evidence synthesis, and there is a lack of detail when reporting on the level of qualitative evidence and its grade recommendations’.

Availability of data and materials

All data generated or analyzed are included in this published article.

Abbreviations

WHO:

World Health Organization

GRC:

Guidelines Review Committee

CQ:

Clinical Questions

SIGN:

Scottish Intercollegiate Guidelines Network

NGC:

National Guideline Clearinghouse

NICE:

The National Institute for Health and Care Excellence

RNAO:

Registered Nurses’ Association of Ontario

CNKI:

China National Knowledge Infrastructure

CBM:

Chinese BioMedical Literature Database

VIP:

VIP Database for Chinese Technical Periodicals

AGREE II:

The Appraisal of Guidelines for Research and Evaluation

IQR:

Interquartile Range

ICC:

Intraclass Correlation Coefficient

GRADE:

The Grading of Recommendations Assessment, Development and Evaluation

ICU:

Intensive Care Unit

CASP:

The Critical Appraisal Skills Programme

FCC:

Family-Centered Care

JBI:

The Joanna Briggs Institute

GRADE-CERQual:

Grading of Recommendations Assessment, Development and Evaluation-Confidence in the Evidence from Reviews of Qualitative research

References

  1. Murphy E, Dingwall R, Greatbatch D, et al. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998;2:iii.

    CAS  PubMed  Google Scholar 

  2. Payne JB, Dance KV, Farone M, et al. Patient and caregiver perceptions of lymphoma care and research opportunities: A qualitative study. Cancer. 2019;125(22):4096–104.

    Article  PubMed  Google Scholar 

  3. Moses C, Flegg K, Dimaras H. Patient knowledge, experiences and preferences regarding retinoblastoma and research: A qualitative study. Health Expect. 2020;10.1111/hex.13043. https://doi.org/10.1111/hex.13043. [Published online ahead of print, 2020 Feb 29].

  4. WHO. Handbook for Guideline Development (2nd ed). Available at: https://www.who.int/publications/guidelines/guidelines_review_committee/en/.

  5. Tan TP, Stokes T, Shaw EJ. Use of qualitative research as evidence in the clinical guideline program of the National Institute for Health and Clinical Excellence. Int J Evid Based Healthc. 2009;7(3):169–72.

    Article  PubMed  Google Scholar 

  6. NICE. Developing NICE Guidelines the Manual. 2017. Available at: https://www.nice.org.uk/process/pmg20/chapter/introduction-andoverview.

    Google Scholar 

  7. Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the Optimize MNH guidance process. Implement Sci. 2016;11:98.

    Article  PubMed  PubMed Central  Google Scholar 

  8. SIGN. A guideline developer’s handbook. 2015. Available at: http://www.sign.ac.uk/sign-50.html.

    Google Scholar 

  9. Carroll C. Qualitative evidence synthesis to improve implementation of clinical guidelines. BMJ. 2017;356:j80.

    Article  PubMed  Google Scholar 

  10. Kelson M, Akl EA, Bastian H, et al. Integrating values and consumer involvement in guidelines with the patient at the center: article 8 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):262–8.

    Article  PubMed  Google Scholar 

  11. Armstrong MJ, Mullins CD, Gronseth GS, et al. Impact of patient involvement on clinical practice guideline development: a parallel group study. Implement Sci. 2018;13(1):55.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Mudge S, Hart A, Murugan S, et al. What influences the implementation of the New Zealand stroke guidelines for physiotherapists and occupational therapists? Disabil Rehabil. 2017;39(5):511–8.

    Article  PubMed  Google Scholar 

  13. Lewin S, Glenton C. Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(1):126.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement, the PRISMA GROUP. PLoS Med. 2009;6(6):e1000097 https://doi.org/10.1371/journal/pmed.1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011. Available at: http://www.nap.edu/catalog/13058.html.

    Google Scholar 

  16. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

    Article  PubMed  Google Scholar 

  17. AGREE Next Steps Consortium. The AGREE II Instrument [Electronic version]. 2017. Available at: http://www.agreecollaboration.org.

    Google Scholar 

  18. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.

    Article  CAS  PubMed  Google Scholar 

  19. Kramer MS, Feinstein AR. Clinical biostatistics: LIV. The biostatistics of concordance. Clin Pharmacol Ther. 1981;29(1):111–23.

    Article  CAS  PubMed  Google Scholar 

  20. Shrout PE, Fleiss JL. Intraclass correlaions: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8.

    Article  CAS  PubMed  Google Scholar 

  21. SIGN. Management of epithelial ovarian cancer. 2018. Available at: https://www.sign.ac.uk/sign-135-management-of-epithelial-ovarian-cancer.

    Google Scholar 

  22. SIGN. Diagnosis and management of epilepsy in adults. 2018. Available at: https://www.sign.ac.uk/sign-143-diagnosis-and-management-of-epilepsy-in-adults.

    Google Scholar 

  23. SIGN. Management of stable angina. 2018. Available at: https://www.sign.ac.uk/sign-151-stable-angina.

    Google Scholar 

  24. SIGN. Cardiac rehabilitation. 2017. Available at: https://www.sign.ac.uk/sign-150-cardiac-rehabilitation.

    Google Scholar 

  25. SIGN. Assessment, diagnosis and interventions for autism spectrum disorders. 2016. Available at: https://www.sign.ac.uk/sign-145-assessment,-diagnosis-and-interventions-for-autism-spectrum-disorders.

    Google Scholar 

  26. SIGN. Management of chronic heart failure. 2016. Available at: https://www.sign.ac.uk/sign-147-management-of-chronic-heart-failure.

    Google Scholar 

  27. SIGN. Acute coronary syndrome. 2016. Available at: https://www.sign.ac.uk/sign-148-acute-coronary-syndrome.

    Google Scholar 

  28. SIGN. British guideline on the management of asthma. 2016. Available at: https://www.sign.ac.uk/sign-153-british-guideline-on-the-management-of-asthma.

    Google Scholar 

  29. SIGN. Glaucoma referral and safe discharge. 2015. Available at: https://www.sign.ac.uk/sign-144-glaucoma-referral-and-safe-discharge.

    Google Scholar 

  30. SIGN. Brain injury rehabilitation in adults. 2013. Available at: https://www.sign.ac.uk/sign-130-brain-injury-rehabilitation-in-adults.

    Google Scholar 

  31. SIGN. Management of hepatitis C. 2013. Available at: https://www.sign.ac.uk/sign-133-management-of-hepatitis-c.

    Google Scholar 

  32. SIGN. Management of chronic pain. 2013. Available at: https://www.sign.ac.uk/sign-136-management-of-chronic-pain.

    Google Scholar 

  33. SIGN. Management of adult testicular germ cell tumours. 2011. Available at: https://www.sign.ac.uk/sign-124-management-of-adult-testicular-germ-cell-tumours.

    Google Scholar 

  34. SIGN. Diagnosis and management of colorectal cancer. 2011. Available at: https://www.sign.ac.uk/sign-126-diagnosis-and-management-of-colorectal-cancer.

    Google Scholar 

  35. RNAO. Implementing supervised injection services. 2018. Available at: https://rnao.ca/bpg/guidelines/implementing-supervised-injection-services.

    Google Scholar 

  36. RNAO. Promoting and supporting the initiation, exclusivity, and continuation of breastfeeding for newborns, infants, and young children. 2018. Available at: https://rnao.ca/bpg/guidelines/breastfeeding-promoting-and-supporting-initiation-exclusivity-and-continuation-breast.

    Google Scholar 

  37. RNAO. Adult asthma care: Promoting control of asthma (Second Edition). 2017. Available at: https://rnao.ca/bpg/guidelines/adult-asthma-care.

    Google Scholar 

  38. RNAO. Crisis intervention for adults using a trauma-informed approach: Initial four weeks of management (Third Edition). 2017. Available at: https://rnao.ca/bpg/guidelines/crisis-intervention.

    Google Scholar 

  39. RNAO. Delirium, dementia, and depression in older adults: Assessment and care (Second Edition). 2016. Available at: https://rnao.ca/bpg/guidelines/assessment-and-care-older-adults-delirium-dementia-and-depression.

    Google Scholar 

  40. RNAO. Person- and family-centred care. 2015. Available at: https://rnao.ca/bpg/guidelines/person-and-family-centred-care.

    Google Scholar 

  41. RNAO. Care transitions. 2014. Available at: https://rnao.ca/bpg/guidelines/care-transitions.

    Google Scholar 

  42. RNAO. Preventing and addressing abuse and neglect of older adults: Person-centred, collaborative, system-wide approaches. 2014. Available at: https://rnao.ca/bpg/guidelines/abuse-and-neglect-older-adults.

    Google Scholar 

  43. RNAO. Primary prevention of childhood obesity (Second Edition). 2014. Available at: https://rnao.ca/bpg/guidelines/primary-prevention-childhood-obesity.

    Google Scholar 

  44. RNAO. Assessment and management of foot ulcers for people with diabetes (Second Edition). 2013. Available at: https://rnao.ca/bpg/guidelines/assessment-and-management-foot-ulcers-people-diabetes-second-edition.

    Google Scholar 

  45. RNAO. Promoting safety: Alternative approaches to the use of restraints. 2012. Available at: https://rnao.ca/bpg/guidelines/promoting-safety-alternative-approaches-use-restraints.

    Google Scholar 

  46. NICE. Depression in children and young people: identification and management. 2019. Available at: https://www.nice.org.uk/guidance/ng134.

    Google Scholar 

  47. NICE. Pancreatic cancer in adults: diagnosis and management. 2018. Available at: https://www.nice.org.uk/guidance/ng85.

    Google Scholar 

  48. NICE. Antimicrobial stewardship: changing risk related behaviours in the general population. 2017. Available at: https://www.nice.org.uk/guidance/ng63.pdf.

    Google Scholar 

  49. NICE. Eating disorders: recognition and treatment. 2017. Available at: https://www.nice.org.uk/guidance/ng69.

    Google Scholar 

  50. NICE. Healthcare-associated infections: prevention and control in primary and community care. 2017. Available at: https://www.nice.org.uk/guidance/cg139.

    Google Scholar 

  51. NICE. Hip fracture: management. 2017. Available at: https://www.nice.org.uk/guidance/cg124.

    Google Scholar 

  52. NICE. Immunisations: reducing differences in uptake in under 19s. 2017. Available at: https://www.nice.org.uk/guidance/ph21.

    Google Scholar 

  53. NICE. Intermediate care including reablement. 2017. Available at: https://www.nice.org.uk/guidance/ng74.

    Google Scholar 

  54. NICE. Suspected cancer: recognition and referral. 2017. Available at: https://www.nice.org.uk/guidance/ng12.

    Google Scholar 

  55. NICE. Coexisting severe mental illness and substance misuse: community health and social care services. 2016. Available at: https://www.nice.org.uk/guidance/ng58.

    Google Scholar 

  56. NICE. Oral health for adults in care homes. 2016. Available at: https://www.nice.org.uk/guidance/ng48.

    Google Scholar 

  57. NICE. Skin cancer prevention. 2016. Available at: https://www.nice.org.uk/guidance/ph32.

    Google Scholar 

  58. NICE. Maternal and child nutrition. 2014. Available at: https://www.nice.org.uk/guidance/ph11.

    Google Scholar 

  59. NICE. Needle and syringe programmes. 2014. Available at: https://www.nice.org.uk/guidance/ph52.

    Google Scholar 

  60. NICE. Physical activity: brief advice for adults in primary care. 2013. Available at: https://www.nice.org.uk/guidance/ph44.

    Google Scholar 

  61. NICE. Service user experience in adult mental health: improving the experience of care for people using adult NHS mental health services. 2011. Available at: https://www.nice.org.uk/guidance/cg136.

    Google Scholar 

  62. NICE. Type 2 diabetes prevention: population and community-level interventions. 2011. Available at: https://www.nice.org.uk/guidance/ph35.

    Google Scholar 

  63. WHO. WHO recommendations: intrapartum care for a positive childbirth experience. 2018. Available at: https://www.who.int/reproductivehealth/publications/intrapartum-care-guidelines/en.

    Google Scholar 

  64. WHO. Guidelines for managing advanced HIV disease and rapid initiation of antiretroviral therapy. 2017. Available at: https://www.who.int/hiv/pub/guidelines/advanced-HIV-disease/en.

    Google Scholar 

  65. WHO. Protecting, promoting and supporting breastfeeding in facilities providing maternity and newborn services. 2017. Available at: http://www.who.int/nutrition/publications/guidelines/breastfeeding-facilities-maternity-newborn/en.

    Google Scholar 

  66. WHO. Guidelines on HIV self-testing and partner notification: supplement to consolidated guidelines on HIV testing services. 2016. Available at: https://www.who.int/hiv/pub/vct/hiv-self-testing-guidelines/en.

    Google Scholar 

  67. WHO. WHO recommendations on antenatal care for a positive pregnancy experience. 2016. Available at: https://www.who.int/reproductivehealth/publications/maternal_perinatal_health/anc-positive-pregnancy-experience/en.

    Google Scholar 

  68. WHO. Health worker roles in providing safe abortion care and post-abortion contraception. 2015. Available at: https://www.who.int/reproductivehealth/publications/unsafe_abortion/abortion-task-shifting/en/-33k.

    Google Scholar 

  69. WHO. WHO recommendations on health promotion interventions for maternal and newborn health. 2015. Available at: https://www.who.int/maternal_child_adolescent/documents/health-promotion-interventions/en/-70k.

    Google Scholar 

  70. WHO. WHO recommendations Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting. 2012. Available at: https://www.who.int/reproductivehealth/publications/maternal_perinatal_health/978924504843/en.

    Google Scholar 

  71. National Center for Cardiovascular Diseases (NCCD), Cardiovascular Committee of Beijing Nursing Association (CCBNA), Research Center of Nursing Theory and Practice Chinese Academy of Medical Sciences & Peking Union Medical College (RC-NTPC-AMSPUMC), Evidence-based Medical Center of Lanzhou University (EBMCLU). Construction of Nursing Practice Guideline for Emergency Percutaneous Coronary Intervention. Chin J Nurs. 2019;54:36–41.

    Google Scholar 

  72. Committee of Breast Cancer Society (CBCS), Committee Specialist of Breast Surgeon (CSBS). Expert consensus on breast tumor plastic surgery and breast reconstruction (2018 Edition). China Oncol. 2018;28(6):439–80.

    Google Scholar 

  73. Maternity Hospital of Fudan University (MHFU), School of Nursing Fudan University (SNFU), Shanghai Evidence-based Nursing Center (SEBNC). Gestational diabetes mellitus clinical nursing practice guideline. 2018. Available at: http://nursing.ebn.fudan.edu.cn/WebContent_list.aspx?BID=263&SID=279.

    Google Scholar 

  74. Pediatric Hospital of Fudan University (PHFU), School of Nursing Fudan University (SNFU), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU), Shanghai Evidence-based Nursing Center (SEBNC). Evidence-based guidelines for breastfeeding of hospitalized newborns. 2017. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2018/7/201807042226014857_1.pdf.

    Google Scholar 

  75. Tian L, Li HL, Tao M, et al. Construction of clinical nursing guideline on cancer related fatigue in adults. Nurs Res China. 2017;31:1564–8.

    Google Scholar 

  76. Guangdong Medical Association Accelerated Rehabilitation Surgeons Branch (GD-MAARSB). Clinical application of anaesthesia in accelerated rehabilitation surgery of colorectal surgery in lingnan Expert consensus on operation specification (2016 Edition). J Dig Oncol (Electronic Edition). 2016;8(4):209–19.

    Google Scholar 

  77. Shanghai Public Health Clinical Center (SPHCC), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). HIV/AIDS nursing clinical practice guidelines. 2016. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/3/201903112057079741_1.pdf.

    Google Scholar 

  78. National Center for Cardiovascular Diseases (NCCD), Heart Failure Committee of Chinese Medical Association (HFCCMA), Beijing Nursing Society (BNS). Nursing practice guideline of acute heart failure. Chin Nurs Manage. 2016;16:1179–88.

    Google Scholar 

  79. JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU), Shanghai Evidence-based Nursing Center (SEBNC), Pediatric Hospital of Fudan University (PHFU). Clinical nursing practice guideline for enteral nutrition for infants with congenital heart disease. 2016. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/1/201901242006519494_1.pdf.

    Google Scholar 

  80. East China Hospital of Fudan University (ECHFU), School of Nursing Fudan University (SNFU). Clinical Practice guideline for nasogastric tube feeding among adult patients. 2015. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/2/201902281629196565_1.pdf.

    Google Scholar 

  81. Fudan University Shanghai Cancer Center (FUSCC), School of Nursing Fudan University (SNFU), JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). Clinical practice guidelines of peripherally inserted central catheter (PICC) catheterization. 2014. Available at: http://nursing.ebn.fudan.edu.cn/Upfile/OtherInfo/2019/7/201907150909333584_1.pdf.

    Google Scholar 

  82. Zhou YF, Qin W, Liu RY, et al. The construction of an evidence-based practice guideline on prevention and management of medication errors in hospitalized adult patients. J Nurs Sci. 2014;29:1–4.

    Google Scholar 

  83. Tang HT. Development of clinical practice guideline for oral care on critically ill patients with endotracheal intubation. Shanghai: School of Nursing Fudan University; 2013. p. 1–100.

  84. JBI Evidence-based Nursing Cooperation Center of Fudan University (JBI-EBNCCFU). Clinical practice guideline on inpatient fall prevention. 2011. Available at: http://nursing.ebn.fudan.edu.cn/xzzx/UploadFiles/file/201510/1612274173499.pdf.

    Google Scholar 

  85. JBI. Levels of evidence FAME. Available at: http://joannabriggs.org/Levels%20of%20Evidence%20%20FAME.

  86. JBI. Grades of Recommendation. Available at: http://joannabriggs.org/Grades%20of%20Recommendation.

  87. The GRADE Working Group. What is GRADE? [EB/OL]. Available at: http://www.gradeworkinggroup.org.

  88. Saleh RR, Majeed H, Tibau A, et al. Undisclosed financial conflicts of interest among authors of American Society of Clinical Oncology clinical practice guidelines. Cancer. 2019;125(22):4069–75.

  89. Grindal AW, Rishad K, Scaffidi MA, et al. Financial Conflicts of Interest in Inflammatory Bowel Disease Guidelines. Inflamm Bowel Dis. 2019;25(4):642–5.

  90. Wang Y, Yu S, Wang L, et al. The methodology for developing nursing clinical practice guidelines over recent decades in China: A critical appraisal using AGREE II. J Nurs Manag. 2020;28(4):976–97.

  91. Paschalis G, Alan A, Roberts Keith J, et al. Appraisal of the current guidelines for management of cholangiocarcinoma-using the Appraisal of Guidelines Research and Evaluation II (AGREE II) Instrument. Hepatobiliary Surg Nutr. 2020;9:126–35.

    Article  Google Scholar 

  92. Angela B, Adrienne Y, Alison M, et al. EXploring practice gaps to improve PERIoperativE Nutrition CarE (EXPERIENCE Study): a qualitative analysis of barriers to implementation of evidence-based practice guidelines. Eur J Clin Nutr. 2019;73(1):94–101.

  93. Lind S, Wallin L, Brytting T, et al. Implementation of national palliative care guidelines in Swedish acute care hospitals: a qualitative content analysis of stakeholders' perceptions. Health Policy. 2017;121(11):1194–201.

  94. Flemming KA, Booth A, Garside R, et al. Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Glob Health. 2019;4(Suppl 1):e000882.

  95. Moore G, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

  96. Lewin S, Glenton C, Munthe-Kaas H, et al. Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

None.

Funding

None.

Author information

Authors and Affiliations

  1. Center for Evidence-Based and Translational Medicine, Zhongnan Hospital of Wuhan University, Wuhan, China

    Yun-Yun Wang, Yue Cao, Cheng Fang, Di Huang & Ying-Hui Jin

  2. Department of Evidence-Based Medicine and Clinical Epidemiology, The Second Clinical College, Wuhan University, Wuhan, China

    Yun-Yun Wang, Yue Cao, Cheng Fang, Di Huang & Ying-Hui Jin

  3. Center for Evidence-Based and Translational Medicine, Wuhan University, Wuhan, China

    Yun-Yun Wang, Yue Cao, Cheng Fang, Di Huang & Ying-Hui Jin

  4. School of Nursing and Health, Henan University, Kaifeng, China

    Dan-Dan Liang

  5. Department of surgery, Huaihe Hospital of Henan University, Kaifeng, China

    Dan-Dan Liang

  6. Center for Evidence-based Medicine, Institute of Evidence-Based Medicine and Knowledge Translation, Henan University, Kaifeng, China

    Dan-Dan Liang

  7. Emergency Department, Tianjin TEDA hospital, Tianjin, China

    Cui Lu

  8. School of Nursing, Peking University, Beijing, China

    Yue-Xian Shi

  9. First Clinical College of Wuhan University, Wuhan, China

    Jing Zhang

Authors
  1. Yun-Yun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dan-Dan Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cui Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yue-Xian Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yue Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cheng Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Di Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ying-Hui Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YYW and YHJ designed the study and formulated inclusion criteria. CL,YXS and JZ searched and selected eligible guidelines. DDL,YC and CF extracted significant information. YHJ examined the data extraction forms. YYW and DH evaluated the quality of the guidelines. YYW, and YHJ contributed to the analysis of the data and discussed the findings. YYW developed the final manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Ying-Hui Jin.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

The process of data extraction.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, YY., Liang, DD., Lu, C. et al. An exploration of how developers use qualitative evidence: content analysis and critical appraisal of guidelines. BMC Med Res Methodol 20, 160 (2020). https://doi.org/10.1186/s12874-020-01041-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-020-01041-8

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us