Recommendations to conduct and report systematic reviews in medical literature: a scoping review

Abstract

Background

This scoping review aims to identify, review and characterize the published recommendations to conduct and/or to report a systematic review in medical interventions area.

Methods

A search was carried out in PubMed, EMBASE and Cochrane Library databases, using systematic reviews search filters. The search comprises all recommendations to conduct and/or report a systematic review. Data on methods were extracted from each recommendation. A descriptive analysis was performed.

Results

Eighty-three recommendations were identified. Approximately 60% of retrieved references were published in the last 6 years. Recommendations to both conduct and report a systematic review were issued in 47% studies. The guidance presented in each recommendation to conduct and/ or report a systematic review varied. Almost 96% of the recommendations offer guidance on systematic review methods section. The need and time for updating was only recommended in 29% of recommendations. Forty percent of recommendations endorsed their methods to any subject related to medical interventions. Half of the studies did not specify the design of studies to be included in a systematic review.

Conclusions

Several recommendations to conduct and/or report a systematic review were published and offered different guidance. Further research on the impact of such heterogeneity can improve systematic reviews quality.

Background

A systematic review aims to collect evidence from the research literature, using systematic and explicit methods, to answer a clearly formulated research question [1, 2]. It is a rigorous methodology to identify, select, assess methodological quality, analyze and discuss relevant studies [1, 2]. These characteristics distinguish a systematic review from other type of reviews, since the systematic appraisal of studies based on methodological quality can provide useful information to the clinical decision process, regulatory decisions, and clinical guidelines [1,2,3].

The first clinical systematic review was published in 1955 in the Journal of the American Medical Association (JAMA) [4]. At the end of 80s, the first systematic review and meta-analysis in the health field, entitled ‘Effective Care during Pregnancy and Childbirth’, was published [5]. In the year of 2015, approximately 950 reviews were published only on the Cochrane Database of Systematic Reviews [6].

Several groups have been dedicated to develop and improve systematic reviews methodology. In the 90s, the Cochrane Collaboration was created with the goal of “prepare, maintain, and disseminate systematic reviews” [7]. Later, this group published the “Cochrane Handbook for Systematic Reviews of Interventions” [8]. Other main groups developed their own guidance on systematic reviews, such as the Centre for Reviews and Dissemination (CRD) which published “Systematic Reviews CRD’s guidance for undertaking reviews in healthcare” [9]; and The Joanna Briggs Institute (JBI) which developed the “The JBI Reviewer’s Manual” [10]. In 1999, a group developed guidance on the reporting of meta-analysis (the QUOROM, QUality Of Reporting Of Meta-analyses) [11]. Ten years later, this guidance was updated and included recommendations on the reporting of systematic reviews (PRISMA, Preferred Reporting Items of Systematic reviews and Meta-Analyses) [11]. Since then, the PRISMA group has been developing specific recommendations on the reporting of systematic reviews, such as abstracts, equity, harms, diagnostic test accuracy, among others [11]. The PRISMA Statement has been endorsed by several scientific journals as the recommend guidance to report a systematic review [11].

The selection of a methodology will depend on the research question and type of review [8,9,10]. It is recognized that the majority of the recommendations to conduct a systematic review follow four primary steps: 1) review of the literature; 2) selection of criteria to include studies for analysis; 3) extraction of the data from the selected studies; and 4) analysis of the extracted data [8,9,10]. Since several recommendations are available to help to conduct and/or reporting a systematic review, the knowledge of each recommendation along with the specificity and individuality of each area (for instance by disease or by type of intervention) could define the best methodology to adopt on the conduct and/or report of a systematic review.

The objective of this scoping review is to identify, review and characterize the recommendations available in healthcare literature to conduct and/or to report a systematic review.

Methods

This scoping review was developed according to the recommendations of The Joanna Briggs Institute Reviewer’s Manual – Methodology for Scoping Reviews [10].

Literature search and data selection

A search was carried out in PubMed (https://www.ncbi.nlm.nih.gov/pubmed/), EMBASE (https://www.embase.com/), and Cochrane Library (http://cochranelibrary-wiley.com/cochranelibrary/search/advanced) databases. The databases were searched since its inception until July 17, 2018. The search terms comprised systematic reviews methodology. A filter was applied to restrict the search to English articles. The reference list of all identified articles was also hand searched for additional studies. The literature search and search strategy are listed in the Appendix 1.

Articles were selected for inclusion if they meet the following selection criteria: published in English language; conducted in humans; and were recommendations to conduct and/or to report a systematic review of healthcare interventions (e.g., drugs, medical, surgical, behavioral and occupational therapy, and diagnostic testing). Articles describing exclusively the use of qualitative evidence were excluded. Articles such as editorials, letters, commentaries, and abstracts from congresses and articles describing recommendations on how to read or to interpret a systematic review were also excluded. A recommendation could be described in a series of articles or in a single article, this is, in one or more references.

Two researchers independently screened by hand the titles and abstracts and selected full articles for inclusion. Disagreement was resolved by discussion and consensus.

Data extraction

The following information was extracted independently from each article:

• A. Reference, including authors’ names and year of publication

• B. Methodological design used to develop a recommendation to conduct and/or report a systematic review, classified between review or consensus study;

• C. Name attributed to the recommendation, if applicable;

• D. Type of recommendations: to conduct and/or report a systematic review. A recommendation to conduct a systematic review describes the steps to perform it; instead of a recommendation to report a systematic review which describes on how it should be write;

• E. Suggested methodology to conduct and/or report a systematic review; the methodological recommendations analyzed were divided into the following sections: 1) introduction; 2) identification of the research question; 3) definition of research protocol; 4) definition of eligibility criteria; 5) execution of literature search; 6) identification of sources of information; 7) data selection; 8) data extraction; 9) risk of bias/methodological quality assessment; 10) data analysis; 11) presentation of results; 12) interpretation of results; 13) discussion/conclusion of results; 14) need and time for updating; 15) helpful material;

• F. The subject of the methods issued, for example by disease or study area;

• G. Type of studies to be included in the systematic review, for instance, randomized controlled trials (RCT), observational studies, among others;

• H. Study group’s name, this is who issued the methodology, if applicable.

Data analysis

Data were analyzed using descriptive statistics. Statistical analyses were conducted with Microsoft Excel 2010 (Microsoft Corporation, Santa Rosa, CA).

Results

A total of 3034 potentially relevant references were yielded from literature search. Twenty additional references were identified. Based on the inclusion criteria, 210 references were selected for full-text further inclusion. A final sample of 131 references covering 83 different recommendations met the inclusion criteria (some recommendations were described in a series of references, this is, in more than one article). The selection of references is shown in Fig. 1. The references of the included and excluded studies are listed in the Additional file 1.

Fig. 1

Flow diagram of Literature search

Sixty-three percent (n = 83/131) of retrieved articles were published since 2012 (Appendix 2).

Sixty out of 83 (72%) recommendations were developed through a review study, whereas 23 (28%) as a consensus study.

Guidance to conduct and report a systematic review were issued in 39 (47%) recommendations. Only 10% (n = 8/83) of recommendations described guidance on how to report a systematic review. The type of recommendations is described in Fig. 2. A detailed description of the recommendations is presented in Appendix 2.

Fig. 2

Type of recommendations, to conduct and/or report a systematic review

Table 3 describes the methods’ sections to conduct and/or report a systematic review for which the recommendations offered guidance.

Table 3 Methods to conduct and/or report a systematic review as suggested in each study

Methodological steps to analyze data, assess risk of bias/ methodological quality and define eligibility criteria were addressed by most of the recommendations (between 98 and 99% of recommendations). The definition of an a priori research protocol was less often recommended (59 (71%) recommendations). An orientation about how to prepare an introduction, including the background and purpose of systematic review, was only comprised in 38 (46%) recommendations. Guidance about the need and time for updating the systematic review were included in only 24 (29%) recommendations. Thirty (36%) recommendations made available supportive material to conduct and/or report a systematic review, such as tables, graphics, methodological quality assessment scales, and/or flowcharts.

Thirty-three (39.8%) recommendations endorsed their methods to any subject related to healthcare interventions. Twenty (24%) recommendations are specific of a clinical subject, such as cardiology, pain, nephrology, and sports medicine and orthopedic surgery. Eleven (13.2%) recommendations addressed guidance on the conduction and/or report of systematic reviews about investigation procedures. Other subjects, such as economic evaluation, were also studied. The subject of recommendations is presented in Table 4.

Table 4 Subject of the methods issued to conduct and/or report a systematic review by each study

Fifty-two (62.7%) recommendations did not specify the studies’ design to be included in the systematic review. Among those addressing this issue, clinical trials and randomized controlled trials were the type of study preferred to conduct a systematic review (n = 9; 10.8%). The type of studies eligible for inclusion in the systematic review is presented in Table 5.

Table 5 Type of studies eligible for inclusion in the systematic review recommended by each study

Sixty-seven percent of the recommendations were issued by individual groups/authors, without being affiliated with any particular organization researching in methods used in systematic review. Some organizations such as The Joanna Briggs Institute, The Cochrane Collaboration, The Centre for Reviews and Dissemination, and the Agency for Healthcare Research and Quality also developed, at least one, recommendation to conduct and/or report a systematic review. The distribution of the recommendations by study groups is presented in Table 6.

Table 6 Study groups, who issued the methods to conduct and/or report a systematic review

PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analysis; AHRQ: Agency for Healthcare Research and Quality; JGIM: Journal of General Internal Medicine; COSMIN: Consensus-based Standards for the selection of health Measurement Instruments.

Discussion

In the last years, several organizations and individual groups have published recommendations to conduct and/or report a systematic review. In general, they can be applied to study any healthcare intervention, combining different types of evidence. The recommendations focus on the methods of the systematic review.

In the present study, only orientations about systematic review were identified, characterized and reviewed; however, some of the recommendations include guidance to conduct and/or to report a meta-analysis.

More than half of the recommendations evaluated in this study were published in the last 6 years. The volume of information and new studies is growing. In MEDLINE, one of the largest databases of medical literature, more than 8 million articles were indexed in 23 years [12]. In addition, systematic reviews synthesize several types of study, such as network meta-analysis, adverse events, economic studies, among others [8,9,10]. The need for specific recommendations addressing this type of studies was also increased over time. This can explain the growth of certain organizations such as the Cochrane Collaboration or the JBI and the development of such specific recommendations [8,9,10]. Moreover, regulatory authorities required a compilation of various individual studies in the health technology assessment, for instance in market access [13], in its re-evaluation and to monitoring its benefit-risk ratio [14]. A systematic review becomes a recognized need to support informed decisions in medicine. A study by Bastian et al. estimated that 11 systematic reviews are published per day [15]. Therefore, guidance on how to conduct and/or report a systematic review of any kind becomes essential [16].

According to The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, the development of a recommendation should include a review of all existing evidence on the research question and an evaluation of this data by a panel of experts. After that, a consensus is achieved on which steps a recommendation should follow [17]. However, among the recommendations analyzed in this study, approximately 70% were published under a review study, and only 30% were developed based on a consensus of a panel of experts.

Some organizations were created to study and develop guidance on the best synthesis of different types of information and, thereafter, the elaboration of systematic reviews [8,9,10]. In this review, nearly 32% of the recommendations were issued by these organizations. The other 68% were issued by individual groups/authors. The growth in publication of scientific studies reflects the need to conduct methodological well/structured reviews of the literature [12]. This may also result in the increase of recommendations to conduct and/or report systematic reviews, particularly recommendations for a specific area, such as safety or economic evaluations.

A recommendation to conduct and/or report a systematic review should list and detail all fundamental steps to help authors to write, or scientific journals to appraise a systematic review. From the recommendations characterized in this review, between 88 and 99% developed guidance on methods (from the definition of eligibility criteria to data analysis). The elaboration and publication of a systematic review protocol improves transparency and avoids duplication of work [8]. In this review, only half of the recommendations addressed the elaboration of a protocol. Nowadays, there are several ways to publish a systematic review protocol, such as registration in PROSPERO (International prospective register of systematic reviews) or publication of the protocol in peer-reviewed journals [3]. However, a significant proportion (44%) of the recommendations issuing the elaboration of a protocol was published during the last 4 years (since 2015). This becomes interesting since it shows the importance of continuing to address this step in the recommendations to conduct and/or report a systematic review. Despite the majority of the recommendations offer orientations on how to define the research question, the elaboration of the Introduction was the step less described. Describing the background and the purpose of the systematic review may help readers to understand the research question and make a most properly judgement of the results, increasing the systematic review quality [3, 12]. Approximately 76% of the recommendations present guidance on the interpretation and discussion of the results. This proportion seems to be low since systematic reviews’ main goals are to inform and help the interested parties in the decision-making processes. ‘Need and time for updating’ was the step less recommended. Systematic reviews are constantly out of date with new evidence published every day [18]. Recently, the panel for updating guidance for systematic reviews (PUGs) group had illustrated the importance of update systematic reviews and developed some guidance that can help authors and readers to understand when to update a systematic review [19]. Moreover, in 2014, the concept of living systematic review emerged [20]. This intended to continually updated a systematic review (of any type). It predisposes a periodic search and the constant update since new data arises [20]. Nonetheless, despite some guidance for updating systematic reviews are available, it is still necessary to include this step in the recommendations to conduct and/ or report a systematic review.

In this review, all recommendations presented some differences in methodology which may lead to some bias, such as reporting bias. These bias reflects the influence on their reporting, which can lead to a misunderstanding of the results [21]. If the recommended methodology to conduct a systematic review is not clear enough, the results and conclusions of the produced systematic review could be flawed, limiting its importance and objective [12]. Such methodological impairments may compromise the comparability of systematic reviews addressing the same research question, eligibility criteria, search criteria, and time of research, but which follow different recommendations [22]. Therefore, their results, based on the same studies, may be biased and presented in different ways. Thus, each systematic review could present its own conclusions, introducing confounding in health decision-making [22].

Eight recommendations were specifically developed to address the reporting of systematic reviews. Seven of these recommendations were developed by PRISMA working group. Currently, PRISMA has becoming a wide-scale adopted guideline, used by authors to report and by scientific journals to appraise a systematic review [11]. The PRISMA, a guideline created to increase the quality of reporting a systematic review, aims at enhancing transparency, reliability, and ease of reading [11]. Several studies demonstrated the poor quality of systematic reviews when they are not compliant with a reporting guideline [11]. Despite the publication and dissemination of PRISMA, there are several studies showing the suboptimal compliance to this guidance when reporting a systematic review [23, 24]. Moreover, one of the PRISMA extensions, PRISMA of Diagnostic Test Accuracy (PRISMA-DTA, has recently being update in order to improve the reporting in systematic reviews of this type [25].

The safety of healthcare interventions is of major importance. The knowledge of their safety profile should be continuously updated to keep healthcare professionals, consumers, and healthcare regulators informed [8,9,10]. To characterize the safety profile, several types of information provided by distinct sources need to be consulted. In opposite to efficacy data, safety data is mainly obtained from post-marketing surveillance data sources, which comprises several types of studies, such as post-marketing clinical trials, observational studies, case reports, and spontaneous reports of adverse events [26]. Combining evidence from these several sources presupposes some specific methodology in conducting and/or reporting systematic reviews. This review identified four recommendations addressing how to conduct and/or report systematic reviews of adverse events.

Some other relevant areas related to healthcare interventions were also taken into consideration, such as economic evaluations of healthcare interventions. Expenditure with pharmaceuticals may account for a significant amount of health spending, depending on the countries [27]. Thereafter, pharmacoeconomic studies become essential in supporting the appraisal of medical interventions, medicines, and their market access [13]. A systematic review of these studies is important for healthcare policymaking [28]. However, the role of systematic review to synthesize economic evaluations has been questioned [29]. Not only due to the specific design of economic evaluations, such as type of analysis, perspective adopted, among others, but also because economic evaluations already synthesize information [29]. Thus, the elaboration of specific recommendations to conduct and/or report a systematic review of economic evaluations may be valuable.

Almost half of the analyzed recommendations did not specify the design of studies to be included in a systematic review. Some recommendations only endorse the inclusion of randomized controlled trials, because of its classification such as the highest level of evidence [30]. Nonetheless, the type of studies selected must reflect the objective of the systematic review. In a systematic review evaluating the effectiveness of investigation procedures, such as diagnostic tests, studies evaluating the accuracy of diagnostic tests must be chosen [31]. Naturally, the methodological quality level of the evidence chosen will be varied. However, an evaluation of the risk of bias or methodological quality of the studies included must be conducted. Hereafter, the results of this evaluation must be included in the interpretation of the results of the systematic review [3, 8].

This scoping review has several limitations. An a priori protocol was not previously published. The search was conducted according to the PubMed, EMBASE and Cochrane Library databases indexed terms for studies about systematic reviews’ methodology. These indexed terms may not comprise all recommendations published in literature. Despite the combination of these terms with free terms such as “methods”, the search strategy may not be comprehensive and some references may not have been included. References from other languages than English were not analyzed. In addition, grey literature was not searched. This could lead to the exclusion of some recommendations. Therefore, the results must be interpreted carefully. This review offers an overview of what is published and does not intend to address criticism or influence the choice of a specific recommendation. The preliminary results of this study were presented at ISPOR Europe 2018: New Perspectives for Improving twenty-first Century Health Systems [32].

Conclusions

Several recommendations to conduct and/or report a systematic review are available to combine evidence from diverse healthcare areas. Such recommendations differ in some methodological aspects. Further research on the implications of such heterogeneity seems important, in order to guarantee systematic review transparency, quality and its role in healthcare.

Appendices

Appendix 1

Table 1 Search literature and Search strategy

Appendix 2

Table 2 Detailed description of the recommendations to conduct and/or report a systematic review