Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

A descriptive analysis of non-Cochrane child-relevant systematic reviews published in 2014

BMC Medical Research Methodology volume 18, Article number: 99 (2018) Cite this article

Abstract

Background

Consumers, clinicians, policymakers and researchers require high quality evidence to guide decision-making in child health. Though Cochrane systematic reviews (SRs) are a well-established source of evidence, little is known about the characteristics of non-Cochrane child-relevant SRs. To complement published descriptions of Cochrane SRs, we aimed to characterize the epidemiologic, methodological, and reporting qualities of non-Cochrane child-relevant SRs published in 2014.

Methods

English-language child-relevant SRs of quantitative primary research published outside the Cochrane Library in 2014 were eligible for this descriptive analysis. A research librarian searched MEDLINE, CINAHL, Web of Science, and PubMed in August 2015. A single reviewer screened articles for inclusion; a second verified the excluded studies. Reviewers extracted: general characteristics of the review; included study characteristics; methodological approaches. We performed univariate analyses and presented the findings narratively.

Results

We identified 1598 child-relevant SRs containing a median (IQR) 19 (11, 33) studies. These originated primarily from high-income countries (n = 1247, 78.0%) and spanned 47 of the 53 Cochrane Review Groups. Most synthesized therapeutic (n = 753, 47.1%) or epidemiologic (n = 701, 43.8%) evidence. Though 39.3% (n = 628) of SRs included evidence related to children only, few were published in pediatric-specific journals (n = 283, 17.7%). Reporting quality seemed poor based on the items we assessed; few reviews mentioned an a-priori protocol (n = 246, 15.4%) or registration (n = 111, 6.9%), and only 23.4% (n = 374) specified a primary outcome. Many SRs relied solely on evidence from non-RCTs (n = 796, 49.8%). Less than two-thirds (n = 953, 59.6%) appraised the quality of included studies and assessments of the certainty of the body of evidence were rare (n = 102, 6.4%).

Conclusions

Child-relevant Cochrane SRs are a known source of high quality evidence in pediatrics. There exists, however, an abundance of evidence from non-Cochrane SRs that may be complementary. Our findings show that high-quality non-Cochrane SRs may not be practical nor easy for knowledge users to find. Improvements are needed to ensure that evidence syntheses published outside of the Cochrane Library adhere to the high standard of conduct and reporting characteristic of Cochrane SRs.

Peer Review reports

Background

Consumers, clinicians, policymakers and researchers require up-to-date evidence to guide decision-making, but keeping up with the rapid accumulation of primary research is a challenge. [1] Systematic reviews (SRs) provide a transparent synthesis of all available evidence on a given research question. [2] They are used to inform healthcare decisions, [3] evidence-based practice guidelines, [4] and can facilitate the identification of knowledge gaps, thereby reducing research waste by guiding where new research is most needed. [5, 6] However, findings of SRs that are outdated, methodologically flawed, or incompletely reported are of limited use. [7]

Although clear standards for conduct and reporting are publicly available to authors and widely endorsed, [8] many reviews fall short [9] of best-practice standards. [10, 11] In a comparison of biomedical SRs published in 2004 [12] and 2014, [9] Page et al. (2016) warned that the rise in popularity of evidence syntheses has been accompanied by a proliferation of substandard SRs. [9] Performance bonuses for publication [13] and the rise of predatory publishers [14, 15] are two suggested contributors. These poorly-conducted reviews run the risk of propagating biased, misleading, or harmful conclusions. [16]

Periodically characterizing all available SRs can increase awareness of the areas in need of improvement. Given the shortage of child-specific evidence for healthcare decision making, [17,18,19] the need for high quality evidence syntheses in pediatrics is clear. In 2009 [20] and 2013, [21] authors from our research group undertook descriptive analyses of all child-relevant SRs in the Cochrane Library. Reporting quality of the SRs was highly variable and many were out-of-date. [21] Few focused solely on children or conducted child-specific subgroup analyses. [21]

No similar efforts have been made to characterize child-relevant non-Cochrane SRs. Cochrane reviews represent a small minority [9] of the mounting number of available reviews, [16] and are known to differ from non-Cochrane reviews in terms of scope and reporting quality. [9, 12] Thus, the knowledge gap in terms of the quality and quantity of non-Cochrane evidence available in the field of pediatrics is substantial. We aimed to identify all non-Cochrane child-relevant SRs published over 1 year (2014) and describe the characteristics of these SRs, their included studies, and methodological approaches.

Methods

Design

We conducted a cross-sectional descriptive study of all SRs on any child-relevant topic published outside of the Cochrane Library over 1 year. Ethics approval was not required because there were no human participants.

Eligibility criteria

English language child-relevant SRs of quantitative primary research published outside of the Cochrane Library in 2014 were eligible. We used Cochrane [10] standards to define SRs for inclusion, with some flexibility (e.g., we did not require a quality appraisal) to account for the fact that many SRs published outside of the Cochrane library might not meet their stringent criteria nor adhere to published reporting standards such as the PRISMA statement (Preferred Reporting Items for Systematic reviews and Meta-Analyses). [11, 22] Thus, we considered SRs to be those publications that reported an explicitly defined methodology, including: [10, 11, 22] (a) a comprehensive search (i.e., at least two databases or one database and a supplementary method such as hand-searching); (b) a description of study selection, including the number of records identified via the search and primary studies retained for the review; (c) predefined eligibility criteria; (d) a description of the data extraction methodology and/or items extracted; and (e) a summary of findings. We excluded Cochrane SRs (including duplicate or summary publications published outside the Cochrane Library), theses and dissertations, and all other types of reviews (i.e., rapid reviews, scoping reviews, iterative reviews, reviews of qualitative studies, mixed methods reviews, overviews of reviews).

We defined ‘child-relevant’ SRs as those that intended to include children (0 to 18 years) or presented at least one outcome pertaining to children. [20, 21] This included interventions on adults that were intended to improve the health of children (e.g., smoking cessation programs for post-partum mothers [23]), and reviews including both children and adults, even when child-specific subgroup analyses were not conducted. We included SRs on neonates, those that reported on outcomes measured in adults as a result of exposures occurring in childhood (e.g., post-traumatic stress resulting from child abuse [24]) and those relating to pregnancy and breastfeeding with child-relevant outcomes (e.g., birth weight, [25] congenital malformations, [26] outcomes in the child resulting from breastfeeding [27]). We excluded SRs reporting only fetal outcomes, preterm delivery or stillbirth.

Literature search

A research librarian developed a search strategy to locate child-relevant SRs that included subject headings related to the pediatric population, and SRs (Additional file 1). Searches were undertaken on August 17, 2015 in Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations and Ovid Medline(R) 1946 to present, CINAHL via EBSCOhost, the Web of Science Core Collection, and PubMed. We applied a methodological search filter to restrict the search results to SRs. [28] The searches were also restricted to English language SRs in humans published from January 1 to December 31, 2014.

Selection

We employed a two-stage selection process. First, one reviewer screened the titles and abstracts of all records and classified them as ‘include’, ‘exclude’ or ‘unsure’. A second reviewer screened all ‘excludes’. We then retrieved the full text of any record categorized as ‘include’ or ‘unsure’ by either reviewer. We followed the same process for full text screening. Disagreements were resolved by discussion or the involvement of a third reviewer.

Data extraction

One reviewer developed a data extraction form in Microsoft Office Excel (v. 2016, Microsoft Corporation, Redmond, WA). Three reviewers independently pilot tested the form on a sample of 5 to 10 SRs. Then, a single reviewer independently extracted data from the SRs. The three reviewers met regularly to resolve uncertainties.

We extracted three categories of data, similar to previous descriptive analyses of child-relevant Cochrane SRs, [20, 21] to optimize comparability: 1) general SR characteristics i.e., country of the corresponding author and income classification (World Bank [29]), type of journal, type of review question (i.e., therapeutic [treatment and prevention], epidemiological [prevalence, association between exposure and outcome, genetic association reviews], diagnostic/prognostic [accuracy of diagnostic tests, prognostic factors, clinical prediction rules], or other [cost-effectiveness, psychometric properties of measurement tools, others that do not fit any group]), [9] topic of the review based on existing Cochrane Review Groups (CRGs, to allow comparison in the quantity of evidence by topic area), [30] whether the SR was an update, sources of funding, mention of published or registered review protocol and/or registration within a SR registry; 2) characteristics of the included primary studies i.e., primary study designs sought and included, the number of reports of studies and number of human participants included, and the type of participants included; and 3) methodological approaches i.e., statement of an objective and primary outcome, quality appraisal of the included studies, assessment of certainty of body of evidence (defined as “extent of our confidence that the estimates of effect are correct” [31]), and synthesis method. Additional file 2 shows detailed information about the data extraction items. We extracted the data as reported in the SRs, and did not consult authors, search for published protocols or registration, or use other methods to acquire additional information.

We examined the agreement between reviewers via duplicate extraction of a random sample of 10% (n = 160/1598) of the SRs. One of the reviewers involved in data extraction verified all data contained in any data item where agreement was < 85% in the random sample. Any other noted discrepancies were also corrected.

Data analysis

We performed univariate analyses in Excel to describe the general characteristics of the SRs, their included studies and methodological approaches. We calculated counts and frequencies for categorical data, and medians and interquartile ranges for continuous data. We presented the findings narratively.

Results

Selected sample of systematic reviews

We retrieved 5213 unique records via the search and excluded 2070 during title and abstract screening. We inspected the full text of 3143 records and excluded 1545 (Fig. 1). Most full text exclusions were related to publications not meeting our definition of a SR (n = 1258/1545, 81.4%). The final sample included 1598 child-relevant SRs containing a median (IQR) 19 (11 to 33) reports of studies and 2134 (648 to 13,079) participants.

Fig. 1
figure 1

Flow of records through the selection process

Full size image

Characteristics of the overall sample

Tables 1, 2 and 3 show the SR, included primary studies, and methodological characteristics of the sample, overall and by type of review question. The SRs originated almost exclusively from high (n = 1247, 78.0%) and upper-middle (n = 317, 19.8%) income countries. The USA (n = 384, 24.0%), China (n = 201, 12.6%), the UK (n = 201, 12.6%), Canada (n = 150, 9.4%), and Australia (n = 136, 8.5%) were the top child-relevant SR-producing countries.

Table 1 Characteristics of non-Cochrane child-relevant systematic reviews published in 2014, overall and by type of review question
Full size table
Table 2 Included study characteristics for non-Cochrane child-relevant systematic reviews published in 2014, overall and by type of review question
Full size table
Table 3 Methodological characteristics for non-Cochrane child-relevant systematic reviews published in 2014, overall and by type of review question
Full size table

The SRs spanned 47 of the 53 CRGs and the 8 most populated review groups contained 53.2% (n = 849) of the SRs: Developmental, Psychosocial and Learning Problems (n = 216, 13.5%); Metabolic and Endocrine Disorders (n = 132, 8.3%); Pregnancy and Childbirth (n = 124, 7.8%); Common Mental Disorders (n = 110, 6.9%); Airways (n = 70, 4.4%); Anaesthesia, Critical and Emergency Care (n = 69, 4.3%); Public Health (n = 67, 4.2%); and Infectious Diseases (n = 61, 3.8%). Some notable CRGs containing very few SRs were Skin (n = 15, 0.9%), Tobacco Addiction (n = 15, 0.9%), Back and Neck (n = 14, 0.9%), Neonatal (n = 14, 0.9%; neonatal reviews of RCTs only), Eyes and Vision (n = 12, 0.8%), Haematological Malignancies (n = 9, 0.6%), Neuromuscular (n = 7, 0.4%), and Urology (n = 3, 0.2%).

Most often, the SRs synthesized therapeutic (n = 753, 47.1%) or epidemiologic (n = 701, 43.8%) evidence. Although 39.3% (n = 628) of the SRs included evidence specific to children only, fewer were published in pediatric-specific journals (n = 283, 17.7%). Approximately half of SRs (n = 807, 50.5%) declared at least one external source of support; 15.5% (n = 248) had no external funding. The most common sources of support were governments (n = 557, 34.9%), academic or research institutes (n = 228, 14.3%), and foundations (n = 193, 12.1%).

Only three SRs (< 0.01%) included all of the methodological and reporting characteristics that we assessed. Few mentioned a published or registered protocol (n = 246, 15.4%) or SR registration (n = 111, 6.9%). One-third neglected to mention sources of support (n = 543, 34.0%) and more than half did not indicate the total number of included human participants (n = 902, 56.6%). Most (n = 1224, 76.6%) did not specify one or more primary outcome(s). In 9.9% (n = 158) of cases, eligible primary study designs were not mentioned, and in 9.2% (n = 147) the type(s) included in the SR was not discernable. Few SRs were identified as updates (n = 32, 2.0%). Some were labelled as updates by their authors, but appeared to be new.

Authors rarely searched only for RCTs (n = 203, 12.7%). Most commonly, both RCTs and other primary study designs were eligible (n = 765, 47.9%). This was reflected in the included primary studies, where most SRs relied solely on evidence from non-RCTs (n = 796, 49.8%) or a mix of designs (n = 420, 26.3%). Less than two-thirds of SRs (n = 953, 59.6%) included an assessment of the quality of the included studies and very few assessed the certainty of the body of evidence (n = 102, 6.4%; using Grading of Recommendations Assessment, Development and Evaluation (GRADE) (n = 72, 4.5%) or another method (n = 30, 1.9%)). The synthesis method used in SRs was roughly equally split between narrative (n = 741, 46.4%) and statistical (n = 857, 53.6%). There were 5 network meta-analyses and 12 individual patient data meta-analyses.

Characteristics of the sample by type of review question

Epidemiological SRs included the largest number of primary studies (median (IQR), 23 (13 to 40)) and human participants (11,038 (2098 to 63,897)). Compared to the overall sample, a larger proportion of diagnostic/prognostic SRs were published in Asia (n = 25/91, 27.5%), and therapeutic SRs in North America (n = 287/753, 38.1%). The Injuries CRG was the most strongly represented group among diagnostic/prognostic SRs, while the Developmental, Psychosocial, and Learning Problems CRG was predominant across all other review types.

Compared to the overall sample, epidemiological SRs more often included mixed samples of children and adults (n = 384/701, 54.8%) and less often included only children (n = 212/701, 30.2%), but the type of journal in which they were published did not differ. In contrast, the ‘other’ SRs were more often published in pediatric journals (n = 16/53, 30.1%), though the sample of these was small. The highest proportion of unfunded SRs was in the therapeutic group (n = 130/753, 17.3%). Industry funding was most common among the diagnostic/prognostic (n = 5/91, 5.5%) and ‘other’ (n = 3/53, 5.7%) SRs.

Reporting of the items of interest that we collected was most complete for the therapeutic SRs and least for the epidemiological and ‘other’ SRs. The therapeutic SRs most often indicated the type of primary studies sought (n = 727/753, 96.5%) and included (n = 734/753, 97.5%), reported the total number of human participants (n = 370/753, 49.4%), and identified a primary outcome (n = 281/753, 37.3%). The epidemiological SRs least often identified a published or registered SR protocol (n = 90/701, 12.8%) or registration in a SR registry (n = 35/701, 5.0%), and most often failed to indicate the type of primary studies sought (n = 114/701, 16.3%) and included (n = 111/701, 15.8%). The total number of participants (n = 273/701, 38.9%) and primary outcome (n = 74/701, 10.6%) were relatively infrequently reported among epidemiologic SRs, though the ‘other’ SRs also performed poorly in this respect.

All but one of the SRs seeking to include only RCTs was therapeutic; the epidemiologic, diagnostic/prognostic, and ‘other’ SRs more commonly sought only non-RCTs (49.4%, 52.7%, 54.7%, respectively). Non-RCTs were eligible for inclusion among 69.7% (n = 525/753) of the therapeutic SRs. The quality of the included studies was least frequently assessed in the epidemiologic SRs (n = 325/701, 46.4%), and most frequently in therapeutic (n = 532/753, 70.7%) and diagnostic/prognostic (n = 63/91, 69.2%) SRs. Certainty of the body of evidence was most often assessed in therapeutic SRs (n = 75/753, 10.0%), and rarely in epidemiologic (n = 20/701, 2.8%) or diagnostic/prognostic (n = 3/91, 3.3%) SRs.

Discussion

Though Cochrane SRs are a go-to source for high quality evidence in pediatrics, clinicians must also look elsewhere for evidence on topics that Cochrane reviews do not cover (e.g., epidemiology, and to a lesser extent diagnosis/prognosis). Our findings indicate that outside of the Cochrane Library, an abundance of child-relevant SRs are being produced (i.e., 1598 in 1 year), with a breadth of coverage across almost all CRGs. It appears, however, that publication quantity has outpaced quality. We applied strict inclusion criteria, and sifted through numerous records to identify those that met the minimum standards required to be considered ‘true SRs’ (i.e., the poorest quality reviews would have been excluded). [10, 22] Nevertheless, methodological and reporting deficits among the items that we assessed were prevalent. In the absence of a central repository for high quality child-relevant SR evidence, efforts to ensure that unbiased and accurate child-relevant SRs are readily available and easily identifiable by decision makers are required.

Within Cochrane, the ‘List Project’ is one initiative aiming to align SR priority topics with those that are most likely to impact worldwide health outcomes. [32] To roughly estimate whether the topics covered in non-Cochrane SRs might be aligned with disease burden priorities, we compared the number of publications in various topic areas to the global burden of disease. Similar to child-relevant SRs published in the Cochrane Library, [21] our findings suggest that SRs published outside of the Cochrane Library are diverse but their focus is not necessarily aligned with the global disease burden. Seventy-eight percent of child mortality in 2013 was attributable to lower respiratory infections, other infectious disease (e.g., diarrheal diseases, malaria), conditions related to pregnancy and childbirth, and road injuries. [33] Thus, we would expect the proportion of SRs falling in the Acute Respiratory Infections (n = 42, 2.6%), Infectious Diseases (n = 61, 3.8%) and Injuries (n = 43, 2.7%) CRGs to be higher. Though developing countries face the highest burden of child mortality, [29] few SRs originated from low or lower-middle income countries (n = 34, 2.1%). We acknowledge that the burden of disease is only one factor among many (e.g., political or social context, research capacity, resources) that might influence the topics of published systematic reviews. However, a recommended practice to improve the relevance of SRs would be to conduct prioritization exercises involving all key stakeholders (e.g., patients, researchers, decision makers) [34, 35] and to make the results available to those involved in SR conduct.

While the Cochrane Library is known as the go-to source for high quality evidence syntheses, our sample indicates that SRs published elsewhere can be complementary. While Cochrane SRs have focused on healthcare interventions, [10] and more recently diagnostic accuracy, [21] our non-Cochrane sample included a greater focus on epidemiological (43.8%) and diagnostic/prognostic reviews (5.7%). Similarly, while Cochrane SRs rely heavily on evidence from randomised controlled trials (RCTs), [21] even among the therapeutic reviews in our sample, only 30% restricted their synthesis to RCTs. It has been argued that the inclusion of diverse designs in SRs may not only be desirable, but necessary to accommodate the full breadth of clinical research questions and outcomes. [36] One concern though, is that assessing the quality of non-RCTs is a complex task, and only 55.7% (n = 759/1363) of the SRs that included non-RCTs in our sample conducted any type of quality assessment. There remains a lack of clarity about the credibility of the results of these studies, and their impact on the quality of the body of evidence.

Poor quality evidence syntheses contribute little to advancing research and practice. Though we did not assess all aspects of reporting endorsed by PRISMA, [11, 22] this descriptive analysis, in which we appraised discrete reporting items, adds to numerous studies across various fields of research showing that adherence to widely endorsed reporting guidance (e.g., PRISMA) [8, 11, 22] is often poor. [37] Contributors to poor reporting quality can include inadequate understanding of the guidance, peer reviewers who are unaware of how to appraise reporting quality, or insufficient scrutiny by editorial boards. Indeed, in characterizing biomedical SRs, Page et al. (2016) [9] found that many authors misinterpreted the PRISMA guidelines. The high prevalence of reporting and methodological deficits, based on the few items that we assessed among our sample, speaks to the difficulty that knowledge users may face when attempting to find high quality evidence syntheses outside of the Cochrane Library. The findings further confirm the need for (a) improved dissemination of these guidelines in ways that are easily understood, and (b) increased oversight by editorial organizations.

Decision-makers rely on unbiased and accurate findings from SRs, and must search beyond the Cochrane library to answer clinical questions for which Cochrane SRs are unavailable. Our analysis indicates that even among those with expertise in evidence synthesis, high quality SRs published outside of the Cochrane Library are challenging and time consuming to find. Despite using a validated search filter, [28] we spent considerable time sifting through records indexed as ‘systematic reviews’ that in reality did not meet the minimum SR criteria, as defined by Cochrane. [10] Only a handful of SRs in the final sample included all of the methodological and reporting features that might be expected (as informed by the aspects of PRISMA reporting standards that we assessed). [11, 22] Available tools to appraise the quality of evidence syntheses [38] require expertise to use and cannot be quickly nor easily applied in practice. With limited guidance available outside of Cochrane, [39, 40] SR updates are difficult for authors to design and publish, and can be challenging for readers to identify. The end result is an increasingly overwhelming, potentially redundant, and confusing body of research that cannot easily be used.

Strengths and limitations

We have identified key areas in need of improvement in the methodology and reporting of child-relevant SRs. This assessment was based on several reporting standards of interest, but we did not formally appraise the SRs against specific guidance for methods or reporting. In the future, an in-depth characterization of the included SRs against established guidance (e.g., PRISMA, [11, 22] AMSTAR 2 [41]) would add value. One limitation of our sample is that it included only English-language SRs of quantitative data published in 1 year. Other types of reviews fell outside the scope, and SR topics will vary over time. Related to the inconsistent, and overall poor reporting, it was often difficult to extract key information from the SRs. Also, one reviewer extracted data from the SRs, which increased the risk of error compared to duplicate extraction. [42] We mitigated these risks by pilot testing the data extraction form, meeting regularly to address concerns, and re-evaluating data items with poor agreement between reviewers. As we did not seek additional information about the SRs, it was not possible to separate methodological from reporting deficits.

Conclusion

Knowledge users require high quality pediatric-specific evidence to guide decision-making in child health. Though Cochrane SRs remain the go-to source for high quality SRs, knowledge users may search elsewhere for evidence syntheses that suit their needs. We have shown that non-Cochrane evidence syntheses may complement those within the Cochrane Library (e.g., greater focus on epidemiological reviews). However, a high prevalence of methodological and reporting shortfalls means that finding high quality evidence syntheses outside of the Cochrane Library may not be easy nor practical. Improvement is needed to ensure easy access to high quality evidence syntheses outside of the Cochrane Library.

Abbreviations

GRADE:

Grading of Recommendations Assessment

Development and Evaluation

IQR:

inter-quartile range

RCT:

randomised controlled trial

SR:

systematic review

References

  1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326.

    Article  Google Scholar 

  2. Egger M, Smith GD, O'Rourke K. Introduction: rationale, potentials, and promise of systematic reviews. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. 2nd ed. London, UK: BMJ Publishing Group; 2008. p. 1–19.

    Google Scholar 

  3. Cook DJ, Mulrow CD, Haynes R. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

    Article  CAS  Google Scholar 

  4. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–E42.

    Article  Google Scholar 

  5. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

    Article  Google Scholar 

  6. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.

    Article  Google Scholar 

  7. Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

    Article  Google Scholar 

  8. PRISMA. PRISMA endorsers. 2015. http://www.prisma-statement.org/Endorsement/PRISMAEndorsers.aspx. Accessed 14 Dec 2017.

  9. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028. https://doi.org/10.1371/journal.pmed.1002028.

    Article  Google Scholar 

  10. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions, version 5.1.0. London, UK: The Cochrane Collaboration; 2011. Available from: http://handbook-5-1.cochrane.org/. Accessed 14 Dec 2017.

  11. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100 https://doi.org/10.1371/journal.pmed.1000100.

    Article  Google Scholar 

  12. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78–13 https://doi.org/10.1371/journal.pmed.0040078.

    Article  Google Scholar 

  13. Franzoni C, Scellato G, Stephan P. Changing incentives to publish. Science. 2011;333(6043):702–3.

    Article  CAS  Google Scholar 

  14. Bartholomew RE. Science for sale: the rise of predatory journals. J R Soc Med. 2014;107(10):384–5.

    Article  Google Scholar 

  15. Clark J, Smith R. Firm action needed on predatory journals. BMJ. 2015;350:h210. https://doi.org/10.1136/bmj.h210.

    Article  Google Scholar 

  16. Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

    Article  Google Scholar 

  17. Cramer K, Wiebe N, Moyer V, et al. Children in reviews: methodological issues in child-relevant evidence syntheses. BMC Pediatr. 2005;5(1):38.

    Article  Google Scholar 

  18. Klassen TP, Hartling L, Craig JC, Offringa M. Children are not just small adults: the urgent need for high-quality trial evidence in children. PLoS Med. 2008;5(8):e172. https://doi.org/10.1371/journal.pmed.0050172.

    Article  Google Scholar 

  19. Zylke JW, Rivara FP, Bauchner H. Challenges to excellence in child health research: call for papers. JAMA. 2012;308(10):1040–1.

    Article  CAS  Google Scholar 

  20. Bow S, Klassen J, Chisholm A, et al. A descriptive analysis of child-relevant systematic reviews in the Cochrane database of systematic reviews. BMC Pediatr. 2010;10(1):34. https://doi.org/10.1186/1471-2431-10-34.

    Article  Google Scholar 

  21. Crick K, Thomson D, Fernandes RM, et al. Descriptive analysis of Cochrane child-relevant systematic reviews: an update and comparison between 2009 and 2013. BMC Pediatr 2017;17(1):155. doi: https://doi.org/10.1186/s12887-017-0908-7.

  22. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097.

    Article  Google Scholar 

  23. Su A, Buttenheim AM. Maintenance of smoking cessation in the postpartum period: which interventions work best in the long-term? Matern Child Health J. 2014;18(3):714–28.

    Article  Google Scholar 

  24. Lindert J, Von Ehrenstein OS, Grashow R, Gal G, Braehler E, Weisskopf MG. Sexual and physical abuse in childhood is associated with depression and anxiety over the life course: systematic review and meta-analysis. Int J Public Health. 2014;59(2):359–72.

    Article  Google Scholar 

  25. Murphy MM, Stettler N, Smith KM, Reiss R. Associations of consumption of fruits and vegetables during pregnancy with infant birth weight or small for gestational age births: a systematic review of the literature. Int J Womens Health. 2014;6:899–912.

    Article  Google Scholar 

  26. Shi QY, Zhang JB, Mi YQ, Song Y, Ma J, Zhang YL. Congenital heart defects and maternal fever: systematic review and meta-analysis. J Perinatol. 2014;34(9):677–82.

    Article  CAS  Google Scholar 

  27. Dogaru CM, Nyffenegger D, Pescatore AM, Spycher BD, Kuehni CE. Breastfeeding and childhood asthma: systematic review and meta-analysis. Am J Epidemiol. 2014;179(10):1153–67.

    Article  Google Scholar 

  28. Montori VM, Wilczynski NL, Morgan D, Haynes RB, For the hedges team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330(7482):68. https://doi.org/10.1136/bmj.38336.804167.47.

    Article  Google Scholar 

  29. The World Bank Group. World Bank country and lending groups. 2017. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519. Accessed 13 September 2017.

  30. The Cochrane Collaboration. Review groups. 2017. Available from: http://www.cochrane.org/ca/contact/review-groups. Accessed 13 September 2017.

  31. Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE working group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13.

    Article  Google Scholar 

  32. Cochrane Community. Prioritization list project. 2018 Available from: https://community.cochrane.org/review-production/production-resources/prioritization-list-project. Accessed 11 Sept 2018.

  33. Global Burden of Disease Pediatrics Collaboration. Global and national burden of diseases and injuries among children and adolescents between 1990 and 2013: findings from the global burden of disease 2013 study. JAMA Pediatr. 2016;170(3):267–87.

    Article  Google Scholar 

  34. Doyle J, Waters E, Yach D, McQueen D, De Francisco A, Stewart T, et al. Global priority setting for Cochrane systematic reviews of health promotion and public health research. J Epidemiol Community Health. 2005;59(3):193–7.

    Article  CAS  Google Scholar 

  35. Viergever RF, Terry R, Matsoso M. Health research prioritization at WHO: an overview of methodology and high level analysis of WHO led health research priority setting exercises. Geneva: World Health Organization; 2010.

    Google Scholar 

  36. Peinemann F, Tushabe DA, Kleijnen J. Using multiple types of studies in systematic reviews of health care interventions – a systematic review. PLoS One. 2013;8(12):e85035. https://doi.org/10.1371/journal.pone.0085035.

    Article  Google Scholar 

  37. Samaan Z, Mbuagbaw L, Kosa D, Debono VB, Dillenburg R, Zhang S, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

    PubMed  PubMed Central  Google Scholar 

  38. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

    Article  Google Scholar 

  39. Pieper D, Mathes T. Survey of instructions for authors on how to report an update of a systematic review: guidance is needed. Evid Based Med. 2017;22(2):45–8.

    Article  Google Scholar 

  40. Garner P, Hopewell S, Chandler J, MacLehose H, Akl EA, Beyene J, et al. When and how to update systematic reviews: consensus and checklist. BMJ. 2016;354:i3507. https://doi.org/10.1136/bmj.i3507.

    Article  Google Scholar 

  41. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

    Article  Google Scholar 

  42. Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

    Article  Google Scholar 

  43. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.

    Article  Google Scholar 

Download references

Acknowledgments

We thank Dr. Sanja Schreiber, PhD for early work on the project, Robin Featherstone, MLIS for planning and performing the search, and Dr. Allison Gates, PhD for screening assistance and acting as a third reviewer to aid in achieving consensus when needed. Those acknowledged have no conflicts of interest or funding sources to declare.

Funding

The Child Health Field has received support from the Canadian Institutes of Health Research and Cochrane. LH is supported by a New Investigator Award from the Canadian Institutes of Health Research.

Availability of data and materials

The datasets analysed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

  1. Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, AB, Canada

    Michelle Gates, Sarah A Elliott, Cydney Johnson, Denise Thomson & Lisa Hartling

  2. Cochrane Child Health, Department of Pediatrics, University of Alberta, Edmonton, AB, Canada

    Sarah A Elliott, Denise Thomson, Katrina Williams, Ricardo M Fernandes & Lisa Hartling

  3. School of Pediatrics, Royal Children’s Hospital, Melbourne, VIC, Australia

    Katrina Williams

  4. Department of Pediatrics, Santa Maria Hospital, Lisbon Academic Center, Lisbon, Portugal

    Ricardo M Fernandes

  5. Clinical Pharmacology and Therapeutics, Instituto de Medicina Molecular, Universidade de Lisboa, Lisbon, Portugal

    Ricardo M Fernandes

  6. School of Pediatrics, Royal Children’s Hospital, 50 Flemington Road, Parkville, VIC, 20152, Australia

    Katrina Williams

  7. Department of Pediatrics, Hospital Santa Maria (CHLN), Lisbon Academic Center, Lisbon, Portugal

    Ricardo M Fernandes

Authors
  1. Michelle Gates
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah A Elliott
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cydney Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Denise Thomson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Katrina Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ricardo M Fernandes
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lisa Hartling
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MG, DT, KW, RMF, and LH contributed to conception and design of the work; MG, SE, and CJ collected, analyzed and interpreted the data; MG drafted the manuscript; all authors revised drafts of the manuscript for critically important intellectual content, approve of the version submitted, and agree to be accountable for all aspects of the work.

Corresponding author

Correspondence to Lisa Hartling.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Search strategy. This file documents the search strategy used to locate included studies (PDF 193 kb)

Additional file 2:

Data extraction guide for a descriptive analysis of non-Cochrane child-relevant systematic reviews. This file provides a detailed description of the data extracted for analysis. (PDF 423 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gates, M., Elliott, S.A., Johnson, C. et al. A descriptive analysis of non-Cochrane child-relevant systematic reviews published in 2014. BMC Med Res Methodol 18, 99 (2018). https://doi.org/10.1186/s12874-018-0562-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-018-0562-2

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us