A descriptive analysis of non-Cochrane child-relevant systematic reviews published in 2014

Background Consumers, clinicians, policymakers and researchers require high quality evidence to guide decision-making in child health. Though Cochrane systematic reviews (SRs) are a well-established source of evidence, little is known about the characteristics of non-Cochrane child-relevant SRs. To complement published descriptions of Cochrane SRs, we aimed to characterize the epidemiologic, methodological, and reporting qualities of non-Cochrane child-relevant SRs published in 2014. Methods English-language child-relevant SRs of quantitative primary research published outside the Cochrane Library in 2014 were eligible for this descriptive analysis. A research librarian searched MEDLINE, CINAHL, Web of Science, and PubMed in August 2015. A single reviewer screened articles for inclusion; a second verified the excluded studies. Reviewers extracted: general characteristics of the review; included study characteristics; methodological approaches. We performed univariate analyses and presented the findings narratively. Results We identified 1598 child-relevant SRs containing a median (IQR) 19 (11, 33) studies. These originated primarily from high-income countries (n = 1247, 78.0%) and spanned 47 of the 53 Cochrane Review Groups. Most synthesized therapeutic (n = 753, 47.1%) or epidemiologic (n = 701, 43.8%) evidence. Though 39.3% (n = 628) of SRs included evidence related to children only, few were published in pediatric-specific journals (n = 283, 17.7%). Reporting quality seemed poor based on the items we assessed; few reviews mentioned an a-priori protocol (n = 246, 15.4%) or registration (n = 111, 6.9%), and only 23.4% (n = 374) specified a primary outcome. Many SRs relied solely on evidence from non-RCTs (n = 796, 49.8%). Less than two-thirds (n = 953, 59.6%) appraised the quality of included studies and assessments of the certainty of the body of evidence were rare (n = 102, 6.4%). Conclusions Child-relevant Cochrane SRs are a known source of high quality evidence in pediatrics. There exists, however, an abundance of evidence from non-Cochrane SRs that may be complementary. Our findings show that high-quality non-Cochrane SRs may not be practical nor easy for knowledge users to find. Improvements are needed to ensure that evidence syntheses published outside of the Cochrane Library adhere to the high standard of conduct and reporting characteristic of Cochrane SRs. Electronic supplementary material The online version of this article (10.1186/s12874-018-0562-2) contains supplementary material, which is available to authorized users.

Highlight for discussion if it is too difficult or if it seems like the review could fit in more than one group, write key words in the next column.

Open text
This column only needs to be used if you cannot decide on a CRG.
Extract key words relating to the clinical area. Example: "Disclosure and non-disclosure of concussion and concussion symptoms in athletes: Review and application of the socio-ecological framework". Key words provided by the authors included: Policy, reporting, sports, traumatic brain injury.
Since the purpose of extracting key words is to easier identify a CRG the review would likely fall under, here we would most likely want to extract "traumatic brain injury". Therapeutic: Includes treatment and prevention (interventions would typically fall in this group) Epidemiology: Includes prevalence reviews and those looking at the association between an exposure and an outcome (e.g., studies of etiology) Diagnosis/prognosis: reviews of diagnostic test accuracy (e.g., sensitivity, specificity, false +/-), clinical prediction rules Other: psychometric properties (e.g., reliability and validity) of tools, cost of illness, and other topics that would not fit into the other categories Was the review an update?

□ Yes □ No
Yes: the authors identify the review as an update of an existing review. Authors must have incorporated the data from the previous Existence of an a-priori protocol a,b □Yes □No Yes: the authors indicated that a protocol was developed a priori. No: the authors indicated that protocol was not developed a priori.

Field
Response Options Instructions □Not mentioned Not mentioned: the authors do not mention whether a protocol was developed.
Note: If mentioned that the review was registered in PROSPERO, this means there is a protocol Registration of the review a,b □Yes □No □Not mentioned Yes: the authors indicated that a review was registered, or indicate that the protocol was registered (which implies registration of the review) No: the authors indicated that a review was not registered. Not mentioned: the authors did not mention whether a review was registered.

CHARACTERISTICS OF INCLUDED STUDIES Study designs sought (part of eligibility criteria a,b )
□Only RCTs □Only non-RCTs □RCTS and other designs □Unclear/unreported Only RCTs: randomized controlled trials (parallel group or crossover designs) Only non-RCTs (other designs): e.g., observational or epidemiological studies -cohort, cross sectional, before-after, case-control, time series RCTs and other designs: Searched for any study design (this should be stated -e.g., searched for 'all studies' on a particular topic). Also, e.g., if they only excluded case control studies or reviews, you can choose this as they can be assumed to have included all other designs Unclear/unreported: there is no mention of what type of design was sought -e.g., searched for 'studies' with no mention of design Note: Do not guess or attempt to infer, choose unclear whenever the type of design sought (or excluded) is not clearly stated Study designs included (part of adequate description of included studies a,b ) □Only RCTs □Other designs □RCTS and other designs □Unclear/unreported As above Note: Do not guess or attempt to infer, choose unclear whenever the type of designs obtained are not clearly stated (e.g., in results text or tables). Do not search reference lists.  Yes: the authors explicitly report on the total number of the participants included (or a summary of groups, if there was a comparison of 2+ groups such as intervention and control). This can typically be found in the results, and sometimes a PRISMA chart. Please calculate ONLY IF a summary value is given (e.g., for two groups such as control vs. treatment). No: the number of the participants is not obviously reported. DO NOT calculate across included studies.

Number
If a review's population members of naturally occurring groups, such as families or classes, enter the number of individuals not the groups If a number was given as an estimate (e.g. >2 million, or approximately 1 million, etc.), enter the estimated lower number in digits using no decimals, spaces or commas (e.g. 2000000 or 1000000) NA: If there are no studies in the review, the authors cannot report on the number of participants.

Field Response Options Instructions METHODOLOGICAL APPROACHES Objective stated a,b □Yes □No
Yes: the authors provide one or more objectives/purpose for the review, even if this is only mentioned in the abstract. Ideally would be based on PICO but this is not a requirement for this project.
No: no objectives specified anywhere in the paper (before results) Primary outcome specified

□Yes □No
Yes: There is a specific statement e.g., "the primary outcome is/was…" -must explicitly state what the primary outcome is (could also call this a dependent variable or something similar) (note: for SRs there can be several primary outcomes, so long as they are identified); just mentioning 'outcomes' is not enough. No: There is no explicit statement about the primary outcome (even if there is only 1) Outcomes of interest listed or can be inferred a,b □Yes □No □NA Yes: a-priori outcomes (at least one) of interest are listed or can be inferred from statements in the introduction, objectives, hypotheses or methods (e.g., inclusion/exclusion, data extraction) No: Cannot tell the intended outcomes from the introduction, objectives, hypotheses or methods (or intended outcomes are not discrete and instead are only vague categories e.g., 'behavioural outcomes', so you cannot tell what exactly they are looking for) NA: If a primary outcome is stated, then this item is not applicable Was the quality of included studies formally assessed? a,b □Yes □No □NA Yes: a tool was used to appraise quality of included studies (e.g., Cochrane risk of bias tool, Newcastle-Ottawa scale, or one developed by the authors for use in the SR) No: there was no formal quality appraisal NA: no studies were found, thus the quality could not be assessed Was the quality of evidence assessed using GRADE? a □Yes □No □Other method □NA This question pertains to the quality of evidence usually assessed by GRADE, and not quality of the included studies (above). Yes: used the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) assessment to grade the quality of the evidence. No: the GRADE tool was not used Other tool: some tool/method other than GRADE was used to assess the quality of the evidence (specify in comments)

Field
Response Options Instructions NA: no studies were found, thus the quality could not be assessed How was the evidence synthesized?
□Narratively only □Statistically Narratively only: results were narratively summarized the findings of multiple studies are explained. Some authors call this a 'qualitative synthesis'. There is no statistical analysis. Statistically: results were analyzed using a statistical approach (e.g. meta-analysis or network meta-analysis). If the evidence was synthesized statistically, what method was used?
□Meta-analysis □Network meta-analysis □Individual patient data meta-analysis □NA Indicate method used from the 3 choices. Mixed: multiple methods were used in the review (list in comments) NA: the results were not synthesized statistically a The item is a PRISMA reporting item [3,4] b The item may be used to appraise systematic review quality using AMSTAR 2 [5]