Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Outcome choice and definition in systematic reviews leads to few eligible studies included in meta-analyses: a case study

BMC Medical Research Methodology volume 20, Article number: 30 (2020) Cite this article

Abstract

Background

There is broad recognition of the importance of evidence in informing clinical decisions. When information from all studies included in a systematic review (“review”) does not contribute to a meta-analysis, decision-makers can be frustrated. Our objectives were to use the field of eyes and vision as a case study and examine the extent to which authors of Cochrane reviews conducted meta-analyses for their review’s pre-specified main outcome domain and the reasons that some otherwise eligible studies were not incorporated into meta-analyses.

Methods

We examined all completed systematic reviews published by Cochrane Eyes and Vision, as of August 11, 2017. We extracted information about each review’s outcomes and, using an algorithm, categorized one outcome as its “main” outcome. We calculated the percentage of included studies incorporated into meta-analyses for any outcome and for the main outcome. We examined reasons for non-inclusion of studies into the meta-analysis for the main outcome.

Results

We identified 175 completed reviews, of which 125 reviews included two or more studies. Across these 125 reviews, the median proportions of studies incorporated into at least one meta-analysis for any outcome and for the main outcome were 74% (interquartile range [IQR] 0–100%) and 28% (IQR 0–71%), respectively. Fifty-one reviews (41%) could not conduct a meta-analysis for the main outcome, mostly because fewer than two included studies measured the outcome (21/51 reviews) or the specific measurements for the outcome were inconsistent (16/51 reviews).

Conclusions

Outcome choice during systematic reviews can lead to few eligible studies included in meta-analyses. Core outcome sets and improved reporting of outcomes can help solve some of these problems.

Peer Review reports

Background

There is broad recognition of the importance of evidence in determining clinical decision-making [1]. For evidence-based healthcare, decision-makers (e.g., patients, clinicians, guideline developers) increasingly rely on systematic reviews (“reviews”) [1]. Reviews identify primary studies, such as clinical trials and observational studies, that have addressed the research question of interest. This research question typically defines the population, interventions, and comparators; these defined aspects in turn help delineate the primary studies eligible for the review.

Reviews may or may not include quantitative syntheses of data across studies (“meta-analyses”). When appropriately conducted, meta-analyses provide decision-makers with summary estimates (e.g., relative risks) and accompanying estimates of uncertainty (e.g., 95% confidence intervals) that convey information about treatment effectiveness or safety succinctly [2]. Often, however, meta-analyses cannot be conducted because the studies address somewhat different clinical questions, assess different outcomes than the systematic reviewer (“reviewer”) had pre-specified, are methodologically heterogeneous, or are poorly-reported (e.g., inadequate information about results). In these circumstances, a study may be eligible for the review, but may not contribute to a meta-analysis [3]. When a review includes multiple studies, but these studies cannot be included in the meta-analysis, both doers (i.e., reviewers) and users of reviews (i.e., decision-makers) can be frustrated. Decision-makers want to know how treatments compare quantitatively; they may not be able to get reliable information about this when only some included studies contribute data to the meta-analysis or when no meta-analysis is possible [4].

Outcomes are measures or events used to assess the effectiveness and/or safety of clinical interventions [5]. A frequent reason for non-conduct of meta-analyses is that the studies assess different outcomes or assess the same outcomes, but do so differently. These scenarios can occur even among high-quality studies.

Although outcomes are fundamental to reviews of interventions, outcomes are typically not considered when determining the eligibility of a primary study in such reviews [6]. This is because outcomes inform meta-analyses, not whether the primary study is eligible for the review. Consistent with guidance in the Cochrane Handbook for Systematic Reviews of Interventions [6], we believe that studies that address the population, interventions, and comparators of interest should be included and cataloged in systematic reviews even if they do not report outcomes of interest. Outcome choice in a review is crucial because: (1) outcomes serve as yardsticks for basing conclusions about treatments; and (2) which outcomes are chosen and how they are defined can impact how many meta-analyses can be done and how many studies can be included in them [7,8,9,10,11].

Outcomes may be assessed differently in different studies because an “outcome” (a seemingly monolithic entity) actually comprises five elements: domain, e.g., visual acuity; specific measurement, e.g., Snellen chart; specific metric, e.g., ≥3 lines of vision lost; method of aggregation, e.g., proportion; and time-points, e.g., 6 months [9, 12]. Another example of the application of this five-element framework to clearly specify a particular data point of interest related to the outcome of “anxiety” is mean (method of aggregation) change (specific metric) in anxiety (domain) measured through the Hamilton Anxiety Rating Scale (specific measurement) from baseline to 1 year (time-point) [9, 12].

We previously demonstrated, through case studies in the fields of eyes and vision [11] and HIV/AIDS [10], that reviewers and clinical trialists addressing the same research question often examine different outcomes. In addition, inconsistency in outcome reporting across eligible studies prevents incorporation of all eligible studies into meta-analyses. For instance, a 2017 Cochrane systematic review comparing non-steroidal anti-inflammatory drugs (NSAIDs) with corticosteroids for inflammation after cataract surgery [13] included 48 trials, none of which reported data for the review’s pre-specified primary outcome, “proportion of patients with intraocular inflammation at 1 week after surgery.”

To document the extent and determinants of this problem, we embarked on the current case study in the field of eyes and vision. Our objectives were to examine the extent to which Cochrane reviews in eyes and vision conducted meta-analyses for the main outcome domain and the reasons why some otherwise eligible studies were not incorporated into meta-analyses.

Methods

Reviews examined

We examined all completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews as of August 11, 2017. We excluded reviews that were still in the protocol stage.

Data extraction

We developed a data extraction form in the Systematic Review Data Repository (SRDR), an open-source platform for extracting and archiving data [14, 15]. Using a pilot-tested form, two individuals (from among SM, HK, BTS, and IJS) independently extracted data, resolving discrepancies through discussion. We extracted the following data: year published, population (i.e., eye function/region affected), and types of interventions and comparators. We extracted the numbers of primary, secondary, and other, i.e., non-primary and non-secondary, outcome domains. We also extracted the number of studies included in the review and in ≥1 meta-analysis for any, any primary, any secondary, and any other domain.

“Main” outcome domains

We categorized one domain from each review as its “main” outcome domain (Table 1). For reviews that named only one primary outcome domain, we categorized it as the main outcome domain; for reviews that named more than one primary outcome domain (or named more than one secondary outcome domain), we categorized the primary outcome domain (or secondary outcome domain) with the highest number of included studies as the main outcome domain. For reviews that did not name any primary or secondary outcome domains, we categorized the “other”, i.e., nonprimary and non-secondary, outcome domain with the highest number of included studies as the main outcome domain.

Table 1 Algorithm for categorizing the “main” outcome domain for each systematic review
Full size table

For each main outcome domain, we extracted the other four elements specified: specific measurement, specific metric, method of aggregation, and time-points. For the main outcome domain, we also extracted the numbers of studies that reported measuring it, reported any data, reported any meta-analyzable data, and were incorporated into ≥ 1 meta-analysis. We considered data for a given outcome from a given study to be “meta-analyzable” if the study reported adequate information so that it could be incorporated into a meta-analysis. For categorical outcomes, meta-analyzable meant that either of these conditions were met: (1) total number of participants and number of participants with the outcome were reported for each study arm; and (2) the between-group treatment effect (e.g., relative risk) and an uncertainty estimate (e.g., 95% confidence interval) were reported. For continuous and time-to-event outcomes, meta-analyzable meant that either of these conditions were met: (1) mean and uncertainty estimates were reported for each study arm; and (2) the between-group treatment effect (e.g., mean difference) and an uncertainty estimate were reported.

Results

Reviews examined

We identified 175 completed systematic reviews published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews (Table 2). The reviews were published between January 1, 2005 and August 11, 2017 (median = 2014). The most common populations were patients with retinal/choroidal disease (35 reviews; 20%) and visual impairment/low vision (33 reviews; 19%). The most common types of interventions/comparators were drugs (74 reviews; 42%) and surgeries (67 reviews; 38%).

Table 2 Characteristics of systematic reviews examined
Full size table

Incorporation of studies into meta-analyses for any outcome domain

The 175 included reviews examined a median of 6 total outcome domains, including a median of 1 primary outcome domain, 4 secondary outcome domains, and 1 other outcome domain.

The 175 reviews included a median of 3 studies (IQR 1–9); 125 reviews (71%) included ≥2 studies. For these 125 reviews, Fig. 2 plots the percentage of studies incorporated into a meta-analysis for any outcome domain (blue line) and for the main outcome domain (red bars). Among these reviews, 44/125 reviews (35%) incorporated every included study into ≥1 meta-analysis (for any outcome domain). Conversely, 33/125 reviews (26%) did not incorporate any study into any meta-analysis for any outcome, i.e., they did not conduct any meta-analysis. The remaining 48/125 reviews (38%) incorporated only a subset of their studies into ≥1 meta-analysis. These 48 reviews included a median of 12.5 studies (IQR 6–22), and the meta-analyses in these reviews incorporated a median of 6.5 studies (IQR 4–13).

Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 74% (IQR 0–100%). Among the 92 reviews that conducted a meta-analysis, the median proportion of studies incorporated into ≥1 meta-analysis for any outcome was 93% (IQR 64–100%).

Characteristics of main outcome domains

Almost all reviews (172/175 reviews; 98%) named ≥1 primary outcome domain (Table 1). Three in four reviews (131/175 reviews; 75%) each named exactly one primary outcome domain, which we categorized as their main outcome domain. The most frequent main outcome domains across the 175 reviews were visual acuity (31%) and intraocular pressure (6%) (Table 3). Thirty-eight outcome domains were main outcome domains in just one review each. The main outcome was categorical in 70% and continuous in 29% of reviews. Most main outcome domains (98%) were efficacy outcomes, i.e., not safety outcomes.

Table 3 Characteristics of main outcome domains in all 175 systematic reviews examined
Full size table

Incorporation of studies into meta-analyses for the main outcome domain

Among the 125 reviews including ≥2 studies, only 18 reviews (14%) incorporated all their studies into a meta-analysis for the main outcome domain. Conversely, 51/125 reviews (41%) did not incorporate any study into the meta-analysis for the main outcome domain, i.e., they did not conduct any meta-analysis for the main outcome domain. The remaining 56/125 reviews (45%) incorporated only a subset of their studies into the meta-analysis for the main outcome domain. These 56 reviews included a median of 12 studies each, and the meta-analyses for the main outcome domain in these reviews incorporated a median of 4 studies each.

Among the 125 reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median proportion of studies incorporated into ≥1 meta-analysis for the main outcome domain was 28% (IQR 0–71%). Among the 74 reviews that conducted meta-analyses for the main outcome domain, the median proportion of studies incorporated was 67% (IQR 39–91%).

Meta-analysis conduct for the main outcome domain

Figure 1 illustrates a cascading effect of loss of information as regards the main outcome domain in the 175 reviews. Thirty-five reviews (20%) included no studies, i.e., were empty reviews, and 15 (9%) included one study each (Fig. 1). Of the 125 reviews including ≥2 studies, i.e., those in which a meta-analysis could theoretically be done for the main outcome if ≥2 studies reported meta-analyzable data, only 74 reviews (59%) conducted a meta-analysis for the main outcome.

Fig. 1
figure 1

Conduct of meta-analyses for the main outcome domain

Full size image

Reasons for non-conduct of meta-analyses for the main outcome domain

Among the 125 reviews including ≥2 studies, 51 reviews (41%) did not conduct a meta-analysis for the main outcome domain. For 21/51 reviews (41%), fewer than two studies measured the review’s main outcome (Table 4). When ≥2 studies reported meta-analyzable data, there were numerous reasons why reviewers did not conduct a meta-analysis, most frequently due to inconsistency in outcome elements among the included studies. Specifically, data could not be meta-analyzed because the specific measurements used (16/51 reviews; 31%) and time-points examined (9/51 reviews; 18%) were inconsistent among studies.

Table 4 Reasons for non-conduct of a meta-analysis for the systematic review’s main outcome even when ≥2 studies were included in the systematic review (N = 51 of 125 reviews that included ≥2 studies)
Full size table

Figure 2 demonstrates that the loss of information for the main outcome domain (red bars) was similar in pattern to the loss of information when considering any outcome domain (blue line).

Fig. 2
figure 2

Percentage of studies included in the review that were incorporated into a meta-analysis for any outcome (blue line) and for the review’s main outcome (red bars)

Notes: This Figure excludes the 50 systematic reviews in whom a meta-analysis was not possible: 35 systematic reviews that each included 0 studies (i.e., “empty reviews”) and 15 systematic reviews included that each included only 1 study. When the blue line is non-0 but the red bars are 0, it implies that the systematic review did not conduct a meta-analysis for the main outcome, but did so for ≥1 of the remaining outcomes

Full size image

Discussion

Through a case study of all Cochrane reviews in the field of eyes and vision, the current work demonstrates three major areas that need improvement.

First, primary studies addressing similar research questions should align their outcomes better. Studies often could not be incorporated into meta-analyses because the outcomes were not aligned, either because the domains or ≥1 of the other four outcome elements did not overlap. Among the reviews including ≥2 studies, only 59 and 74% could conduct a meta-analysis for the main outcome and for any outcome, respectively. In other words, even when reviews included ≥2 studies, 41 and 26% of reviews missed opportunities to conduct a meta-analysis to succinctly convey information regarding the main outcome and any outcome, respectively.

Second, reviews and primary studies should align their outcomes better. When looking at reviews that could have conducted a meta-analysis, i.e., those including ≥2 studies, the median percentages of included studies incorporated into the meta-analysis for the main outcome and for any outcome were 28 and 74%, respectively. This suggests that, approximately 7 in 10 studies that reviewers include are not incorporated into the meta-analysis for the main outcome, and 1 in 4 studies are not incorporated into the meta-analysis for any outcome. In previous work, we demonstrated poor overlap between outcomes in clinical trials and reviews, and possible differences in the types of outcomes they examine [10, 11]. For HIV/AIDS, we demonstrated that reviewers examined more long-term clinical outcomes and patient-centered outcomes than did clinical trialists. Such differences may arise because: (1) reviews may more directly inform clinical practice guidelines, and (2) reviewers may be less affected by common constraints faced by clinical trialists, e.g., costs and sample size [10].

Our findings beg the question of who should prioritize outcomes for measurement and reporting in research. It has aptly been stated that achieving consensus in outcome use across research “cannot be left to serendipity.” [16] One deliberate and fundamental aspect of the solution to the problem of outcome inconsistency is the development of “core outcome sets.” A core outcome set is a minimum set of outcomes that should be measured and reported in all clinical trials addressing a given condition [17]. Core outcome sets are increasingly common in various health fields; a 2018 systematic review identified 307 core outcomes sets [18]. However, outcome inconsistency remains widespread; 40% of recent (2019) published Cochrane reviews explicitly noted this problem [19].

We [10, 11] and others [20] have argued that, as stakeholders in a given field, systematic reviewers should both participate in the development of and adopt core outcome sets for that field. By broadening the participation in outcome prioritization efforts, this could potentially help ensure that the outcomes that are measured and reported in research are widely relevant and important. Two aspects of core outcome sets are worthy of clarification. First, core outcome sets do not stifle innovation; they are simply meant to represent a minimum set of outcomes that should be reported. Once a core outcome set exists for a given topic, clinical trialists working in that topic area should explicitly specify the intention to measure and report the outcomes in the set. Second, core outcome sets are not static; they can and should be updated as the field advances and new knowledge emerges.

The third major area in need of improvement that our study demonstrates is the reporting of outcomes in primary studies. Results data from primary studies were often not meta-analyzable even when outcomes might have been aligned. In addition, outcome domains were frequently not reported in primary studies or ≥1 of the outcome elements were frequently missing or inadequately reported (e.g., “worsening of disease” without clarification of how “worsening” was defined). It is possible that the studies measured these outcomes, but did not report measuring them or reported them inadequately. If such selective reporting, either non-reporting or inadequate reporting, of outcomes in the included studies occurred as a function of the direction of the outcome’s results, it would be suggestive of outcome reporting bias [21]. In this case study, we relied on the reviewers’ reporting of the extent to which the primary studies reported the outcomes. Because we did not examine the reports of the primary studies (or their protocols), we are unable to comment definitively on whether non-reporting of the outcomes indicates outcome reporting bias. However, outcome reporting bias in primary studies has been documented to be a widespread problem across reviews [22,23,24,25,26], and, as such, is a likely explanation for some outcomes not being reported.

Implications

For the evidence-based medicine paradigm to work, decision-makers must be able to rely on systematic reviews, which in turn rely on the results of primary studies. For results of primary studies to be actionable, there (1) needs to be alignment in outcomes considered important to both primary study researchers and reviewers, and (2) those outcomes need to be reported completely. Important discussions need to be had regarding who should choose outcomes for the field and how such choices should be made. We, in conjunction with others, suggest that these discussions should include, at the least, clinicians, patients, clinical trialists, systematic reviewers, regulators, and other decision-makers [27].

We have demonstrated that the choice of outcomes for systematic reviews may have led to loss of information through non-incorporation of results from included studies into meta-analyses. The most substantial drops in the percentage of reviews conducting meta-analyses for the main outcome domain appeared to be due to inadequate numbers of studies reporting the outcome and, when there were adequate numbers of studies for a meta-analysis (i.e., ≥2 studies), differences in the specific measurements and time-points used.

Our findings also demonstrate that even when focusing on reviews that conducted meta-analyses for their main outcome domain, only about 2 in 3 studies were incorporated into those meta-analyses. As such, non-incorporation of included studies into meta-analyses represents two main problems. First, it represents missed opportunities for using research to inform decision-making through evidence synthesis. This contributes considerably towards research waste [28,29,30]. Second, non-incorporation of included studies into meta-analyses represents a failed obligation on the part of the researchers (both trialists and reviewers) [31]. As a community of researchers, both parties have a solemn obligation to research participants to ensure that their participation will lead to a useful contribution to science; failing to agree upon outcomes that should be collected and adequately reported likely violates this obligation.

Other solutions

Core outcome sets are integral to solving the problems this study illustrates. Other parts of the solution are worth discussing. We agree with existing recommendations against studies being excluded from systematic reviews solely on the basis of the lack of relevant outcome data [3]. Thankfully, such recommendations have been associated with a reduction in the number of reviews excluding studies solely on the basis of outcome data [32]. As the current study demonstrates, the review team’s choice of outcomes may not align with that of the primary studies. This may be particularly true for eyes and vision, a field with few core outcome sets [4, 18]. We also encourage reviewers to report an outcome matrix [23, 24], a transparent and simple way to indicate all fully-reported, partially-reported, or non-reported outcomes in each included study.

Large numbers of empty reviews and reviews including only one study

Twenty-percent of the reviews we examined were empty and 9% included only one study each. While such reviews are useful in driving primary research, the possible reasons for the paucity of studies in them are worth exploring. One possibility is that these represent topics that primary researchers have not yet studied. Another is that only observational studies addressing these topics may exist; Cochrane reviews typically include only randomized trials. It also is possible that these topics reflect the priorities of Cochrane Eyes and Vision and the authors of these reviews, rather than of the field at-large.

Limitations

Our study has certain limitations. First, we focused on Cochrane reviews within one field. Loss of information due to the choice of review outcomes could be a bigger, similar, or smaller problem in non-Cochrane systematic reviews in eyes and vision or systematic reviews in other fields. Second, we analyzed in-depth the extent of incorporation of included studies into meta-analyses only for the main outcome domain. Meta-analyses of other primary, secondary, and other outcome domains may have incorporated higher percentages of included studies. However, Fig. 2 suggests that this is likely not the case. It is possible that our algorithm for categorizing the “main” outcome for each review could have impacted our findings. But, in reviews where more than one outcome domain could have served as the main outcome, we categorized as the main outcome the outcome that the highest number of included studies had reported. Our results thus represent the best-case scenario. Third, most outcome domains (98%) were efficacy outcomes. Selective outcome reporting has also been reported to be a problem for safety outcomes [33]. Fourth, we relied on the reviews to determine whether or not each included study did the following for the main outcome domain: reported measuring it, reported any results for it, and reported meta-analyzable data for it. Related to this, we did not examine the appropriateness or feasibility of the reviewers’ being able to conduct meta-analyses when the included studies reported data in a format different from what the reviewers were interested. As such, our results document what was actually done in the reviews.

Conclusions

This case study of all Cochrane systematic reviews addressing an entire field (eyes and vision) demonstrates that only 59 and 74% of the reviews including ≥2 studies could conduct a meta-analysis for the main outcome and for any outcome, respectively. In evidence-based healthcare, such loss of information represents missed opportunities and a failed obligation by researchers to research participants to ensure that their participation will lead to a useful contribution to science. Core outcome sets and improved outcome reporting can help solve some of these problems.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

IQR:

Interquartile range

NSAID:

Non-steroidal anti-inflammatory drug

SRDR:

Systematic Review Data Repository

References

  1. Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011.

  2. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111.

    Article  Google Scholar 

  3. Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. Version 1.02 ed. London: Cochrane; 2016.

    Google Scholar 

  4. Saldanha IJ, Le JT, Solomon SD, Repka MX, Akpek EK, Li T. Choosing Core outcomes for use in clinical trials in ophthalmology: perspectives from three ophthalmology outcomes working groups. Ophthalmology. 2019;126(1):6–9.

    Article  Google Scholar 

  5. Meinert CL. Clinical trials dictionary: terminology and usage recommendation. 2nd ed Hoboken, NJ: Wiley; 2012.

  6. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester: Wiley; 2019.

    Google Scholar 

  7. Mayo-Wilson E, Li T, Fusco N, et al. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol. 2017;91:95–110.

    Article  Google Scholar 

  8. Mayo-Wilson E, Li T, Fusco N, Dickersin K. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Res Synth Methods. 2018;9(1):2–12.

    Article  Google Scholar 

  9. Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400.

    Article  Google Scholar 

  10. Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;84:85–94.

    Article  Google Scholar 

  11. Saldanha IJ, Lindsley K, Do DV, et al. Comparison of clinical trial and systematic review outcomes for the 4 Most prevalent eye diseases. JAMA Ophthalmology. 2017;135(9):933–40.

    Article  Google Scholar 

  12. Zarin DA, Tse T, Williams RJ, Califf RM, ide NC. The ClinicalTrials.gov results database--update and key issues. N Engl J Med. 2011;364(9):852–60.

    Article  CAS  Google Scholar 

  13. Juthani VV, Clearfield E, Chuck RS. Non-steroidal anti-inflammatory drugs versus corticosteroids for controlling inflammation after uncomplicated cataract surgery. Cochrane Database Syst Rev. 2017;7:Cd010516.

    PubMed  Google Scholar 

  14. Ip S, Hadar N, Keefe S, et al. A web-based archive of systematic review data. Syst Rev. 2012;1:15.

    Article  Google Scholar 

  15. Li T, Vedula SS, Hadar N, Parkin C, Lau J, Dickersin K. Innovations in data collection, management, and archiving for systematic reviews. Ann Intern Med. 2015;162(4):287–94.

    Article  Google Scholar 

  16. Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8:39.

    Article  Google Scholar 

  17. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.

    Article  Google Scholar 

  18. Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. PLoS One. 2018;13(12):e0209869.

    Article  CAS  Google Scholar 

  19. Williamson PR, Oliveira RD, Clarke M, et al. The relevance and uptake of core outcome sets in Cochrane systematic reviews Submitted.

  20. Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5:11.

    Article  Google Scholar 

  21. Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–40.

    Article  Google Scholar 

  22. Dwan K, Kirkham JJ, Williamson PR, Gamble C. Selective reporting of outcomes in randomised controlled trials in systematic reviews of cystic fibrosis. BMJ Open. 2013;3(6).

    Article  Google Scholar 

  23. Kirkham JJ, Altman DG, Chan AW, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ. 2018;362:k3802.

    Article  Google Scholar 

  24. Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.

    Article  Google Scholar 

  25. Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179–95.

    Article  Google Scholar 

  26. Page MJ, McKenzie JE, Kirkham J, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;(10):Mr000035.

  27. Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280.

    Article  Google Scholar 

  28. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet (London, England). 2014;383(9912):156–65.

    Article  Google Scholar 

  29. Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014;9(7):e102670.

    Article  Google Scholar 

  30. Macleod MR, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet (London, England). 2014;383(9912):101–4.

    Article  Google Scholar 

  31. Law A, Lindsley K, Rouse B, Wormald R, Dickersin K, Li T. Missed opportunity from randomised controlled trials of medical interventions for open-angle glaucoma. Br J Ophthalmol. 2017;101(10):1315–7.

    Article  Google Scholar 

  32. Dwan KM, Williamson PR, Kirkham JJ. Do systematic reviews still exclude studies with “no relevant outcome data”? BMJ. 2017;358:j3919.

    Article  Google Scholar 

  33. Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501.

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge Elizabeth Clearfield and Dr. Roy Chuck for their contributions to the concepts presented in this work.

Funding

The National Institutes of Health (Grant Number UG EY020522) funded this work. The funder played no role in the design of the study; collection, analysis, and interpretation of the data, or writing of the manuscript.

Author information

Authors and Affiliations

  1. Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice (Primary), Department of Epidemiology (Secondary), Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI, 02903, USA

    Ian J. Saldanha

  2. Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Room Str. 6.127, Utrecht, GA, 3508, Netherlands

    Kristina B. Lindsley

  3. ISA Group, 201 North Union Street, Suite 300, Alexandria, VA, 22314, USA

    Sarah Money

  4. Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI, 02903, USA

    Hannah J. Kimmel

  5. Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI, 02903, USA

    Bryant T. Smith

  6. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Baltimore, MD, 21205, USA

    Kay Dickersin

Authors
  1. Ian J. Saldanha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kristina B. Lindsley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sarah Money
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hannah J. Kimmel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bryant T. Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kay Dickersin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

IJS, KL, and KD conceptualized and designed the study. IJS and KD analyzed and interpreted the data. IJS drafted the manuscript. All authors provided critical feedback to the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ian J. Saldanha.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

Drs. Saldanha and Dickersin and Ms. Lindsley and Money declare affiliation with Cochrane Eyes and Vision during conduct of the work related to this manuscript. Ms. Kimmel and Mr. Smith were research assistants on this study. All of the reviews examined in this manuscript were produced by Cochrane Eyes and Vision. No other disclosures are reported.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saldanha, I.J., Lindsley, K.B., Money, S. et al. Outcome choice and definition in systematic reviews leads to few eligible studies included in meta-analyses: a case study. BMC Med Res Methodol 20, 30 (2020). https://doi.org/10.1186/s12874-020-0898-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-020-0898-2

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us