Skip to content

Advertisement

Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review

  • Timor Faber1, 2,
  • Philippe Ravaud1, 2, 3, 4, 5,
  • Carolina Riveros1, 2,
  • Elodie Perrodeau1, 2 and
  • Agnes Dechartres1, 2, 3, 4Email author
BMC Medical Research MethodologyBMC series – open, inclusive and trusted201616:35

https://doi.org/10.1186/s12874-016-0136-0

Received: 14 October 2015

Accepted: 15 March 2016

Published: 22 March 2016

Abstract

Background

There is an increasing number of meta-analyses including data from non-randomized studies for therapeutic evaluation. We aimed to systematically assess the methods used in meta-analyses including non-randomized studies evaluating therapeutic interventions.

Methods

For this methodological review, we searched MEDLINE via PubMed, from January 1, 2013 to December 31, 2013 for meta-analyses including at least one non-randomized study evaluating therapeutic interventions. Etiological assessments and meta-analyses with no comparison group were excluded. Two reviewers independently assessed the general characteristics and key methodological components of the systematic review process and meta-analysis methods.

Results

One hundred eighty eight meta-analyses were selected: 119 included both randomized controlled trials (RCTs) and non-randomized studies of interventions (NRSI) and 69 only NRSI. Half of the meta-analyses (n = 92, 49 %) evaluated non-pharmacological interventions. “Grey literature” was searched for 72 meta-analyses (38 %). An assessment of methodological quality or risk of bias was reported in 135 meta-analyses (72 %) but this assessment considered the risk of confounding bias in only 33 meta-analyses (18 %). In 130 meta-analyses (69 %), the design of each NRSI was not clearly specified. In 131 (70 %), whether crude or adjusted estimates of treatment effect for NRSI were combined was unclear or not reported. Heterogeneity across studies was assessed in 182 meta-analyses (97 %) and further explored in 157 (84 %). Reporting bias was assessed in 127 (68 %).

Conclusions

Some key methodological components of the systematic review process—search for grey literature, description of the type of NRSI included, assessment of risk of confounding bias and reporting of whether crude or adjusted estimates were combined—are not adequately carried out or reported in meta-analyses including NRSI.

Keywords

Meta-analysesTherapeutic evaluationNon-randomized studiesReporting

Background

Randomized controlled trials (RCTs) are considered the gold standard for evidence-based medicine because they are designed to minimize the risk of bias [1]. However, the applicability of their results has been criticized because of restrictive selection criteria, with, commonly, exclusion of older adults and people with co-morbidities or severe disease [24]. Also, conducting an RCT is sometimes impossible or inappropriate (eg, when studying rare or long-term events) [1, 3, 5], which results in critical information gaps.

In contrast, observational studies, the overarching term for all non-experimental non-randomized studies (including cohort, case–control, and cross-sectional studies) [6], generally are more likely to reflect clinical practice in real life because of their broader range of participants, longer follow-up time, and lower costs than RCTs [710]. With the aim of generating evidence that will guide healthcare decisions, the field of comparative effectiveness research (CER) emphasizes the need to incorporate data from observational studies to complement RCTs [8, 1116]. A comprehensive assessment in 2009 indicated that 54 % of CER studies had an observational study design [17]. Therefore, an increasing number of systematic reviews and meta-analyses are including data from non-randomized studies to assess therapeutic interventions.

Similar to systematic reviews of RCTs, reviews including non-randomized studies are expected to follow the general recommendations for good conduct, such as retrieving all relevant studies and assessing their risk of bias. However, some elements should be adapted specifically to the inclusion of non-randomized studies because their study designs inherently differ from RCTs [7, 9, 14, 1823]. Lacking randomization, they are likely subject to confounding bias, which results in an imbalance in prognostic factors associated with the outcome of interest that may severely compromise the validity of their results [24].

Previous methodological reviews evaluating systematic reviews including observational studies exist [2528]. However, these studies have a different objective. One assessed the main characteristics of all systematic reviews indexed in Medline on November 2004 whatever the type of included studies (ie, therapeutic, epidemiological, prognostic or diagnostic studies) [27]. Two others focused on the methods and reporting of harms in systematic reviews of adverse events [26, 28]. The last one was in the field of psychiatry and did not concern therapeutic evaluation but assessment of prevalence or association [25]. Further, none of these previous reviews has evaluated the specific methodological problems raised by the inclusion of non-randomized studies.

In this study, we performed a methodological review of meta-analyses including non-randomized studies of interventions (NRSI) to evaluate key methodological components common to all meta-analyses and those specifically related to the inclusion of non-randomized studies.

Methods

Study design

This is a methodological review of meta-analyses including NRSI for therapeutic evaluation. For clarity and consistency, we refer to this article as a “methodological review”, the systematic reviews with meta-analyses included in this methodological review as “meta-analyses”, and the studies included in these meta-analyses as “studies”.

Search strategy

Our goal was not to create an exhaustive list of all meta-analyses that include NRSI but rather to identify a relatively representative sample of recently published meta-analyses that a health professional would most likely encounter when searching for meta-analyses. We therefore searched MEDLINE via PubMed because of its wide use among health professionals, combining keywords and MeSH terms for NRSI, systematic reviews, and meta-analyses (Appendix 1). The search was conducted on January 7, 2014 and restricted to the year 2013.

Eligibility criteria

To be eligible, a meta-analysis had to examine a therapeutic or preventive intervention (such as vaccines) for efficacy or safety, include data from at least one NRSI, and be published in 2013. We excluded meta-analyses that included studies without a comparison group and meta-analyses of etiological assessment. When it was difficult to distinguish an etiological from a therapeutic evaluation, we agreed to include the former if the authors considered the inclusion of RCTs in their meta-analysis. To illustrate: a meta-analysis that investigated the association of the use of statins and risk of cancer would was considered a therapeutic evaluation if the authors planned to include RCTs. Individual patient data meta-analyses were also excluded, as were non-randomized studies that conducted a meta-analysis of the literature as secondary analysis. Finally, we did not include meta-analyses published in a language other than English or those for which the full text was not available.

Selection of relevant meta-analyses

The selection of relevant meta-analyses was conducted in 2 steps. In the first step, one reviewer (CR) excluded clearly irrelevant studies based on the title, abstract, and full text, then, a second reviewer (TF) performed the final selection, discussing all doubtful cases with a third reviewer (AD).

Data extraction

The data extraction form for this methodological review was developed from the MOOSE statement for reporting meta-analyses that include observational studies [29], the PRISMA statement for reporting systematic reviews and meta-analyses of studies evaluating healthcare interventions [30, 31], and the AMSTAR measurement tool for assessing the methodological quality of systematic reviews [32]. The data extraction form was tested by one reviewer (TF) with 10 studies before data extraction commenced.

Two reviewers (TF, CR) independently extracted all data in duplicate, resolving discrepancies with a third reviewer (AD) if necessary. The following characteristics were extracted from the full text and online appendix of each meta-analysis:
  • General characteristics: We collected whether the journal was a specialty or a general journal, the location of the corresponding author, and the medical area. We verified whether the meta-analysis was registered on the international prospective register of systematic reviews by the University of York’s Centre for Reviews and Dissemination (PROSPERO). We collected whether epidemiologists or statisticians were involved, relying on the definition given by Delgado-Rodriguez et al.[33] and assessed whether the authors reported the funding sources and declared conflict of interests. We assessed whether the meta-analyses evaluated a pharmacological or non-pharmacological intervention. Non-pharmacological interventions were classified as surgical procedures or other interventions. We also assessed the type of studies included: only NRSI or both NRSI and RCTs.

  • Systematic review methods:
    • ▪Search strategy: We collected how many and which electronic databases were searched, and whether the search strategy for at least one database was provided. We collected whether reference lists and journals were hand-searched and whether the authors searched for grey literature, and if yes, how: search of registries (eg, ClinicalTrials.gov), conference abstracts, or contacting experts. We assessed whether the authors restrict their searches by language.

    • ▪Study selection and data extraction process: We assessed whether study selection and data extraction were conducted in duplicate.

    • ▪Contact of the study authors: We noted whether it was mentioned that study authors were contacted for clarification or additional results.

    • ▪Methodological quality/risk of bias assessment: We assessed whether methodological quality or risk of bias assessment was conducted, what tools were used, and whether the assessment was conducted in duplicate.

  • Meta-analysis methods:
    • ▪Studies combined: We assessed the types of NRSI included. NRSI were categorized as concurrent (prospective) cohort, nonconcurrent (retrospective) cohort, case–control, or historically controlled studies according to the definition by Ioannidis et al. [21]. We also assessed whether the authors combined the results from NRSI and RCTs and whether they combined results from different types of NRSI (eg, cohort and case–control studies).

    • ▪Meta-analysis model: We collected whether the authors used crude or adjusted estimates for NRSI and whether they used fixed- or random-effects models to pool the data. For adjusted estimates, we also collected whether the confounding factors taken into account were listed.

    • ▪Assessment and exploration of heterogeneity: We collected whether and how the authors assessed heterogeneity and whether they conducted meta-regression, subgroup, or sensitivity analyses to explore heterogeneity.

    • ▪Assessment of reporting bias: We collected information on whether the authors assessed reporting bias, and how.

Statistical analysis

The analysis of the data consisted of descriptive statistics, providing numbers and percentages for qualitative variables and median (minimum, maximum, or interquartile range) for quantitative variables. The results were stratified for meta-analyses including only NRSI and those including both NRSI and RCTs. We did not assess statistical differences between these strata. Statistical analysis involved use of R 3.0.2. (R Core Team [2013]. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL: http://www.R-project.org/).

Results

Study selection

Our MEDLINE search identified 3602 citations; Among the 341 potentially relevant meta-analyses, 188 were eligible for this review (Fig. 1). Complete references for the included meta-analyses and meta-analyses excluded are in Appendixes 2 and 3, respectively.
Figure 1
Fig. 1

Study selection flowchart

General characteristics (Table 1)

Table 1

General characteristics of therapeutic meta-analyses published in 2013 and including non-randomized studies of intervention (n = 188)

Characteristic

Included NRSI only

Included NRSI&RCTs

All meta-analyses

(n = 69)

(n = 119)

(n = 188)

No (%)

No (%)

No (%)

Journal type

   

 Specialty

59 (86)

99 (83)

158 (84)

 General

10 (14)

20 (17)

30 (16)

Location of corresponding author

   

 Asia

26 (38)

38 (32)

64 (34)

 Europe

22 (32)

39 (33)

61 (32)

 North-America

18 (26)

33 (28)

51 (27)

 Other

3 (4)

9 (7)

12 (7)

Medical area

   

 Surgery

21 (30)

28 (24)

49 (26)

 Cardiology

8 (12)

25 (21)

33 (18)

 Oncology

6 (9)

19 (16)

25 (13)

 Other

34 (49)

47 (39)

81 (43)

Type of intervention

   

 Pharmacological

36 (52)

56 (47)

92 (49)

 Non-pharmacological

32 (46)

60 (50)

92 (49)

   Surgical procedures

30 (43)

44 (37)

74 (39)

   Other

2 (3)

16 (13)

18 (10)

 Both pharmacological and non-pharmacological

1 (1)

3 (3)

4 (2)

Epidemiologists or statisticians involved

17 (25)

19 (16)

36 (19)

Declaration of conflict of interests

64 (93)

102 (86)

166 (88)

 Potential conflict of interests

9 (13)

17 (14)

26 (14)

Funding source

   

 Public

30 (43)

36 (30)

66 (35)

 No specific funding

10 (14)

29 (24)

39 (21)

 Private

2 (3)

7 (6)

9 (5)

 Both public and private

3 (4)

2 (2)

5 (3)

 Not reported/unclear

24 (35)

45 (38)

69 (37)

Registration on PROSPERO

0

3 (3)

3 (2)

NRSI non-randomized studies of intervention; RCTs randomized controlled trials

Among the 188 included meta-analyses, 49 (26 %) were of surgery, 33 (18 %) cardiology, and 25 (13 %) oncology. Half of the meta-analyses assessed non-pharmacological interventions (n = 92, 49 %); 74 involved surgical procedures. Approximately one third (n = 69, 37 %) included only NRSI, and two thirds included both NRSI and RCTs (n = 119, 63 %).

In total, 36 meta-analyses (19 %) involved epidemiologists or statisticians. Conflict of interest was declared in 166 (88 %), with 26 reporting a potential conflict of interest. About one-third of the meta-analyses did not report a source of funding (n = 69, 37 %).

Systematic review methods (Table 2)

Table 2

Systematic review methods of therapeutic meta-analyses published in 2013 and including non-randomized studies (n = 188)

Characteristic

Included NRSI only

Included NRSI&RCTs

All Meta-analyses

(n = 69)

(n = 119)

(n = 188)

No (%)

No (%)

No (%)

Search strategy

   

No. of electronic databases searched, median [IQR]

3 [24]

3 [3, 4]

3 [3, 4]

  > 2 databases searched

51 (74)

96 (81)

147 (78)

Electronic databases searcheda

   

 Medline/PubMed/Ovid

69 (100)

118 (99)

187 (99)

 EMBASE

52 (75)

97 (82)

149 (79)

 Cochrane

45 (65)

81 (68)

126 (67)

 Web of science/web of knowledge

14 (20)

22 (18)

36 (19)

Time-frame of search strategy reported

62 (90)

109 (92)

171 (91)

Time end of search-publication date (months), median [min-max]

10 [0–24]

9 [1–51]

10 [0–51]

Search strategy provided for at least one database

18 (26)

44 (37)

62 (33)

Searched reference lists for relevant studies

57 (83)

105 (88)

162 (86)

Hand searched journals

5 (7)

7 (6)

12 (6)

Searched for grey literaturea

23 (33)

49 (41)

72 (38)

 Conference abstracts

14 (20)

27 (23)

41 (22)

 Registries (eg, ClinicalTrials.gov)

8 (12)

25 (21)

33 (18)

 Experts

6 (9)

9 (8)

15 (8)

No restriction by language reported

20 (29)

62 (52)

82 (44)

Study selection

   

Duplicate study selection

47 (68)

84 (71)

131 (70)

Methodological quality/risk of bias

   

Assessment of MQ/RoB

47 (68)

88 (74)

135 (72)

Tools used for RCTsa

   

 Cochrane RoB Tool [41]

NA

42 (35)

NA

 Jadad scale [42]

NA

22 (18)

NA

 GRADE [43]

NA

9 (8)

NA

 Other

NA

21 (18)

NA

 Authors made their own tool

NA

3 (3)

NA

 Unclear

NA

7 (6)

NA

Tools used for NRSIa

   

 Newcastle-Ottawa scale [44]

25 (36)

43 (36)

68 (36)

 GRADE [43]

5 (7)

8 (7)

13 (7)

 Cochrane RoB Tool [41]

3 (4)

7 (6)

10 (5)

 Other tools

13 (19)

24 (20)

37 (20)

 Authors made their own tool

6 (9)

6 (5)

12 (6)

 Unclear

3 (4)

9 (8)

12 (6)

Confounding taken into account in risk of bias

13 (19)

20 (17)

33 (18)

Same tool used for RCTs and NRSI

NA

27 (23)

NA

Duplicate MQ/RoB assessment

30/47 (64)

57/88 (65)

87/135 (64)

Data extraction

   

 Duplicate data extraction

52 (75)

86 (72)

138 (73)

 Contact authors for clarification or additional results

18 (26)

43 (36)

61 (32)

Data are no. (%), unless indicated. IQR interquartile range; MQ methodological quality; NRSI non-randomized studies of interventions; RCTs randomized controlled trials; Cochrane RoB Cochrane risk of bias tool; NA Not applicable

aMore than one item could have been reported for each meta-analysis

Literature search

Overall, all but one of the meta-analyses reported the search of at least 1 electronic database and 147 (78 %) reported the search of > 2 electronic databases. One third provided the search strategy for each database (n = 62, 33 %). MEDLINE, Embase, and the Cochrane Library were most frequently searched (187 [99 %], 149 [79 %], and 126 (67 %) meta-analyses, respectively). In addition to the search of electronic databases, 162 meta-analyses (86 %) reported screening the reference lists of included studies, and 12 (6 %) reported hand-searching journals. About one-third of the meta-analyses (n = 72, 38 %) reported searching for grey literature: 41 (22 %) conference abstracts, 33 (18 %) registries, and 15 (8 %) contacted experts. For 82 meta-analyses (44 %), the authors reported that they did not restrict their searches by language.

Methodological quality/risk of bias assessment

Methodological quality or risk of bias of included studies was assessed in 135 (72 %) meta-analyses.

For the 119 meta-analyses including RCTs and NRSI, risk of bias was assessed in 88 (74 %) with 4 assessing risk of bias for RCTs only. RCTs were assessed with the Cochrane Risk of Bias tool in 42 (35 %) meta-analyses. The assessment of risk of bias involved the same tool for both RCTs and NRSI in 27 (23 %) meta-analyses. For the assessment of NRSI, a variety of tools were used. The most frequently used tool was the Newcastle-Ottawa Scale (n = 68). GRADE and the Cochrane Collaboration Risk of Bias Tool were used in 13 and 10 meta-analyses, respectively. In 37 meta-analyses, authors used other tools; in 12, they developed their own tools; and in 12, they were unclear about the methods used for assessing methodological quality/risk of bias. Overall, the authors have considered the risk of confounding bias in their risk of bias assessment in 33 meta-analyses (18 %). Of the 135 meta-analyses with an assessment of risk of bias, 87 (64 %) reported having performed it in duplicate.

Meta-analysis methods (Table 3)

Studies combined

For 130 meta-analyses (69 %), the authors did not clearly report the design for each individual study. Among the meta-analyses that included both NRSI and RCTs (n = 119), for 88 (74 %), the results of NRSI and RCTs were combined.
Table 3

Meta-analysis methods of therapeutic meta-analyses published in 2013 and including non-randomized studies (n = 188)

Characteristic

Included NRSI only

Included NRSI&RCTs

All meta-analyses

(n = 69)

(n = 119)

(n = 188)

No (%)

No (%)

No (%)

Studies combined

   

Type of studies included

   

 Only prospective cohort studies

2 (3)

3 (3)

5 (3)

 Only cohort studies

18 (26)

34 (29)

52 (28)

 Including also case–control studies

18 (26)

28 (23)

46 (24)

 Including all types of NRSI

5 (7)

18 (15)

23 (12)

 Other

28 (41)

39 (33)

67 (36)

 “Observational studies”

6 (9)

22 (18)

28 (15)

 “Prospective and retrospective studies”

11 (16)

12 (10)

23 (12)

 “Retrospective studies”

11 (16)

5 (4)

16 (9)

Did not clearly report design for each study

43 (62)

87 (73)

130 (69)

Meta-analysis combining results of NRSI and RCTs

NA

88 (74)

NA

 Results also presented separately

NA

44/88 (50)

NA

Meta-analysis combining studies of different designs

39 (57)

61 (51)

100 (53)

Meta-analysis model

   

Network meta-analysis

0

2 (2)

2 (1)

Crude or adjusted estimates used

   

 Combined crude and adjusted estimates

10 (14)

12 (10)

22 (12)

 Adjusted

8 (12)

13 (11)

21 (11)

 Both crude and adjusted estimates separately

4 (6)

4 (3)

8 (4)

 Crude

2 (3)

4 (3)

6 (3)

 Not reported/unclear

45 (65)

86 (72)

131 (70)

If adjusted, list of confounding factors

15/22 (68)

19/29 (66)

34/51 (67)

Type of model

   

 Random-effects model

39 (57)

56 (47)

95 (50)

 Fixed-effects model, unless high heterogeneity, then random-effects model

19 (28)

33 (28)

52 (28)

 Both fixed- and random-effects models

5 (7)

21 (18)

26 (14)

 Fixed-effects model

3 (4)

6 (5)

9 (5)

 Not reported/unclear

3 (4)

3 (3)

6 (3)

Heterogeneity assessment & exploration

   

Heterogeneity assesseda

66 (96)

116 (97)

182 (97)

 I2

57 (83)

107 (90)

164 (87)

 Cochran Q χ2 test

41 (59)

74 (62)

115 (61)

 τ2

4 (6)

2 (2)

6 (3)

Heterogeneity exploreda

54 (82)

103 (87)

157 (84)

 Meta-regression analysis

13 (19)

21 (18)

34 (18)

 Subgroup analysis

44 (64)

82 (69)

126 (67)

 Sensitivity analysis

35 (51)

74 (62)

109 (58)

Reporting bias assessmenta

49 (71)

78 (66)

127 (68)

 Standard funnel plot

41 (59)

70 (59)

111 (59)

 Egger’s test

32 (46)

36 (30)

68 (36)

 Begg’s test

18 (26)

24 (20)

42 (22)

Among MAs with ≥10 studies, reporting bias assessed

24/29 (83)

58/76 (76)

82/105 (78)

MAs meta-analyses; NRSI non-randomized studies of intervention; RCTs randomized controlled trials

aMore than one item could have been reported for each meta-analysis

Concerning NRSI combined, 52 meta-analyses (28 %) included only cohort studies and 5 only prospective cohort studies; 46 meta-analyses (24 %) combined cohort and case–control studies, and 23 (12 %) included all types of NRSI. The other 67 meta-analyses (36 %) included “observational studies” (without further details) (n = 28, 15 %), “prospective and retrospective studies” (n = 23, 12 %), and only “retrospective studies” (n = 16, 9 %).

Crude or adjusted estimates used for NRSI

For 131 meta-analyses (70 %), whether crude or adjusted estimates of treatment effect from the NRSI were used for the meta-analysis was unclear or not reported. For the remaining meta-analyses, the authors reported combining crude and adjusted estimates for 22 (12 %), only adjusted estimates for 21 (11 %), and only crude estimates for 6 (3 %). For 8 meta-analyses (4 %), the authors extracted both the crude and adjusted estimates and used them separately in 2 meta-analyses. Among the 51 meta-analyses involving adjusted estimates, 17 (33 %) did not report the confounding factors adjusted for.

Meta-analysis model

A random-effects model was used for half of the meta-analyses (n = 95). For 52 (28 %), a fixed-effects model was used primarily but then replaced with a random-effects model if high heterogeneity was observed in the model. For 26 meta-analyses (14 %), the authors used both fixed- and random-effects models, and for 9 (5 %), a fixed-effects model. The type of model was not reported or was unclear for 6 meta-analyses (3 %). We found 2 network meta-analyses (1 %).

Heterogeneity assessment

Almost all meta-analyses assessed heterogeneity (n = 182, 97 %). The I2 statistic was used in 164 meta-analyses (87 %), Cochran Q χ2 test in 115 (61 %), and between-study variance τ2 in 6 (3 %). Heterogeneity was explored in 157 meta-analyses (84 %) by subgroup analyses (n = 126, 67 %), sensitivity analyses (n = 109, 58 %) and meta-regression analyses (n = 34, 18 %).

For 44 of 88 (50 %) meta-analyses combining results from RCTs and NRSI, a subgroup or sensitivity analysis was based on the type of study (RCT vs NRSI). For 28 meta-analyses (15 %), subgroup or sensitivity analyses were based on the type of NRSI included.

Reporting bias assessment

Reporting bias was assessed in 127 meta-analyses (68 %) by standard funnel plots (n = 111, 59 %), Egger’s test (n = 68, 36 %), or Begg’s test (n = 42, 22 %). Overall, 82 of the 105 meta-analyses (78 %) including 10 or more studies reported having assessed reporting bias.

Discussion

We systematically assessed key methodological components of a large sample of therapeutic meta-analyses including NRSI in a variety of medical areas. Our results highlight some important methodological shortcomings. Only 38 % of the meta-analyses reported having searched for grey literature. Specific points related to the inclusion of NRSI raise concerns, with 69 % of the meta-analyses not reporting the study design of the included NRSI, and 70 % not reporting whether crude or adjusted estimates were combined.

Strengths and limitations of study

To the best of our knowledge, no previous study has comprehensively assessed both key methodological components common to all systematic reviews and elements specific to the inclusion of non-randomized studies. Other studies that previously evaluated methods or reporting of systematic reviews including NRSI concentrated on the reporting of harms [26, 28] and on systematic reviews in psychiatric epidemiology [25].

Our study has some limitations. The representativeness of our sample could be debated because we searched for studies in only one online database (MEDLINE), and limited our selection to meta-analyses in English. In addition, for the assessment of the methods, we depended completely on the reporting; we did not assess protocols or contact the authors if methods were not clearly reported. Even though poor reporting does not necessarily reflect poor conduct, it may severely limit the reader’s comprehension of the systematic review process [34].

Before being able to apply the results of any meta-analysis to patient care, health professionals need to evaluate the credibility of the methods of the meta-analysis [35]. One of the key methodological elements is the search for relevant studies. Because not all studies (and particularly those with negative results) are published in scientific journals, a meta-analysis must involve a search for grey literature to try to avoid such publication bias (a type of reporting bias) [24, 35]. However, we found that only 38 % of our meta-analyses reported having searched for grey literature. Because of no mandatory registration for NRSI as for RCTs, most NRSI are not registered, so searching for grey literature of NRSI is difficult [36]. However, a recent study found that for 32 % of the observational studies registered at ClinicalTrials.gov, unpublished results could be retrieved [37]. In contrast, we found that many meta-analyses assessed reporting bias (68 %). Reviewers may have compensated for the absence of searching for grey literature by assessing reporting bias. Evaluating reporting bias does not exempt the reviewers from searching for grey literature because the assessment of Funnel plot asymmetry may be subjective and statistical methods to test for asymmetry of the plot may lack power [38, 39].

Another critical part of the systematic review process is assessing the methodological quality or risk of bias of the studies included, because the validity of the meta-analysis could be questionable with problems in the design and conduct of individual studies [40]. We found that 72 % of our meta-analyses reported having assessed the methodological quality or risk of bias but only 33 (18 %) considered the risk of confounding bias in their assessment. The Cochrane Collaboration has recognized the need to improve the assessment of risk of bias for NRSI and is currently developing a tool for this.

Finally, we found specific issues related to the inclusion of NRSI. In 69 % of the meta-analyses, the study design for each included study was unclear. The risk of bias may vary depending on the type of NRSI, with case–control studies generally considered as having a higher risk of bias than cohort studies. A description of the type of studies included in the meta-analysis is crucial. In addition, NRSI are prone to confounding: an imbalance in prognostic factors associated with the outcome of interest [24]. NRSI are expected to at least present adjusted estimates from multivariate analyses [3, 4]. Many of our meta-analyses (70 %) did not report or were unclear about whether the crude or adjusted estimates of NRSI were combined. Among the meta-analyses involving adjusted estimates, 33 % did not report the confounding factors adjusted for. This information was likely poorly reported in the individual studies, but then the reviewers should contact the authors for clarification or report it clearly in the meta-analysis.

Conclusions

Some key methodological components of the systematic review process – search for grey literature, description of the type of NRSI included, assessment of risk of confounding bias and reporting of whether crude or adjusted estimates were combined—are not adequately reported in meta-analyses including NRSI. Attention should be paid to improving these elements in such meta-analyses to have an increased confidence in their results.

Ethics

Not applicable. This article reports a meta-research study.

Consent

Not needed. This study does not include human participants.

Availability of supporting data

Data are available upon request for academic researchers.

Abbreviations

CER: 

comparative effectiveness research

NRSI: 

non-randomized studies of interventions

RCT: 

randomized controlled trial

RoB Tool: 

Risk of Bias tool

Declarations

Acknowledgements

Philippe Ravaud is director of the French EQUATOR Centre and a member of the EQUATOR Network Steering Group.

Sources of funding

The researchers did not receive external sources of funding.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, APHP, Paris, France
(2)
Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France
(3)
Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France
(4)
Cochrane France, Paris, France
(5)
Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA

References

  1. Castillo RC, Scharfstein DO, MacKenzie EJ. Observational studies in the era of randomized trials: finding the balance. J Bone Joint Surg Am. 2012;94 Suppl 1:112–7.View ArticlePubMedGoogle Scholar
  2. Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med. 2005;142(12 Pt 2):1090–9.View ArticlePubMedGoogle Scholar
  3. Grootendorst DC, Jager KJ, Zoccali C, Dekker FW. Observational studies are complementary to randomized controlled trials. Nephron. 2010;114(3):c173–177.PubMedGoogle Scholar
  4. Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trials. Diabetes Res Clin Pract. 2010;88 Suppl 1:S3–9.View ArticlePubMedGoogle Scholar
  5. Black N. Why we need observational studies to evaluate the effectiveness of health care. Bmj. 1996;312(7040):1215–8.View ArticlePubMedPubMed CentralGoogle Scholar
  6. The Cochrane Collaboration Glossary [http://www.cochrane.org/glossary/5#letters]
  7. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86.View ArticlePubMedGoogle Scholar
  8. Goulart BH, Ramsey SD, Parvathaneni U. Observational study designs for comparative effectiveness research: an alternative approach to close evidence gaps in head-and-neck cancer. Int J Radiat Oncol Biol Phys. 2014;88(1):106–14.View ArticlePubMedGoogle Scholar
  9. Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.View ArticlePubMedGoogle Scholar
  10. Hannan EL. Randomized clinical trials and observational studies: guidelines for assessing respective strengths and limitations. JACC Cardiovasc Interv. 2008;1(3):211–7.View ArticlePubMedGoogle Scholar
  11. Alemayehu D, Cappelleri JC. Revisiting issues, drawbacks and opportunities with observational studies in comparative effectiveness research. J Eval Clin Pract. 2013;19(4):579–83.View ArticlePubMedGoogle Scholar
  12. Garabedian LF, Chu P, Toh S, Zaslavsky AM, Soumerai SB. Potential bias of instrumental variable analyses for observational comparative effectiveness research. Ann Intern Med. 2014;161(2):131–8.View ArticlePubMedGoogle Scholar
  13. Marko NF, Weil RJ. The role of observational investigations in comparative effectiveness research. Value Health. 2010;13(8):989–97.View ArticlePubMedGoogle Scholar
  14. Roche N, Reddel H, Martin R, Brusselle G, Papi A, Thomas M, et al. Quality standards for real-world research. Focus on observational database studies of comparative effectiveness. Ann Am Thorac Soc. 2014;11 Suppl 2:S99–104.View ArticlePubMedGoogle Scholar
  15. Shrier I, Boivin JF, Steele RJ, Platt RW, Furlan A, Kakuma R, et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol. 2007;166(10):1203–9.View ArticlePubMedGoogle Scholar
  16. Norris SL, Atkins D, Bruening W, Fox S, Johnson E, Kane R, et al. Observational studies in systematic [corrected] reviews of comparative effectiveness: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011;64(11):1178–86.View ArticlePubMedGoogle Scholar
  17. Holve E, Pittman P. A first look at the volume and cost of comparative effectiveness research in the United States [monograph]. Washington, DC: Academy Health; 2009.Google Scholar
  18. Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. Journal of Clinical Epidemiology. 2010;63(3):238–45.View ArticlePubMedGoogle Scholar
  19. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92.View ArticlePubMedPubMed CentralGoogle Scholar
  20. Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med. 2011;8(5):e1001026.View ArticlePubMedPubMed CentralGoogle Scholar
  21. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. Jama. 2001;286(7):821–30.View ArticlePubMedGoogle Scholar
  22. Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet. 2004;363(9422):1728–31.View ArticlePubMedGoogle Scholar
  23. Zhang Z, Ni H, Xu X. Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine. J Clin Epidemiol. 2014;67(8):932–9.View ArticlePubMedGoogle Scholar
  24. Higgins JPT, Green S, editors: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011, Available from www.cochrane-handbook.orgGoogle Scholar
  25. Brugha TS, Matthews R, Morgan Z, Hill T, Alonso J, Jones DR. Methodology and reporting of systematic reviews and meta-analyses of observational studies in psychiatric epidemiology: systematic review. Br J Psychiatry. 2012;200(6):446–53.View ArticlePubMedGoogle Scholar
  26. Golder S, Loke Y, McIntosh HM. Room for improvement? A survey of the methods used in systematic reviews of adverse effects BMC Med Res Methodol. 2006;6:3.PubMedGoogle Scholar
  27. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, et al. Quality of reporting in systematic reviews of adverse events: systematic review. Bmj. 2014;348:f7668.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. Jama. 2000;283(15):2008–12.View ArticlePubMedGoogle Scholar
  30. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology. 2009;62(10):e1–34.View ArticlePubMedGoogle Scholar
  31. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open Med. 2009;3(3):e123–130.PubMedPubMed CentralGoogle Scholar
  32. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.View ArticlePubMedPubMed CentralGoogle Scholar
  33. Delgado-Rodriguez M, Ruiz-Canela M, De Irala-Estevez J, Llorca J, Martinez-Gonzalez A. Participation of epidemiologists and/or biostatisticians and methodological quality of published controlled clinical trials. J Epidemiol Community Health. 2001;55(8):569–72.View ArticlePubMedPubMed CentralGoogle Scholar
  34. Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. Bmj. 2013;346:f1798.View ArticlePubMedGoogle Scholar
  35. Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. Jama. 2014;312(2):171–9.View ArticlePubMedGoogle Scholar
  36. Dal-Re R, Ioannidis JP, Bracken MB, Buffler PA, Chan AW, Franco EL, et al. Making prospective registration of observational research a reality. Sci Transl Med. 2014;6(224):224cm221.View ArticleGoogle Scholar
  37. Baudart M, Ravaud P, Baron G, Dechartres A, Haneef R, Boutron I. Public availability of results of observational studies evaluating an intervention registered at ClinicalTrials.gov. BMC Med. 2016;14(1):7.View ArticlePubMedPubMed CentralGoogle Scholar
  38. Ioannidis JP, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Cmaj. 2007;176(8):1091–6.View ArticlePubMedPubMed CentralGoogle Scholar
  39. Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58(9):894–901.View ArticlePubMedGoogle Scholar
  40. Hopewell S, Boutron I, Altman DG, Ravaud P. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013;3(8):e003342.View ArticlePubMedPubMed CentralGoogle Scholar
  41. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Bmj. 2011;343:d5928.View ArticlePubMedPubMed CentralGoogle Scholar
  42. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12.View ArticlePubMedGoogle Scholar
  43. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Bmj. 2004;328(7454):1490.View ArticlePubMedGoogle Scholar
  44. Wells GA, Shea BJ, O'Connell D, Peterson J, Welch V, Losos M et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. 2013.

Copyright

© Faber et al. 2016

Advertisement