Skip to main content

Database combinations to retrieve systematic reviews in overviews of reviews: a methodological study



When conducting an Overviews of Reviews on health-related topics, it is unclear which combination of bibliographic databases authors should use for searching for SRs. Our goal was to determine which databases included the most systematic reviews and identify an optimal database combination for searching systematic reviews.


A set of 86 Overviews of Reviews with 1219 included systematic reviews was extracted from a previous study. Inclusion of the systematic reviews was assessed in MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. The mean inclusion rate (% of included systematic reviews) and corresponding 95% confidence interval were calculated for each database individually, as well as for combinations of MEDLINE with each other database and reference checking.


Inclusion of systematic reviews was higher in MEDLINE than in any other single database (mean inclusion rate 89.7%; 95% confidence interval [89.0–90.3%]). Combined with reference checking, this value increased to 93.7% [93.2–94.2%]. The best combination of two databases plus reference checking consisted of MEDLINE and Epistemonikos (99.2% [99.0–99.3%]). Stratification by Health Technology Assessment reports (97.7% [96.5–98.9%]) vs. Cochrane Overviews (100.0%) vs. non-Cochrane Overviews (99.3% [99.1–99.4%]) showed that inclusion was only slightly lower for Health Technology Assessment reports. However, MEDLINE, Epistemonikos, and reference checking remained the best combination. Among the 10/1219 systematic reviews not identified by this combination, five were published as websites rather than journals, two were included in CINAHL and Embase, and one was included in the database ERIC.


MEDLINE and Epistemonikos, complemented by reference checking of included studies, is the best database combination to identify systematic reviews on health-related topics.

Peer Review reports


What is already known

  • Overviews of Reviews summarize data from several systematic reviews, thus providing a broad picture of relevant evidence.

  • Few empirical studies exist that support methodological approaches to producing Overviews of Reviews, including the search process for systematic reviews.

What is new

  • The present empirical study has shown that in a sample of 1219 systematic reviews, 99.2% of records are included in a combination of the electronic databases MEDLINE via PubMed and Epistemonikos, supplemented by checking reference lists of eligible studies retrieved by these databases.

Potential impact

  • A resource-efficient search process for systematic reviews should be based on the free-of-charge electronic databases MEDLINE via PubMed and Epistemonikos, supplemented by reference checking.


Overviews of Reviews (Overviews) are used to summarize data from across multiple systematic reviews (SRs) on the same topic, thus providing a broader picture of evidence relevant for decision-making [1]. Generally, they aim to provide a comprehensive assessment of multiple interventions, populations, or outcomes for a defined health problem using SR-level evidence.

Some methodological components of preparing Overviews are analogous to SRs, such as defining the review question structured into a Population, Intervention, Comparator, Outcomes, and Study types (PICOS) format, setting eligibility criteria, and defining a search strategy, though nuances clearly exist. Other methodological components such as the critical appraisal of included SRs, extraction and analysis of data, and presentation of findings differ more markedly between Overviews and SRs [2,3,4].

Many specific steps for conducting an Overview are not yet based on empirical research [5], and some steps contain few empirical studies which may guide future recommendations including selection of electronic databases to use when searching for SRs.

Current guidance recommends searching the Cochrane Database of Systematic Reviews (CDSR) to locate Cochrane SRs [1, 6], and a structured search process to locate and select non-Cochrane reviews, (e.g. in MEDLINE [7] and Embase [8]). Guidance also recommends searching additional regional and subject-specific databases (e.g. LILACS [9], CINAHL [10], PsycINFO [11]), major repositories of SRs, and the PROSPERO register [1, 2, 12]. In practice, the majority of Overviews in healthcare-related fields published to date searched at least two databases [13, 14], in addition to reference checking (i.e. looking through the reference list of included studies), and consultation with the relevant Cochrane group [14].

Rathbone et al. have evaluated retrieval of a set of 400 SRs of interventions for hypertension in seven electronic databases [15]. They searched for SRs in the Cochrane Library, DARE, Embase, Epistemonikos, MEDLINE, PubMed Health, and TRIP. For each database, sensitivity, precision, numbers missed, and unique records identified were calculated. Embase had the highest sensitivity (169/400, or 69% of SRs) and retrieved the largest number of unique SR records, i.e. records found by their search in only one database (34/400, or 8.5% of SRs). The authors conclude that searching should not be restricted to one or two major databases, but that all databases should be searched.

In a situation where resources are limited, it remains unclear which combination of electronic resources Overview authors should use for searching for SRs. Therefore, the aims of the present methodological study were: (1) to assess the inclusion of SRs in six electronic databases: MEDLINE [7], CINAHL [10], Embase [8], Epistemonikos [16], PsycINFO [11], and TRIP [17] without limiting the SR topic; (2) to evaluate whether a single database combined with manual reference checking is sufficient to comprehensively retrieve SRs in the context of preparing Overviews; (3) to identify a combination of databases plus reference checking that may be recommended for searching SRs; and (4) to explore possible contexts (for example, for certain Overview types, or for healthcare interventions compared to other fields of healthcare research) in which the recommendation result of objective #3 produces inadequate retrieval of SRs, defined as lower than 95% retrieval [18]. It was not our aim to evaluate search strategies.


Study design

A methodological study was conducted to analyse the rate of inclusion of SRs in six selected electronic resources, and in combinations of two databases with reference checking. We did not write a protocol for this study.

Search methods

A set of Overviews, including Health Technology Assessment (HTA) reports, Cochrane Overviews, and non-Cochrane Overviews, was obtained from a preceding methodological study, as described in detail in Pieper et al. (2014) [19]. Briefly, a search for overviews was conducted in MEDLINE via PubMed, Embase via [8], CINAHL via EBSCOhost [10], PEDro [20], CDSR, DARE, CENTRAL, CMR, HTA, and NHS-EED via the Cochrane Library, and the websites of 127 HTA agencies from inception to May 2012. Search terms were text words related to Overviews and SRs and the search algorithms can be found in Pieper et al. (2014) [19]. The search was limited to Overviews published in English or German between 2009 and 2011.

Eligibility criteria

Overviews were defined as systematic reviews for which the unit of searching, inclusion and data analysis is the systematic review rather than the primary study [1]. Thus, we included all Overviews that had searched explicitly and systematically for SRs in at least one electronic database, included at least one SR (Overviews including both SRs and primary studies were eligible if the evidence synthesis relied at least in part on SRs, e.g., by combining primary studies and SRs in the evidence synthesis), and critically appraised all included SRs and additional primary studies. A HTA report was defined as “a method of evidence synthesis that considers evidence regarding clinical effectiveness, safety, cost-effectiveness and, when broadly applied, includes social, ethical, and legal aspects of the use of health technologies” [21].

Inclusion criteria

  • Searched for SRs in at least one electronic database;

  • Included at least one SR in their evidence synthesis;

  • Critically appraised included SRs and primary studies; and

  • Full text publication was available.

Exclusion criteria

  • Overviews with a methodological focus; and

  • Published in a language other than English or German.

The set of included Overviews will be henceforth called the “reference set”.

Description of electronic databases selected in this study

Six databases were selected to assess inclusion of systematic reviews as described in the section “data collection”, below, namely MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. The key features of these databases are described in Table 1.

Table 1 Description of electronic databases and resources

The sources scraped by Epistemonikos include, or have included, CDSR, PubMed, Embase, CINAHL, PsycINFO, LILACS, DARE, HTA database, The Campbell Collaboration online library, JBI Database of Systematic Reviews and Implementation Reports, EPPI-Centre Evidence Library. Updates of algorithms in February and April 2019 have led to an expansion in the dataset by more than a factor of 1.5. TRIP widely collects references from sources of SRs (including Cochrane Library and DARE), guidelines, regulatory agencies (FDA, EMA, NICE, IQWIG), HTA databases, NHS EED, literature databases (PubMed), journals, as well as PROSPERO and clinical trial registries.

Data collection

From the full text of each included Overview, the following data were extracted into MS Excel (2016): citation, publication title, number of databases searched, name of each database searched, searched in social science/economics databases (yes/no; i.e. EconLit, HEED, NHS EED, IBSS, Social Sciences Citation Index, Social SciSearch, the Campbell Collaboration Database, Social Sciences Abstracts, Social Services Abstracts, Applied Social Science Index and Abstracts, Social Service Information Gateway), searched in additional sources (‘other sources’ yes/no; i.e. reference lists of included studies, queries to experts, Google, Google Scholar, internal departmental files, hand-searching or electronically searching journals, clinical trial or study registries (e.g., PROSPERO), publishers’ databases (e.g. Springer, ScienceDirect, Thieme, Wolters Kluwer), HTA agencies’ websites (e.g.,,, number of SRs included, Overview type (Cochrane Overview, HTA report, or non-Cochrane Overview), intervention/non-intervention Overview, and mental health- or psychology-related topic (yes/no).

For each Overview, the included SRs were extracted and tagged with the Overview from which they originated. Primary studies were not extracted. HTA reports are usually structured into sections entitled clinical effectiveness, safety, cost-effectiveness, social, ethical, or legal. For HTAs, we only included SRs from the clinical effectiveness section of the report.

The database searches for SRs were performed in April 2019. A stepwise process was followed to identify whether a SR was included in an electronic database, found by reference checking, or included in a database combination:

  1. (A)

    From the sample of SRs extracted from the Overviews, we determined which of six databases each SR was included in, namely MEDLINE, CINAHL, Embase, Epistemonikos, PsycINFO, or TRIP.

  2. (B)

    We then determined which database contained the largest overall number of included SRs. This database was identified as the ‘reference database’. We set the reference database to MEDLINE as it had the highest inclusion rate. The SRs included in MEDLINE will be henceforth called the “MEDLINE-included SRs”.

  3. (C)

    A list of all SRs not included in MEDLINE was compiled. These SRs are called the ‘MEDLINE-non-included SRs’.

  4. (D)

    For each ‘MEDLINE-non-included SR’ obtained in step C, we then manually checked the reference lists of the ‘MEDLINE-included SRs’ that were cited in the same Overview as the ‘MEDLINE-non-included SR’. The purpose of this step was to find out if each SR not included in MEDLINE could have been identified by reference checking of SRs identified in MEDLINE on the same topic, rather than by additional database searching. The SRs found in the reference lists/bibliographies are henceforth called ‘biblio SRs’.

Finally, we constructed five combined sets of SRs by merging the ‘MEDLINE-included SRs’, ‘biblio-SRs’, and SRs obtained in step A for CINAHL, Embase, Epistemonikos, PsycINFO, and TRIP. For each of these five combined sets, we calculated a combined mean inclusion rate (see statistical analysis). This was done to evaluate whether searching more than one database would expand the study pool.

Statistical analysis

The dataset generated and analysed during the current study is made available in Additional file 1. For each Overview, we calculated:

  1. (A)

    the mean inclusion rate (% of included SRs) and corresponding 95% confidence interval (95% CI) separately for each database.

  2. (B)

    the mean inclusion rate for the reference database (as defined above) combined with reference checking (as described above).

  3. (C)

    the mean inclusion rates for combinations of the reference database, reference checking, and each of the other five databases.

The Overview-level inclusion rates obtained in statistical analysis steps A to C were then aggregated for the entire dataset by calculating weighted mean inclusion rates and corresponding 95% confidence intervals (95% CI). Weighting was based on the number of SRs included in each Overview.


The goals of stratification were to generate hypotheses on contexts where the results-based recommendations would apply, and to identify situations where retrieval would be inadequate and further searches may be necessary. Inadequate retrieval of SRs was defined as lower than 95% retrieval. Thus, we: (1) investigated whether searching large numbers of databases offers added value, (2) gauged the magnitude of effect when using ‘other sources’ (as defined above in section ‘data collection’), (3) examined whether different Overview types require searching different electronic resources, (4) explored differences in database inclusion between healthcare interventions and other fields of healthcare research, and (5) evaluated the role of specialist databases when such databases exist in the area of the Overview topic, using PsycINFO as an example.

To answer the above objectives (1) to (5), respectively, exploratory analyses were performed for the following strata: (1) number of databases searched (1–3 / ≥4), (2) other sources searched (yes / no), (3) Overview type (Cochrane Overview, HTA report, or non-Cochrane Overview), (4) intervention/non-intervention Overviews, and (5) mental health- or psychology-related topic (yes/no).

Stratification analysis was performed only for strata containing ≥3 Overviews for analysis. For each stratification analysis, the weighted mean inclusion rate with 95% CI was calculated for combinations of the reference database and reference checking with each of the other databases. For analyses with two strata, the weighted difference in means and corresponding p-value were calculated using a two-sample weighted t-test (Welch) computed in R version 3.5.1 (2018-07-02) using the R package ‘weights’ [26, 27]. The significance level for each individual test αi was adjusted for multiple testing using the Bonferroni correction, i.e. αi = αg/n = 0.0017 for a global significance level of αg = 0.05 and n = 30 tests (6 databases and 5 stratified analyses).

Qualitative analysis of missed SRs

All SRs that were not included in a combination of the reference database, reference checking, and the best additional database were analysed qualitatively. Features investigated were the topics of these SRs, whether they were located on websites, included in the other five databases that were investigated in this study, listed in a publisher’s database (e.g. ScienceDirect, Wiley Online Library, Springerlink, De Gruyter), or in Google Scholar.


Overviews included in the study

The literature search yielded a reference set of 86 Overviews [28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113], of which 73 had been identified in electronic databases and 13 on HTA websites [19]. The characteristics of the included Overviews are summarised in Table 2, and detailed in Table S1 (see Additional file 2).

Table 2 Summary characteristics of N = 86 included Overviews

A total of 1219 SRs were included in the 86 Overviews overall, with a median of eight SRs in an Overview. Between one and 25 (median 6) databases had been searched. Among them, the most widely used were MEDLINE and Embase. More than half of the Overviews had searched DARE or the Cochrane Library, which up to 2018 also contained all DARE records. None had searched the international SR-focused database Epistemonikos, which was first launched internationally in August 2012, and was therefore not available at the time of publication of the included Overviews [22].

A wide range of subjects were covered by the included Overviews [13]. Sixty-nine percent were intervention Overviews, defined as Overviews of health-related interventions, i.e. actions taken with the intent of modifying a defined outcome, usually to treat or cure a health condition or change behaviour. Non-intervention Overviews focused, for example, on diagnosis, incidence/prevalence, or risk factors for certain outcomes.

Thirty-five percent (n = 30/86) of Overviews searched PsycINFO, but of these, only 30% (n = 9/30) had topics related to mental health (i.e. autism, depression, psychosis, or schizophrenia), or psychology (i.e. psychological or psychosocial interventions (such as behavioural therapy)).

Inclusion in individual databases and database combinations

Among the 1219 SRs retrieved, 90% (n = 1093/1219) were included in MEDLINE. The inclusion rates in other databases were lower (Table 3, Fig. 1). Therefore, MEDLINE was used as the reference database.

Table 3 Proportion of SRs included in individual databases and their combination with MEDLINE and reference checking
Fig. 1
figure 1

Inclusion rates in individual databases and their combination with MEDLINE and reference checking. MEDL = MEDLINE, CIN = CINAHL, EMB = Embase, PSI = PsycINFO, EPIS = Epistemonikos, DB = database

In addition to the SRs included in MEDLINE, a further 4% of SRs (n = 49/1219) were identified by reference checking (see Methods, step B). Combinations of MEDLINE and reference checking with the addition of one database (Epistemonikos, PsycINFO, and TRIP) led to inclusion rates above 95% (n = 1209/1219, 1159/1219, and 1174/1219, respectively). Among these databases, the inclusion rate was highest in the combination with Epistemonikos (Table 3, Fig. 1).


Table 4 presents the results obtained for each of the strata investigated. For the combination of MEDLINE, Epistemonikos and reference checking, inclusion rates and their 95% confidence intervals were above the 95% retrieval threshold for all individual strata. Therefore, no context was identified in which this combination produced inadequate retrieval of SRs.

Table 4 Stratified inclusion rates in database combinations with MEDLINE and reference checking (mean [95% CI])

Databases searched

The Overviews in the reference set had searched for SRs in 1–25 databases. The inclusion rates calculated for Overviews searching 1–3 databases was roughly equal to that calculated for Overviews searching 4 or more databases.

Other sources searched

Other sources that were searched by Overview authors include reference lists of included studies, queries to experts, Google, Google Scholar, internal, non-public departmental files from the institutions of Overview authors, hand-searching journals, trial registries, publishers’ databases, journals, institutions, and HTA agencies. Because this represents a larger dataset containing SRs that may not all be included in databases, one may expect that database inclusion rates would be smaller in this stratum.

A total of 74% of Overviews (n = 64/86) reported searching at least one other source. In any database combination, the mean inclusion of SRs for Overviews that had searched other sources was only slightly lower than for Overviews that had not searched other sources. For example, the difference was 0.7% for the combination of MEDLINE, Epistemonikos and reference checking.

Overview types

Overviews were classified into three types, i.e. Cochrane Overviews (n = 3), HTA reports (n = 14), and non-Cochrane Overviews (n = 69). Inclusion of SRs from Cochrane Overviews was complete for combinations of MEDLINE plus reference checking and each of the databases Embase, TRIP, and Epistemonikos (Fig. 2). In contrast, inclusion rates were lower for SRs originating from HTA reports in most database combinations. MEDLINE, Epistemonikos, and reference checking remained the best combination, reaching a mean rate of 97.7% included SRs [96.5 to 98.9]. For non-Cochrane Overviews, which made up 80% (n = 69/86) of the total set and contributed 89% (n = 1087/1219) of SRs, the results were similar to the results for the total set. Thus, the SR inclusion rate for MEDLINE, Epistemonikos, and reference checking was 99.3% [99.1 to 99.4%].

Fig. 2
figure 2

Inclusion rates in database combinations with MEDLINE and reference checking, stratified by Overview type

Intervention/non-intervention overviews

Intervention Overviews represented 69% of the overall sample. The difference between SRs originating from intervention vs. non-intervention Overviews was minimal for all databases.

Mental health- or psychology-related topics

The present dataset included only 10% (n = 9/86) Overviews in the stratum with a mental health- or psychology-related topic, corresponding to 8% (n = 98/1219) of included SRs. In this stratum, the inclusion rate in PsycINFO combined with MEDLINE and reference checking was slightly higher than in the stratum without psychology-related topics. However, the combination of the non-specialist databases Epistemonikos and MEDLINE with reference checking was superior, because 100% of SRs with a mental health- or psychology-related topic were included in this combination.

Qualitative analysis of missed SRs

Ten SRs (n = 10/1219, 0.8%) were not identified by the best database combination (Table 5). Of these, one was incorrectly cited in the original Overview [116]. The correct citation [126] was included in the databases CINAHL, Epistemonikos, and Embase. Two were evidence syntheses published on the web only, for which alternative versions were found by database searching [124, 125]. Thus, the evidence for 3/10 missed records would have been included in alternative form, resulting in a total coverage of 99.43% (n = 1212/1219) of SRs.

Table 5 Qualitative analysis of SRs not included in MEDLINE, Epistemonikos and reference checking

Three missed references (n = 3/1219, 0.24%) were websites not included in any of the databases investigated in the present study [119, 122, 123]. Two missed SRs (n = 2/1219, 0.16%) were included in both CINAHL and Embase [120, 121]. One (n = 1/1219, 0.08%) was listed on the publisher’s website (ScienceDirect) and found via Google Scholar [117]. Finally, one was not found via any of the routes explored in this study [118]. However, it was included in the Education Resources Information Center (ERIC) database, sponsored by the Institute of Education Sciences of the U.S. Department of Education [127], which had been searched by the Overview authors [93].


We identified MEDLINE/Epistemonikos, complemented by reference checking, as the best combination to retrieve SRs for an Overview of Reviews on health-related topics. This study fill a gap in the evidence around the evaluation of methods in overviews as identified in the MoOR framework by Lunny et al. [5, 128]. Our investigation mapped to the option ‘select the types of databases to search’ in the MoOR framework of all methods used in overviews, and provides clear guidance for authors around which databases to choose when planning an overview.

No single database was found to be sufficiently comprehensive by itself for a systematic search for SRs in the context of preparing Overviews. MEDLINE emerged as the best single source for retrieval of SRs (inclusion rate 89.7%). MEDLINE using the PubMed interface has a good inclusion rate for SRs, and is free of charge and open access.

Among databases assessed in combination with MEDLINE and reference checking (i.e. checking the citation lists of SRs identified in MEDLINE), Epistemonikos gave the highest pooled inclusion rate with 99.2%, followed by TRIP (96.3%). Both Epistemonikos and TRIP are resources specialising in the retrieval of SRs. In spite of the fact that the inclusion rate of SRs was relatively high in Embase (inclusion rate 83.7%), the pooled inclusion rate of MEDLINE, Embase and reference checking was below 95%, possibly due to greater overlap with MEDLINE. The subject-specific databases CINAHL (nursing) and PsycINFO (psychology) had pooled inclusion rates similar to Embase (94.6 and 95.1%, respectively).

Analysis of the different strata did not identify any context in which the combination of MEDLINE, Epistemonikos and reference checking was inadequate.

The difference between the number of SRs retrieved from Overviews that had searched in 4 or more databases was minimal compared to Overviews that had searched in 1–3 databases, casting doubt on the value of searching large numbers of databases without specific justification. When looking at the SRs not covered by our optimal database combination, an additional 0.16% of SRs were included in both CINAHL and Embase, and 0.08% in ERIC. The effect of searching other sources beyond electronic databases (e.g. Google Scholar and publishers’ databases) was found to be similarly small (0.5 and 0.16%, respectively, of studies found in addition to the recommended combination). Nevertheless, a limited number of SRs are published on the web only and not identified by searching the databases investigated in this study (0.24% in this dataset).

The recommended database combination included slightly more SRs from Cochrane and non-Cochrane Overviews than from HTA reports. The better rate of SRs included in Cochrane Overviews (100% of included SRs) is likely a result of guidance for Cochrane Overview authors to search for, and include, only Cochrane SRs [1]. SRs from Cochrane Overviews are more frequently included than from non-Cochrane Overviews (99.3% of included SRs), because all Cochrane Reviews are included in PubMed and other electronic databases [6]. The inclusion rate using the recommended combination was slightly lower for SRs from HTA reports (97.7% of included SRs). This may be because authors of HTA reports typically search more widely, and might thus identify more studies, than authors of Cochrane or non-Cochrane Overviews (median 8 databases searched, compared to 2 databases for Cochrane Overviews and 5 for non-Cochrane Overview; all searches in HTA reports complemented by other methods).

Only one empirical study was found that investigated the types of databases to search [15]. The recall rates in the Rathbone study (88% recall for Cochrane Library and Embase; and 83% for Cochrane Library, Epistemonikos and MEDLINE) are substantially lower than in our study (99.2% indexing in MEDLINE, Epistemonikos and reference checking). Rathbone et al. examined bibliographic database performance in identifying SRs using an approach that was different to ours in several key aspects. First, the study was topic specific and related to interventions for hypertension. Second, the reference set of SRs was generated by searching in the same electronic databases that were later used to analyse retrieval of SRs, and was thus a relative recall analysis based on databases alone [129]. They performed systematic literature searches in seven electronic databases and used search filters for SRs to increase precision. Therefore, their approach primarily evaluated the relative performance of search strategies in bibliographic databases compared to each other, in terms of precision and sensitivity.

In contrast to the Rathbone study [15], our sample was not topic-specific, and included both intervention and non-intervention SRs. We evaluated database coverage independently of the search. Our sample of SRs was larger (1219 vs. 400), and was obtained from published Overviews which had used various methods to identify relevant literature. Our study investigating database inclusion of SRs, as opposed to evaluating the performance of search strategies in different databases.

These methodological differences may explain the different results obtained. In addition, our last search was performed after a major update in the Epistemonikos machine learning algorithms. As a result, a larger number of SRs were included in this resource and retrieved by our search.

Rathbone et al. concluded that “a search of all databases should be performed”. This conclusion is not supported by our results. First, the high database inclusion rate of over 99% that we observed for a combination of two databases and reference checking shows that when a sensitive search strategy is employed, this combination is sufficient for retrieving SRs. Second, in the context of limited resources, it may not be efficient to search additional databases as we found no relevant differences when we stratified inclusion of SRs by Overviews that had searched in 4 or more databases compare with 1 to 3 databases (99.3% vs. 98.8% inclusion, respectively).

Strengths and limitations of the study

The present methodological study is based on a large sample of 1219 SRs. The SRs were extracted from published Overviews, and were found in diverse electronic databases, often complemented by reference checking, queries to experts, searches of the web (e.g. Google, Google Scholar, government reports), or hand-searching journals, thus making our sample representative of searches in a variety of overviews. We analysed the inclusion rate of SR in databases, and have not used any search strategies for the retrieval of SRs. Therefore, our results are independent of the effectiveness of any search strategy employed.

The results of this study apply only to database coverage. Real-world retrieval of SRs will usually be lower because search strategies can be unreliable and less than 100% sensitive. Another limitation relates to the choice of sample. The lack of recency of the dataset of SRs dated from 1982 to 2011 may be limiting, and future efforts should be made to validate this research in a more up-to-date set of SRs. The reference set of SRs was not generated by hand-searching all relevant resources, which may be considered the gold standard, but relies on relative recall [129]. Some Overview authors may have searched for SRs less extensively or less effectively than others, resulting in a possible bias for studies included in databases. This may affect the absolute values for means and confidence intervals generated within this study. In the worst case, if some Overview authors have missed harder-to-find SRs in their searches, this may tend to even out observable differences between databases and bias the results towards higher overall inclusion rates.

The inclusion of SRs in the six databases is likely to change over time, as the terminology used to describe systematic reviews changes, and as tagging of SRs and algorithms for retrieval of SRs improves. For example, the National Library of Medicine has established a new MeSH publication type “Systematic Reviews” in January 2019, and SRs have retrospectively been re-indexed in MEDLINE [130]. A more general search filter, designed to include SRs that had not yet been indexed, is also available [131]. Other databases may decline in inclusion rates, such as DARE and the NHS EED database (not updated since 2015). The reference set of Overviews was limited to those published in English or German, so that the results are not applicable to SRs published in other languages. In addition, some SRs may have been included in more than one Overview, the effect of which was not assessed. The ‘biblio-SRs’, i.e. those obtained by reference checking in this study, may also have been retrievable by other means. Overview authors were not contacted to clarify the origin of these ‘biblio-SRs’. Also, the results may depend on factors not studied, for example, specific diseases, drugs, or other interventions that are more likely to be included in one database than in another. Furthermore, the present study was not designed primarily to quantify differences between strata, so that the results of the stratification analysis should be considered exploratory. Finally, inclusion in databases was not assessed in duplicate, leading to potential errors in extractions and calculations.

Future research

Future research should be directed at validating our findings using an up-to-date reference set, assessing whether it is applicable to languages not evaluated in the present study, and exploring the relative contribution of other methods of study retrieval.

The Cochrane Handbook [1] and the Joanna Briggs Institute [132] currently recommend that a structured search process using multiple databases (e.g. MEDLINE and Embase, and additional regional and subject-specific databases, such as LILACS, CINAHL, PsycINFO) be used to retrieve systematic reviews. This guidance will need to be re-evaluated based on our findings that MEDLINE and Epistemonikos, with reference checking of included studies, is the best strategy for retrieval of systematic reviews.


The present study shows that the literature search for Overviews should be performed in MEDLINE and Epistemonikos, complemented by reference checking.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.



Confidence interval




Health Technology Assessment


Interquartile range


Population, Intervention, Comparator, Outcomes, and Study types


Systematic review


  1. Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews. In: JPT H, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated March 2020): Cochrane; 2020. Available from

  2. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Int J Evid Based Healthc. 2015;13(3):132–40. Epub 2015/09/12. PubMed PMID: 26360830.

    Article  PubMed  Google Scholar 

  3. Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evid Based Mental Health. 2018;21(3):95–100. Epub 2018/07/15. PubMed PMID: 30006442.

    Article  Google Scholar 

  4. Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L. What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Syst Rev. 2016;5(1):190. Epub 2016/11/16. PubMed PMID: 27842604; PubMed Central PMCID: PMCPMC5109841.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231. Epub 2017/11/23. PubMed PMID: 29162130; PubMed Central PMCID: PMCPMC5698938.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Cochrane Library: About the Cochrane Database of Systematic Reviews. Accessed 6 Sept 2019.

  7. NCBI PubMed. Accessed 23 Mar 2019.

  8. Embase. Accessed 23 Mar 2019.

  9. Literatura Latino-Americana e do Caribe em Ciências da Saúde [Latin American and Caribbean Center on Health Sciencies Information]: LILACS. Accessed 23 Mar 2019.

  10. Cumulative Index for Nursing and Allied Health Literature (CINAHL). Accessed 23 Mar 2019.

  11. EBSCO PsycINFO. Accessed 23 Mar 2019.

  12. International prospective register of systematic reviews (PROSPERO). Accessed 23 Mar 2019.

  13. Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–73 Epub 2012/09/11. PubMed PMID: 22959594.

    Article  PubMed  Google Scholar 

  14. Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One. 2012;7(11):e49667 Epub 2012/11/21. PubMed PMID: 23166744; PubMed Central PMCID: PMCPMC3499476.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Rathbone J, Carter M, Hoffmann T, Glasziou P, et al. Syst Rev. 2016;5:27 Epub 2016/02/11. PubMed PMID: 26862061; PubMed Central PMCID: PMCPMC4748526.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Epistemonikos: Database of the best of Evidence-Based Health Care. Accessed 23 Mar 2019.

  17. Trip medical database. Accessed 23 Mar 2019.

  18. Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):245 Epub 2017/12/07. PubMed PMID: 29208034; PubMed Central PMCID: PMCPMC5718002.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Pieper D, Antoine SL, Mathes T, Neugebauer EA, Eikermann M. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol. 2014;67(4):368–75. Epub 2014/03/04. PubMed PMID: 24581293.

    Article  PubMed  Google Scholar 

  20. Physiotherapy Evidence Database (PEDro). Accessed 23 Mar 2019.

  21. Luce BR, Drummond M, Jonsson B, Neumann PJ, Schwartz JS, Siebert U, et al. EBM, HTA, and CER: clearing the confusion. Milbank Q. 2010;88(2):256–76. Epub 2010/06/29. PubMed PMID: 20579285; PubMed Central PMCID: PMCPMC2980346.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Rada G, Perez D, Capurro D. Epistemonikos: a free, relational, collaborative, multilingual database of health evidence. Stud Health Technol Inform. 2013;192:486–90. Epub 2013/08/08. PubMed PMID: 23920602.

    PubMed  Google Scholar 

  23. El-Khayat YM. Epistemonikos (Resource Review). J Med Libr Assoc. 2017;105(4):431–2 Epub 2017 Oct 1. PubMed Central PMCID: PMCPMC5624441.

    Article  PubMed Central  Google Scholar 

  24. MEDLINE (R): Description of the database. Accessed 27 Aug 2019.

  25. Sources searched by Trip. In: Trip database blog. Accessed 7 Mar 2019.

  26. R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for statistical computing; 2018. URL

    Google Scholar 

  27. Josh Pasek, with some assistance from Alex Tahk, some code modified from R-core; Additional contributions by Gene Culter and Marcus Schwemmle. weights: Weighting and Weighted Statistics. R package version 1.0. 2018.

    Google Scholar 

  28. Andersen JH, Fallentin N, Thomsen JF, Mikkelsen S. Risk factors for neck and upper extremity disorders among computers users and the effect of interventions: an overview of systematic reviews. PLoS One. 2011;6(5):e19691. Epub 2011/05/19. PubMed PMID: 21589875; PubMed Central PMCID: PMCPmc3093401.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Australia Medical Services Advisory Committee. Positron emission tomography for lymphoma: assessment report: Medical Services Advisory Committee; 2009. ISBN: 978-1-74241-066-1. Available from:

    Google Scholar 

  30. Australia Medical Services Advisory Committee. Human papillomavirus triage test for women with possible or definite low-grade squamous intra-epithelial lesions. 2009. ISBN: 978-1-74241-019-7. Available from:

    Google Scholar 

  31. Australia Medical Services Advisory Committee. Positron emission tomography for cervical cancer : assessment report: Medical Services Advisory Committee. 2010. ISBN: 978-1-74241-155-2. Available from:

    Google Scholar 

  32. Australia Medical Services Advisory Committee. Positron emission tomography for glioma: Medical Services Advisory Committee. 2010. ISBN: 978-1-74241-151-4. Available from:

    Google Scholar 

  33. Australia Medical Services Advisory Committee, Khafagi F, Atherton J, Gracanin A, Kearney B, Pitman A. Positron emission tomography for myocardial viability: assessment report: Medical Services Advisory Committee. 2010. ISBN: 978-1-74241-157-6. Available from:

    Google Scholar 

  34. Beattie M, Taylor J. Silver alloy vs. uncoated urinary catheters: a systematic review of the literature. J Clin Nurs. 2011;20(15–16):2098–108. Epub 2011/03/23. PubMed PMID: 21418360.

    Article  PubMed  Google Scholar 

  35. Berkhof M, van Rijssen HJ, Schellart AJ, Anema JR, van der Beek AJ. Effective training strategies for teaching communication skills to physicians: an overview of systematic reviews. Patient Educ Couns. 2011;84(2):152–62. Epub 2010/08/03. PubMed PMID: 20673620.

    Article  PubMed  Google Scholar 

  36. Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011;8(1):e1000387. Epub 2011/01/27. PubMed PMID: 21267058; PubMed Central PMCID: PMCPmc3022523.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Brouwers MC, Garcia K, Makarski J, Daraz L. The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews. Implement Sci. 2011;6:130. Epub 2011/12/22. PubMed PMID: 22185329; PubMed Central PMCID: PMCPmc3284444.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Clar CBK, Cummins E, Royle P, Waugh N. Self-monitoring of blood glucose in type 2 diabetes: systematic review. Health Technol Assess (Winchester, England). 2010;14(12).

  39. Damen A, Remmen R, Wens J, Paulus D. Evidence based post graduate training. A systematic review of reviews based on the WFME quality framework. BMC Med Educ. 2011;11:80. Epub 2011/10/08. PubMed PMID: 21977898; PubMed Central PMCID: PMCPmc3200166.

    Article  PubMed  PubMed Central  Google Scholar 

  40. De Niet GJ, Tiemens BG, Kloos MW, Hutschemaekers GJ. Review of systematic reviews about the efficacy of non-pharmacological interventions to improve sleep quality in insomnia. Int J Evid Based Healthc. 2009;7(4):233–42. Epub 2009/12/01. PubMed PMID: 21631864.

    Article  PubMed  Google Scholar 

  41. de Vet E, de Ridder DT, de Wit JB. Environmental correlates of physical activity and dietary behaviours among young people: a systematic review of reviews. Obes Rev. 2011;12(5):e130–42 Epub 2010/07/16. PubMed PMID: 20630024.

    Article  PubMed  Google Scholar 

  42. Diener MK, Voss S, Jensen K, Buchler MW, Seiler CM. Elective midline laparotomy closure: the INLINE systematic review and meta-analysis. Ann Surg. 2010;251(5):843–56. Epub 2010/04/17. PubMed PMID: 20395846.

    Article  PubMed  Google Scholar 

  43. Hagen A, Gorenoi V, Schönermark MP. Spezifische Immuntherapie (SIT) zur Behandlung der allergischen Rhinitis. Schriftenreihe Health Technology Assessment; 2010. p. Bd. 96.

    Google Scholar 

  44. Gorenoi V, Schönermark MP, Hagen A. Perkutane Koronarinterventionen zusätzlich zur optimalen medikamentösen Therapie bei stabiler Angina Pectoris Schriftenreihe Health Technology Assessment; 2011. p. Bd. 115.

    Google Scholar 

  45. Schnell-Inderst P, Hunger T, Hintringer K, Schwarzer R, Seifert-Klauss V, Gothe H, et al. Individuelle Gesundheitsleistungen. Schriftenreihe Health Technology Assessment; 2011. p. Bd. 113.

    Google Scholar 

  46. Ekeland AG, Bowes A, Flottorp S. Effectiveness of telemedicine: a systematic review of reviews. Int J Med Inform. 2010;79(11):736–71. Epub 2010/10/05. PubMed PMID: 20884286.

    Article  PubMed  Google Scholar 

  47. Ernst E, Lee M. Ayurvedic medicine: an overview of systematic reviews. Perfusion. 2010;23:132–40.

    Google Scholar 

  48. Ernst E, Lee MS. Acupressure: an overview of systematic reviews. J Pain Symptom Manag. 2010;40(4):e3–7. Epub 2010/09/04. PubMed PMID: 20813496.

    Article  Google Scholar 

  49. Ernst E, Lee MS. Acupuncture for rheumatic conditions: an overview of systematic reviews. Rheumatology (Oxford, England). 2010;49(10):1957–61. Epub 2010/07/02. PubMed PMID: 20591833.

    Article  Google Scholar 

  50. Ernst E, Lee MS. How effective is yoga? A concise overview of systematic reviews. Focus Altern Complement Ther. 2010;15(4).

  51. Ernst E, Lee MS, Choi TY. Acupuncture for depression?: a systematic review of systematic reviews. Eval Health Prof. 2011;34(4):403–12. Epub 2010/12/09. PubMed PMID: 21138913.

    Article  CAS  PubMed  Google Scholar 

  52. Ernst E, Lee MS, Choi TY. Acupuncture for insomnia? An overview of systematic reviews. Eur J Gen Pract. 2011;17(2):116–23. Epub 2011/04/06. PubMed PMID: 21463162.

    Article  PubMed  Google Scholar 

  53. Ernst E, Posadzki P. Complementary and alternative medicine for rheumatoid arthritis and osteoarthritis: an overview of systematic reviews. Curr Pain Headache Rep. 2011;15(6):431–7. Epub 2011/10/08. PubMed PMID: 21979101.

    Article  PubMed  Google Scholar 

  54. Ernst E, Posadzki P, Lee MS. Complementary and alternative medicine (CAM) for sexual dysfunction and erectile dysfunction in older men and women: an overview of systematic reviews. Maturitas. 2011;70(1):37–41. Epub 2011/07/26. PubMed PMID: 21782365.

    Article  PubMed  Google Scholar 

  55. Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev. 2011;(7):Cd009255. Epub 2011/07/08. PubMed PMID: 21735443; PubMed Central PMCID: PMCPmc4204491.

  56. Gibson M, Petticrew M, Bambra C, Sowden AJ, Wright KE, Whitehead M. Housing and health inequalities: a synthesis of systematic reviews of interventions aimed at different pathways linking housing and health. Health Place. 2011;17(1):175–84. Epub 2010/12/17. PubMed PMID: 21159542; PubMed Central PMCID: PMCPmc3098470.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Greaves CJ, Sheppard KE, Abraham C, Hardeman W, Roden M, Evans PH, et al. Systematic review of reviews of intervention components associated with increased effectiveness in dietary and physical activity interventions. BMC Public Health. 2011;11:119. Epub 2011/02/22. PubMed PMID: 21333011; PubMed Central PMCID: PMCPmc3048531.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Heighes PT, Doig GS, Sweetman EA, Simpson F. An overview of evidence from systematic reviews evaluating early enteral nutrition in critically ill patients: more convincing evidence is needed. Anaesth Intensive Care. 2010;38(1):167–74. Epub 2010/03/03. PubMed PMID: 20191793.

    Article  CAS  PubMed  Google Scholar 

  59. Hillberg T, Hamilton-Giachritsis C, Dixon L. Review of meta-analyses on the association between child sexual abuse and adult mental health difficulties: a systematic approach. Trauma Violence Abuse. 2011;12(1):38–49. Epub 2011/02/04. PubMed PMID: 21288934.

    Article  PubMed  Google Scholar 

  60. Hopton A, MacPherson H. Acupuncture for chronic pain: is acupuncture more than an effective placebo? A systematic review of pooled data from meta-analyses. Pain Practi. 2010;10(2):94–102. Epub 2010/01/15. PubMed PMID: 20070551.

    Article  Google Scholar 

  61. Huguet F, Girard N, Guerche CS, Hennequin C, Mornex F, Azria D. Chemoradiotherapy in the management of locally advanced pancreatic carcinoma: a qualitative systematic review. J Clin Oncol. 2009;27(13):2269–77. Epub 2009/03/25. PubMed PMID: 19307501.

    Article  CAS  PubMed  Google Scholar 

  62. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Nutzenbewertung nichtmedikamentöser Behandlungsstrategien bei Patienten mit essenzieller Hypertonie: Kochsalzreduktion. Rapid Rep. 2009:A05–21B. Available from:

  63. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Positronenemissionstomographie (PET und PET/CT) bei malignen Lymphomen. IQWiG-Berichte; 2009. p. Nr. 47. Available from:

    Google Scholar 

  64. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Positronenemissionstomographie (PET) und PET/CT bei malignem Melanom. IQWiG-Berichte; 2011. p. Nr. 86. Available from:

    Google Scholar 

  65. Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Positronenemissionstomographie (PET) und PET/CT bei Kopf- und Halstumoren. IQWiG-Berichte; 2011. p. Nr. 82. Kopf-Halstumoren.pdf.

    Google Scholar 

  66. Jepson RG, Harris FM, Platt S, Tannahill C. The effectiveness of interventions to change six health behaviours: a review of reviews. BMC Public Health. 2010;10:538. Epub 2010/09/10. PubMed PMID: 20825660; PubMed Central PMCID: PMCPmc2944371.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Kamioka H, Tsutani K, Okuizumi H, Mutoh Y, Ohta M, Handa S, et al. Effectiveness of aquatic exercise and balneotherapy: a summary of systematic reviews based on randomized controlled trials of water immersion therapies. J Epidemiol. 2010;20(1):2–12. Epub 2009/11/03. PubMed PMID: 19881230; PubMed Central PMCID: PMCPmc3900774.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Kang HS, Jeong D, Kim DI, Lee MS. The use of acupuncture for managing gynaecologic conditions: an overview of systematic reviews. Maturitas. 2011;68(4):346–54. Epub 2011/03/08. PubMed PMID: 21376483.

    Article  PubMed  Google Scholar 

  69. Keus F, Gooszen HG, van Laarhoven CJ. Open, small-incision, or laparoscopic cholecystectomy for patients with symptomatic cholecystolithiasis. An overview of Cochrane Hepato-Biliary Group reviews. Cochrane Database Syst Rev. 2010;(1):Cd008318. Epub 2010/01/22. PubMed PMID: 20091665.

  70. Khambalia AZ, Dickinson S, Hardy LL, Gill T, Baur LA. A synthesis of existing systematic reviews and meta-analyses of school-based behavioural interventions for controlling and preventing obesity. Obe Rev:214–33. Epub 2011/11/11. PubMed PMID: 22070186.

  71. Kumar A, Galeb S, Djulbegovic B. Treatment of patients with multiple myeloma: an overview of systematic reviews. Acta Haematol. 2011;125(1–2):8–22 Epub 2010/12/15. PubMed PMID: 21150183.

    Article  CAS  PubMed  Google Scholar 

  72. Lakke SE, Soer R, Takken T, Reneman MF. Risk and prognostic factors for non-specific musculoskeletal pain: a synthesis of evidence from systematic reviews classified into ICF dimensions. Pain. 2009;147(1–3):153–64. Epub 2009/10/06. PubMed PMID: 19800735.

  73. Fischer S, Zechmeister I. Sakralnervenstimulation bei fäkaler Inkontinenz. Rapid Assessment. Wien: Ludwig Boltzmann Institut für Health Technology Assessment; 2011. p. Nr. 004. ISSN: 1996–935. Available from:

    Google Scholar 

  74. Lee MS, Kim JI, Ernst E. Is cupping an effective treatment? An overview of systematic reviews. J Acupunct Meridian Studies. 2011;4(1):1–4. Epub 2011/03/29. PubMed PMID: 21440874.

    Article  Google Scholar 

  75. Lenz M, Richter T, Mühlhauser I. Morbidität und Mortalität bei Übergewicht und Adipositas im Erwachsenenalter. Dtsch Arztebl Int. 2009;106(40):641–8.

    PubMed  PubMed Central  Google Scholar 

  76. List T, Axelsson S. Management of TMD: evidence from systematic reviews and meta-analyses. J Oral Rehabil. 2010;37(6):430–51. Epub 2010/05/05. PubMed PMID: 20438615.

    Article  CAS  PubMed  Google Scholar 

  77. Maniglio R. The impact of child sexual abuse on health: a systematic review of reviews. Clin Psychol Rev. 2009;29(7):647–57. Epub 2009/09/08. PubMed PMID: 19733950.

    Article  PubMed  Google Scholar 

  78. Maniglio R. Child sexual abuse in the etiology of depression: a systematic review of reviews. Depress Anxiety. 2010;27(7):631–42. Epub 2010/03/26. PubMed PMID: 20336807.

    Article  PubMed  Google Scholar 

  79. Matheson SL, Green MJ, Loo C, Carr VJ. Quality assessment and comparison of evidence for electroconvulsive therapy and repetitive transcranial magnetic stimulation for schizophrenia: a systematic meta-review. Schizophr Res. 2010;118(1–3):201–10. Epub 2010/02/02. PubMed PMID: 20117918.

    Article  CAS  PubMed  Google Scholar 

  80. Matheson SL, Shepherd AM, Laurens KR, Carr VJ. A systematic meta-review grading the evidence for non-genetic risk factors and putative antecedents of schizophrenia. Schizophr Res. 2011;133(1–3):133–42. Epub 2011/10/18. PubMed PMID: 21999904.

  81. Matyas E, Jeitler K, Horvath K, Semlitsch T, Hemkens LG, Pignitter N, et al. Benefit assessment of salt reduction in patients with hypertension: systematic overview. J Hypertens. 2011;29(5):821–8. Epub 2011/04/09. PubMed PMID: 21475042.

    Article  CAS  PubMed  Google Scholar 

  82. Melnik T, Soares BG, Puga ME, Atallah AN. Efficacy and safety of atypical antipsychotic drugs (quetiapine, risperidone, aripiprazole and paliperidone) compared with placebo or typical antipsychotic drugs for treating refractory schizophrenia: overview of systematic reviews. Sao Paulo Med J. 2010;128(3):141–66. Epub 2010/10/22. PubMed PMID: 20963366.

    Article  PubMed  Google Scholar 

  83. Mikton C, Butchart A. Child maltreatment prevention: a systematic review of reviews. Bull World Health Organ. 2009;87(5):353–61. Epub 2009/06/25. PubMed PMID: 19551253; PubMed Central PMCID: PMCPmc2678770.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Minozzi S, Davoli M, Bargagli AM, Amato L, Vecchi S, Perucci CA. An overview of systematic reviews on cannabis and psychosis: discussing apparently conflicting results. Drug Alcohol Rev. 2010;29(3):304–17. Epub 2010/06/23. PubMed PMID: 20565524.

    Article  PubMed  Google Scholar 

  85. Moe RH, Kjeken I, Uhlig T, Hagen KB. There is inadequate evidence to determine the effectiveness of nonpharmacological and nonsurgical interventions for hand osteoarthritis: an overview of high-quality systematic reviews. Phys Ther. 2009;89(12):1363–70. Epub 2009/10/24. PubMed PMID: 19850713.

    Article  PubMed  Google Scholar 

  86. Monasta L, Batty GD, Cattaneo A, Lutje V, Ronfani L, Van Lenthe FJ, et al. Early-life determinants of overweight and obesity: a review of systematic reviews. Obes Rev. 2010;11(10):695–708. Epub 2010/03/25. PubMed PMID: 20331509.

    Article  CAS  PubMed  Google Scholar 

  87. Muangpaisan W, Brayne C. Systematic review of statins for the prevention of vascular dementia or dementia. Geriatr Gerontol Int. 2010;10(2):199–208. Epub 2010/01/27. PubMed PMID: 20100290.

    PubMed  Google Scholar 

  88. O'Connor EA, Whitlock EP, Gaynes B, Beil TL. U.S. Preventive Services Task Force Evidence Syntheses, formerly Systematic Evidence Reviews. Screening for Depression in Adults and Older Adults in Primary Care: An Updated Systematic Review. Rockville: Agency for Healthcare Research and Quality (US); 2009. Available from:

    Google Scholar 

  89. Oestergaard S, Moldrup C. Improving outcomes for patients with depression by enhancing antidepressant therapy with non-pharmacological interventions: a systematic review of reviews. Public Health. 2011;125(6):357–67. Epub 2011/05/24. PubMed PMID: 21600619.

    Article  CAS  PubMed  Google Scholar 

  90. Onakpoya IJ, Wider B, Pittler MH, Ernst E. Food supplements for body weight reduction: a systematic review of systematic reviews. Obesity (Silver Spring, Md). 2011;19(2):239–44 Epub 2010/09/04. PubMed PMID: 20814412.

    Article  Google Scholar 

  91. Oxman AD, Fretheim A. Can paying for results help to achieve the millennium development goals? Overview of the effectiveness of results-based financing. J Evid Based Med. 2009;2(2):70–83. Epub 2009/05/01. PubMed PMID: 21348993.

    Article  PubMed  Google Scholar 

  92. Palmateer N, Kimber J, Hickman M, Hutchinson S, Rhodes T, Goldberg D. Evidence for the effectiveness of sterile injecting equipment provision in preventing hepatitis C and human immunodeficiency virus transmission among injecting drug users: a review of reviews. Addiction (Abingdon, England). 2010;105(5):844–59 Epub 2010/03/12. PubMed PMID: 20219055.

    Article  Google Scholar 

  93. Peters LW, Kok G, Ten Dam GT, Buijs GJ, Paulussen TG. Effective elements of school health promotion across behavioral domains: a systematic review of reviews. BMC Public Health. 2009;9:182. Epub 2009/06/16. PubMed PMID: 19523195; PubMed Central PMCID: PMCPMC2702385.

    Article  PubMed  PubMed Central  Google Scholar 

  94. Posadzki P, Ernst E. Spinal manipulation: an update of a systematic review of systematic reviews. N Z Med J. 2011;124(1340):55–71. Epub 2011/09/29. PubMed PMID: 21952385. Available from:

    PubMed  Google Scholar 

  95. Posadzki P, Ernst E. Systematic reviews of spinal manipulations for headaches: an attempt to clear up the confusion. Headache. 2011;51(9):1419–25 Epub 2011/04/02. PubMed PMID: 21453329.

    Article  PubMed  Google Scholar 

  96. Ryan R, Santesso N, Hill S, Lowe D, Kaufman C, Grimshaw J. Consumer-oriented interventions for evidence-based prescribing and medicines use: an overview of systematic reviews. Cochrane Database Syst Rev. 2011;(5):Cd007768 Epub 2011/05/13. PubMed PMID: 21563160.

  97. Safron M, Cislak A, Gaspar T, Luszczynska A. Micro-environmental characteristics related to body weight, diet, and physical activity of children and adolescents: a systematic umbrella review. Int J Environ Health Res. 2011;21(5):317–30. Epub 2011/05/07. PubMed PMID: 21547807.

    Article  PubMed  Google Scholar 

  98. Safron M, Cislak A, Gaspar T, Luszczynska A. Effects of school-based interventions targeting obesity-related behaviors and body weight change: a systematic umbrella review. Behav Med (Washington, DC). 2011;37(1):15–25. Epub 2011/02/25. PubMed PMID: 21347906.

    Article  Google Scholar 

  99. Salvo F, Fourrier-Reglat A, Bazin F, Robinson P, Riera-Guardia N, Haag M, et al. Cardiovascular and gastrointestinal safety of NSAIDs: a systematic review of meta-analyses of randomized clinical trials. Clin Pharmacol Ther. 2011;89(6):855–66. Epub 2011/04/08. PubMed PMID: 21471964.

    Article  CAS  PubMed  Google Scholar 

  100. Savard LA, Thompson DR, Clark AM. A meta-review of evidence on heart failure disease management programs: the challenges of describing and synthesizing evidence on complex interventions. Trials. 2011;12:194. Epub 2011/08/19. PubMed PMID: 21846340; PubMed Central PMCID: PMCPMC3174117.

    Article  PubMed  PubMed Central  Google Scholar 

  101. Schneider Chafen JJ, Newberry SJ, Riedl MA, Bravata DM, Maglione M, Suttorp MJ, et al. Diagnosing and managing common food allergies: a systematic review. JAMA. 2010;303(18):1848–56. Epub 2010/05/13. PubMed PMID: 20460624.

    Article  CAS  Google Scholar 

  102. Seida JK, Ospina MB, Karkhaneh M, Hartling L, Smith V, Clark B. Systematic reviews of psychosocial interventions for autism: an umbrella review. Dev Med Child Neurol. 2009;51(2):95–104. Epub 2009/02/05. PubMed PMID: 19191842.

    Article  PubMed  Google Scholar 

  103. Smith CA, Cochrane S. Does acupuncture have a place as an adjunct treatment during pregnancy? A review of randomized controlled trials and systematic reviews. Birth (Berkeley, Calif). 2009;36(3):246–53. Epub 2009/09/15. PubMed PMID: 19747272.

    Article  Google Scholar 

  104. Smith V, Devane D, Begley CM, Clarke M, Higgins S. A systematic review and quality assessment of systematic reviews of randomised trials of interventions for preventing and treating preterm birth. Eur J Obstet Gynecol Reprod Biol. 2009;142(1):3–11. Epub 2008/11/11. PubMed PMID: 18996637.

    Article  PubMed  Google Scholar 

  105. Smith CA, Carmady B. Acupuncture to treat common reproductive health complaints: an overview of the evidence. Auton Neurosci. 2010;157(1–2):52–6. Epub 2010/05/21. PubMed PMID: 20483671.

    Article  PubMed  Google Scholar 

  106. Spearing NM, Connelly LB. Is compensation “bad for health”? A systematic meta-review. Injury. 2011;42(1):15–24. Epub 2010/01/12. PubMed PMID: 20060524.

    Article  PubMed  Google Scholar 

  107. Swinkels A, Cochrane K, Burt A, Johnson L, Lunn T, Rees AS. Exercise interventions for non-specific low back pain: an overview of systematic reviews. Phys Ther Rev. 2009;14(4):247–59.

    Article  Google Scholar 

  108. Tamayo-Velazquez MI, Simon-Lorda P, Villegas-Portero R, Higueras-Callejon C, Garcia-Gutierrez JF, Martinez-Pecino F, et al. Interventions to promote the use of advance directives: an overview of systematic reviews. Patient Educ Couns. 2010;80(1):10–20. Epub 2009/11/03. PubMed PMID: 19879090.

    Article  PubMed  Google Scholar 

  109. Terry R, Perry R, Ernst E. An overview of systematic reviews of complementary and alternative medicine for fibromyalgia. Clin Rheumatol. 2012;31(1):55–66 Epub 2011/05/27. PubMed PMID: 21614472.

    Article  PubMed  Google Scholar 

  110. Trudeau M, Green E, Cosby R, Charbonneau F, Easty T, Ko Y, et al. Key components of intravenous chemotherapy labeling: a systematic review and practice guideline. J Oncol Pharm Pract. 2011;17(4):409–24. Epub 2010/09/30. PubMed PMID: 20876081.

    Article  CAS  PubMed  Google Scholar 

  111. van Achterberg T, Huisman-de Waal GG, Ketelaar NA, Oostendorp RA, Jacobs JE, Wollersheim HC. How to promote healthy behaviours in patients? An overview of evidence for behaviour change techniques. Health Promot Int. 2011;26(2):148–62. Epub 2010/08/27. PubMed PMID: 20739325; PubMed Central PMCID: PMCPMC3090154.

    Article  PubMed  Google Scholar 

  112. van der Feltz-Cornelis CM, Sarchiapone M, Postuvan V, Volker D, Roskar S, Grum AT, et al. Best practice elements of multilevel suicide prevention strategies: a review of systematic reviews. Crisis. 2011;32(6):319–33 Epub 2011/09/29. PubMed PMID: 21945840; PubMed Central PMCID: PMCPMC3306243.

    Article  PubMed  PubMed Central  Google Scholar 

  113. Zwicker JG, Mayson TA. Effectiveness of treadmill training in children with motor impairments: an overview of systematic reviews. Pediatr Physical Ther. 2010;22(4):361–77. Epub 2010/11/12. PubMed PMID: 21068636.

    Article  Google Scholar 

  114. NIHR Centre for Reviews and Dissemination (CRD): Database of Abstracts of Reviews and Effects (DARE). .

  115. Canadian Agency for Drugs and Technologies in Health. Accessed 25 Feb 2020.

  116. Brunton G, Harden A, Rees R, Kavanagh J, Oliver S, Oakley A. Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention and qualitative studies. London: EPPI Centre; 2003.

    Google Scholar 

  117. Büssing A, Schnepp W, Ostermann T, Neugebauer E. Impact of yoga on chronic pain – a review. Eur J Integr Med. 2009;1:236–7.

    Article  Google Scholar 

  118. Christopher FS. Adolescent Pregnancy Prevention. Family Relations, Vol. 44, No. 4, Helping Contemporary Families (Oct., 1995), pp. 384–391. Accessed 25 Apr 2019.

  119. Clamp S, Keen J. The value of electronic health records: A literature review. Final report, December 2005. Yorkshire Centre for Health Informatics, University of Leeds. Available from: Accessed 25 Apr 2019.

  120. Erny-Albrecht K, Valentine WJ, Christensen J, Vestenbaek U, Palmer AJ. Sublingual immunotherapy in allergic rhinitis and asthma: a review of recent clinical evidence. J Appl Res. 2007;7(1):17–31.

    Google Scholar 

  121. Evans S, Subramanian S, Sternlieb D. Yoga as treatment for chronic pain conditions: a literature review. Int J Disabil Hum Dev. 2008;7:25–32.

    Article  Google Scholar 

  122. Horta BL, Bahl R, Martinés JC, Victora CG. Evidence on the long-term effects of breastfeeding : systematic review and meta-analyses: World Health Organization; 2007. Accessed 25 Apr 2019.

    Google Scholar 

  123. Institute for Safe Medication Practices US. ‘Principles of Designing a Medication Label for Injectable Syringes for Patient Specific, Inpatient Use’, original website (accessed by the Overview authors in March 2008; no longer accessible); updated website Accessed 27 Feb 2018.

  124. Nelson H, Westhoff C, Piepert J, Berg A. Screening for ovarian cancer: brief evidence update: AHRQ; 2004. Accessed 25 Apr 2019.

  125. Shekelle PG, Goldzweig CL. Costs and benefits of health information technology: an updated systematic review. Southern California Evidence-based Practice Centre RAND Corporation. Accessed 25 Apr 2019.

  126. Brunton G, Thomas J, Harden A, Rees R, Kavanagh J, Oliver S, et al. Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Health Educ J. 2005;64(4):323–38.

    Article  Google Scholar 

  127. ERIC - Education Resources Information Center.

  128. Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev. 2018;7(1):159. Epub 2018/10/14. PubMed PMID: 30314530; PubMed Central PMCID: PMCPMC6186052.

    Article  PubMed  PubMed Central  Google Scholar 

  129. Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol. 2006;6:33. Epub 2006/07/20. PubMed PMID: 16848895; PubMed Central PMCID: PMCPMC1557524.

    Article  PubMed  PubMed Central  Google Scholar 

  130. MEDLINE Data Changes - 2019. NLM Technical Bulletin, 2018 Nov-Dec. Accessed 6 Sept2019.

  131. National Library of Medicine: Search Strategy Used to Create the PubMed Systematic Reviews Filter. Accessed 20 Feb 2020.

  132. Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: Umbrella Reviews. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer's Manual: The Joanna Briggs Institute; 2017. Available from

    Google Scholar 

Download references


Zemzem Cinar and Jelena Heitz are gratefully acknowledged for contributing to the assessment of database inclusion. We thank Camilo Vergara for insights into the Epistemonikos search and classification methods.


This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations



SH performed database searches and collected data. KG designed the analysis methods, analysed and interpreted the data. KG and CL co-wrote the manuscript. DP conceived and supervised the research, and revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Käthe Goossen.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Dataset generated and analysed during the current study.

Additional file 2: Table S1.

Characteristics of included Overviews.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goossen, K., Hess, S., Lunny, C. et al. Database combinations to retrieve systematic reviews in overviews of reviews: a methodological study. BMC Med Res Methodol 20, 138 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: