This article has Open Peer Review reports available.
Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality
© Betrán et al; licensee BioMed Central Ltd. 2005
Received: 06 August 2004
Accepted: 28 January 2005
Published: 28 January 2005
Failure to be comprehensive can distort the results of a systematic review. Conversely, extensive searches may yield unmanageable number of citations of which only few may be relevant. Knowledge of usefulness of each source of information may help to tailor search strategies in systematic reviews.
We conducted a systematic review of prevalence/incidence of maternal mortality and morbidities from 1997 to 2002. The search strategy included electronic databases, hand searching, screening of reference lists, congress abstract books, contacting experts active in the field and web sites from less developed countries. We evaluated the effectiveness of each source of data and discuss limitations and implications for future research on this topic.
Electronic databases identified 64098 different citations of which 2093 were included. Additionally 487 citations were included from other sources. MEDLINE had the highest yield identifying about 62% of the included citations. BIOSIS was the most precise with 13.2% of screened citations included. Considering electronic citations alone (2093), almost 20% were identified uniquely by MEDLINE (400), 7.4% uniquely by EMBASE (154), and 5.6% uniquely by LILACS (117). About 60% of the electronic citations included were identified by two or more databases.
This analysis confirms the need for extending the search to other sources beyond well-known electronic databases in systematic reviews of maternal mortality and morbidity prevalence/incidence. These include regional databases such as LILACS and other topic specific sources such as hand searching of relevant journals not indexed in electronic databases. Guidelines for search strategies for prevalence/incidence studies need to be developed.
The importance of comprehensive search strategies to identify 'all relevant articles' when conducting systematic reviews has been long documented . Comprehensive strategies include systematic searching of multiple databases as well as hand searching and contacting relevant experts. These strategies, however, may yield thousands of citations from which only a small number is eventually included in the review. Knowing which sources yield a reasonable number of relevant studies on a health topic may help those planning and conducting systematic reviews in that particular area.
At the World Health Organization (WHO), we have conducted a systematic review of prevalence/incidence of maternal mortality and morbidities from 1997 to 2002. The primary objective of this review is to assist in mapping the burden of reproductive ill-health by providing a comprehensive, standardized and reliable tabulation of data on the prevalence/incidence of maternal morbidity and mortality . This article evaluates the usefulness of different sources in identifying data for the systematic review. We also discuss limitations of the databases and implications for future systematic reviews of observational studies.
Results of the various types of searches
Number unique included*
All Electronic databases**
We downloaded all citations identified by electronic databases into Reference Manager® software. Selection of eligible studies involved two stages; the first stage consisted of screening of title and/or abstracts from the citations downloaded. Citations were excluded if any of the following applied: (i) data collection dates were not reported, (ii) data were collected only before 1990, (iii) part of the data were collected before 1980 and disaggregation by year was not possible (in order to exclude data before 1990), (iv) number of study participants was less than 200, (v) the study design was case-control and incidence/prevalence estimates from the defined population cannot be calculated, (vi) the methodology was not described. The second stage consisted of full-text evaluation of those citations that were not excluded in the first step applying the same criteria. For practical reasons, studies identified by other sources were only entered in Reference Manager® if they were eventually included in the review.
We recorded the source or sources of each citation. For each source, we calculated number of citations identified, included, and number included that were unique to the source. The sensitivity of each source was defined as the number of included citations identified by the source over the total number included. The precision was defined as the number of included citations identified by a source over the number of both included and excluded citations identified by that source. This review had no language restrictions and we recorded the language in which the report was written. Detailed results regarding languages will be reported separately.
Two reviewers performed the screening for all citations. In order to estimate the level of disagreement between the two reviewers within 2.5% of the true value, they independently screened title/abstracts for a sample of citations (560). This sample size assumes a 95% confidence interval and that the level of disagreement between the two reviewers will not exceed 10% . The percentage of agreement was 88.9% (95% CI 86.0% to 91.4%). The inter-observer agreement beyond chance was calculated using the Kappa statistics and found to be 0.60 (95% CI 0.52 to 0.69). This value corresponds to moderate to substantial agreement between the reviewers.
For the time period from 1997 to 2002, 64098 different citations from electronic databases were identified of which 2093 were included. Additionally, 487 citations were included from other sources. There were 92 citations for which we could not obtain full text and, therefore, they could not be assessed (2% of those that required full-text evaluation).
Table 1 shows a breakdown of the results by source including, for each source of data, number of citations identified, number included, number included unique to each source, sensitivity and precision. Overall, electronic databases identified four fifths (81.1%) of the included citations. MEDLINE and EMBASE which identified about 62% and 44% of the included citations respectively were most sensitive. Considering electronic citations alone (2093), almost 20% were identified uniquely by MEDLINE (400), 7.4% uniquely by EMBASE (154), and 5.6% uniquely by LILACS (117). About 60% of the electronic citations included were identified by 2 or more databases.
Overall, in terms of precision, we included 1 in 25 screened citations (4%). The most precise database was BIOSIS where 13.2% of the screened citations were included; IMEMR, LILACS and Popline followed this with 10.1%, 9.4% and 9.1%, respectively (see Table 1). MEDLINE and EMBASE were similarly precise, 4.1% and 4.9%, respectively.
Preliminary analysis regarding languages revealed that about 20% of the included studies were published in other languages than English. Spanish and French were, after English, the most used languages for reporting.
Failure to identify relevant information in systematic reviews can result in bias . The importance of including other sources of data in addition to electronic databases in general and MEDLINE in particular has been documented, especially for clinical or randomised controlled trials [1, 9–11]. On the other hand, search strategies for systematic reviews of observational data on morbidities are less precise, more difficult to narrow the focus and have been studied to a lesser extent . This led us to perform for our systematic review an extensive search strategy that is highly sensitive but barely precise. From 64098 citations identified only 2580 were included which represents 4% of the scrutinized articles.
Although MEDLINE identified about 62% of all citations and 76% of electronic citations relevant for this review, sources of data other than the major electronic databases are confirmed to be crucial. Some 487 (one fifth) citations were identified by reference lists of articles, expert contacts, congress proceedings, abstract books, hand searching of journals available in libraries that are not indexed in electronic databases, and other emerging databases in developing countries. As expected, there has been a large overlap between databases: 60% were identified by two or more databases and about 44% were identified by MEDLINE and EMBASE together. These two databases also provided the largest number of unique citations and both are considered necessary. PAIS International and Econlit only identified 3 and 1 citations, respectively, that were not identified by any other database and they could probably be disregarded in future reviews.
The nature of this systematic review with its focus on settings where burden of disease is highest necessitates extensive searching of developing country sources. However, literature from developing countries is difficult to access and it is not well represented in MEDLINE or other well-known electronic databases [13, 14]. An editorial by Zielinksi in 1995 stated that only 2% of the journals indexed in MEDLINE or the Science Citation Index were from developing countries . In 2004, the situation was similar. We calculated the number of journals published in developing countries and also indexed in MEDLINE to be about 6%. In 1996, the whole Latin American continent accounted for 0.39% of the total number of articles included in MEDLINE, down from a high of 2.03% in 1966 . One of the reasons for this is the indexing of journals on a priority system where the impact factor of a journal influences its chances of being indexed. This results in country bias since western journals have in general higher impact factors, and they are therefore more likely to be indexed than those from developing countries.
The value of LILACS database to improve the quality of systematic reviews has been previously reported . Our analysis confirms LILACS as a unique source of information for the Latin America and the Caribbean region that is not covered in other databases (117 unique citations included). Unfortunately, specific databases for other less developed regions like Asia and Africa, are just emerging or their access and functioning limited (e.g. AIM, IMEMR, IndMED, HELLIS.ORG). Although these regional databases are included in the review, the results are not presented individually but under 'other' in Table 1. With these databases, we experienced language barriers, difficulty in obtaining abstracts and full-text reports, inconsistencies, lack of essential information from the citation (e.g. year or title missing) and other technical problems. We believe that the low number of citations identified by IMEMR (see Table 1) is due more to the limitations mentioned above than to lack of data. These regional databases provide unique relevant citations and incomplete access limits their usefulness. Strengthening the functionality and improving the search facility of these databases could provide substantial relevant information.
A limiting factor for identifying citations is related to late indexing of journals in electronic databases. Search strategies for this review were conducted in early 2003 to identify articles published in 2002 or earlier. While only few articles published in 2002 could be expected not to be in the databases by 2003, some articles published in 1997 were only appearing in the databases as late as 2003. Traditionally, EMBASE has been found to index faster than MEDLINE, thus supporting the argument to search multiple databases . Furthermore, each database producer has a particular schedule that the searcher needs to be aware of. For example, MEDLINE available through OVID, due to the updating of the MeSH terms by the National Library of Medicine, will cease entry of new citations in November and only update the database in January of the following year. These factors need to be considered in assessing the yield from different databases. It is necessary to determine how to capture these 'late indexed' citations whether by delaying the running of the search or building into follow-up studies the need to capture these citations. The electronic search for this review would have probably captured more citations had it been run in 2004.
This systematic review involved significant financial and human resources over a 3-year period . Screening of a large number of citations and retrieving the full text of about 5000 articles have resource implications that need to be balanced with the benefits of the results. For this type of reviews, decisions on the extent of the comprehensiveness of the search strategy should take the resource implications into account. A careful selection of databases to be used and tailored search strategies for each database would help to maximise the benefits compared to costs.
1. In contrast with RCTs, guidelines for search strategy of systematic reviews of observational studies in general, and incidence/prevalence studies in particular, need to be developed.
2. Searching beyond the major electronic databases such as MEDLINE or EMBASE is necessary to identify studies in journals from less developed countries.
3. Regional databases indexing citations from local journals not reaching MEDLINE are especially relevant for this type of systematic review. They need to be fully functioning and made worldwide accessible. A network for accessing the full text of these articles would be helpful.
We thank Dr Paul Van Look for his comments and support.
- Dickersin K, Scherer R, Lefebvre C: Systematic reviews: Identifying relevant studies for systematic reviews. BMJ. 1994, 309: 1286-1291.View ArticlePubMedPubMed CentralGoogle Scholar
- Villar J, Betrán AP, Gülmezoglu AM, Say L: WHO leads global effort on systematic reviews. Int J Epidemiol. 2003, 32: 164-165. 10.1093/ije/dyg119.View ArticlePubMedGoogle Scholar
- Gülmezoglu AM, Say L, Betrán AP, Villar J, Piaggio G: WHO systematic review of maternal mortality and morbidity: methodological issues and challenges. BMC Medical Research Methodology. 2004, 4: 16-10.1186/1471-2288-4-16.View ArticlePubMedPubMed CentralGoogle Scholar
- Index Medicus for the Eastern Mediterranean Region. [http://www.who.int/library/country/regional/imemr]
- African Index Medicus. [http://www.who.int/library/country/regional/aim]
- IndMed. [http://indmed.nic.in]
- HELLIS.ORG. [http://www.hellis.org]
- Machin D, Campbell M, Fayers P, Pinol A: Observer agreement studies. Sample size tables for clinical studies. 1997, Oxford, Blackwell Science Ltd, 11: 287-195. Second editionGoogle Scholar
- Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P: Should meta-analysts search EMBASE in addition to Medline?. J Clin Epidemiol. 2003, 56: 943-955. 10.1016/S0895-4356(03)00110-0.View ArticlePubMedGoogle Scholar
- Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M: A comparison of handsearching versus MEDLINE searching to identified reports of randomized controlled trials. Stat Med. 2002, 21: 1625-1634. 10.1002/sim.1191.View ArticlePubMedGoogle Scholar
- Robinson KA, Dickersin K: Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol. 2002, 31: 150-153. 10.1093/ije/31.1.150.View ArticlePubMedGoogle Scholar
- Dickersin K: Systematic reviews in epidemiology: why are we so far behind?. Int J Epidemiol. 2002, 31: 6-12. 10.1093/ije/31.1.6.View ArticlePubMedGoogle Scholar
- Smith R: Publishing research from developing countries. Stat Med. 2002, 21: 2869-2877. 10.1002/sim.1291.View ArticlePubMedGoogle Scholar
- Horton R: North and South: bridging the information gap. Lancet. 2000, 356: 1036-View ArticleGoogle Scholar
- Zielinski C: New equities of information in an electronic age. BMJ. 1995, 310: 1481-View ArticleGoogle Scholar
- Rosselli D: Latin America biomedical publications: the case of Colombia in Medline. Med Educ. 1998, 32: 274-277. 10.1046/j.1365-2923.1998.00214.x.View ArticlePubMedGoogle Scholar
- Camara Clark OA, Araujo Castro A: Searching the Literatura Latino Americana e do Caribe em Ciências da Saúde (LILACS) database improves systematic reviews. Int J Epidemiol. 2002, 31: 112-114. 10.1093/ije/31.1.112.View ArticleGoogle Scholar
- Biomedical Literature Searching: A comparison of BIOSIS Previews, EMBASE, and Medline. [http://www.biosis.org/training-support/documentation/bp_medl_embase.doc]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/5/6/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.