Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review
© Crumley et al; licensee BioMed Central Ltd. 2005
Received: 23 February 2005
Accepted: 10 August 2005
Published: 10 August 2005
Systematic reviewers seek to comprehensively search for relevant studies and summarize these to present the most valid estimate of intervention effectiveness. The more resources searched, the higher the yield, and thus time and costs required to conduct a systematic review. While there is an abundance of evidence to suggest how extensive a search for randomized controlled trials (RCTs) should be, it is neither conclusive nor consistent. This systematic review was conducted in order to assess the value of different resources to identify trials for inclusion in systematic reviews.
Seven electronic databases, four journals and Cochrane Colloquia were searched. Key authors were contacted and references of relevant articles screened. Included studies compared two or more sources to find RCTs or controlled clinical trials (CCTs). A checklist was developed and applied to assess quality of reporting. Data were extracted by one reviewer and checked by a second. Medians and ranges for precision and recall were calculated; results were grouped by comparison. Meta-analysis was not performed due to large heterogeneity. Subgroup analyses were conducted for: search strategy (Cochrane, Simple, Complex, Index), expertise of the searcher (Cochrane, librarian, non-librarian), and study design (RCT and CCT).
Sixty-four studies representing 13 electronic databases met inclusion criteria. The most common comparisons were MEDLINE vs. handsearching (n = 23), MEDLINE vs. MEDLINE+handsearching (n = 13), and MEDLINE vs. reference standard (n = 13). Quality was low, particularly for the reporting of study selection methodology. Overall, recall and precision varied substantially by comparison and ranged from 0 to 100% and 0 to 99%, respectively. The trial registries performed the best with median recall of 89% (range 84, 95) and median precision of 96.5% (96, 97), although these results are based on a small number of studies. Inadequate or inappropriate indexing was the reason most cited for missing studies. Complex and Cochrane search strategies (SS) performed better than Simple SS.
Multiple-source comprehensive searches are necessary to identify all RCTs for a systematic review, although indexing needs to be improved. Although trial registries demonstrated the highest recall and precision, the Cochrane SS or a Complex SS in consultation with a librarian are recommended. Continued efforts to develop CENTRAL should be supported.
The aim of systematic reviews is to present the most valid estimate of the effectiveness of the intervention in question. To do so, the identification of relevant studies must be comprehensive and unbiased. Systematic reviews usually include a comprehensive summary of data from both randomized (RCT) and controlled trials (CCT), although other study designs are sometimes incorporated. There is an ongoing debate about the number and type of resources that should be used to identify trials for systematic reviews [1–3]. These resources include electronic databases, the Internet, handsearching, checking relevant article references, and personal communication with experts in the field. Reviewers are encouraged to search numerous resources in order to identify as many relevant studies as possible without systematically introducing bias . However, searching more resources typically results in a higher yield; thus, more time and resources are required to conduct the review . Consequently, determining the relative value of different sources of trials is critical to enhance the efficiency of systematic reviews, while maintaining their validity.
The Cochrane Collaboration Reviewers' Handbook notes that MEDLINE, EMBASE, and CENTRAL are the three electronic bibliographic databases generally considered as the richest sources of trials . The Cochrane Collaboration maintains that handsearching is vital to the credibility and success of systematic reviews . Hopewell et al. conducted a systematic review of studies that compared handsearching to searching an electronic database for RCTs . In the 34 included studies, they found that complex searches of electronic databases recalled 65% of relevant RCTs; the other 35% were retrieved in other ways including handsearching. They concluded that "handsearching still has a valuable role to play in identifying reports of randomized controlled trials for inclusion in systematic reviews of health care interventions, particularly in identifying trials reported as abstracts, letters and those published in languages other than English, together with all the reports published in journals not indexed in electronic databases" .
Our research question was: Does resource-specific searching retrieve RCT/CCTs for systematic reviews with the same recall and precision as those searches which combine two or more distinct resources? Our primary goal was to identify and quantitatively review studies comparing two or more different resources (e.g., databases, Internet, handsearching) used to identify RCTs and CCTs for systematic reviews. Specifically, we were interested in determining the value (in terms of identifying unique citations) of searching key resources (e.g., EMBASE, CENTRAL, PsycINFO, handsearching) in addition to the key resource MEDLINE.
Seven electronic databases (MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, Web of Science, Cochrane Library) were searched from their inception to April 2004. The MEDLINE search strategy was tailored as necessary for each database (Appendix 1). Four journals were handsearched from 1990 to 2004: Health Information & Libraries Journal (Health Libraries Review), Hypothesis, Journal of the Medical Library Association (Bulletin of the Medical Library Association), Medical Reference Services Quarterly. All abstracts presented at Cochrane Colloquia (1993–2003) were handsearched. In addition, key authors were contacted via email and references of relevant articles were screened. The searches were not limited by language or date of publication. Searches are available upon request from the corresponding author.
Two reviewers independently screened the yield from the searches to identify potentially relevant studies. The full text of these studies was obtained and two reviewers independently applied inclusion/exclusion criteria using a standard form. Any differences were resolved through discussion. Studies were included if they compared two or more sources to find RCTs or CCTs (e.g., one or more resources compared against a "gold standard"; handsearch versus MEDLINE; and, EMBASE versus MEDLINE). Inclusion was not limited by the topic/content area in the individual studies.
A study was excluded if: it only compared different search strategies within the same database; it only included non-randomized trials; or, if the resource is not currently accessible. If authors searched for all study designs including trials, it was included only if data were reported separately for RCTs or CCTs.
RCTs were defined as an experiment in which eligible patients are assigned to two or more study groups using an appropriate method of randomization . CCTs were defined similarly except that the method of allocation is not necessarily random. When authors did not provide definitions, we accepted their classification/indication of study design.
Assessment of quality
We developed a checklist to assess the quality of reporting of the included studies. The quality items were chosen based on threats to the validity of comparative studies that have been empirically supported in the literature . The items assessed reporting in four key areas:
Was there an adequate description of what the search was attempting to identify (e.g., type of studies, content area, standard inclusion/exclusion criteria);
Was there an adequate description of the methods used to search (e.g., resource(s), words/subject headings used, time period covered, date of search);
Was there an adequate description of the reference standard (e.g., how many references) and how it was derived (e.g., sources searched and methods used); and,
Was bias avoided in the selection of relevant studies (e.g., was there an independent assessment of studies for inclusion by more than one researcher).
Two reviewers independently applied the checklist to the included studies. Discrepancies were resolved through discussion.
Data were extracted independently by one investigator and checked by a second independent investigator. A standard form was used to extract the following information: language of publication, country where study was conducted, year of publication, study design and objective(s), resources being compared, topic being searched, years the search covered, search strategies used, results (yield, recall, precision, reasons studies were missed), and author's conclusions. When data were not available, authors were contacted and asked to supply the missing information.
Data were analyzed using Splus 6.2 (Insightful Corporation 2003). Recall and precision were expressed as percentages with 95% confidence intervals which were calculated using exact methods . Recall is the percentage of relevant studies that were identified by the search. Precision is the percentage of studies identified by the search that were relevant. Results were grouped by comparison (e.g., MEDLINE versus handsearching, MEDLINE vs. other reference standard). Meta-analysis was not performed due to large heterogeneity. Comparisons, however, were summarized using medians and ranges. With regards to the independence of the results, we conducted a sensitivity analysis around the inclusion of duplicate topics. Duplicate topics for exclusion were randomly selected and the median and range summaries were re-calculated.
Possible sources of heterogeneity were explored with numerical summaries both by within-study and between-study analyses. Within-study analyses are direct analyses and they occur when two searches are conducted with the same known conditions (i.e., strategy, expertise of the searcher, topic of the search, type of design) and some unknown conditions except for the condition or variable of interest. These are also called direct analyses. Between-study analyses are indirect analyses and are of a lower grade  in that there is a stronger potential for known variables (i.e., topic, strategy, author of search, topic of search, type of design) to confound results. The variables of interest we explored were: search strategy (Cochrane, Simple, Complex, Index), expertise of the searcher (Cochrane, librarian, non-librarian), and study design (RCT and CCT).
Searches were divided into the following four categories (modified from Hopewell et al. ): Complex (using a combination of types of search terms); Cochrane (the Cochrane Highly Sensitive Search Strategy (HSSS)); Simple (using five or fewer search terms which may include a combination of MeSH, Publication Type, keywords); and, Index (using one or two search terms (usually author's name or article title) to check/verify if the study is in a database).
Description of included studies
The included studies searched a variety of topics which fell into four major categories: journal (e.g., Lancet, BMJ), disease/condition/state (e.g., hepatitis, rheumatoid arthritis), specialty/sub-specialty (e.g., rehabilitation, pediatrics) and methodology (e.g., search strategies) [see Additional file 1]. Generally, the objectives of the studies were: to compare different searches (e.g., handsearch vs. database); to determine recall/precision of a search; to handsearch a journal and check to see if trials were in a database; or to develop a trial register.
The reference standards varied (e.g., handsearching, handsearching plus MEDLINE, MEDLINE plus EMBASE plus other databases). The specific study design for which authors were searching varied by study: RCTs only (n = 27); RCTs and CCTs (n = 28); and RCTs, CCTs, and other designs (n = 9).
Number of Studies†
Number of Comparisons
Number of Studies†
Number of Comparisons
MEDLINE vs. Handsearching
53 (7,97) 58 (7,97)
35 (0.03,99) 31 (0.03,78)
MEDLINE vs. MEDLINE + Handsearching
70 (18,90) 75 (18,90)
49 (13,83) 40 (13,83)
MEDLINE vs. Reference Standard
59 (17,98) 58 (17,98)
27 (1,91) 30 (1,91)
EMBASE vs. Handsearching
EMBASE vs. Reference Standard
72 (13,100) 68 (13,100)
28 (0,48) 26 (0,48)
PsycINFO vs. Handsearching
PsycINFO vs. Reference Standard
50 (0,65) (38,61)
39 (36,47) (36,47)
Trial Registries vs. Reference Standard
CINAHL vs. Reference Standard
Biosis vs. Reference Standard
CancerLIT vs. Handsearching
Cabnar vs. Reference Standard
CENTRAL vs. Reference Standard
Chirolars vs. Reference Standard
HealthStar vs. Reference Standard
Internet vs. Reference Standard
SciCitIndex vs. Reference Standard
Methodological quality of included studies
All studies indicated the type of study design for which they were searching and all but one specified the topic area. Most (70%) stated their inclusion/exclusion criteria. Eighty percent of the studies described (or indicated they were available) reproducible search strategies/methods. Half of the studies stated who developed the search strategies and of these only 2 did not provide reproducible information about their search strategy. Eighty-five percent of articles fully described how the reference standard was compiled. Twenty-five percent reported that at least 2 people independently screened searches and in 35% at least 2 people also independently applied eligibility criteria.
Table 1 summarizes the results of the comparisons (e.g., MEDLINE versus handsearching, MEDLINE vs. other reference standard). Thirteen databases (including the Internet) were included in the numerical results. The results from 7 studies could not be used in the data analysis for the following reasons: did not use same journals for handsearch and MEDLINE ; insufficient data available (usually because it was an abstract) [14–18]; reporting flaw(s) which could not be clarified by author(s) .
Forty-nine studies had usable data for the MEDLINE comparisons. Three comparisons were analyzed: MEDLINE versus handsearching (41 comparisons, 23 studies), MEDLINE versus MEDLINE plus handsearching (16, 13), and MEDLINE versus other reference standards (24, 13). Estimates of both recall and precision for all three comparisons varied substantially, ranging from 7 to 98% and 0.03 to 99%, respectively. The estimates for MEDLINE versus handsearching and MEDLINE versus a reference standard were comparable: median of 53 versus 59% for recall and 35 versus 27% for precision. Median recall and precision for MEDLINE versus MEDLINE plus handsearching were somewhat larger (70 and 49%, respectively).
Eleven studies had usable data for the EMBASE comparisons. Two comparisons were analyzed: EMBASE versus handsearching (2 comparisons, 2 studies) and EMBASE versus a reference standard (14, 9). Individual study estimates ranged from 13 to 100% for recall and 0 to 48% for precision. Summarizing all studies, medians were 65 and 72% for recall and, 13 and 28% for precision.
Four studies contained data for the PsycINFO comparisons. Two comparisons were analyzed: PsycINFO versus handsearching (2 comparisons, 2 studies), and PsycINFO versus a reference standard (4, 2). Recall ranged from 0 to 70%. Precision ranged from 8 to 47%. Medians were 69 and 50% for recall and 9 and 39% for precision.
Two studies investigated CINAHL and trial registers. Only one study had usable data from other databases (i.e., BIOSIS, CancerLit, CABNAR, CENTRAL, Chirolars, HealthSTAR, the Internet, SciCitIndex). The results from the trial registries versus a reference standard were consistent and high: 89% for median recall and 97% for median precision. The remaining comparisons ranged from 0 to 92% for median recall and 0 to 17% for median precision. Regardless of topic, there were too few included studies in these comparisons for this data to be representative.
Direct Subgroup Analyses
Recall (95% CI)
Precision (95% CI)
Simple versus Complex
Simple versus Cochrane
McDonald unpub a
McDonald unpub b
Complex versus Cochrane
Author versus Librarian
There were five direct comparisons of a Simple search strategy versus the Cochrane HSSS. Again, all but one of the comparisons  had larger sensitivities for the Cochrane search strategy. None of these four comparisons reported precision. Fergusson  found negligible differences for both recall and precision.
Indirect Subgroup Analyses: Recall
MEDLINE vs. Handsearching
MEDLINE vs. MEDLINE + Handsearching
MEDLINE vs. Reference Standard
Indirect Subgroup Analyses: Precision
MEDLINE vs. Handsearching
MEDLINE vs. MEDLINE + Handsearching
MEDLINE vs. Reference Standard
Only one study  directly compared search strategies from two different authors (i.e., a librarian versus a non-librarian). In this one example, the librarian's Complex search had a recall of 53% and the non-librarian's Complex search had a recall of 34. For the indirect subgroup comparisons, the librarians did not systematically outperform the non-librarians on either median recall or median precision; however the Cochrane HSSS (as author) did outperform the librarians and non-librarians on median recall (67, 81 and 78%).
No studies directly compared searching for RCTs versus CCTs. Based on indirect comparisons, the three MEDLINE comparisons showed no systematic difference in median recall and precision between design types.
A sensitivity analysis excluding duplicate topics was performed due to the concern for non-independence between studies. Studies or comparisons searching on the same topic may include some of the same relevant studies. We picked one comparison randomly from each topic area and eliminated it from the main quantitative results. The results are shown in Table 1. We found that recall had similar ranges and medians signifying that non-independence was not distorting our results.
Reasons for missed trials
Reasons why studies were missed by electronic search and handsearch
Studies citing reason (n = 42)
Inadequate indexing in general
Journal not indexed in resource
Not indexed with proper methodologic terms
Not indexed as RCT/CCT
Not indexed for any of the methodologic terms used by searchers
Database does not contain relevant index terms
Not indexed with any of the subheadings used in searches
Insufficient or restricted search strategy
Keywords or methodology not reported by author
Issue(s) omitted from resource
Time lag in updating resource
Article(s) omitted or missing from resource
Published before database was created or beyond coverage years of database
Published as abstracts, books, book reviews, brief reports, letters, proceedings, supplements, etc.
• Wrong abstract assigned to reference
• Journals do not encourage authors to explicitly report methodology
• Length and complexity of search is limited using search engines
• Ongoing or recently finished studies
• Found through references
• Missed most Japanese reports
• Some personal communication did not get responses; organizations and people must be willing to supply trial information
• Not all abstracts are published in full
• Authors misspelled gingko
• Many RCTs are not identifiable in database
Handsearchers not trained properly
Methodologic terms "hidden" in article
Searcher fatigue, boredom
Journal not handsearched
Article not an RCT or CCT
Different topic than what handsearchers were looking for
In this study, seven studies performed Index searches. Six comparisons using the Index searches were found for MEDLINE versus handsearching. The median recall was 93% and the range was 41 to 100%. On average, 7% of studies were not indexed adequately. One study compared MEDLINE to a reference standard, their Index search produced 66% of the included studies. Two further Index searches were performed: EMBASE versus handsearching and PsycINFO versus handsearching; their recalls were 52 and 97%, respectively.
For handsearching, few authors provided information for why trials were missed. Handsearches had high precision and some studies did not miss any references through their handsearches. In the 3 studies where handsearchers missed studies, authors reported the reasons for missing studies were the handsearchers were not trained properly or they had fatigue/boredom. In two studies where trials were missed, authors reported that the journal was not handsearched, yet a database was used to search for this same journal. One of the missed articles was misclassified by a handsearcher as an RCT/CCT and one had a different topic than what handsearchers were meant to identify. Results from the MEDLINE versus MEDLINE plus handsearching comparisons quantify the percentage of trials missed by handsearching. In 13 studies, the median percentage found in MEDLINE but not by handsearching was 6% (range 1 to 15%).
For certain topics trial registries may be sufficient (e.g., perinatology, Japanese), however, the median recall estimates (Table 1) were not large enough to support single-source searches. These data highlight the importance of searching multiple sources when conducting a systematic review. Initiatives to compile references from different sources, such as CENTRAL and other trial registries, need to receive continued encouragement and support in order to eliminate the need for multiple-source search endeavors.
Over and above the recalls, the median precisions are quite low and indicate a need for improved indexing in databases. Efforts to improve and standardize the indexing of various databases need to be supported. Guidelines for journals and authors regarding the reporting of key methodological or subject terms when publishing studies would facilitate these efforts . In addition to very poor precisions, the authors of our included studies reported precisions or the data necessary to calculate precision only 40% (47/117 comparisons) of the time.
Most of the research has involved MEDLINE and EMBASE, two of the major databases that the systematic review community recommends reviewers search. However, searching multiple databases can be difficult, time consuming and costly. For example, although MEDLINE is available freely on the Internet through PubMed, EMBASE is very costly and many institutions do not subscribe to it. This is of particular concern as studies have demonstrated that there is 17 to 75% overlap between MEDLINE and EMBASE [3, 23, 24] indicating that EMBASE may yield a substantial number of unique articles. To date, the gold standard for conducting systematic reviews still remains searching multiple bibliographic databases and hand searching. In addition, using only MEDLINE for systematic reviews still results in important trials being missed thereby compromising the external validity of the review.
Optimally, it would be most efficient to search few resources, retrieve a maximum yield of relevant trials, and retrieve a minimum yield of irrelevant trials. The Cochrane Collaboration is trying to achieve this with the Controlled Trials Register (CENTRAL) available through the Cochrane Library. The register now includes over 420,000 RCTs and CCTs. While there are numerous studies that discuss the vast amount of trials that have been added to CENTRAL through handsearching efforts, there are very few studies evaluating whether CENTRAL can be searched exclusively for RCT/CCTs. If one resource (e.g., CENTRAL) can be searched to identify RCT/CCTs, this would substantially reduce the time and costs associated with searching.
There was extensive heterogeneity among topics investigated in the studies included in this review. For the comparisons which had many studies, the values for both recall and precisions covered most of the possible range (e.g., 0–100). Thus, the topic searched may be the strongest determinant of the results. Topics are indexed differentially within and across various sources. Due to the between-study heterogeneity, very little can be concluded about the indirect subgroup results.
Over the 20 years that this review covers, it was noted that the older studies were conducted prior to indexing improvements in resources, especially MEDLINE. While there have been numerous changes in search technology in the past two decades, upon conducting post-hoc sub-group analyses, no difference was found. In addition, a sub-group analysis was done of recent studies and no difference was found when compared to the results of older studies. As mentioned above, this may be due to topic heterogeneity, not changing search technology. Thus, including the older studies did not confound our results and did not lead to the conclusion that there is one sufficient resource which identifies RCT/CCTs. Unfortunately, the more recent studies are not showing results which significantly differ from the ones obtained 20 years ago.
We found that, generally, both Complex and Cochrane search strategies performed better in recall than did Simple search strategies without loss in precision. However, the indirect subgroup results for recall showed support for this finding for the Cochrane search strategies, but not for the Complex search strategies. The Cochrane search strategy precisions were poorer in the indirect subgroup results, however data were sparse. Little direct evidence was available comparing searchers with different expertise. No direct evidence was available comparing searches for different design types. Without supporting direct subgroup evidence, conclusions from the indirect subgroup evidence would be too speculative. Other reasons that may explain the heterogeneity include: 1) the time period covered by the search (indexing as well as other search technologies have progressed over time which would affect the accuracy of searches); and 2) the methodological quality of the study. For example the rigor with which handsearching was done, searches were screened, or inclusion criteria applied (e.g., having 2 people independently perform each step) would affect the comparability of results across studies.
The quality of the included studies varied and, in most cases, the poor quality result was due to the lack of rigor in the reporting of the selection methodology. One-third reported that standard inclusion and exclusion criteria were developed and applied to each database/method at the relevance stage. Almost all studies did not indicate that at least 2 people independently screened the searches for potentially relevant studies. In addition, two-thirds of the included studies did not indicate that at least 2 people independently applied eligibility criteria to identify relevant studies. Quality of these studies can be improved by adopting more stringent methodology and reporting its use.
Post-hoc, we looked at how the calculated results of recall and precision may have improved over the last two decades considering the changes in search technology (in particular, indexing). Within our three MEDLINE comparisons, we found no pattern of association between year of publication and results. The correlations ranged from -0.91 to 1. As mentioned above, this quantitative analysis may be too dilute due to topic heterogeneity. We suggest a within topic analysis to robustly test for improvements in search technology.
This paper provides the most current and comprehensive review of the existing evidence comparing any electronic database against any other source or combination of sources. This and previous reviews demonstrate that there is a dearth of evidence regarding the use of different databases to retrieve RCTs, with the notable exception of handsearching and MEDLINE. Therefore, searching multiple resources to retrieve RCTs cannot be ruled out based upon this evidence. There needs to be more research done on major databases such as: EMBASE, CENTRAL, PsycINFO and trial registries in order to gather more information about the value of these databases in identifying RCT/CCTs for various topical areas. This review is more comprehensive than previous work in this area  and reflects the different ways that searches can be conducted (i.e., using a variety of databases and types of searches). Moreover, while a previous review focused upon between-study subgroup comparisons  (e.g., Complex versus Simple search strategies), we also systematically examined the within-study subgroup comparisons which provides more valid information . However, similar to negative clinical trials, it is important recognize the limitations of current resources and the implications for decision-making.
There are several limitations to this study. Foremost, there is a lack of a validated quality score for this type of study (i.e., comparative). Reference standards are difficult to compare as they are generally different and may not be reported in enough detail to be reproducible. As well, the topic chosen to search can determine the success of the strategy. In addition, there are limitations to using precision and recall which are addressed by Kagolovsky and Moehr .
Implications for practice
Since recall is low with single resources, multiple-source comprehensive searches continue to be necessary. The Cochrane search strategy or Complex search strategy in consultation with a librarian are recommended.
Implications for research
Efforts to enhance and build CENTRAL, a large trial registry, need to be continued. A number of the resources used to find trials for CENTRAL (e.g., journals, grey literature) are not indexed in MEDLINE, therefore CENTRAL has a significant amount of unique information not found in any other source. CENTRAL is also free for researchers in developing countries and available in CD-ROM and on the internet. Based upon the results of why studies were missed, indexing efforts also need to improve. Guidelines should be provided for authors to include MeSH terms and keywords in their abstracts which can then be used by indexers. Other resources that need to be studied include: Trial registries, LILACS, PsycINFO, Science Citation Index, BIOSIS, CABNAR and CINAHL. In addition, those researchers studying searches need to report precision results.
Appendix 1: MEDLINE Search Strategy
(psyclit or psycinfo or psychlit or psychinfo).mp.
"web of science".mp.
"system for information on grey literature in europe".mp.
"science citation index".mp.
"science citation abstracts".mp.
("cochrane controlled trials register" or central or cctr).mp.
"database of abstracts of reviews of effectiveness".mp.
"world wide web".mp.
"institute of scientific information".mp.
"inside information plus".mp.
"international pharmaceutical abstracts".mp.
(dare and cochrane).mp.
(handsearch$ or "hand search$").mp.
(controlled adj2 trial$).mp.
(randomized controlled trial$ or randomised controlled trial$).mp.
(rct or cct).mp.
List of abbreviations used
controlled clinical trial
randomized controlled trial
We are indebted to ARCHE who provided financial assistance for this project when ETC and NW formerly worked with them. Thank you to Research Assistants Michelle Tubman and Philip Berry for assisting with data collection and study retrieval.
- Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P: Should meta-analysis search EMBASE in addition to MEDLINE?. J Clin Epidemiol. 2003, 56: 943-955. 10.1016/S0895-4356(03)00110-0.View ArticlePubMedGoogle Scholar
- Royle P, Waugh N: Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports (TAR) carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess. 2003, 7: 1-64.View ArticleGoogle Scholar
- Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C: Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000, 21: 476-487. 10.1016/S0197-2456(00)00067-2.View ArticlePubMedGoogle Scholar
- Locating and selecting studies for reviews In: The Cochrane Library, Issue 1, 2004. Cochrane Reviewers' Handbook 4.2.2 [updated December 2003]. Edited by: Alderson P, Green S, Higgins JPT. 2004, Chichester, UK: John Wiley & Sons, Ltd
- Allen IE, Olkin I: Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999, 282: 634-635. 10.1001/jama.282.7.634.View ArticlePubMedGoogle Scholar
- Locating and selecting studies for reviews In: The Cochrane Library, Issue 1, 2004. Cochrane Reviewers' Handbook 4.2.2 [updated December 2003]. Edited by: Alderson P, Green S, Higgins JPT. 2004, Chichester, UK: John Wiley & Sons, Ltd
- Dickersin K, Manheimer E, Wieland S, Robinson KA, Lefebvre C, McDonald S: Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials. Eval Health Prof. 2002, 25: 38-64. 10.1177/0163278702025001004.View ArticlePubMedGoogle Scholar
- Hopewell S, Clarke M, Lefebvre C, Scherer R: Handsearching versus electronic searching to identify reports of randomized trials (Cochrane Methodology Review). The Cochrane Library. 2004, Chichester, UK: John Wiley & Sons, Ltd, 3
- Jadad AR, Moore RA, Carroll D: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.View ArticlePubMedGoogle Scholar
- Lijmer JC, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM: Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999, 282: 1061-1066. 10.1001/jama.282.11.1061.View ArticlePubMedGoogle Scholar
- Fleiss JL: Statistical Methods for Rates and Proportions. 1981, New York, NY: John Wiley & Sons LtdGoogle Scholar
- Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278. 10.1016/0895-4356(91)90160-B.View ArticlePubMedGoogle Scholar
- Silagy C: Developing a register of randomised controlled trials in primary care. BMJ. 1993, 306: 897-900.View ArticlePubMedPubMed CentralGoogle Scholar
- Tsutani K, Sakuma A: How to access RCTs in Japan through network access from abroad. [poster]. Proceedings of the 2nd Annual Cochrane Colloquium. October 1994; Canada
- Thomas B, Gubitz G, McInnes A, Krabshuis J, Counsell C: A multifile search of five bibliographic databases for stroke trials. [poster]. Proceedings of the 6th Annual Cochrane Colloquium. October 1998; USA
- Hedger NA, Croft AMJ, Rowe M: Handsearching the Journal of the Royal Naval Medical Service for trials. J R Nav Med Serv. 1999, 85: 108-111.PubMedGoogle Scholar
- Veldhuyzen van Zanten SJO, Cleary C, Talley NJ, Peterson TC, Nyren O, Bradley LA, Verlinden M, Tytgat GNJ: Drug treatment of functional dyspepsia: a systematic analysis of trial methodology with recommendations for design of future trials. Am J Gastroenterol. 1996, 91: 660-673.PubMedGoogle Scholar
- Helmer D, Savoie I, Green C, Kazanjian A: Evidence-based practice: extending the search to find material for the systematic review. Bull Med Libr Assoc. 2001, 89: 346-352.PubMedPubMed CentralGoogle Scholar
- Kennedy G, Rutherford G: Identifying randomized controlled trials in the journal Aids. Proceedings of the 8th Annual Cochrane Colloquium. October 2000; South Africa
- Fergusson D, Laupacis A, Salmi LR, McAlister FA, Huet C: What should be included in meta-analyses? An exploration methodological issues using the ISPOT meta-analyses. International Journal of Technology Assessment in Health Care. 2000, 16: 1109-1119. 10.1017/S0266462300103150.View ArticlePubMedGoogle Scholar
- Kirpalani H, Schmidt B, McKibbon KA, Haynes RB, Sinclair JC: Searching MEDLINE for randomized clinical trials involving care of the newborn. Pediatrics. 1989, 83: 543-546.PubMedGoogle Scholar
- Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB: Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000, 283: 2008-2012. 10.1001/jama.283.15.2008.View ArticlePubMedGoogle Scholar
- Minozzi S, Pistotti V, Forni M: Searching for rehabilitation articles on MEDLINE and EMBASE. An example with cross-over design. Arch Phys Med Rehabil. 2000, 81: 720-722.View ArticlePubMedGoogle Scholar
- Topfer LA, Parada A, Menon D, Noorani H, Perras C, Serra-Prat M: Comparison of literature searches on quality and costs for health technology assessment using the MEDLINE and EMBASE databases. Int J Technol Assess Health Care. 1999, 15: 297-303.PubMedGoogle Scholar
- Kagolovsky Y, Moehr JR: Current status of the evaluation of information retrieval. J Med Syst. 2003, 27: 409-424. 10.1023/A:1025603704680.View ArticlePubMedGoogle Scholar
- Adams CE, Power A, Frederick K, Lefebvre C: An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994, 24: 741-748.View ArticlePubMedGoogle Scholar
- Bender JS, Halpern SH, Thangaroopan M, Jadad AR, Ohlsson A: Quality and retrieval of obstetrical anaesthesia randomized controlled trials. Can J Anaesth. 1997, 44: 14-18.View ArticlePubMedGoogle Scholar
- Dickersin K, Hewitt P, Mutch L, Chalmers I, Chalmers TC: Perusing the Literature: Comparison of MEDLINE searching with a perinatal trials database. Control Clin Trials. 1985, 6: 306-317. 10.1016/0197-2456(85)90106-0.View ArticlePubMedGoogle Scholar
- Marson AG, Chadwick DW: How easy are randomized controlled trials in epilepsy to find on MEDLINE? The sensitivity and precision of two MEDLINE searches. Epilepsia. 1996, 37: 377-380. 10.1111/j.1528-1157.1996.tb00575.x.View ArticlePubMedGoogle Scholar
- Brand M, Gonzalez J, Aguilar C: Identifying RCTs in MEDLINE by publication type and through the Cochrane Strategy: the case in hypertension. 1988, poster B22-Google Scholar
- McDonald SJ, Lefebvre C, Clarke MJ: Identifying reports of controlled trials in the BMJ and The Lancet. BMJ. 1996, 313: 1116-1117.View ArticlePubMedPubMed CentralGoogle Scholar
- Bernstein F: The retrieval of randomized clinical trials in liver diseases from the medical literature: manual versus MEDLARS searches. Control Clin Trials. 1988, 9: 23-31. 10.1016/0197-2456(88)90006-2.View ArticlePubMedGoogle Scholar
- Clarke M, McDonald S, Lefebvre C: Identifying reports of randomized controlled trials (RCTs) in the Lancet: the contribution of the Cochrane Collaboration. [poster]. Proceedings of the 4th Annual Cochrane Colloquium. October 1996; Australia
- Cullum N: Identification and analysis of randomised controlled trials in nursing: a preliminary study. Qual Health Care. 1997, 6: 2-6.View ArticlePubMedPubMed CentralGoogle Scholar
- Dickersin K, Scherer R, Lefebvre C: Identifying relevant studies for systematic reviews. BMJ. 1994, 309: 1286-1291.View ArticlePubMedPubMed CentralGoogle Scholar
- Duggan LM: Prevalence study of the randomized controlled trials in the Journal of Intellectual Disability Research: 1957–1994. J Intellect Disabil Res. 1997, 41: 232-237. 10.1046/j.1365-2788.1997.02626.x.View ArticlePubMedGoogle Scholar
- Dumbrigue HB, Esquivel JF, Jones JS: Assessment of MEDLINE search strategies for randomized controlled trials in prosthodontics. J Prosthodont. 2000, 9: 8-13. 10.1111/j.1532-849X.2000.00008.x.View ArticlePubMedGoogle Scholar
- Jadad AR, Carroll D, Moore A, McQuay H: Developing a database of published reports of randomised clinical trials in pain research. Pain. 1996, 66: 239-246. 10.1016/0304-3959(96)03033-3.View ArticlePubMedGoogle Scholar
- Kleijnen J, Knipschild P: The comprehensiveness of MEDLINE and EMBASE computer searches. Searches for controlled trials of homoeopathy, ascorbic acid for common cold and ginkgo biloba for cerebral insufficiency and intermittent claudication. Pharm Weekbl Sci. 1992, 14: 316-320.View ArticlePubMedGoogle Scholar
- Marti J, Bonfill X, Urrutia G, Lacalle JR, Bravo R: [The identification and description of clinical trials published in Spanish journals of general and internal medicine during the period of 1971–1995]. [Spanish]. Medicina Clinica. 1999, 112 (Suppl 1): 28-34.PubMedGoogle Scholar
- Nwosu CR, Khan KS, Chien PW: A two-term medline search strategy for identifying randomized trials in obstetrics and gynecology. Obstet Gynecol. 1998, 91: 618-622. 10.1016/S0029-7844(97)00703-5.PubMedGoogle Scholar
- Poynard T, Conn HO: The retrieval of randomized clinical trials in liver disease from the medical literature: a comparison of MEDLARS and manual methods. Control Clin Trials. 1985, 6: 271-279. 10.1016/0197-2456(85)90103-5.View ArticlePubMedGoogle Scholar
- Schlomer G: RCTs and systematic reviews in nursing literature: a comparison of German and international nursing research [German]. Pflege, 12(4):250-8, 1999 Aug. 1999, 12: 250-258.View ArticleGoogle Scholar
- Watson RJ, Richardson PH: Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of EMBASE, MEDLINE and PsycINFO bibliographic databases. Br J Med Psychol. 1999, 72: 535-542. 10.1348/000711299160220.View ArticlePubMedGoogle Scholar
- Bassler D, Galandi D, Forster J, Antes G: Handsearching in German paediatric journals.[poster]. Proceedings of the 8th Annual Cochrane Colloquium. October 2000; South Africa
- Bereczki D, Gesztelyi G: A Hungarian example for handsearching specialized national healthcare journals of small countries for controlled trials. Is it worth the trouble?. Health Libr Rev. 2000, 17: 144-147. 10.1046/j.1365-2532.2000.00280.x.View ArticlePubMedGoogle Scholar
- Galandi D, Bassler D, Antes G: Identifying randomized controlled trials published in German general health care journals using Medline and Embase: How useful is the controlled vocabulary? [poster]. Proceedings of the 8th Annual Cochrane Colloquium. October 2000; South Africa
- Jadad AR, McQuay HJ: A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials. 1993, Doc No 33: 3973-Google Scholar
- Adetugbo K, Williams H: How well are randomized controlled trials reported in the dermatology literature?. Arch Dermatol. 2000, 136: 381-384. 10.1001/archderm.136.3.381.View ArticlePubMedGoogle Scholar
- Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M: A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 2002, 21: 1625-1634. 10.1002/sim.1191.View ArticlePubMedGoogle Scholar
- Olesen V, Engell L, Jensen KL, Gotzsche PC: Randomised clinical trials in the Scandinavian Journal of Rheumatology. Scand J Rheumatol. 2000, 29: 349-351. 10.1080/030097400447534.View ArticlePubMedGoogle Scholar
- Pasternack I, Varonen H, Juva K, Lehtinen E, Jormanainen V, Makela M: Hand Searching of Finnish Medical Journals. [poster]. Proceedings of the 5th Annual Cochrane Colloquium. October 1997; Netherlands
- Solomon MJ, Laxamana A, Devore L, McLeod RS: Randomized controlled trials in surgery. Surgery. 1994, 115: 707-712.PubMedGoogle Scholar
- Smith PJ, Moffatt ME, Gelskey SC, Hudson S, Kaita K: Are community health interventions evaluated appropriately? A review of six journals. J Clin Epidemiol. 1997, 50: 137-146. 10.1016/S0895-4356(96)00338-1.View ArticlePubMedGoogle Scholar
- Brazier H, Murphy AW, Lynch C, Bury G, Wilson S: Searching for the evidence in pre-hospital care: a review of randomised controlled trials. J Accid Emerg Med. 1999, 16: 18-23.View ArticlePubMedPubMed CentralGoogle Scholar
- Gotzsche PC, Lange B: Comparison of search strategies for recalling double-blind trials from MEDLINE. Dan Med Bull. 1991, 38: 476-478.PubMedGoogle Scholar
- Langham J, Thompson E, Rowan K: Identification of randomized controlled trials from the emergency medicine literature: comparison of hand searching versus MEDLINE searching. Ann Emerg Med. 1999, 34: 25-34.View ArticlePubMedGoogle Scholar
- Gluud C, Nikolova D: Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol. 1998, 29: 321-327. 10.1016/S0168-8278(98)80021-4.View ArticlePubMedGoogle Scholar
- Watson RJD, Richardson PH: Accessing the literature on outcome studies in group psychotherapy: the sensitivity and precision of Medline and PsycINFO bibliographic database searching. Br J Med Psychol. 1999, 72: 127-134. 10.1348/000711299159763.View ArticlePubMedGoogle Scholar
- Hay PJ, Adams CE, Lefebvre C: The efficiency of searches for randomized controlled trials in the International Journal of Eating Disorders: a comparison of handsearching, EMBASE and PsycLIT. Health Libr Rev. 1996, 13: 91-96. 10.1046/j.1365-2532.1996.1320091.x.View ArticleGoogle Scholar
- Avenell A, Handoll HHG, Grant AM: Lessons for search strategies from a systematic review, in the Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr. 2002, 73: 505-510.Google Scholar
- Ahmed I, Souza Soares KV, Seifas R, Adams CE: Randomized controlled trials in Archives of General Psychiatry (1959–1995): a prevalence study. Arch Gen Psychiatry. 1998, 55: 754-755. 10.1001/archpsyc.55.8.754.View ArticlePubMedGoogle Scholar
- Croft AM, Vassallo DJ, Rowe M: Handsearching the Journal of the Royal Army Medical Corps for Trials. J R Army Med Corps. 1999, 145: 86-88.View ArticlePubMedGoogle Scholar
- Eysenbach G, Tuische J, Diepgen TL: Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Med Inform (Lond). 2001, 26: 203-218.Google Scholar
- Furukawa TA, Inada T, Adams CE, McGuire H, Inagaki A, Nozaki S: Are the Cochrane group registers comprehensive? A case study of Japanese psychiatry trials. BMC Med Res Methodol. 2002, 2:Google Scholar
- Neal B, Rodgers A, Mackie MJ, MacMahon S: Forty years of randomised trials in the New Zealand Medical Journal. N Z Med J. 1996, 109: 372-373.PubMedGoogle Scholar
- Langham J, Thompson E, Rowan K: Randomised controlled trials from the critical care literature: identification and assessment of quality. Clin Intensive Care. 2002, 13: 73-83.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/5/24/prepub