- Research article
- Open Access
- Open Peer Review
Optimizing literature search in systematic reviews – are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders?
BMC Medical Research Methodologyvolume 16, Article number: 161 (2016)
When conducting systematic reviews, it is essential to perform a comprehensive literature search to identify all published studies relevant to the specific research question. The Cochrane Collaborations Methodological Expectations of Cochrane Intervention Reviews (MECIR) guidelines state that searching MEDLINE, EMBASE and CENTRAL should be considered mandatory. The aim of this study was to evaluate the MECIR recommendations to use MEDLINE, EMBASE and CENTRAL combined, and examine the yield of using these to find randomized controlled trials (RCTs) within the area of musculoskeletal disorders.
Data sources were systematic reviews published by the Cochrane Musculoskeletal Review Group, including at least five RCTs, reporting a search history, searching MEDLINE, EMBASE, CENTRAL, and adding reference- and hand-searching. Additional databases were deemed eligible if they indexed RCTs, were in English and used in more than three of the systematic reviews. Relative recall was calculated as the number of studies identified by the literature search divided by the number of eligible studies i.e. included studies in the individual systematic reviews. Finally, cumulative median recall was calculated for MEDLINE, EMBASE and CENTRAL combined followed by the databases yielding additional studies.
Deemed eligible was twenty-three systematic reviews and the databases included other than MEDLINE, EMBASE and CENTRAL was AMED, CINAHL, HealthSTAR, MANTIS, OT-Seeker, PEDro, PsychINFO, SCOPUS, SportDISCUS and Web of Science. Cumulative median recall for combined searching in MEDLINE, EMBASE and CENTRAL was 88.9% and increased to 90.9% when adding 10 additional databases.
Searching MEDLINE, EMBASE and CENTRAL was not sufficient for identifying all effect studies on musculoskeletal disorders, but additional ten databases did only increase the median recall by 2%. It is possible that searching databases is not sufficient to identify all relevant references, and that reviewers must rely upon additional sources in their literature search. However further research is needed.
Systematic reviews (SR) and meta-analyses (MA) are key elements in both evidence-based healthcare  and evidence-based research  By synthesizing the available evidence, SRs support clinicians in making well-informed decisions about health care  and researchers in deciding which topics are the most relevant for new research . When conducting SRs, it is essential to perform a comprehensive literature search to identify all published studies relevant to the research question as a failure to do so can result in selection bias and distort the conclusion of the review by potentially over- or underestimating of the treatment effect [3, 5]. One of the recommended methods to identify scientific literature in health science is searching electronic databases [3, 6]. However, when doing a high quality search two main questions arise; which databases is necessary to searched, and how many? According to the Cochrane Collaborations Methodological Expectations of Cochrane Intervention Reviews (MECIR) three main databases: MEDLINE, EMBASE and CENTRAL are mandatory electronic databases to search when performing a Cochrane Review . Several studies indicate that searching MEDLINE identifies the highest number of studies [8–10] and others that the gains from searching beyond MEDLINE and in particular EMBASE are modest , however multiple studies have found that searching MEDLINE alone is not sufficient [8–10, 12–19]. In addition, when analysing the use of databases in Cochrane reviews, it was found that between 1 and 27 different databases was used , even though some studies indicates that searching no more than 3–5 databases seems to be sufficient,[8, 16, 17] and searching only one database would not be enough . Even though MEDLINE, EMBASE and other major medical databases yield a high proportion of relevant studies, some studies found it necessary to include other sources such as reference- and citation search, browsing conference proceedings, asking experts and alike to identify all the relevant studies [9, 14, 17, 19]. The difference between the results from these studies could be due to their evaluation of different areas or due to the methods used to search the different databases; hence some of the above mentioned studies construct a new search strategy to identify studies in a given area thereby making the evaluation be dependent on not just the database, but also the quality/accuracy of the search strategy constructed [21–23]. The great variations thus indicate a need to evaluate if the MECIR guideline recommendations to search MEDLINE, EMBASE and CENTRAL combined would be enough when performing a literature search or whether additional databases should be added to this list.
In order to focus this evaluation, we choose to concentrate on musculoskeletal disorders. The area of musculoskeletal injuries and diseases is the leading cause of long-term pain and physical disability [24–26] and are associated with 130 million health care encounters and estimated to cost over $50 billion annually in the United States . In addition, the Cochrane Musculoskeletal Review Group (CMSG) is among the largest review groups in the Cochrane Collaboration, responsible for more than 200 SRs.
The aim of this study is therefore to evaluate the relative recall in the databases recommended by MECIR for systematic literature searches within the area of musculoskeletal disorders. Furthermore, this study addressed the question: What is the increase in recall when searching additional databases to searching MEDLINE, EMBASE and CENTRAL combined?
Selection of systematic reviews
All SRs from the Cochrane Database of Systematic Reviews (CDSR) published by CMSG were obtained . SRs were excluded if they: (i) did not include at least five randomised controlled trials (RCT), as the consequence of missing one study in reviews with few studies included would affect the overall percentage more than with a higher total of included studies. (ii) had been withdrawn, (ii) did not report any search history in the SR (iii) did not search all of the following sources: MEDLINE, EMBASE, CENTRAL, reference- and hand searching, as recommended by The Cochrane Handbook  and MECIR guidelines . This strategy was defined in order to identify systematic reviews, which had included all (or close to all) relevant studies related to at certain research question by both searching electronic databases and other sources.
The recall was used to evaluate the ability of a search strategy to identify all relevant studies . Recall can be defined as the percentage of relevant records retrieved divided by the total number of included studies in the individual systematic reviews. However, to estimate true recall one need to know the total amount of relevant records in a database, which is not an easy (if not impossible) task. Thus often, relative recall is estimated by firstly defining a pool of relevant records (the included studies in a SR) and then determines what proportion of this pool the literature search retrieves.[30, 31] In this study we therefore used the included studies in each of the included SRs as the pool of relevant records.
Identification of databases
From the pool of SRs, a list of databases used was created. Databases were ranked in descending order according to how many SRs that had searched the database. Databases other than MEDLINE, EMBASE and CENTRAL were deemed eligible if they (i) was indexing RCTs, (ii) was in English, (iii) was used by at least three SRs.
The following data were extracted for each included SR: (i) details of the search strategy as described in the review (ii) date of when the search was performed or updated (iii) full bibliographic details of all primary studies included in the SR (i.e. title of the study, author names, journal title, publications year etc.).
Searching individual databases
The search strategy/strategies reported in each SR was replicated and used for searching all the databases included in the SR. For databases with no reported search strategy, the MEDLINE search strategy was replicated and searched in all included databases. MEDLINE syntax (i.e. fields, truncation, adjacency) was modified to the individual database. When possible, the exact search dates from the SR was used for each database. However, CENTRAL for instance, only allows specification by month and year. End Note ×7.5.3 software (Thomson Reuters™) was used to manage records retrieved from searches of electronic databases.
Relative recall for each of the included database was calculated separately and for each of the included SR. Relative median recall for each database was calculated for all included SRs combined. Cumulative median relative recall was estimated for searches in MEDLINE, EMBASE and CENTRAL adding databases in descending order (based on how often the databases was searched in the SRs published by CMSG). Data managing was performed using Microsoft Excel 2016 and data analysis was performed using STATA version 13.1 (StataCorp, College Station, Texas) software package.
Subgroup analysis was pre-specified and planned to assess the cumulative median recall for subgroups, rehabilitation, medicine or other content (surgery, lifestyle intervention, electrical stimulation etc.) as previous studies have found differences in recommendations depending on the topics searched [32, 33]. One post-hoc sensitivity analysis was conducted to address to what extend the inclusion of SRs with a cut-off of three included RCTs instead of five would affect the result.
Eligible databases and systematic reviews
A timeline is displayed in Fig. 1. A set of 164 SRs where identified and obtained from the CMSG on March 3, 2013 and revisited for an update on July 3 2013 by the first author (Fig. 2). Of the 164 SRs assessed for eligibility by title and abstract, 10 were excluded. Nine for being withdrawn and one for being an overview of reviews. Of the remaining 154 SRs assessed in full-text, 114 were excluded, as they did not search one or more of the following sources: MEDLINE-, CENTRAL-, EMBASE, or reference- and hand searching. 11 SRs were excluded, as they did not include five or more studies in their analysis. Six were excluded for not reporting any search strategy in the SR, neither reporting where one could be acquired. A final set of 23 SRs finally met all inclusion criteria [34–56], (Table 1).
The generated list of databases other than MEDLINE, EMBASE and CENTRAL included a total of 58 databases identified in the 23 included SRs. Of these 58 databases, 48 databases were excluded; 10 did not index RCTs (i.e. trial registry etc.), nine where included in other databases (i.e. Premedline in MEDLINE etc.), 28 were used in less than three SRs, and one database was not in English. The following 10 databases met the inclusion criteria: AMED (via EBSCOhost), CINAHL (via EBSCOhost), HealthSTAR (via OVID), MANTIS (via OVID), OT-Seeker, PEDro, PsychINFO (via OVID), SCOPUS, SportDISCUS (via EBSCOhost) and Web of Science. Searching MEDLINE was performed using the host specified in the SR (i.e. via OVID or Pubmed), EMBASE via the OVID and CENTRAL via the Wiley InterScience portal.
Characteristics of the included systematic reviews
The 23 SRs included a total of 365 primary studies. Each review included from 5 to 103 studies (median 10) (Table 1). The number of search strategies reported in the SRs ranged from 1 to 7 (median 2). Eleven out of 23 SRs reported only one search strategy; 4 of which reported a “standard search strategy” that was adapted to other databases searched, while 7 reported the search strategy used for MEDLINE (Table 1). Of the 23 SRs, the intervention in 5 was classified as rehabilitation, 14 as medicine and 4 was classified as other content (Table 1).
Synthesis of results
Table 2 displays the median relative recall for the combined search in MEDLINE, EMBASE and CENTRAL and for the additional 10 databases included. Data shown are the median recall and interquartile range (IQR) from the total number of SRs and for the three subgroups separately. MEDLINE, EMBASE and CENTRAL combined yielded a median recall of 88.9% (IQR 81.6–100%), followed by SCOPUS (85.7%) and HealthSTAR (83.3%) (Table 2).
Results of the overall cumulative median analysis on relative recall are displayed in Fig. 3 and Table 3. The most exhaustive search (i.e. the minimum number of databases required to be searched to retrieve the maximum number of studies) involved searching MEDLINE, EMBASE and CENTRAL with the addition of SCOPUS and CINAHL. Results show that adding these databases to MEDLINE, EMBASE and CENTRAL increased the median recall by 2.0 percentage points, from 88.9% to a median recall of 90.9% (IQR 83.3–100%). Adding the remaining 8 databases did not increase the median recall.
Subgroup analyses according to content area demonstrated some variations. The most exhaustive search for the rehabilitation group involved searching MEDLINE, EMBASE and CENTRAL with the addition of SCOPUS and CINAHL for a cumulative median recall of 100% (IQR 60–100%) (Table 4), the medicine group with the addition of SCOPUS for a cumulative median recall of 87.3% (IQR 83.3–97.7%) (Table 4), and the other content groups with the addition of Scopus and CINAHL for a cumulative median recall of 100% (IQR 97.7–100%) (Table 4).
Post-hoc sensitivity analysis showed that with the inclusion of SRs with at least three included RCTs, 4 SRs would be added to the analyses [57–60]. The analyses showed that the cumulative median recall increased when adding these SRs, however the IQR remained unchanged (Table 5).
This study supports the recommendations by Cochrane Collaboration to prioritize MEDLINE, EMBASE and CENTRAL as the basic databases for literature search to locate RCTs in the musculoskeletal area. Secondly, this study indicates that besides MEDLINE, EMBASE and CENTRAL a literature search to locate RCTs in the musculoskeletal area could also consider SCOPUS and CINAHL. Finally, this study indicates that even with the addition search of 10 other often used databases median recall is not improved noticeably.
Thirteen different databases were not enough to identify all relevant references. Searching MEDLINE, EMBASE and CENTRAL retrieved 88.9%, and searches in 10 additional databases increased the median recall by only 2 percentage point. Thus, results from this study could be interpreted, as an indication that searching databases is not sufficient to identify all relevant references and that other sources must be included in the literature search in order to achieve a larger recall. This study does not evaluate which source that may be the most important. Savoie et al. and Helmer et al. [61, 62] found that 29.2% of all items retrieved for two SRs could be uncovered by extended systematic search methods; searching subject- specific or specialized databases (including trial registries), hand-searching, scanning reference lists, and communicating personally with experts. Yet Robinson et al.  recently showed that researchers do not cite all possible previous trials, and that less than half (38%) of RCTs could be identified by citation network searching.
It therefore remains to be evaluated how much higher recall could be achieved by supplementing the database search with reference and/or citation search, and which impact if any these additional sources have on the pooled estimate effect.
Searching SCOPUS and CINAHL increased the median recall by 2 point. Yet, as results from the subgroup analysis showed, each database contributed differently depending on the field groups searched. SCOPUS increased the median recall slightly in the Other group, and had some effect on the IQR in all three groups. This could be due to the fact that SCOPUS is a generic database containing studies from a wide range of subject fields. The large increase in median recall in the rehabilitation group searching CINAHL could be because CINAHL is a database including research from health care professionals often involved with rehabilitation. The fact that CINAHL only increased the recall in the area of rehabilitation are supported by Beckles et al. who strongly recommends that the database should be relegated too selective rather than routine searching due to a very low proportion of unique references .
Results from the post-hoc sensitivity analyses showed not surprisingly that the inclusion of studies with a low proportion of included studies could introduce high risk of bias of the results. Adding only four more studies increased the median recall to 100% and by 10% compared to the main results. Yet, as the IQR of the results are unchanged, this reinforce the notion, that searching additional databases is less likely to add more studies.
To our knowledge only one earlier study concluded that one database was enough in order to achieve full recall. Kelly et al.  concluded that MEDLINE was enough to identify all relevant studies for their specific question. However, they concluded that to fully capture the complete body of available literature on other subjects might require searching multiple databases . This is strongly supported by a number of other studies [8–13, 15, 16, 18–22, 33, 66–78]. Studies evaluating this question within the musculoskeletal field make the same conclusion: searching more than one database is necessary [14, 17, 23, 32, 79–81]. Based on results from earlier studies and the results from the present study, recommendation for an optimal systematic literature search to locate RCTs within the musculoskeletal area may be to use the three generic databases: MEDLINE; EMBASE and CENTRAL, and an additional two or more other databases. However, this search should ad other sources such as reference- and citation search, grey literature, conference proceedings, and contact experts within the area as results from this study suggest that 10% could be missed when only searching electronical databases to identify relevant studies.
Limitations of this study
This study has some limitations. An important limitation of this study and other studies evaluating the recall of a systematic literature search is the definition of the true number of studies that should be identified. In this study we defined this as the number of studies deemed relevant (i.e. included) in a SR, yet as a SR seek to answer a well-defined question, there are some limitations to whether the included SRs in this study fully represent the various areas of the musculoskeletal field. Another limitation of any study reproducing an original search strategy at a later date is that the contents and indexing of databases change over time, and not all of these changes can be rewound.
Another limitation to this study is the underlining assumption that the systematic literature search strategy used in each SR did capture all relevant studies in the database searched. The result from the present study does not evaluate this question, yet Sampson et al.  found that errors in electronic search strategies reduce the effectiveness of electronic search strategies. Further research is needed to evaluate not only the recall of studies retrieved using a search strategy, but also comparing this to the recall of studies indexed in a database by bibliographic verification: searching for known items , thereby addressing the key question, what is indexed in a database? and what is actually retrieved when searching this database?
The aim of this study was to evaluate the relative recall in the databases recommended by MECIR for systematic literature searches to locate RCTs within the area of musculoskeletal disorders. The use and limitations of the method to the musculoskeletal area thus clearly limits the general conclusion from this study. However, our results are in line with other studies evaluating literature search in electronic databases.
The strengths of this study lies in the systematic approach of selecting Cochrane SRs of the highest quality, and combining results of the literature search from these SRs in a way that make SRs with a high number of studies included equal to those with low number of studies included. This is also one of few studies [20, 23, 84] that have combined multiple databases using cumulative analysis, thereby accepting what researchers have urged in the past, that searching one database is not enough but investigating what a combined search would yield.
Searching MEDLINE, EMBASE and CENTRAL is not sufficient for identifying all effect studies within the area of musculoskeletal disorders. Literature searches in ten additional databases only increases the median recall by 2 percentage point.
It remains to be evaluated how much higher the relative recall could be achieved by supplementing the database search with reference and citation search. Further studies where the same methods are applied on different content areas needs to be performed, to see if the assumption that the way to perform a search depends on the content area is true or not. It is possible that searching databases is not sufficient to identify all relevant references, and that reviewers must rely upon additional sources in their literature search, but further research on these additional sources is needed.
Cochrane database of systematic reviews
Cochrane musculoskeletal review group
- M + E + C:
MEDLINE + EMBASE + CENTRAL
- M + E + C + S:
MEDLINE + EMBASE + CENTRAL + SCOPUS
- M + E + C + S + Ci:
MEDLINE + EMBASE + CENTRAL + SCOPUS+ Cinahl
Methodological expectations of cochrane intervention reviews
Randomised controlled trials
Gray JAM, Booth A, Booth S, Ison E. Evidence-based healthcare. Edinburgh: Churchill Livingstone; 2001.
Lund H, Brunnhuber K, Juhl C, Robinson K, Leenaars M, Dorch BF, Jamtvedt G, Nortvedt MW, Christensen R, Chalmers I. Towards evidence based research. BMJ. 2016;355:i5440.
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons; 2008.
Chalmers I, Nylenna M. A new network to promote evidence-based research. Lancet. 2014;384(9958):1903–4.
Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, Alonso-Coello P, Djulbegovic B, Atkins D, Falck-Ytter Y, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011;64(12):1277–82.
Klassen TP, Jadad AR, Moher D. Guides for reading and interpreting systematic reviews: I. Getting started. Arch Pediatr Adolesc Med. 1998;152(7):700–4.
Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention Reviews. The Cochrane Unit. 2013;2:3.
Bayliss SE, Davenport CF, Pennant ME. Where and how to search for information on the effectiveness of public health interventions--a case study for prevention of cardiovascular disease. Health Info Libr J. 2014;31(4):303–13.
Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: Added value or diminishing returns? Complement Ther Med. 2004;12(4):228–32.
Rollin L, Darmoni S, Caillard JF, Gehanno JF. Searching for high-quality articles about intervention studies in occupational health - What is really missed when using only the Medline database? Scand J Work Environ Health. 2010;36(6):484–7.
Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. J Clin Epidemiol. 2015;68(9):1076–84.
Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Inj Prev. 2008;14(6):401–4.
Minozzi S, Pistotti V, Forni M. Searching for rehabilitation articles on MEDLINE and EMBASE. An example with cross-over design. Arch Phys Med Rehabil. 2000;81(6):720–2.
Murphy LS, Reinsch S, Najm WI, Dickerson VM, Seffinger MA, Adams A, Mishra SI. Spinal palpation: The challenges of information retrieval using available databases. J Manipulative Physiol Ther. 2003;26(6):374–82.
Sampson M, Cogo E, Ajiferuke I, Manheimer E, Campbell K, Daniel R, Moher D. Searching for controlled trials of complementary and alternative medicine: A comparison of 15 databases. Evid Based Complement Alternat Med. 2011;2011:858246.
Shariff SZ, Sontrop JM, Iansavichus AV, Haynes RB, Weir MA, Gandhi S, Cuerden MS, Garg AX. Availability of renal literature in six bibliographic databases. Clin Kidney J. 2012;5(6):610–7.
Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic Databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.
Vickers AJ. Bibliometric analysis of randomized trials in complementary medicine. Complement Ther Med. 1998;6(4):185–9.
Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357.
Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: Rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.
Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.
Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.
Slobogean GP, Verma A, Giustini D, Slobogean BL, Mulpuri K. MEDLINE, EMBASE, and Cochrane index most primary studies but not abstracts included in orthopedic meta-analyses. J Clin Epidemiol. 2009;62(12):1261–7.
Woolf AD, Pfleger B. Burden of major musculoskeletal conditions. Bull World Health Organ. 2003;81(9):646–56.
Horton R. GBD 2010: understanding disease, injury, and risk. Lancet. 2012;380(9859):2053–4.
HSE: The health and safety executive statistics 2010/11. In.: http://www.hse.gov.uk/statistics/overall/hssh1011.pdf Accessed 31 Jan 2016.
Utterback DF, Schnorr TM: Use of workers’ compensation data for occupational safety and health: proceedings from June 2012 workshop. In US Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health. In. http://www.cdc.gov/niosh/topics/workercomp/cwcs/publications.html: Assessed 25 Aug 2016.
Cochrane Musculoskeletal - Our Reviews. In. http://musculoskeletal.cochrane.org/our-reviews: Accessed 3 Jul 2013
Harter SP. Online information retrieval : concepts, principles and techniques. Orlando: Academic; 1986.
Sampson M, Zhang L, Morrison A, Barrowman NJ, Clifford TJ, Platt RW, Klassen TP, Moher D. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Med Res Methodol. 2006;6:33.
Vincent S, Greenley S, Beaven O. Clinical Evidence diagnosis: Developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Info Libr J. 2003;20(3):150–9.
Michaleff ZA, Costa LOP, Moseley AM, Maher CG, Elkins MR, Herbert RD, Sherrington C. CENTRAL, PEDro, PubMed, and EMBASE are the most comprehensive databases indexing randomized controlled trials of physical therapy interventions. Phys Ther. 2011;91(2):190–7.
Watson RJD, Richardson PH. Identifying randomized controlled trials of cognitive therapy for depression: Comparing the efficiency of embase, medline and PsycINFO bibliographic databases. Psychol Psychother Theory Res Pract. 1999;72(4):535–42.
Adie S, Kwan A, Naylor JM, Harris IA, Mittal R. Cryotherapy following total knee replacement. Cochrane Database Syst Rev. 2012;9, CD007911.
Bartels EM, Lund H, Hagen KB, Dagfinrud H, Christensen R, Danneskiold-Samsoe B. Aquatic exercise for the treatment of knee and hip osteoarthritis. Cochrane Database Syst Rev. 2007;4, CD005523.
Bellamy N, Campbell J, Robinson V, Gee T, Bourne R, Wells G. Viscosupplementation for the treatment of osteoarthritis of the knee. Cochrane Database Syst Rev. 2006;2, CD005321.
Bellamy N, Campbell J, Robinson V, Gee T, Bourne R, Wells G. Intraarticular corticosteroid for treatment of osteoarthritis of the knee. Cochrane Database Syst Rev. 2006;2, CD005328.
Coghlan JA, Buchbinder R, Green S, Johnston RV, Bell SN. Surgery for rotator cuff disease. Cochrane Database Syst Rev. 2008;1, CD005619.
Colebatch AN, Marks JL, Edwards CJ. Safety of non-steroidal anti-inflammatory drugs, including aspirin and paracetamol (acetaminophen) in people receiving methotrexate for inflammatory arthritis (rheumatoid arthritis, ankylosing spondylitis, psoriatic arthritis, other spondyloarthritis). Cochrane Database Syst Rev. 2011;11, CD008872.
Cranney A, Welch V, Adachi JD, Homik J, Shea B, Suarez-Almazor ME, Tugwell P, Wells G. Calcitonin for the treatment and prevention of corticosteroid-induced osteoporosis. Cochrane Database Syst Rev. 2000;2, CD001983.
De Morton NA, Keating JL, Jeffs K. Exercise for acutely hospitalised older medical patients. Cochrane Database Syst Rev. 2007;1, CD005955.
Fidelix TS, Soares BG, Trevisani VF. Diacerein for osteoarthritis. Cochrane Database Syst Rev. 2006;1, CD005117.
Karjalainen K, Malmivaara A, Van Tulder M, Roine R, Jauhiainen M, Hurri H, Koes B. Multidisciplinary rehabilitation for fibromyalgia and musculoskeletal pain in working age adults. Cochrane Database Syst Rev. 2000;2, CD001984.
Katchamart W, Trudeau J, Phumethum V, Bombardier C. Methotrexate monotherapy versus methotrexate combination therapy with non-biologic disease modifying anti-rheumatic drugs for rheumatoid arthritis. Cochrane Database Syst Rev. 2010;4, CD008495.
Khan F, Ng L, Gonzalez S, Hale T, Turner-Stokes L. Multidisciplinary rehabilitation programmes following joint replacement at the hip and knee in chronic arthropathy. Cochrane Database Syst Rev. 2008;2, CD004957.
Lethaby A, Lopez-Olivo MA, Maxwell L, Burls A, Tugwell P, Wells GA. Etanercept for the treatment of rheumatoid arthritis. Cochrane Database Syst Rev. 2013;5, CD004525.
Nuesch E, Rutjes AW, Husni E, Welch V, Juni P. Oral or transdermal opioids for osteoarthritis of the knee or hip. Cochrane Database Syst Rev. 2009;4, CD003115.
Osiri M, Shea B, Robinson V, Suarez-Almazor M, Strand V, Tugwell P, Wells G. Leflunomide for treating rheumatoid arthritis. Cochrane Database Syst Rev. 2003;1, CD002047.
Richards BL, Whittle SL, Buchbinder R. Antidepressants for pain management in rheumatoid arthritis. Cochrane Database Syst Rev. 2011;11, CD008920.
Richards BL, Whittle SL, Buchbinder R. Muscle relaxants for pain management in rheumatoid arthritis. Cochrane Database Syst Rev. 2012;1, CD008922.
Ruiz Garcia V, Jobanputra P, Burls A, Cabello JB, Galvez Munoz JG, Saiz Cuenca ES, Fry-Smith A. Certolizumab pegol (CDP870) for rheumatoid arthritis in adults. Cochrane Database Syst Rev. 2011;2, CD007649.
Rutjes AW, Nuesch E, Sterchi R, Juni P. Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database Syst Rev. 2010;1, CD003132.
Rutjes AW, Nuesch E, Sterchi R, Kalichman L, Hendriks E, Osiri M, Brosseau L, Reichenbach S, Juni P. Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database Syst Rev. 2009;4, CD002823.
Wajon A, Ada L, Edmunds I. Surgery for thumb (trapeziometacarpal joint) osteoarthritis. Cochrane Database Syst Rev. 2005;4, CD004631.
Whittle SL, Richards BL, Husni E, Buchbinder R. Opioid therapy for treating rheumatoid arthritis pain. Cochrane Database Syst Rev. 2011;11, CD003113.
Winzenberg TM, Powell S, Shaw KA, Jones G. Vitamin D supplementation for improving bone mineral density in children. Cochrane Database Syst Rev. 2010;10, CD006944.
Brosseau L, Judd MG, Marchand S, Robinson VA, Tugwell P, Wells G, Yonge K. Transcutaneous electrical nerve stimulation (TENS) for the treatment of rheumatoid arthritis in the hand. Cochrane Database Syst Rev. 2003;3, CD004377.
O’Donnell S, Cranney A, Wells GA, Adachi JD, Reginster JY. Strontium ranelate for preventing and treating postmenopausal osteoporosis. Cochrane Database Syst Rev. 2006;3, CD005326.
Richards BL, Whittle SL, Buchbinder R. Neuromodulators for pain management in rheumatoid arthritis. Cochrane Database Syst Rev. 2012;1, CD008921.
Rome K, Ashford RL, Evans A. Non-surgical interventions for paediatric pes planus. Cochrane Database Syst Rev. 2010;7, CD006311.
Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond Medline: reducing bias through extended systematic review search. Int J Technol Assess Health Care. 2003;19(1):168–78.
Helmer D, Savoie I, Green C, Kazanjian A. Evidence-based practice: extending the search to find material for the systematic review. Bull Med Libr Assoc. 2001;89(4):346–52.
Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014;67(7):793–9.
Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, Alderson P. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.
Kelly L, St Pierre-Hansen N. So many databases, such little clarity: Searching the literature for the topic aboriginal. Can Fam Physician. 2008;54(11):1572–1573.e1575.
Adams CE, Power A, Frederick K, Lefebvre C. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.
Aker PD, McDermaid C, Opitz BG, White MW. Searching chiropractic literature: A comparison of three computerized databases. J Manip Physiol Ther. 1996;19(8):518–24.
Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L. Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Med Res Methodol. 2005;5:24.
Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994;309(6964):1286–91.
Haafkens J, Moerman C, Schuring M, Van Dijk F. Searching bibliographic databases for literature on chronic disease and work participation. Occup Med. 2006;56(1):39–45.
Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, Yao L, Yang K. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.
Lorenzetti DL, Topfer LA, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.
Lovarini M, Wallen M, Imms C. Searching for evidence in pediatric occupational therapy using free versus subscription databases: A comparison of outcomes. Phys Occup Ther Pediatr. 2006;26(3):19–38.
Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, Pill R, Russell I, Stott N, Wilkinson C. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.
Royle PL, Bain L, Waugh NR. Sources of evidence for systematic reviews of interventions in diabetes. Diabet Med. 2005;22(10):1386–93.
Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.
Woodman J, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C. Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. J Med Libr Assoc. 2010;98(2):140–6.
Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies required health, social science, and grey literature databases. J Clin Epidemiol. 2014;67(7):800–10.
Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.
Moseley AM, Sherrington C, Elkins MR, Herbert RD, Maher CG. Indexing of randomised controlled trials of physiotherapy interventions: a comparison of AMED, CENTRAL, CINAHL, EMBASE, Hooked on Evidence, PEDro, PsycINFO and PubMed. Physiotherapy. 2009;95(3):151–6.
Ramos-Remus C, Suarez-Almazor M, Dorgan M, Gomez-Vargas A, Russell AS. Performance of online biomedical databases in rheumatology. J Rheumatol. 1994;21(10):1912–21.
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.
Schoones JW, Vissers T. Re: “using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions”. J Clin Epidemiol. 2016.
Bramer WM, Giustini D, Kramer BM. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev. 2016;5:39.
We wish to acknowledge and thank Videncentret, the Medical Library, Odense University Hospital and University Library of Southern Denmark for great and important help preparing this work. This work has been prepared as part of the SEARCH research group (Synthesis of Evidence And ResearCH). SEARCH is a Danish research group aiming at increase use of systematic review when prioritizing research, interpreting research results, and improving clinical practice. SEARCH also contributes to the development of methods for preparing systematic reviews, and evaluating the use of systematic reviews in research practice (meta-research).
The author(s) declare that they have received no funding.
Availability of data and materials
The datasets analysed during the current study are available from the corresponding author on reasonable request.
TAA, CJ and HL generated the concept and design of the study. TAA screened reviews for eligibility, undertook the data collection and analysis. All authors participated in the development of the manuscript from its early stages. All authors contributed to, read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
About this article
- Information retrieval
- MECIR guidelines
- Evidence-based medicine
- Relative recall
- Literature searching
- Systematic review
- Musculoskeletal area