Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough

Background Recent research indicates a high recall in Google Scholar searches for systematic reviews. These reports raised high expectations of Google Scholar as a unified and easy to use search interface. However, studies on the coverage of Google Scholar rarely used the search interface in a realistic approach but instead merely checked for the existence of gold standard references. In addition, the severe limitations of the Google Search interface must be taken into consideration when comparing with professional literature retrieval tools. The objectives of this work are to measure the relative recall and precision of searches with Google Scholar under conditions which are derived from structured search procedures conventional in scientific literature retrieval; and to provide an overview of current advantages and disadvantages of the Google Scholar search interface in scientific literature retrieval. Methods General and MEDLINE-specific search strategies were retrieved from 14 Cochrane systematic reviews. Cochrane systematic review search strategies were translated to Google Scholar search expression as good as possible under consideration of the original search semantics. The references of the included studies from the Cochrane reviews were checked for their inclusion in the result sets of the Google Scholar searches. Relative recall and precision were calculated. Results We investigated Cochrane reviews with a number of included references between 11 and 70 with a total of 396 references. The Google Scholar searches resulted in sets between 4,320 and 67,800 and a total of 291,190 hits. The relative recall of the Google Scholar searches had a minimum of 76.2% and a maximum of 100% (7 searches). The precision of the Google Scholar searches had a minimum of 0.05% and a maximum of 0.92%. The overall relative recall for all searches was 92.9%, the overall precision was 0.13%. Conclusion The reported relative recall must be interpreted with care. It is a quality indicator of Google Scholar confined to an experimental setting which is unavailable in systematic retrieval due to the severe limitations of the Google Scholar search interface. Currently, Google Scholar does not provide necessary elements for systematic scientific literature retrieval such as tools for incremental query optimization, export of a large number of references, a visual search builder or a history function. Google Scholar is not ready as a professional searching tool for tasks where structured retrieval methodology is necessary.


Appendix 1. Search of the CCDANCTR-Studies Register
CCDANCTR-Studies -searched on 24 September 2007 Intervention = (Antidepress* or "Monoamine Oxidase Inhibitors" or "Selective Serotonin Reuptake Inhibitors" or "Tricyclic Drugs" or Acetylcarnitine or Alaproclate or Amersergide or Amiflamine or Amineptine or Amitriptyline or Amoxapine or Befloxatone or Benactyzine or Brofaromine or Bupropion or Butriptyline or Caroxazone or Chlorpoxiten or Cilosamine or Cimoxatone or Citalo-pram or Clomipramine or Clorgyline or Clorimipramine or

(X) Heather 1996
Heather N, Rollnick S, Bell A, Richmond R. Effects of brief counselling among male heavy drinkers identified on general hospital wards. Drug and alcohol review 1996;15: 29-38.

(X) Vadhan-Raj 2004
Vadhan-Raj S, Skibber JM, Crane C, Buesos-Ramos CE, Rodriguez-Bigas MA, Feig BW, et al.Randomized, double-blind, placebo-controlled trial of epoetin alfa (Procrit) in patients with rectal and gastric cancer undergoing chemo-radiotherapy (CT/RT) followed by surgery: early termination of the trial due to increased incidence of thromboembolic events (TEE

Search strategies
Search strategies for IPD meta-analysis update

(X) Galanis 1998
Galanis DJ, Kolonel LN, Lee J, Nomura A. Intakes of selected foods and beverages and the incidence of gastric cancer among the Japanese residents of Hawaii: a prospective study. International Journal of Epidemiology 1998;27(2):173-80.

(X) Suzuki 2004
Suzuki Y, Tsubono Y, Nakaya N, Suzuki Y, Koizumi Y, Tsuji I. Green tea and the risk of breast cancer: pooled analysis of two prospective studies in Japan. British Journal of Cancer 2004;90 (7):1361-3.

(X) Davies 2008
Davies H, Marion S, Teschke K. The impact of hearing conservation programs on incidence of noise-induced hearing loss in Canadian workers. American Journal of Industrial Medicine 2008;51:923-31.

(X) Zlotkin 2003
Zlotkin S, Antwi KY, Schauer C, Yeung G. Use of microencapsulated iron(II) fumarate sprinkles to prevent recurrence of anaemia in infants and young children at high risk.