This article has Open Peer Review reports available.
The implications of biomarker evidence for systematic reviews
© Choong and Tsafnat; licensee BioMed Central Ltd. 2012
Received: 7 August 2012
Accepted: 3 November 2012
Published: 22 November 2012
In Evidence-Based Medicine, clinical practice guidelines and systematic reviews are crucial devices for medical practitioners in making clinical decision. Clinical practice guidelines are systematically developed statements to support health care decisions for specific circumstances whereas systematic reviews are summaries of evidence on clearly formulated clinical questions. Biomarkers are biological measurements (primarily molecular) that are used to diagnose, predict treatment outcomes and prognosticate disease and are increasingly used in randomized controlled trials (RCT).
We search PubMed for systematic reviews, RCTs, case reports and non-systematic reviews with and without mentions of biomarkers between years 1990–2011. We compared the frequency and growth rate of biomarkers and non-biomarkers publications. We also compared the growth of the proportion of biomarker-based RCTs with the growth of the proportion of biomarker-based systematic reviews.
With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers. The annual growth rate of biomarkers publications is consistently higher than non-biomarkers publications, showing the growth in biomarkers research. From 20 years of systematic review publications indexed in PubMed, we identified a bias in systematic reviews against the inclusion of biomarker-based RCTs.
With the realisation of genome-based personalised medicine, biomarkers are becoming important for clinical decision making. The bias against the inclusion of biomarkers in systematic reviews leads to medical practitioners deprive of important information they require to address clinical questions. Sparse or weak evidence and lack of genetic training for systematic reviewers may contribute to this trend.
Evidence-Based Medicine (EBM) stipulates the use of the best available external scientific evidence in clinical decision making. Systematic Reviews (SRs) are conducted through a well defined process , and published as robust answers to clinical questions given the best available evidence . Randomized Controlled Trials (RCTs) are trusted above all other primary evidence types . EBM is thus implemented by following clinical practice guidelines (CPGs) or relying on SRs whenever possible when making clinical decisions. Guideline and SR development is a slow and complex process or translating research that can take more than a year  to complete. In this process it is critical to include all relevant evidence . It was previously claimed that as much as 80% of all published reviews are not updated and only 3% of systematic reviews published in peer-reviewed journals had been updated within two years of their publication . Only a small proportion of relevant trials are incorporated into systematic reviews .
The trend toward genome-based personalised medicine, and the rapid advancement of sequencing and high-throughput technologies, has dramatically reshaped disease research. Genomic data can now be obtained expeditiously and inexpensively. This has led to the use of a range of analytical tools to assess biological parameters .
Biomarkers that distinguish among subtypes of disease are now standard practice in biomedical research. There is an overwhelming interest in biomarker research reflected in a large number of research grants awarded and academic publications . However, only a limited number of biomarkers have been incorporated into clinical guidelines , and the anticipation that biomarker research will revolutionize medical practice has so far not been realized. The ambiguity of the term “biomarker” has prompted the US National Institutes of Health (NIH) to define the terminology. Biomarkers are defined by the NIH Biomarkers Definitions Working Group as “a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention” . Biomarkers are used as tools in disease diagnostic, early detection, staging, prognosis or prediction of treatment outcome. They can guide individualized treatment and improve patient care . Biomarkers can provide the basis for design, improve the safety and efficiency, and explain empirical results of clinical trials .
The translation of biomarkers into clinical practice follows the regular translational pathways from discovery to preclinical, clinical, and post-approval trials and implementation. However, some translational crossroads are unique to biomarkers [10, 11]. In particular, the diversity and lengthy process of biomarker assay development, and limited industry support were identified as bottlenecks for translating biomarker research into clinical practice. In this study, we examine the validation to implementation phase (also called T2 ). As there are cases of exaggeration in the effect sizes of many highly cited biomarkers studies (which lead to overestimated findings) , we measured the inclusion of RCTs of biomarkers in SRs. We believe that prospective evaluations of biomarkers in RCTs could provide more reliable results about their effects and clinical utility.
How does the output frequency of publications that include biomarkers compare to those that don’t?
Is the growth rate of publication of biomarkers RCTs in line with the non-biomarkers publications?
To which extent biomarkers RCTs are being included in SRs as compared to non-biomarkers RCTs?
Publication type search based on publication types of PubMed
Publication type in PubMed
Systematic reviews (SR/MA)
“Systematic Reviews” OR “Meta-Analysis”
Controlled Trial (RCT/CCT)
“Randomized Controlled Trial” OR “Controlled Clinical Trial”
There are ambiguities in the naming and terminology of the term “biomarker” within the literature . As we are interested in all biomarkers, we have included as many terms with overlapping meaning of “biomarker” as possible, such as biological marker/s, molecular marker/s, genetic marker/s, DNA marker/s, cytogenetic marker/s, proteomics marker/s and biochemical marker/s. We repeated each search with (biomarker OR "biological marker" OR “biological markers” OR "molecular marker" OR "molecular markers" OR "genetic marker" OR "genetic markers" OR "DNA marker" OR "DNA markers" OR "cytogenetic marker" OR "cytogenetic markers" OR "proteomics marker" OR "proteomics markers" OR "biochemical marker" OR "biochemical markers") helped restrict the results to those biomarkers publications only, and (all[sb] NOT (biomarker OR "biological marker" OR “biological markers” OR "molecular marker" OR "molecular markers" OR "genetic marker" OR "genetic markers" OR "DNA marker" OR "DNA markers" OR "cytogenetic marker" OR "cytogenetic markers" OR "proteomics marker" OR "proteomics markers" OR "biochemical marker" OR "biochemical markers")) for non-biomarkers publications. “All publications” is the summation of the above two complementary groups. We limited our search to human subjects and to documents in English. All searches were done on the 18th of October 2012.
In order to validate the search strategy, we randomly chose 110 search results from each group (biomarkers and non-biomarkers) of RCT/CCT (“Randomized Controlled Trial” or “Controlled Clinical Trial”) and manually read their abstracts and if needed their full text, to determine if the trial is indeed about biomarkers or using biomarkers as outcome measures.
where x i is the current year's total publications and N=22 is the total number of years included in the study.
In order to assess the extent of inclusions of clinical trials publications in systematic reviews, we normalized the systematic reviews samples by the number of published clinical trials.
The validation of search results on RCT/CCT yields an accuracy of 0.82, with sensitivity of 0.76 and specificity of 0.94.
While only a minority of trials has been included in systematic reviews , this study found that the number of trials involving biomarkers assessed in systematic reviews is even smaller. With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers.
Total publications and average annual growth rates of biomarkers and non-biomarkers publications from year 1990-2011
Total publications (1990–2011)
Annual growth rate (%)
Total publications (1990–2011)
Annual growth rate (%)
Biomedical research articles often include poor reporting of statistical methods. Incomplete reporting of statistical analysis limits or prevents the use of these studies in the systematic reviews. This could explain why only a small proportion of RCTs are included in systematic reviews. Further study is needed to test if biomarkers RCTs are more prone to bad statistical methods reporting than other RCTs.
Our results show discrepancy between the rate of publications of biomarkers RCTs and biomarkers reviews. Possible explanations for this include:
The frequent proposals of new biomarkers and assays which have complicated the translation and commercialization processes .
Independent investigations in biomarkers show contradicting results. A biomarker is ready for clinical testing only when several retrospective tests consistently confirm its performance . As such, weak and sparse evidence could not lead to any conclusion.
Systematic reviewers shy away from a domain in which their training is lacking .
Genetic disorders, where specific genetic variants causally associated with common diseases, account for only a fraction of cases , which result in smaller cohorts for studies.
RCTs, observational studies or cross-sectional diagnostic studies are designed to answer generalized clinical questions and ignore outliers. Biomarker evidence might be too sensitive to outliers  to provide low p-values.
A few outliers might also cause small effects even in large cohort studies  which would result in exclusion from systematic reviews as they will provide no clear difference from previous summarized evidence.
As systematic reviews are the main source for guideline development, the lack of effort on systematically reviewing studies in this area can explain why only a limited number of molecular markers have been incorporated into clinical guidelines. There are still many relevant trials not being assessed or not included in systematic reviews and guidelines. This seems to be stronger in biomarker evidence as we have shown.
Biomarkers are becoming increasingly valuable in clinical settings, whether to diagnose, prognosticate or to guide treatment. It is important to fast track the research and translation process. We identified the need for systematic reviewers to include more biomarkers and proposed several possible explanations of why this has not been done yet including lacking education of systematic reviewers on molecular biology concepts and the low predictive power of biomarkers. We propose that specific search technologies can support the review process.
The authors would like to thank Professor Paul Glasziou for his feedback on an early version of this manuscript.
MKC was supported by National Health and Medical Research Council Program Grant 568612. GT was supported by the New South Wales Health Capacity Building Infrastructure Grant. The article-processing charge for the manuscript was supported by Career Advancement Fund from UNSW.
- Sackett DL, Strauss SE, Richardson WS, Rosenberg WMC, Haynes RB: 2000, Edinburg, Scotland: Evidence-based medicine: how to practice and teach EBM, Churchill LivingstoneGoogle Scholar
- Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312 (7023): 71-72. 10.1136/bmj.312.7023.71.View ArticlePubMedPubMed CentralGoogle Scholar
- Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, Dröschel D, Beck L, Abalos E, El-Jardali F, Gilson L, et al: Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012, 9 (3): e1001185-10.1371/journal.pmed.1001185.View ArticlePubMedPubMed CentralGoogle Scholar
- Cohen AM, Adams CE, Davis JM, Yu C, Yu PS, Meng W, Duggan L, McDonagh M, Smalheiser NR: Proceedings of the 1st ACM international health informatics symposium. Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools. 2010, Virginia, USA: ACM, Arlington, 376-380.Google Scholar
- Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007, 4 (3): e78-10.1371/journal.pmed.0040078.View ArticlePubMedPubMed CentralGoogle Scholar
- Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: How will We ever keep Up?. PLoS Med. 2010, 7 (9): e1000326-10.1371/journal.pmed.1000326.View ArticlePubMedPubMed CentralGoogle Scholar
- Atkinson AJ, Colburn WA, DeGruttola VG, DeMets DL, Downing GJ, Hoth DF, Oates JA, Peck CC, Schooley RT, Spilker BA, et al: Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework*. Clin Pharmacol Ther. 2001, 69 (3): 89-95.View ArticleGoogle Scholar
- Ptolemy AS, Rifai N: What is a biomarker? Research investments and lack of clinical integration necessitate a review of biomarker terminology and validation schema. Scand J Clin Lab Invest. 2010, 70 (S242): 6-14. 10.3109/00365513.2010.493354.View ArticleGoogle Scholar
- Choong MK, Tsafnat G: Genetic and epigenetic biomarkers of colorectal cancer. Clin Gastroenterol Hepatol. 2012, 10 (1): 9-15. 10.1016/j.cgh.2011.04.020.View ArticlePubMedGoogle Scholar
- Bast RC, Lilja H, Urban N, Rimm DL, Fritsche H, Gray J, Veltri R, Klee G, Allen A, Kim N: Translational crossroads for biomarkers. Clin Cancer Res. 2005, 11 (17): 6103-6108. 10.1158/1078-0432.CCR-04-2213.View ArticlePubMedGoogle Scholar
- Lee JW, Figeys D, Vasilescu J: Biomarker assay translation from discovery to clinical studies in cancer drug development: quantification of emerging protein biomarkers. Adv Cancer Res. 2006, 96: 269-298.View ArticleGoogle Scholar
- Sung NS, Crowley JWF, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, et al: Central challenges facing the national clinical research enterprise. JAMA. 2003, 289 (10): 1278-1287. 10.1001/jama.289.10.1278.View ArticlePubMedGoogle Scholar
- Ioannidis J, Panagiotou OA: Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA. 2011, 305 (21): 2200-10.1001/jama.2011.713.View ArticlePubMedGoogle Scholar
- de Solla DJ P: Little science, big science. 1963, New York and London: Columbia University PressGoogle Scholar
- Butte AJ: Translational bioinformatics: coming of age. J Am Med Inform Assoc. 2008, 15 (6): 709-714. 10.1197/jamia.M2824.View ArticlePubMedPubMed CentralGoogle Scholar
- Burke W, Emery J: Genetics education for primary-care providers. Nat Rev Genet. 2002, 3 (7): 561-566. 10.1038/nrg845.View ArticlePubMedGoogle Scholar
- Khoury MJ, Little J, Gwinn M, Ioannidis JP: On the synthesis and interpretation of consistent but weak gene–disease associations in the era of genome-wide association studies. Int J Epidemiol. 2007, 36 (2): 439-445. 10.1093/ije/dyl253.View ArticlePubMedGoogle Scholar
- de Leon J: Evidence-based medicine versus personalized medicine: Are they enemies?. J Clin Psychopharmacol. 2012, 32 (2): 153-164. 10.1097/JCP.0b013e3182491383. 110.1097/JCP.1090b1013e3182491383View ArticlePubMedGoogle Scholar
- Siontis GC, Ioannidis JP: Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011, 40 (5): 1292-1307. 10.1093/ije/dyr099.View ArticlePubMedGoogle Scholar
- Ioannidis JPA: Effect of formal statistical significance on the credibility of observational associations. Am J Epidemiol. 2008, 168 (4): 374-383. 10.1093/aje/kwn156.View ArticlePubMedGoogle Scholar
- Panagiotou OA, Ioannidis JPA: Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol. 2012, 65 (7): 740-747. 10.1016/j.jclinepi.2012.01.008.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/176/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.