- Research article
- Open Access
- Open Peer Review
The implications of biomarker evidence for systematic reviews
BMC Medical Research Methodology volume 12, Article number: 176 (2012)
In Evidence-Based Medicine, clinical practice guidelines and systematic reviews are crucial devices for medical practitioners in making clinical decision. Clinical practice guidelines are systematically developed statements to support health care decisions for specific circumstances whereas systematic reviews are summaries of evidence on clearly formulated clinical questions. Biomarkers are biological measurements (primarily molecular) that are used to diagnose, predict treatment outcomes and prognosticate disease and are increasingly used in randomized controlled trials (RCT).
We search PubMed for systematic reviews, RCTs, case reports and non-systematic reviews with and without mentions of biomarkers between years 1990–2011. We compared the frequency and growth rate of biomarkers and non-biomarkers publications. We also compared the growth of the proportion of biomarker-based RCTs with the growth of the proportion of biomarker-based systematic reviews.
With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers. The annual growth rate of biomarkers publications is consistently higher than non-biomarkers publications, showing the growth in biomarkers research. From 20 years of systematic review publications indexed in PubMed, we identified a bias in systematic reviews against the inclusion of biomarker-based RCTs.
With the realisation of genome-based personalised medicine, biomarkers are becoming important for clinical decision making. The bias against the inclusion of biomarkers in systematic reviews leads to medical practitioners deprive of important information they require to address clinical questions. Sparse or weak evidence and lack of genetic training for systematic reviewers may contribute to this trend.
Evidence-Based Medicine (EBM) stipulates the use of the best available external scientific evidence in clinical decision making. Systematic Reviews (SRs) are conducted through a well defined process , and published as robust answers to clinical questions given the best available evidence . Randomized Controlled Trials (RCTs) are trusted above all other primary evidence types . EBM is thus implemented by following clinical practice guidelines (CPGs) or relying on SRs whenever possible when making clinical decisions. Guideline and SR development is a slow and complex process or translating research that can take more than a year  to complete. In this process it is critical to include all relevant evidence . It was previously claimed that as much as 80% of all published reviews are not updated and only 3% of systematic reviews published in peer-reviewed journals had been updated within two years of their publication . Only a small proportion of relevant trials are incorporated into systematic reviews .
The trend toward genome-based personalised medicine, and the rapid advancement of sequencing and high-throughput technologies, has dramatically reshaped disease research. Genomic data can now be obtained expeditiously and inexpensively. This has led to the use of a range of analytical tools to assess biological parameters .
Biomarkers that distinguish among subtypes of disease are now standard practice in biomedical research. There is an overwhelming interest in biomarker research reflected in a large number of research grants awarded and academic publications . However, only a limited number of biomarkers have been incorporated into clinical guidelines , and the anticipation that biomarker research will revolutionize medical practice has so far not been realized. The ambiguity of the term “biomarker” has prompted the US National Institutes of Health (NIH) to define the terminology. Biomarkers are defined by the NIH Biomarkers Definitions Working Group as “a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention” . Biomarkers are used as tools in disease diagnostic, early detection, staging, prognosis or prediction of treatment outcome. They can guide individualized treatment and improve patient care . Biomarkers can provide the basis for design, improve the safety and efficiency, and explain empirical results of clinical trials .
The translation of biomarkers into clinical practice follows the regular translational pathways from discovery to preclinical, clinical, and post-approval trials and implementation. However, some translational crossroads are unique to biomarkers [10, 11]. In particular, the diversity and lengthy process of biomarker assay development, and limited industry support were identified as bottlenecks for translating biomarker research into clinical practice. In this study, we examine the validation to implementation phase (also called T2 ). As there are cases of exaggeration in the effect sizes of many highly cited biomarkers studies (which lead to overestimated findings) , we measured the inclusion of RCTs of biomarkers in SRs. We believe that prospective evaluations of biomarkers in RCTs could provide more reliable results about their effects and clinical utility.
We address the following questions:
How does the output frequency of publications that include biomarkers compare to those that don’t?
Is the growth rate of publication of biomarkers RCTs in line with the non-biomarkers publications?
To which extent biomarkers RCTs are being included in SRs as compared to non-biomarkers RCTs?
Although there is no single database reliably showing the true number of SRs or RCTs , we used PubMed as the basis of our search. We searched for systematic reviews, controlled trials, case reports and non-systematic reviews by using the publication type in PubMed's Entrez search engine using the terms shown in Table 1.
There are ambiguities in the naming and terminology of the term “biomarker” within the literature . As we are interested in all biomarkers, we have included as many terms with overlapping meaning of “biomarker” as possible, such as biological marker/s, molecular marker/s, genetic marker/s, DNA marker/s, cytogenetic marker/s, proteomics marker/s and biochemical marker/s. We repeated each search with (biomarker OR "biological marker" OR “biological markers” OR "molecular marker" OR "molecular markers" OR "genetic marker" OR "genetic markers" OR "DNA marker" OR "DNA markers" OR "cytogenetic marker" OR "cytogenetic markers" OR "proteomics marker" OR "proteomics markers" OR "biochemical marker" OR "biochemical markers") helped restrict the results to those biomarkers publications only, and (all[sb] NOT (biomarker OR "biological marker" OR “biological markers” OR "molecular marker" OR "molecular markers" OR "genetic marker" OR "genetic markers" OR "DNA marker" OR "DNA markers" OR "cytogenetic marker" OR "cytogenetic markers" OR "proteomics marker" OR "proteomics markers" OR "biochemical marker" OR "biochemical markers")) for non-biomarkers publications. “All publications” is the summation of the above two complementary groups. We limited our search to human subjects and to documents in English. All searches were done on the 18th of October 2012.
In order to validate the search strategy, we randomly chose 110 search results from each group (biomarkers and non-biomarkers) of RCT/CCT (“Randomized Controlled Trial” or “Controlled Clinical Trial”) and manually read their abstracts and if needed their full text, to determine if the trial is indeed about biomarkers or using biomarkers as outcome measures.
The average annual growth rates are calculated using the following equation:
where x i is the current year's total publications and N=22 is the total number of years included in the study.
In order to assess the extent of inclusions of clinical trials publications in systematic reviews, we normalized the systematic reviews samples by the number of published clinical trials.
The validation of search results on RCT/CCT yields an accuracy of 0.82, with sensitivity of 0.76 and specificity of 0.94.
While only a minority of trials has been included in systematic reviews , this study found that the number of trials involving biomarkers assessed in systematic reviews is even smaller. With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers.
Figure 1 shows a logarithmic representation of the cumulative number of publications of clinical evidence (SR/MT (“Systematic Reviews” or “Meta-Analysis”), RCT/CCT, case reports and non-systematic reviews). The growth of the publications of clinical evidence appears to support the Prices Law of exponential growth . The total number of publications has increased tremendously, from as low as 691 for SR/MA, 7402 for RCT/CCT, 26,721 for case reports, and 25,261 for non-systematic reviews in 1990 to 19,195, 23,488, 51,019, 74,411 in year 2011 respectively.
Figure 2 and Figure 3 show a logarithmic representation of the total number of publications of SR/MA, RCT/CCT, case reports and non-systematic reviews on biomarkers and non-biomarkers from 1990 to 2011. Figure 2 shows that as the number of published trials, case reports and non-systematic reviews increased for biomarkers publications, so did the number of systematic reviews. Figure 3 shows the same trend for non-biomarker publications. The average annual growth rates of biomarkers and non-biomarkers publications are shown in Table 2. The annual growth rate of case reports and non-systematic reviews are comparatively lower than the SR/MA and RCT/CCT which are at the top two of the hierarchy of clinical evidence. As evidenced by annual growth rate where biomarkers publications are consistently higher than non-biomarkers publications, there is a growth in biomarkers research .
The ratio of systematic reviews over the published clinical trials (ratio SR/RCT) of biomarkers and non-biomarkers are shown in the solid line and dotted line of Figure 4 respectively. The ratio SR/RCT represents how many systematic reviews are published per clinical trials in the same year. In other words, how likely a published clinical trial is to be included in a SR. The ration SR/RCT of both biomarkers and non-biomarkers publications is growing, reflecting the efforts of the review community in assessing the published trials and producing systematic reviews at a growing rate. The ratio SR/RCT of non-biomarkers publications is growing and reached over 0.8, but the increased ratio SR/RCT on biomarkers is comparatively lower and only reached 0.6 (if following the trend of overall publication, there are about 20% to over 100% less systematic reviews on biomarkers over the 22 years). The results show that there are fewer systematic reviews published per published clinical trials in biomarker-domain.
Figure 5 is a plot of ratio SR/RCT of biomarkers versus SR/RCT of non-biomarkers publications. The dotted line in Figure 5 represents the equal ratio of biomarkers and non-biomarkers publications trend. The biomarkers SR/RCT is consistently below that of non-biomarker publications as shown in Figure 4 and Figure 5. With the hypothesis that the likelihood of published clinical trials is being included in systematic reviews for biomarkers and non-biomarkers are the same, it is clear that there is a preference towards non-biomarkers publications.
From Figure 6, we can observe that the percentages biomarkers publications in clinical trials and systematic reviews are increasing. However, the proportion increase in biomarker systematic reviews is comparatively lower (the percentages in early 1990s might not be a good indication as there are only a low number of published trials and systematic reviews). Figure 6 also strengthen the evidence that biomarkers are an active field of research as the proportion of both biomarkers trials and systematic reviews increases. The growth in the number of biomarkers used in clinical trials and systematic reviews has shown its importance.
Biomedical research articles often include poor reporting of statistical methods. Incomplete reporting of statistical analysis limits or prevents the use of these studies in the systematic reviews. This could explain why only a small proportion of RCTs are included in systematic reviews. Further study is needed to test if biomarkers RCTs are more prone to bad statistical methods reporting than other RCTs.
Our results show discrepancy between the rate of publications of biomarkers RCTs and biomarkers reviews. Possible explanations for this include:
The frequent proposals of new biomarkers and assays which have complicated the translation and commercialization processes .
Independent investigations in biomarkers show contradicting results. A biomarker is ready for clinical testing only when several retrospective tests consistently confirm its performance . As such, weak and sparse evidence could not lead to any conclusion.
Systematic reviewers shy away from a domain in which their training is lacking .
Genetic disorders, where specific genetic variants causally associated with common diseases, account for only a fraction of cases , which result in smaller cohorts for studies.
RCTs, observational studies or cross-sectional diagnostic studies are designed to answer generalized clinical questions and ignore outliers. Biomarker evidence might be too sensitive to outliers  to provide low p-values.
A few outliers might also cause small effects even in large cohort studies  which would result in exclusion from systematic reviews as they will provide no clear difference from previous summarized evidence.
As systematic reviews are the main source for guideline development, the lack of effort on systematically reviewing studies in this area can explain why only a limited number of molecular markers have been incorporated into clinical guidelines. There are still many relevant trials not being assessed or not included in systematic reviews and guidelines. This seems to be stronger in biomarker evidence as we have shown.
Biomarkers are becoming increasingly valuable in clinical settings, whether to diagnose, prognosticate or to guide treatment. It is important to fast track the research and translation process. We identified the need for systematic reviewers to include more biomarkers and proposed several possible explanations of why this has not been done yet including lacking education of systematic reviewers on molecular biology concepts and the low predictive power of biomarkers. We propose that specific search technologies can support the review process.
Sackett DL, Strauss SE, Richardson WS, Rosenberg WMC, Haynes RB: 2000, Edinburg, Scotland: Evidence-based medicine: how to practice and teach EBM, Churchill Livingstone
Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312 (7023): 71-72. 10.1136/bmj.312.7023.71.
Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, Dröschel D, Beck L, Abalos E, El-Jardali F, Gilson L, et al: Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012, 9 (3): e1001185-10.1371/journal.pmed.1001185.
Cohen AM, Adams CE, Davis JM, Yu C, Yu PS, Meng W, Duggan L, McDonagh M, Smalheiser NR: Proceedings of the 1st ACM international health informatics symposium. Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools. 2010, Virginia, USA: ACM, Arlington, 376-380.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007, 4 (3): e78-10.1371/journal.pmed.0040078.
Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: How will We ever keep Up?. PLoS Med. 2010, 7 (9): e1000326-10.1371/journal.pmed.1000326.
Atkinson AJ, Colburn WA, DeGruttola VG, DeMets DL, Downing GJ, Hoth DF, Oates JA, Peck CC, Schooley RT, Spilker BA, et al: Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework*. Clin Pharmacol Ther. 2001, 69 (3): 89-95.
Ptolemy AS, Rifai N: What is a biomarker? Research investments and lack of clinical integration necessitate a review of biomarker terminology and validation schema. Scand J Clin Lab Invest. 2010, 70 (S242): 6-14. 10.3109/00365513.2010.493354.
Choong MK, Tsafnat G: Genetic and epigenetic biomarkers of colorectal cancer. Clin Gastroenterol Hepatol. 2012, 10 (1): 9-15. 10.1016/j.cgh.2011.04.020.
Bast RC, Lilja H, Urban N, Rimm DL, Fritsche H, Gray J, Veltri R, Klee G, Allen A, Kim N: Translational crossroads for biomarkers. Clin Cancer Res. 2005, 11 (17): 6103-6108. 10.1158/1078-0432.CCR-04-2213.
Lee JW, Figeys D, Vasilescu J: Biomarker assay translation from discovery to clinical studies in cancer drug development: quantification of emerging protein biomarkers. Adv Cancer Res. 2006, 96: 269-298.
Sung NS, Crowley JWF, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, et al: Central challenges facing the national clinical research enterprise. JAMA. 2003, 289 (10): 1278-1287. 10.1001/jama.289.10.1278.
Ioannidis J, Panagiotou OA: Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA. 2011, 305 (21): 2200-10.1001/jama.2011.713.
de Solla DJ P: Little science, big science. 1963, New York and London: Columbia University Press
Butte AJ: Translational bioinformatics: coming of age. J Am Med Inform Assoc. 2008, 15 (6): 709-714. 10.1197/jamia.M2824.
Burke W, Emery J: Genetics education for primary-care providers. Nat Rev Genet. 2002, 3 (7): 561-566. 10.1038/nrg845.
Khoury MJ, Little J, Gwinn M, Ioannidis JP: On the synthesis and interpretation of consistent but weak gene–disease associations in the era of genome-wide association studies. Int J Epidemiol. 2007, 36 (2): 439-445. 10.1093/ije/dyl253.
de Leon J: Evidence-based medicine versus personalized medicine: Are they enemies?. J Clin Psychopharmacol. 2012, 32 (2): 153-164. 10.1097/JCP.0b013e3182491383. 110.1097/JCP.1090b1013e3182491383
Siontis GC, Ioannidis JP: Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011, 40 (5): 1292-1307. 10.1093/ije/dyr099.
Ioannidis JPA: Effect of formal statistical significance on the credibility of observational associations. Am J Epidemiol. 2008, 168 (4): 374-383. 10.1093/aje/kwn156.
Panagiotou OA, Ioannidis JPA: Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol. 2012, 65 (7): 740-747. 10.1016/j.jclinepi.2012.01.008.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/176/prepub
The authors would like to thank Professor Paul Glasziou for his feedback on an early version of this manuscript.
MKC was supported by National Health and Medical Research Council Program Grant 568612. GT was supported by the New South Wales Health Capacity Building Infrastructure Grant. The article-processing charge for the manuscript was supported by Career Advancement Fund from UNSW.
The authors declare that they have no competing interests.
MKC conceived the study, gathered and analysed data, and drafted the manuscript. GT conceived the study, interpreted the information, and drafted the manuscript. Both authors read and approved the final manuscript.