Evidence at a glance: error matrix approach for overviewing available evidence
© Keus et al. 2010
Received: 1 March 2010
Accepted: 1 October 2010
Published: 1 October 2010
Skip to main content
© Keus et al. 2010
Received: 1 March 2010
Accepted: 1 October 2010
Published: 1 October 2010
Clinical evidence continues to expand and is increasingly difficult to overview. We aimed at conceptualizing a visual assessment tool, i.e., a matrix for overviewing studies and their data in order to assess the clinical evidence at a glance.
A four-step matrix was constructed using the three dimensions of systematic error, random error, and design error. Matrix step I ranks the identified studies according to the dimensions of systematic errors and random errors. Matrix step II orders the studies according to the design errors. Matrix step III assesses the three dimensions of errors in studies. Matrix step IV assesses the size and direction of the intervention effect.
The application of this four-step matrix is illustrated with two examples: peri-operative beta-blockade initialized in relation to surgery versus placebo for major non-cardiac surgery, and antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation. When clinical evidence is deemed both internally and externally valid, the size of the intervention effect is to be assessed.
The error matrix provides an overview of the validity of the available evidence at a glance, and may assist in deciding which interventions to use in clinical practice.
Evidence-based medicine (EBM) was first introduced in 1992 , and its increased application is reflected among others by the growth of The Cochrane Library databases as well as implementation of evidence-based guidelines into clinical practice . EBM underpins that information provided from randomized trials, and systematic reviews of randomized trials represent the most reliable evidence regarding intervention effects [3, 4]. Thanks to the sustained scientific process (Additional file 1: Table S1), we now know that the reliability of what we observe varies due to a whole array of different factors. There are three dimensions that particularly influence the reliability of our observations in clinical research and they are empirically and theoretically well accepted: the risk of systematic error ('bias'), the risk of random error ('play of chance'), and the risk of design error ('wrong design to answer the posed question') [4, 9].
In daily clinical practice, the question of whether sufficient evidence is available to recommend the implementation of a specific intervention as a treatment arises frequently . Depending on the specific clinical question, often an exhaustive list of references is retrieved when using a sensitive search strategy in multiple databases . After the selection of studies, their data must be interpreted [10–13].
Since results may be contradictory and studies may differ in more than one aspect, to draw a clear, practical conclusion from the publications may be problematic .
We aimed at conceptualizing a visual assessment tool, i.e., a matrix for overviewing studies and their data in order to assess the clinical evidence. The matrix is constructed from the three dimensions of errors: systematic error ('bias'), random error ('play of chance'), and design error ('wrong design to answer the question posed' or 'wrong context'). The application of this matrix will be illustrated by two examples: peri-operative beta-blockade initialized in relation to surgery versus placebo for major non-cardiac surgery, and antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation.
When evaluating a clinical study, one should always try to assess its risk of systematic error [3, 4, 9–16]. There is increasing agreement on how trials and studies can be placed in a hierarchy when assessing the risk of systematic error [3, 4, 9–16], depending on the type of research (therapeutic, diagnostic, etiologic, or prognostic) [3, 10, 11, 17]. The risk of systematic error influences the reliability of observed intervention effects [3, 10, 11, 18, 19]. A significant association between inadequate or unclear bias protection and overestimation of beneficial effects and underreporting of adverse effects has been demonstrated [16, 19–23]. Differences in risk of bias are found both between the different levels of evidence and within each level of evidence [4, 16, 20].
For randomized trials, there is empirical evidence that at least six components are associated with systematic error: generation of the allocation sequence , allocation concealment , blinding , incomplete outcome measure reporting , selective outcome measure reporting , and other bias mechanisms (e.g., baseline imbalance, early stopping, vested interests, etc.) [4, 16, 20, 27–29]. The impact of early stopping of trials on bias is largely dependent on how the stopping rules were defined and the level of statistical significance of the interim analysis [30–32]. Trials with one or more systematic error components assessed as inadequate or unclear are considered to be of high risk of bias, while trials with all quality components assessed as adequate are considered to be of low risk of bias [15, 27, 33]. Trials with a low risk of bias are more likely to estimate the 'true' effect of the intervention [16, 20, 27, 33].
Categorization of systematic error (bias) of clinical intervention studies into levels of evidence
Meta-analysis of randomized trials with low risk of bias
Randomized trial with low risk of bias
Meta-analysis of all randomized trials
Randomized trial with high risk of bias
Meta-analysis of cohort studies
Meta-analysis of case-control studies
The risk of random error is the risk of drawing a false conclusion based on sparse data. There are two types of false conclusions: a false rejection of the null hypothesis (type I error; alpha) or a false acceptance of the null hypothesis (type II error; beta). When data are sparse, then the so called 'intervention effect', whether beneficial or harmful, may in fact be caused by randomly skewed variation in prognostic factors between the intervention groups due to sampling error.
and N i being the total number of patients in a trial.
SE depends on the numbers of events and the sample size.
Due to spurious results, incorrect type I error inferences may be drawn. Recent reports indicate that the influence of the 'play of chance' may be much larger than generally perceived . In randomized trials, random error may be one reason for the early stopping of trials at interim analyses when benefit or harm appear to be significant [32, 40]. Increased random error may also play a role in the repeated analyses of accumulating data in both trials and meta-analyses [36, 41–44]. A cumulative meta-analysis subjects accumulating data to repeated testing of the data and is bound to eventually lead to a false rejection of the null hypothesis ('false positive' result) [45, 46]. The random error phenomenon or 'multiplicity' also plays a role in the evaluation of secondary outcome measures . For example, when data on the primary research outcome, on which the sample size calculation was based, may not show statistical significance, while another outcome measure, for which no separate sample size calculation was performed, exhibits statistical significance [47, 48].
Random error may be expressed in a continuous variable using the standard error of for example the log of Peto odds ratios or the log of relative risks.
Types of variables to consider when evaluating the risk of design errors ('context errors') and hence external validity of evidence
Clinical centres or settings including patients
Goal - explanatory or pragmatic
Trial structure - parallel group, crossover, etc
Objective - superiority, equivalence, non-inferiority
Unit of analysis
GRADE has schematically ordered outcomes according to patients' perspective on a categorical scale from 1 to 9, with the most critical outcome, mortality, being graded 9 . Depending on the outcomes, this scale should sometimes be considered nominal and in other situations be considered functional. Moreover, the severity of each outcome may differ as well. A stroke can be minor, while a myocardial infarction may involve a substantial worsening of cardiac function. Grading of outcome measures may also vary according to the clinical question. Therefore, outcomes within a category (i.e., critical, important, or not important) may be interchangeable. However, one can hardly argue that outcomes between categories (i.e., critical, important, or not important) are interchangeable (e.g., mortality is always more important than length of stay in hospital).
Eventually, the design error dimension can be expressed by the priority of outcome measures as an ordinal variable according to GRADE .
A four-step matrix can be constructed, building upon the three dimensions: systematic error, random error, and design error. Matrix step I ranks the identified studies according to the dimensions of systematic errors and random errors. Matrix step II orders the studies according to the design errors. Matrix step III assesses the three dimensions of errors in studies. Here, a 'Manhattan-like' error matrix is constructed where the best evidence is represented by the largest skyscrapers located on the 'upper-west side'. Matrix step IV assesses the size and direction of the intervention effect, e.g., by calculating the number-needed-to-treat to obtain benefit or to harm one patient.
The principle of the matrix approach can be used in different situations. The overall effort in research should be to minimize all three risks of errors before the size and the direction of the intervention effect can be assessed reliably. The 'algorithm' of the matrix approach is generally applicable to all kinds of interventions, although details may differ according to the specific clinical question. Or, the character of the three dimensions remains the same, while according to the specific question details may differ, like: the preferred hierarchy for levels of evidence, the chosen formula for standard error (RR, OR peto , or any other association metric), and the types of outcomes.
The application of this four-step matrix is illustrated with two examples: peri-operative beta-blockade initialized in relation to surgery versus placebo for major non-cardiac surgery, and antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation.
A clinical question in PICOT structure illustrates this model. Is initiating peri-operative beta-blockade effective in patients undergoing major non-cardiac surgery?
Patients: patients undergoing major non-cardiac surgery; Intervention: initiating peri-operative beta blockade; Control: placebo; Outcome measure: mortality, myocardial infarction, and stroke; Time: follow-up of at least 30 days.
We searched in CENTRAL in The Cochrane Library, PubMed, EMBASE, and personal files for all article types up to October 2009, in all languages. Specific searches using the terms 'beta-blockade', 'peri-operative', 'placebo', 'mortality', 'randomised', and 'non-cardiac surgery' were undertaken. The search resulted in multiple publications relevant to our question. References were selected from journals on the basis of importance and relevance [50–58]. We included the publications in our matrix evaluation by extracting information on all-cause mortality, cardiovascular mortality, non-fatal myocardial infarction, and non-fatal stroke. However, the matrix may easily be extended to other outcomes.
Ordering of evidence according to levels of evidence (systematic error), standard error (random error), and outcome measures (design error) in peri-operative beta-blockade versus placebo for major non-cardiac surgery (example 1).
Level of evidence
Non-fatal myocardial infarction
AHA Guidelines 
From Figure 5 it can be concluded at a glance that peri-operative beta blockade does not reduce mortality in patients undergoing major non-cardiac surgery. Peri-operative beta-blockade in these patients seems to increase all-cause mortality. However, peri-operative beta-blockade does reduce non-fatal myocardial infarction on the expense of an increased cardiovascular mortality and an increased rate of non-fatal stroke.
The conclusion of this Cochrane review focuses on the significant increased mortality associated with use of class 1a antiarrhythmics (odds ratio 2.39; 95% confidence interval (CI) 1.03 to 5.59) . The data of this outcome in class 1a antiarrhythmics in this review  as well as in the included randomised trials [60–67] were analysed using the matrix error approach.
Ordering of evidence according to levels of evidence (systematic error), standard error (random error), and outcome measures (design error) in antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation (example 2)
From Figure 7 it can be concluded at a glance that there is both substantial risk of systematic and random error involved in the evidence available so far considering mortality associated with class 1a antiarrhythmics. The best available level of evidence 1c study shows substantial risk of random error (0.43) and the best available level of evidence 1b study shows high risk of random error (0.78). So, the conclusion in the Cochrane review of a significant increased mortality is based on data with high risks for both systematic and random errors, and should therefore be considered unreliable.
The aim of our matrix is to facilitate the overview of evidence in clinical intervention research. The matrix can serve as a tool to provide visual assessment of reliability of observations with respect to systematic error, random error (internal validity), and design error (external validity).
The matrix should not replace the thorough process of systematically reviewing evidence and profound evaluations of data, but could be integrated within these research activities as a tool for overviewing the results. Also, this matrix is not an absolute measure of the risks of errors. The position of studies in relation to each other is relative rather than absolute.
There is a lack of awareness of the importance of the 'play of chance' for the reliability of study findings. Ordering the standard errors of the studies might be a tool for ranking studies according to the level of random error. We have used natural logarithm (ln) transformations for calculating standard errors, although the logarithm with the base 10 may be used without producing different conclusions.
or simple approximations can be very difficult or even impossible to implement for the clinician, since a search for the maximum of the multidimensional posterior may be required for each association . This also includes the asymptotic Bayes factor introduced by Wakefield . In contrast to the Bayes factor, it is possible to calculate the standard error and when available it provides a tool for comparison of the risk of random error between studies of the same intervention.
The aim of minimising error risks according to the three dimensions actually combines the methodological efforts of falsifying any alternative hypothesis in the evaluation of an intervention. Thereby, the matrix concept visualises how far the scientific process has evolved to fulfil Poppers falsification criterion stating that researchers should primarily engage trying to falsify any relevant alternative hypothesis and not only the null hypothesis . The minimisation of systematic errors and random errors, by providing ample room for the null hypothesis, as well as measuring important outcomes is the most audacious attack on any realistic alternative hypothesis. If an array of progressively qualified attacks fails to support the null hypothesis then we can reliably trust the intervention to be either beneficial or harmful.
The conclusion based on an assessment of the evidence using the matrix approach may be implemented into clinical practice or serve as an incentive for new research. The matrix facilitates the identification of lacunae in our knowledge and is likely to benefit the process of developing evidence-based guidelines.
One has to be aware of the multiple forms of bias, potentially present in evidence below level 1 (Table 1). Several examples illustrate that large, apparently beneficial intervention effects from lower level evidence, even from randomized trials [54, 56, 70], may eventually be reversed to harmful effects when new high-quality evidence appears [50, 71]. This is where the three dimensions of error are of central importance in providing a tool for reliability assessment.
Apart from the three error dimensions influencing the reliability of data, other factors play a role in incomparability and uncertainty of inferences. Many reports of studies appear incomplete, and the lack of details raises questions. Incomplete reporting limits interpretation, but more importantly, this reporting factor should be distinguished from the methodological quality of the trial .
Statements like CONSORT , PRISMA , and MOOSE  aim to improve and to maximize the amount and correctness of information to be retrieved from publications. These guidelines also create awareness among researchers about the most important issues to report so that the quality of future research may increase. By following reporting guidelines the yield of the research question is likely to be increased (phase 1 in Figure 1).
Standard error does not consider testing of multiple outcomes and multiple testing on accumulating data, which may also induce risks of random error due to multiplicity as well as correlations.
The division of all outcomes into 'primary' and 'secondary' outcome measures can be helpful as this division sets the standards for the evaluation of interventions. However, this division is artificial, and outcome measures, situated on the border of primary and secondary outcomes, exist. For example, one can argue that quality of life is a primary outcome rather than a secondary outcome. Further, there is also a quantitative aspect in the artificial division into primary and secondary outcomes. Small significant differences in primary outcome measures (e.g., bile duct injuries in patients undergoing cholecystectomy) may be found favouring one intervention, while large differences in secondary outcome measures (e.g., costs) may favour the comparator. Eventually, one may prefer the larger advantages in secondary outcomes to the smaller disadvantages in a primary outcome measure.
Another limitation in the outcome measure dimension is that often outcome measures are correlated and mostly this correlation is ignored. For example when mortality is an outcome measure and complications is another, which again counts deaths as complications, then there is a correlation between the two outcome measures. Authors usually carry out multiple univariate analyses ignoring correlations between outcome measures.
Step IV of the matrix includes the assessment of the size of the intervention effect, e.g., expressed in numbers-needed-to-treat to obtain benefit or to harm one patient with the intervention. This step is the last one since it is irrelevant to consider effect sizes and their directions if a study does not appear to be internally and externally valid.
Another aspect to consider is heterogeneity [76, 77]. Statistical heterogeneity reflects the between trial variance of meta-analytic intervention effect estimates rather than the play of chance . Clinical heterogeneity, however, represents differences in populations, procedures, or interventions in daily practice. All these factors of clinical heterogeneity, together with concordance of in- and exclusion criteria should be considered whenever we want to implement results of available evidence. Assessment and consideration of heterogeneity or diversity, therefore, forms the final step before new evidence is implemented. Assessment of heterogeneity is not included in our matrix.
Assessment of risks of systematic error, random error, and design error are essential factors in evaluating evidence and drawing conclusions. We used the standard error in our matrix to rank studies according to their risk of random error. The risks of these error types were incorporated into a three dimensional matrix to create a schematic overview of the internal and external validity of the evidence, seen at a glance.
There was no funding.
We thank Dimitrinka Nikolova and Kate Whitfield for linguistic assistance, Esther Lutteke for graphical assistance, and Gert-Jan van der Wilt and Rogier Donders for useful suggestions for improvement.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.