Overview of data-synthesis in systematic reviews of studies on outcome prediction models
© van den Berg et al.; licensee BioMed Central Ltd. 2013
Received: 26 September 2012
Accepted: 4 March 2013
Published: 16 March 2013
Many prognostic models have been developed. Different types of models, i.e. prognostic factor and outcome prediction studies, serve different purposes, which should be reflected in how the results are summarized in reviews. Therefore we set out to investigate how authors of reviews synthesize and report the results of primary outcome prediction studies.
Outcome prediction reviews published in MEDLINE between October 2005 and March 2011 were eligible and 127 Systematic reviews with the aim to summarize outcome prediction studies written in English were identified for inclusion.
Characteristics of the reviews and the primary studies that were included were independently assessed by 2 review authors, using standardized forms.
After consensus meetings a total of 50 systematic reviews that met the inclusion criteria were included. The type of primary studies included (prognostic factor or outcome prediction) was unclear in two-thirds of the reviews. A minority of the reviews reported univariable or multivariable point estimates and measures of dispersion from the primary studies. Moreover, the variables considered for outcome prediction model development were often not reported, or were unclear. In most reviews there was no information about model performance. Quantitative analysis was performed in 10 reviews, and 49 reviews assessed the primary studies qualitatively. In both analyses types a range of different methods was used to present the results of the outcome prediction studies.
Different methods are applied to synthesize primary study results but quantitative analysis is rarely performed. The description of its objectives and of the primary studies is suboptimal and performance parameters of the outcome prediction models are rarely mentioned. The poor reporting and the wide variety of data synthesis strategies are prone to influence the conclusions of outcome prediction reviews. Therefore, there is much room for improvement in reviews of outcome prediction studies.
KeywordsReview Systematic Meta-analysis Prediction Prognosis Forecasting Methods
The methodology for prognosis research is still under development. Although there is abundant literature to help researchers perform this type of research [1–5], there is still no widely agreed approach to building a multivariable prediction model . An important distinction in prognosis is made between prognostic factor models, also called explanatory models and outcome prediction models [7, 8]. Prognostic factor studies investigate causal relationships, or pathways between a single (prognostic) factor and an outcome, and focus on the effect size (e.g. relative risk) of this prognostic factor which ideally is adjusted for potential confounders. Outcome prediction studies, on the other hand, combine multiple factors (e.g. clinical and non-clinical patient characteristics) in order to predict future events in individuals, and therefore focus on absolute risks, i.e. predicted probabilities in logistic regression analysis. Methods that can be used to summarize data from prognostic factor studies in a meta-analysis can easily be found in the literature [9, 10], but this is not the case for outcome prediction studies. Therefore, in the present study we focus on how authors of published reviews have synthesized outcome prediction models. The nomenclature to indicate various types of prognosis research is not standardized. We use prognosis research as an umbrella term for all research that might explain or predict a future outcome and prognostic factor and outcome prediction as specific types of prognosis studies.
In 2006, Hayden et al. showed that in systematic reviews of prognosis studies, different methods are used to assess the quality of primary studies . Moreover, when quality is assessed, integration of these quality scores in the synthesis of the review is not guaranteed. For reviews of outcome prediction models, additional characteristics are important in the synthesis of models to reflect choices made in the primary studies, such as which variables are included in statistical models and how this selection was made. These choices therefore also reflect the internal and external validity of a model and influence the predictive performance of the model. In systematic reviews the researchers synthesize results across primary outcome prediction studies which include different variables and show methodological diversity. Moreover, relevant information is not always available, due to poor reporting in the studies. For example, several researchers have found that current knowledge about the recommended number of events per variable, and the coding and selection of variables, among other features, are not always reported in primary outcome prediction research [12–14]. Although improvement in primary studies themselves is needed, reviews that summarize outcome prediction evidence need to consider the current diversity in methodology in primary studies.
In this meta-review we focus on reviews of outcome prediction studies, and how they summarize the characteristics of design and analysis, and the results of primary studies. As there is no guideline nor agreement how primary outcome prediction models in medical research and epidemiology should be summarized in systematic reviews, an overview of current methods helps researchers to improve and develop these methods. Moreover, current methods for outcome prediction reviews are unknown to the research community. Therefore, the aim of this review was to provide an overview on how published reviews of outcome prediction studies describe and summarize the characteristics of the analyses in primary studies, and how the data is synthesized.
Literature search and selection of studies
Systematic reviews and meta-analyses of outcome prediction models that were published between October 2005 and March 2011 were searched. We were only interested in reviews that included multivariable outcome prediction studies. In collaboration with a medical information specialist, we developed a search strategy in MEDLINE, extending on the strategy used by Hayden , by adding recommended other search terms for predictive and prognostic research [15, 16]. The full search strategy is presented in Appendix 1.
Based on title and abstract, potential eligible reviews were selected by one author (TvdB), who in case of any doubt included the review. Another author (MH) checked the set of potential eligible reviews. Ineligible reviews were excluded after consensus between both authors. The full texts of the included reviews were read, and if there was any doubt on eligibility a third review author (HdV) was consulted. The inclusion criteria were met if the study design was a systematic review with or without a meta-analysis, multiple variables were studied in an outcome prediction model, and the review was written in the English language. Reviews were excluded if they were based on individual patient data only, or when the topic was genetic profiling.
A data-extraction form was developed, based on important items to prognosis [1, 2, 12, 13, 17], to assess the characteristics of reviews and primary studies and is available from the first author on request. The items on this data-extraction form are shown in Appendix 2. Before the form was finalized it was pilot-tested by all review authors and minor adjustments were made after discussion about the differences in scores. One review author scored all reviews (TvdB) while other review authors (MH, AV, DV, and SL) collectively scored all reviews. Consensus meetings were held within 2 weeks after a review had been scored to solve disagreements. If consensus was not reached, a third reviewer (MH or HdV) was consulted to make a final decision.
An item was scored ‘yes’ if positive information was found about that specific methodological item, e.g. if it was clear that sensitivity analyses were conducted. If it was clear that a specific methodological requirement was not fulfilled, a ‘no’ was scored, e.g. no sensitivity analyses were conducted. In case of doubt or uncertainty, ‘unclear’ was scored. Sometimes, a methodological item could be scored as ‘not applicable’. The number of reviews within a specific answer category was reported, as well as the proportion.
Literature search and selection process
After completing the data-extraction form for all of the included reviews, most disagreements between review authors were found on items concerning the review objectives, the type of primary studies included, and the method of qualitative data-synthesis. Unclear reporting and, to a lesser degree, reading errors contributed to the disagreements. After consensus meetings only a small proportion of items needed to be discussed with a third reviewer.
Objective and design of the review
Characteristics of the reviews and provided information about the included primary studies
n = 50 reviews
Description of item:
Section 1: Information about the objective and design of the reviews
Type of primary studies included
n = 50 (%)
Only outcome prediction models
Combination of prognostic factor & outcome prediction studies
Is the outcome of interest clearly described?
Is information about quality assessment provided?
Methodological criteria list
Methodological criteria & study design
Was study quality accounted for
Method used *#
n = 23 (%)
Exclusion of poor quality studies (cut-off score used)
Sensitivity analysis based on total quality score
Levels of evidence
Study findings weighted for quality
Section 2: Information about the design and results of the primary studies
Outcomes clearly described
Statistical methods used for variable selection described
Univariable point estimates for all the variables of the primary studies are provided
Univariable estimates for dispersion for all the variables of the primary studies are provided
All variables (starting predictors) used to develop a model are described
Multivariable point estimates for each predictor in the final outcome prediction model are provided
Multivariable estimate of dispersion provided for each predictor in the final outcome prediction model
Model performance is assessed and described
number of events per variable is described
Section 3: Data-analysis and synthesis in the reviews
Heterogeneity between studies described
Qualitative data-synthesis presented
n = 49 (%)
Consistency of findings
Consistency of findings & statistical significance
Available method of defining levels of evidence
Consistency of findings & levels of evidence
Quantitative analysis performed
n = 10 (%)
Random effects model
Fixed effects model
Random & Fixed effects model
n = 10 reviews
Statistical heterogeneity assessed
Method used to assess statistical heterogeneity
n = 4 (%)
I2 & Chi2
n = 50 reviews
Graphic presentation of results provided
n = 8 (%)
Forest plot & scatter plot
Sensitivity analysis performed
n = 6 (%)
Different cut-offs for study quality
Methodological criteria & weights for quality
Including other (excluded) cohorts
Information about the design and results of the primary studies
In Table 1, section 2 shows information provided about the included primary studies. The outcome measures used in the included studies were reported in most of the reviews. Only 2 reviews [28, 52] described the statistical methods that were used in the primary studies to select variables for inclusion of a final prediction model, e.g. forward or backward selection procedures, and 6 others whether and how patients were treated.
A minority of reviews [23, 24, 27, 28] described for all studies the variables that were considered for inclusion in the outcome prediction model and only 5 reviews [36, 37, 39, 48, 55] reported univariable point estimates (i.e.. regression coefficients or odds ratios) and estimates of dispersion (e.g. standard errors) of all studies. Similarly, multivariable point estimates and estimates of dispersion were reported in respectively 11 and 10 of the reviews [21, 26, 27, 31, 33, 37, 44, 52],[55, 64, 65].
With regard to the presentation of univariable and multivariable point estimates, 2 reviews presented both types of results [37, 55], 31 did not report any estimates, and 17 reviews were unclear or reported only univariable or multivariable results [not shown in the table]. Lastly, model performance and number of events per variable were reported in 7 reviews [32, 39, 41, 60, 61, 65, 66] and 4 reviews [40, 48, 58, 61], respectively.
Data-analysis and synthesis in the reviews
Table 1, section 3 illustrates how the results of primary studies were summarized in the reviews. It shows that heterogeneity was described in almost all reviews by reporting differences in the study design and the characteristics of the study population. All but one review  summarized the results of included studies in a qualitative manner. Methods that were mainly used for that purpose were number of statistical significant results, consistency of findings, or a combination of these. Quantitative analysis, i.e. statistical pooling, was performed in 10 of the 50 reviews [25, 28, 31, 36, 37, 44, 45, 57–59]. The quantitative methods used included random effects models and fixed effects models of regression coefficients, odds ratios or hazard ratios. Of these quantitative summaries, 40% assessed the presence of statistical heterogeneity using I2, Chi2, or the Q statistic. In two reviews [25, 59], statistical heterogeneity was found to be present, and subgroup analysis was performed to determine the source of this heterogeneity [results not shown]. In 8 of the reviews there was a graphical presentation of the results, in which a forest plot [25, 28, 36–38, 52, 59], per single predictor, was the frequently used method. Other studies used a barplot  or a scatterplot . In 6 reviews [25, 26, 32, 43, 46, 58] a sensitivity analysis was performed to test the robustness of the choices made such as changing the cut-off value for a high or low quality primary study.
We made an overview of how systematic reviews summarize and report the results of primary outcome prediction studies. Specifically, we extracted information on how the data-synthesis was performed in reviews since outcome prediction models may consider different potential predictors, and include a dissimilar set of variables in the final prediction model, and use a variety of statistical methods to obtain an outcome prediction model.
Currently, in prognosis studies a distinction is made between outcome prediction models and prognostic factor models. The methodology of data synthesis in a review of the latter type of prognosis is comparable to the methodology of aetiological reviews. For that reason, in the present study we only focused on reviews of outcome prediction studies. Nonetheless, we found it difficult to distinct between both review types. Less than half of the reviews that we initially selected for data-extraction in fact seemed to serve an outcome prediction purpose. It appeared that the other reviews summarized prognostic factor studies only, or the objective was unclear. In particular, prognostic factor reviews that investigated more than one variable in addition to non-specific objectives made it difficult to clarify what the purpose of reviews was. As a consequence, we might have misclassified some of the 44 excluded reviews rated as prognostic factor. The objective of a review should also include information about the type of study that is included, that is of outcome prediction studies in this case. However, we found that in reviews aimed at outcome prediction the type of primary study was unclear for two-thirds of the reviews. An example we encountered in a review was that their purpose was “to identify preoperative predictive factors for acute post-operative pain and analgesic consumption” although the review authors included any study that identified one or more potential risk factors or predictive factors. The risk of combining both types of studies, i.e. risk factor or prognostic factor studies and predictive factor studies, is that inclusion of potential covariables in the former type are based on change in regression coefficient of the risk factor while in the latter study type all potential predictor variables are included based on their predictive ability of the outcome. This distinction may lead to: 1) biased results in a meta-analysis or other form of evidence synthesis because a risk factor is not always predictive for an outcome and 2) risk factor studies – if adjusted for potential confounders at all – have a slightly different method to obtain a multivariable model compared to outcome prediction studies which may also lead to biased regression coefficients. The distinction between prognostic factor and outcome prediction studies was already emphasized in 1983 by Copas . He stated that “a method for achieving a good predictor may be quite inappropriate for other questions in regression analysis such as the interpretation of individual regression coefficients”. In other words, the methodology of outcome prediction modelling differs from that of prognostic factor modelling, and therefore combining both types of research into one review to reflect current evidence should be discouraged. Hemingway et al.  appealed for standard nomenclature in prognosis research, and the results of our study underline their plea. Authors of reviews and primary studies should clarify their type of research, for example by using the terms applied by Hayden et al.  ‘prognostic factor modelling’ and ‘outcome prediction modelling’, and give a clear description of their objective.
Studies included in outcome prediction reviews are rarely similar in design and methodology, and this is often neglected when summarizing the evidence. Differences, for instance in the variables studied and the method of analysis for variable selection might explain heterogeneity in results, and should therefore be reported and reflected on when striving to summarize evidence in the most appropriate way. There is no doubt that the methodological quality of primary studies included in reviews is related to the concept of bias [69, 70] and it is therefore important to assess this [11, 69, 70]. Dissemination bias reflects if publication bias is likely to be present, how this is handled and what is done to correct for it . To our knowledge, dissemination bias and especially its consequences in reviews of outcome prediction models are not studied yet. Most likely testimation bias , i.e. the predictors considered and the amount of predictors in relation to the effective sample size influence results more then publication bias. Therefore, we did not study dissemination bias on the review level.
With regard to the reporting of primary study characteristics in the systematic reviews, there is much room for improvement. We found that the methods of model development (e.g. the variables considered and the variable selection methods used) in the primary studies were not, or only vaguely reported in the included reviews. These methods are however important, because variable selection procedures can affect the composition of the multivariable model due to estimation bias, or may result in an increase in model uncertainty [72–74]. Furthermore, the predictive performance of the model can be biased by these methods . We also found that only 5 of the reviews reported what kind of treatment the patients received in the primary studies. Although prescribed treatment is often not considered as a candidate predictor, it is likely to have a considerable impact on prognosis. Moreover, treatment may vary in relation to predictive variables , and although randomized controlled trials provide patients with similar treatment strategies, in cohort studies which are most often seen in prognosis research this is often not the case. Regardless of difficulties in defining groups that receive the same treatment, it is imperative to consider treatment in outcome prediction models. In order to ensure correct data-synthesis of the results, the primary studies not only should provide point estimates and estimates of dispersion of all the included variables, but also for non-significant findings. Whereas the results of positive or favourable findings are more often reported [75–78], the effects of predictive factors that do not reach statistical significance also need to be compared and summarized in a review. Imagine a variable being of statistical significance in one article, but not reported in others because of non-significance. It is likely that this one significant result is a spurious finding or that the others were underpowered. Without information about the non-significant findings in other studies, biased or even incorrect conclusions might be drawn. This means that reporting of the evidence of primary studies should be accompanied by the results of univariable and multivariable associations, regardless of their level of significance. Moreover, confidence intervals, or other estimates of dispersion are also needed in the review, and unfortunately these results were not presented in most of the reviews in our study. Some reviews considered differences in unadjusted and adjusted results, and the results of one review were sensibly stratified according to univariable and multivariable effects . Other reviews merely reported multivariable results , or only univariable results if multivariable results were unavailable . In addition to the multivariable results of a final prediction model, the predictive performance of these models is important for the assessment of clinical usefulness . A prediction model in itself does not indicate how much variance in outcome is explained by the included variables. Unfortunately, in addition to the non-reporting of several primary study characteristics, the performance of the models was rarely reported in the reviews included in our overview.
Different stages can be distinguished in outcome prediction research . Most outcome prediction models evaluated in the systematic reviews appeared to be in a developmental phase. Before implementation in daily practice, confirmation of the results in other studies is needed. With this type of validation studies underway, in future reviews we should acknowledge the difference between externally validated models and models from developmental studies, and analyze them separately.
In systematic reviews data can be combined quantitatively, i.e. a meta-analysis can be performed. This was done in 10 of the reviews. All of them combined point estimates (mostly odds ratios, but also a mix of odds ratios, hazard ratios and relative risks) and confidence intervals for single outcome prediction variables. This made it possible to calculate a pooled point estimate, often complemented with confidence intervals . However, in outcome prediction research we are interested in the estimates of a combination of predictive factors, which makes it possible to calculate absolute risks or probabilities to predict an outcome in individuals . Even if the relative risk of a variable is statistically significant, it does not provide information about the extent to which this variable is predictive for a particular outcome. The distribution of predictor values, outcome prevalence, and correlations between variables also influences the predictive value of variables within a model . Effect sizes also provide no information about the amount of variation in outcomes that is explained. In summary: the current quantitative methods seem to be more of an explanatory way of summarizing the available evidence, instead of quantitatively summarizing complete outcome prediction models.
Medline was the only database that was searched for relevant reviews. Our intention was to provide an overview of recently published reviews and not to include all relevant outcome prediction reviews. Within Medline, some eligible reviews may have been missed if their titles and abstracts did not include relevant terms and information. An extensive search strategy was applied and abstracts were screened thoroughly and discussed in case of disagreement. Data-extraction was performed in pairs to prevent reading and interpretation errors. Disagreements mainly occurred when deciding on the objective of a review and the type of primary studies included, due to poor reporting in most of the reviews. This indicates a lack of clarity, explanation and reporting within reviews. Therefore, screening in pairs is a necessity, and standardized criteria should be developed and applied in future studies focusing on such reviews. Consistency in rating on the data-extraction form was enhanced by one review author rating all reviews, with one of the other review authors as second rater. Several items were scored as “no”, but we did not know whether this was a true negative (i.e. leading to bias) or that no information was reported about a particular item. For review authors it is especially difficult to summarize information about primary studies because there may be a lack of information in the studies [13, 14, 84].
There is still no available methodological procedure for a meta-analysis of regression coefficients of multivariable outcome prediction models. Some authors, such as Riley et al. and Altman [81, 84], are of the opinion that it remains practically impossible, due to poor reporting, publication bias, and heterogeneity across studies. However, a considerable number of outcome prediction studies have been published, and it would be useful to integrate this body of evidence into one summary result. Moreover, there is an increase in the number of reviews that are being published. Therefore, there is a need to find the best strategy to integrate the results of primary outcome prediction studies. Consequently, until a method to quantitatively synthesize results has been developed, a sensible qualitative data-synthesis, which takes methodological differences between primary studies into account, is indicated. In summarizing the evidence, differences in methodological items and model-building strategies should be described and taken into account when assessing the overall evidence for outcome prediction. For example, univariable and multivariable results should be described separately, or subgroup analyses should be performed when they are combined. Other items that, in our opinion should be taken into consideration with regard to the data-synthesis are: study quality, variables used for model development, statistical methods used for variable selection procedures, the performance of models, and sufficient cases and non-cases to guarantee adequate study power. Regardless of whether or not these items are taken into consideration in the data-synthesis, we strongly recommend that in reviews they are described for all primary studies included so that readers can also take them into consideration.
In conclusion, poor reporting of relevant information and differences in methodology occur in primary outcome prediction research. Even the predictive ability of the models was rarely reported. This, together with our current inability to pool multivariable outcome prediction models, challenges review authors to make informative reviews of outcome prediction models.
Search strategy: 01-03-2011
((“systematic review”[tiab] OR “systematic reviews”[tiab] OR “Meta-Analysis as Topic”[Mesh] OR meta-analysis[tiab] OR “Meta-Analysis”[Publication Type]) AND (“2005/11/01”[EDat] : “3000”[EDat]) AND ((“Incidence”[Mesh] OR “Models, Statistical”[Mesh] OR “Mortality”[Mesh] OR “mortality ”[Subheading] OR “Follow-Up Studies”[Mesh] OR “Prognosis”[Mesh:noexp] OR “Disease-Free Survival”[Mesh] OR “Disease Progression”[Mesh:noexp] OR “Natural History”[Mesh] OR “Prospective Studies”[Mesh]) OR ((cohort*[tw] OR course*[tw] OR first episode*[tw] OR predict*[tw] OR predictor*[tw] OR prognos*[tw] OR follow-up stud*[tw] OR inciden*[tw]) NOT medline[sb]))) NOT ((“addresses”[Publication Type] OR “biography”[Publication Type] OR “case reports”[Publication Type] OR “comment”[Publication Type] OR “directory”[Publication Type] OR “editorial”[Publication Type] OR “festschrift”[Publication Type] OR “interview”[Publication Type] OR “lectures”[Publication Type] OR “legal cases”[Publication Type] OR “legislation”[Publication Type] OR “letter”[Publication Type] OR “news”[Publication Type] OR “newspaper article”[Publication Type] OR “patient education handout”[Publication Type] OR “popular works”[Publication Type] OR “congresses”[Publication Type] OR “consensus development conference”[Publication Type] OR “consensus development conference, nih”[Publication Type] OR “practice guideline”[Publication Type]) OR (“Animals”[Mesh] NOT (“Animals”[Mesh] AND “Humans”[Mesh]))).
Items used to assess the characteristics of analyses in outcome prediction primary studies and reviews:
What type of studies are included?
Is(/are) the outcome(s) of interest clearly described?
Is information about the quality assessment method provided?
What method was used?
Did the review account for quality?
What method was used?
Are the outcome measures clearly described?
Is the statistical method used for variable selection described?
Is there a description of treatments received provided?
Are crude univariable associations and estimates of dispersion for all the variables of the primary studies presented?
Are all variables that were used for model development described?
Are the multivariable associations and estimates of dispersions presented?
Is model performance assessed and reported?
Is the number of predictors relative to the number of outcome events described?
Is the heterogeneity of primary studies described?
Is a qualitative synthesis presented?
What method was used?
Are methods for quantitative analysis described?
What method was used?
Is the statistical heterogeneity assessed?
What method is used to assess statistical heterogeneity?
If statistical heterogeneity exists, are sources of the heterogeneity investigated?
What method is used to investigate potential sources of heterogeneity?
Is a graphical presentation of the results provided?
What method was used?
Are sensitivity analysis performed?
On which level?
We thank Ilse Jansma, MSc, for her contributions as a medical information specialist regarding the Medline search strategy. No compensation was received for her contribution.
No external funding was received for this study.
- Harrell FEJ, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996, 15: 361-387. 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.View ArticlePubMedGoogle Scholar
- Hemingway H, Riley RD, Altman DG: Ten steps towards improving prognosis research. BMJ. 2009, 339: b4184-10.1136/bmj.b4184.View ArticlePubMedGoogle Scholar
- Moons KGM, Donders AR, Steyerberg EW, Harrell FE: Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol. 2004, 57: 1262-1270. 10.1016/j.jclinepi.2004.01.020.View ArticlePubMedGoogle Scholar
- Royston P, Altman DG, Sauerbrei W: Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006, 25: 127-141. 10.1002/sim.2331.View ArticlePubMedGoogle Scholar
- Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009, New York: SpringerView ArticleGoogle Scholar
- Royston P, Moons KGM, Altman DG, Vergouwe Y: Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009, 338: b604-10.1136/bmj.b604.View ArticlePubMedGoogle Scholar
- Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG: Prognosis and prognostic research: what, why, and how?. BMJ. 2009, 338: b375-10.1136/bmj.b375.View ArticlePubMedGoogle Scholar
- Hayden JA, Dunn KM, van der Windt DA, Shaw WS: What is the prognosis of back pain?. Best Pract Res Clin Rheumatol. 2010, 24: 167-179. 10.1016/j.berh.2009.12.005.View ArticlePubMedGoogle Scholar
- Hayden JA, Chou R, Hogg-Johnson S, Bombardier C: Systematic reviews of low back pain prognosis had variable methods and results: guidance for future prognosis reviews. J Clin Epidemiol. 2009, 62: 781-796. 10.1016/j.jclinepi.2008.09.004.View ArticlePubMedGoogle Scholar
- Krasopoulos G, Brister SJ, Beattie WS, Buchanan MR: Aspirin “resistance” and risk of cardiovascular morbidity: systematic review and meta-analysis. BMJ. 2008, 336: 195-198. 10.1136/bmj.39430.529549.BE.View ArticlePubMedPubMed CentralGoogle Scholar
- Hayden JA, Cote P, Bombardier C: Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med. 2006, 144: 427-437. 10.7326/0003-4819-144-6-200603210-00010.View ArticlePubMedGoogle Scholar
- Mallett S, Timmer A, Sauerbrei W, Altman DG: Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines. Br J Cancer. 2010, 102: 173-180. 10.1038/sj.bjc.6605462.View ArticlePubMedGoogle Scholar
- Mallett S, Royston P, Waters R, Dutton S, Altman DG: Reporting performance of prognostic models in cancer: a review. BMC Med. 2010, 8: 21-10.1186/1741-7015-8-21.View ArticlePubMedPubMed CentralGoogle Scholar
- Mallett S, Royston P, Dutton S, Waters R, Altman DG: Reporting methods in studies developing prognostic models in cancer: a review. BMC Med. 2010, 8: 20-10.1186/1741-7015-8-20.View ArticlePubMedPubMed CentralGoogle Scholar
- Ingui BJ, Rogers MA: Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001, 8: 391-397. 10.1136/jamia.2001.0080391.View ArticlePubMedPubMed CentralGoogle Scholar
- Wilczynski NL: Natural History and Prognosis. PDQ, Evidence-Based Principles and Practice. Edited by: McKibbon KA, Wilczynski NL, Eady A, Marks S. 2009, Shelton, Connecticut: People’s Medical Publishing HouseGoogle Scholar
- Austin PC, Tu JV: Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol. 2004, 57: 1138-1146. 10.1016/j.jclinepi.2004.04.003.View ArticlePubMedGoogle Scholar
- Lee M, Chodosh J: Dementia and life expectancy: what do we know?. J Am Med Dir Assoc. 2009, 10: 466-471. 10.1016/j.jamda.2009.03.014.View ArticlePubMedGoogle Scholar
- Gravante G, Garcea G, Ong S: Prediction of Mortality in Acute Pancreatitis: A Systematic Review of the Published Evidence. Pancreatology. 2009, 9: 601-614. 10.1159/000212097.View ArticlePubMedGoogle Scholar
- Celestin J, Edwards RR, Jamison RN: Pretreatment psychosocial variables as predictors of outcomes following lumbar surgery and spinal cord stimulation: a systematic review and literature synthesis. Pain Med. 2009, 10: 639-653. 10.1111/j.1526-4637.2009.00632.x.View ArticlePubMedGoogle Scholar
- Wright AA, Cook C, Abbott JH: Variables associated with the progression of hip osteoarthritis: a systematic review. Arthritis Rheum. 2009, 61: 925-936. 10.1002/art.24641.View ArticlePubMedGoogle Scholar
- Heitz C, Hilfiker R, Bachmann L: Comparison of risk factors predicting return to work between patients with subacute and chronic non-specific low back pain: systematic review. Eur Spine J. 2009, 18: 1829-35. 10.1007/s00586-009-1083-9.View ArticlePubMedPubMed CentralGoogle Scholar
- Sansam K, Neumann V, O’Connor R, Bhakta B: Predicting walking ability following lower limb amputation: a systematic review of the literature. J Rehabil Med. 2009, 41: 593-603. 10.2340/16501977-0393.View ArticlePubMedGoogle Scholar
- Detaille SI, Heerkens YF, Engels JA, van der Gulden JWJ, van Dijk FJH: Common prognostic factors of work disability among employees with a chronic somatic disease: a systematic review of cohort studies. Scand J Work Environ Health. 2009, 35: 261-281. 10.5271/sjweh.1337.View ArticlePubMedGoogle Scholar
- Walton DM, Pretty J, MacDermid JC, Teasell RW: Risk factors for persistent problems following whiplash injury: results of a systematic review and meta-analysis. J Orthop Sports Phys Ther. 2009, 39: 334-350.View ArticlePubMedGoogle Scholar
- van Velzen JM, van Bennekom CAM, Edelaar MJA, Sluiter JK, Frings-Dresen MHW: Prognostic factors of return to work after acquired brain injury: a systematic review. Brain Inj. 2009, 23: 385-395. 10.1080/02699050902838165.View ArticlePubMedGoogle Scholar
- Borghuis MS, Lucassen PLBJ, van de Laar FA, Speckens AE, van Weel C, olde Hartman TC: Medically unexplained symptoms, somatisation disorder and hypochondriasis: course and prognosis. A systematic review. J Psychosom Res. 2009, 66: 363-377. 10.1016/j.jpsychores.2008.09.018.View ArticlePubMedGoogle Scholar
- Bramer JAM, van Linge JH, Grimer RJ, Scholten RJPM: Prognostic factors in localized extremity osteosarcoma: a systematic review. Eur J Surg Oncol. 2009, 35: 1030-1036. 10.1016/j.ejso.2009.01.011.View ArticlePubMedGoogle Scholar
- Tandon P, Garcia-Tsao G: Prognostic indicators in hepatocellular carcinoma: a systematic review of 72 studies. Liver Int. 2009, 29: 502-510. 10.1111/j.1478-3231.2008.01957.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Santaguida PL, Hawker GA, Hudak PL: Patient characteristics affecting the prognosis of total hip and knee joint arthroplasty: a systematic review. Can J Surg. 2008, 51: 428-436.PubMedPubMed CentralGoogle Scholar
- Elmunzer BJ, Young SD, Inadomi JM, Schoenfeld P, Laine L: Systematic review of the predictors of recurrent hemorrhage after endoscopic hemostatic therapy for bleeding peptic ulcers. Am J Gastroenterol. 2008, 103: 2625-2632. 10.1111/j.1572-0241.2008.02070.x.View ArticlePubMedGoogle Scholar
- Adamson SJ, Sellman JD, Frampton CMA: Patient predictors of alcohol treatment outcome: a systematic review. J Subst Abuse Treat. 2009, 36: 75-86. 10.1016/j.jsat.2008.05.007.View ArticlePubMedGoogle Scholar
- Paez JIG, Costa SF: Risk factors associated with mortality of infections caused by Stenotrophomonas maltophilia: a systematic review. J Hosp Infect. 2008, 70: 101-108. 10.1016/j.jhin.2008.05.020.View ArticlePubMedGoogle Scholar
- Johnson SR, Swiston JR, Granton JT: Prognostic factors for survival in scleroderma associated pulmonary arterial hypertension. J Rheumatol. 2008, 35: 1584-1590.PubMedGoogle Scholar
- Clarke SA, Eiser C, Skinner R: Health-related quality of life in survivors of BMT for paediatric malignancy: a systematic review of the literature. Bone Marrow Transplant. 2008, 42: 73-82. 10.1038/bmt.2008.156.View ArticlePubMedGoogle Scholar
- Kok M, Cnossen J, Gravendeel L, van der Post J, Opmeer B, Mol BW: Clinical factors to predict the outcome of external cephalic version: a metaanalysis. Am J Obstet Gynecol. 2008, 199: 630-637.View ArticlePubMedGoogle Scholar
- Stuart-Harris R, Caldas C, Pinder SE, Pharoah P: Proliferation markers and survival in early breast cancer: a systematic review and meta-analysis of 85 studies in 32,825 patients. Breast. 2008, 17: 323-334. 10.1016/j.breast.2008.02.002.View ArticlePubMedGoogle Scholar
- Kamper SJ, Rebbeck TJ, Maher CG, McAuley JH, Sterling M: Course and prognostic factors of whiplash: a systematic review and meta-analysis. Pain. 2008, 138: 617-629. 10.1016/j.pain.2008.02.019.View ArticlePubMedGoogle Scholar
- Nijrolder I, van der Horst H, van der Windt D: Prognosis of fatigue. A systematic review. J Psychosom Res. 2008, 64: 335-349. 10.1016/j.jpsychores.2007.11.001.View ArticlePubMedGoogle Scholar
- Williams M, Williamson E, Gates S, Lamb S, Cooke M: A systematic literature review of physical prognostic factors for the development of Late Whiplash Syndrome. Spine (Phila Pa 1976). 2007, 32: E764-E780. 10.1097/BRS.0b013e31815b6565.View ArticleGoogle Scholar
- Willemse-van Son AHP, Ribbers GM, Verhagen AP, Stam HJ: Prognostic factors of long-term functioning and productivity after traumatic brain injury: a systematic review of prospective cohort studies. Clin Rehabil. 2007, 21: 1024-1037. 10.1177/0269215507077603.View ArticlePubMedGoogle Scholar
- Alvarez J, Wilkinson J, Lipshultz S: Outcome Predictors for Pediatric Dilated Cardiomyopathy: A Systematic Review. Prog Pediatr Cardiol. 2007, 23: 25-32. 10.1016/j.ppedcard.2007.05.009.View ArticlePubMedPubMed CentralGoogle Scholar
- Mallen CD, Peat G, Thomas E, Dunn KM, Croft PR: Prognostic factors for musculoskeletal pain in primary care: a systematic review. Br J Gen Pract. 2007, 57: 655-661.PubMedPubMed CentralGoogle Scholar
- Stroke Risk in Atrial Fibrillation Working Group: Independent predictors of stroke in patients with atrial fibrillation: a systematic review. Neurology. 2007, 69: 546-554.View ArticleGoogle Scholar
- Kent PM, Keating JL: Can we predict poor recovery from recent-onset nonspecific low back pain? A systematic review. Man Ther. 2008, 13: 12-28. 10.1016/j.math.2007.05.009.View ArticlePubMedGoogle Scholar
- Tjang YS, van Hees Y, Korfer R, Grobbee DE, van der Heijden GJMG: Predictors of mortality after aortic valve replacement. Eur J Cardiothorac Surg. 2007, 32: 469-474. 10.1016/j.ejcts.2007.06.012.View ArticlePubMedGoogle Scholar
- Pfannschmidt J, Dienemann H, Hoffmann H: Surgical resection of pulmonary metastases from colorectal cancer: a systematic review of published series. Ann Thorac Surg. 2007, 84: 324-338. 10.1016/j.athoracsur.2007.02.093.View ArticlePubMedGoogle Scholar
- Williamson E, Williams M, Gates S, Lamb SE: A systematic literature review of psychological factors and the development of late whiplash syndrome. Pain. 2008, 135: 20-30. 10.1016/j.pain.2007.04.035.View ArticlePubMedGoogle Scholar
- Tas U, Verhagen AP, Bierma-Zeinstra SMA, Odding E, Koes BW: Prognostic factors of disability in older people: a systematic review. Br J Gen Pract. 2007, 57: 319-323.PubMedPubMed CentralGoogle Scholar
- Rassi AJ, Rassi A, Rassi SG: Predictors of mortality in chronic Chagas disease: a systematic review of observational studies. Circulation. 2007, 115: 1101-1108. 10.1161/CIRCULATIONAHA.106.627265.View ArticlePubMedGoogle Scholar
- Belo JN, Berger MY, Reijman M, Koes BW, Bierma-Zeinstra SMA: Prognostic factors of progression of osteoarthritis of the knee: a systematic review of observational studies. Arthritis Rheum. 2007, 57: 13-26. 10.1002/art.22475.View ArticlePubMedGoogle Scholar
- Langer-Gould A, Popat RA, Huang SM: Clinical and demographic predictors of long-term disability in patients with relapsing-remitting multiple sclerosis: a systematic review. Arch Neurol. 2006, 63: 1686-1691. 10.1001/archneur.63.12.1686.View ArticlePubMedGoogle Scholar
- Lamme B, Mahler CW, van Ruler O, Gouma DJ, Reitsma JB, Boermeester MA: Clinical predictors of ongoing infection in secondary peritonitis: systematic review. World J Surg. 2006, 30: 2170-2181. 10.1007/s00268-005-0333-1.View ArticlePubMedGoogle Scholar
- van Dijk GM, Dekker J, Veenhof C, van den Ende CHM: Course of functional status and pain in osteoarthritis of the hip or knee: a systematic review of the literature. Arthritis Rheum. 2006, 55: 779-785. 10.1002/art.22244.View ArticlePubMedGoogle Scholar
- Aalto TJ, Malmivaara A, Kovacs F: Preoperative predictors for postoperative clinical outcome in lumbar spinal stenosis: systematic review. Spine (Phila Pa 1976). 2006, 31: E648-E663. 10.1097/01.brs.0000231727.88477.da.View ArticleGoogle Scholar
- Hauser CA, Stockler MR, Tattersall MHN: Prognostic factors in patients with recently diagnosed incurable cancer: a systematic review. Support Care Cancer. 2006, 14: 999-1011. 10.1007/s00520-006-0079-9.View ArticlePubMedGoogle Scholar
- Bollen CW, Uiterwaal CSPM, van Vught AJ: Systematic review of determinants of mortality in high frequency oscillatory ventilation in acute respiratory distress syndrome. Crit Care. 2006, 10: R34-10.1186/cc4824.View ArticlePubMedPubMed CentralGoogle Scholar
- Steenstra IA, Verbeek JH, Heymans MW, Bongers PM: Prognostic factors for duration of sick leave in patients sick listed with acute low back pain: a systematic review of the literature. Occup Environ Med. 2005, 62: 851-860. 10.1136/oem.2004.015842.View ArticlePubMedPubMed CentralGoogle Scholar
- Bai M, Qi X, Yang Z: Predictors of hepatic encephalopathy after transjugular intrahepatic portosystemic shunt in cirrhotic patients: a systematic review. J Gastroenterol Hepatol. 2011, 26: 943-51. 10.1111/j.1440-1746.2011.06663.x.View ArticlePubMedGoogle Scholar
- Monteiro-Soares M, Boyko E, Ribeiro J, Ribeiro I, Dinis-Ribeiro M: Risk stratification systems for diabetic foot ulcers: a systematic review. Diabetologia. 2011, 54: 1190-1199. 10.1007/s00125-010-2030-3.View ArticlePubMedGoogle Scholar
- Lichtman JH, Leifheit-Limson EC, Jones SB: Predictors of hospital readmission after stroke: a systematic review. Stroke. 2010, 41: 2525-2533. 10.1161/STROKEAHA.110.599159.View ArticlePubMedPubMed CentralGoogle Scholar
- Ronden RA, Houben AJ, Kessels AG, Stehouwer CD, de Leeuw PW, Kroon AA: Predictors of clinical outcome after stent placement in atherosclerotic renal artery stenosis: a systematic review and meta-analysis of prospective studies. J Hypertens. 2010, 28: 2370-2377.View ArticlePubMedGoogle Scholar
- de Jonge RCJ, van Furth AM, Wassenaar M, Gemke RJBJ, Terwee CB: Predicting sequelae and death after bacterial meningitis in childhood: a systematic review of prognostic studies. BMC Infect Dis. 2010, 10: 232-10.1186/1471-2334-10-232.View ArticlePubMedPubMed CentralGoogle Scholar
- Colohan SM: Predicting prognosis in thermal burns with associated inhalational injury: a systematic review of prognostic factors in adult burn victims. J Burn Care Res. 2010, 31: 529-539. 10.1097/BCR.0b013e3181e4d680.View ArticlePubMedGoogle Scholar
- Clay FJ, Newstead SV, McClure RJ: A systematic review of early prognostic factors for return to work following acute orthopaedic trauma. Injury. 2010, 41: 787-803. 10.1016/j.injury.2010.04.005.View ArticlePubMedGoogle Scholar
- Brabrand M, Folkestad L, Clausen NG, Knudsen T, Hallas J: Risk scoring systems for adults admitted to the emergency department: a systematic review. Scand J Trauma Resusc Emerg Med. 2010, 18: 8-10.1186/1757-7241-18-8.View ArticlePubMedPubMed CentralGoogle Scholar
- Montazeri A: Quality of life data as prognostic indicators of survival in cancer patients: an overview of the literature from 1982 to 2008. Health Qual Life Outcomes. 2009, 7: 102-10.1186/1477-7525-7-102.View ArticlePubMedPubMed CentralGoogle Scholar
- Copas JB: Prediction and Shrinkage. J R Stat Soc Ser B (methodological). 1983, 45: 311-354.Google Scholar
- Atkins D, Best D, Briss PA: Grading quality of evidence and strength of recommendations. BMJ. 2004, 328: 1490-View ArticlePubMedGoogle Scholar
- Deeks JJ, Dinnes J, D’Amico R: Evaluating non-randomised intervention studies. Health Technol Assess. 2003, 7: iii-173-View ArticleGoogle Scholar
- Parekh-Bhurke S, Kwok CS, Pang C: Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol. 2011, 64: 349-57. 10.1016/j.jclinepi.2010.04.022.View ArticlePubMedGoogle Scholar
- Steyerberg EW: Selection of main effects. Clinicical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009, New York: SpringerView ArticleGoogle Scholar
- Chatfield C: Model Uncertainty, Data Mining and Statistical Inference. J R Stat Soc Ser A. 1995, 158: 419-466. 10.2307/2983440.View ArticleGoogle Scholar
- Steyerberg EW, Eijkemans MJ, Habbema JD: Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999, 52: 935-942. 10.1016/S0895-4356(99)00103-1.View ArticlePubMedGoogle Scholar
- Altman DG: Systematic reviews of evaluations of prognostic variables. BMJ. 2001, 323: 224-228. 10.1136/bmj.323.7306.224.View ArticlePubMedPubMed CentralGoogle Scholar
- Kyzas PA, Ioannidis JPA, axa-Kyza D: Quality of reporting of cancer prognostic marker studies: association with reported prognostic effect. J Natl Cancer Inst. 2007, 99: 236-243. 10.1093/jnci/djk032.View ArticlePubMedGoogle Scholar
- Kyzas PA, Ioannidis JPA, axa-Kyza D: Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer. 2007, 43: 2559-2579. 10.1016/j.ejca.2007.08.030.View ArticlePubMedGoogle Scholar
- Rifai N, Altman DG, Bossuyt PM: Reporting bias in diagnostic and prognostic studies: time for action. Clin Chem. 2008, 54: 1101-1103. 10.1373/clinchem.2008.108993.View ArticlePubMedGoogle Scholar
- Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JD: Validity of prognostic models: when is a model clinically useful?. Semin Urol Oncol. 2002, 20: 96-107. 10.1053/suro.2002.32521.View ArticlePubMedGoogle Scholar
- Altman DG, Vergouwe Y, Royston P, Moons KGM: Prognosis and prognostic research: validating a prognostic model. BMJ. 2009, 338: b605-10.1136/bmj.b605.View ArticlePubMedGoogle Scholar
- Altman DG: Systematic reviews of evaluations of prognostic variables. Systematic Reviews in Health Care. Edited by: Egger M, Smith GD, Altman DG. 2001, London: BMJ Publishing Group, 228-47.View ArticleGoogle Scholar
- Ware JH: The limitations of risk factors as prognostic tools. N Engl J Med. 2006, 355: 2615-2617. 10.1056/NEJMp068249.View ArticlePubMedGoogle Scholar
- Harrell FE: Multivariable modeling strategies. Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. 2001, New York: Springer,Google Scholar
- Riley RD, Abrams KR, Sutton AJ: Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer. 2003, 88: 1191-1198. 10.1038/sj.bjc.6600886.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/13/42/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.