- Research article
- Open Access
- Open Peer Review
Does updating improve the methodological and reporting quality of systematic reviews?
BMC Medical Research Methodology volume 6, Article number: 27 (2006)
Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions.
We identifed updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ.
We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00).
The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.
A number of papers have been published on the methodological and reporting quality of reviews. A review of 86 English language meta-analyses published between 1950 and 1986 by Sacks  assessed every report on fourteen items covering six content areas believed to be critical in the reporting of meta-analysis. Only 28% of these meta-analyses were found to address all six content areas. An updated survey in 1996 showed little change . Shea  compared the methodological quality of paper-based and, electronic systematic reviews and found little difference and a lot of room for improvement. Assendelft  reviewed 51 reviews and noted that reviews that favoured a given intervention tended to have higher methodological quality scores. In contrast, Jadad and McQuay  reviewed 80 systematic reviews published between 1980 and 1992 and found a disconcerting link between reviews whose results favoured an intervention and poor methodological quality. Jadad found that Cochrane reviews had greater methodological rigor, more frequent updates and higher overall quality scores than those published in peer-reviewed paper journals, though both types were found to contain extensive and serious flaws.
The Cochrane Collaboration  is an international not-for-profit organization that conducts and updates systematic reviews of healthcare studies. With the exception of a few studies completed by Jadad [7, 8] and Shea , little is known about the quality of Cochrane reviews and whether their quality is improving over time and with updating. To our knowledge, the impact of updating on their quality has not been evaluated nor studied.
Studies of the quality of systematic reviews can focus on methodological or reporting quality. Methodological quality is concerned with how well a systematic review was designed and conducted (e.g. literature searching, pooling of data, etc.). Reporting quality considers how well systematic reviewers have reported their methodology and findings.
The purpose of our study was to compare the methodological and reporting quality of Cochrane systematic reviews with that of their updated versions in order to determine whether updating contributed significantly to the improvement of their quality in these two dimensions.
We selected all updated Cochrane systematic reviews from the Cochrane Database of Systematic Reviews 2002 , and the same reviews prior to their update. Updated reviews were chosen following the definition of 'updating' included in the Cochrane handbook .
Based on a previously published study , two instruments chosen to assess the quality of Cochrane systematic reviews for this study were the enhanced Overview Quality Assessment Questionnaire (OQAQ) scale [12–14] (Additional File 1) and the Quality of Reporting of Meta-analsyses (QUOROM) checklist  (Additional File 2).
The OQAQ was selected because it has strong face validity, provided data on several essential elements of its development, and had a published assessment of its construct validity available . In addition, its validity has been thoroughly tested and established using a number of independent measures . However, we noted difficulty applying the questions so we developed an enhanced version of the OQAQ which incorporated guidelines for its' use . The OQAQ scale measures across a continuum using nine questions (items 1–9) designed to assess various aspects of the methodological quality of systematic reviews and one overall assessment question (item 10). When the scale is applied to a systematic review, the first nine items are scored by selecting either yes, no, partial/can't tell. The tenth item requires assessors to assign an overall quality score on a 7-point scale .
The QUOROM statement was chosen for assessing reporting quality. Although, this checklist has not yet been fully validated, extensive work has been conducted and reported . The QUOROM statement is comprised of a checklist and flow diagram and was developed using a consensus process designed to strengthen the reliability of the estimates it yields when applied by different assessors. It estimates the overall reporting quality of systematic reviews. The checklist asks whether authors have provided readers with information on 18 items, including searches, selection, validity assessment, data abstraction, study characteristics, quantitative data syntheses and trial flow. It also asks whether authors have included a flow diagram with information about the number of randomized controlled trials identified, included and excluded, and the reasons for any exclusion. Individual checklist items included in this instrument are answered either yes, no and partial/can't tell .
For each included Cochrane systematic review, we calculated the percentage (with a 95% confidence interval) of 'yes' answers to each item of OQAQ and QUOROM. Individual Cochrane systematic reviews were then compared to their updated versions with respect to the percentage of 'yes' ratings received. A percentage difference (with a 95% confidence interval and p-values) of 'yes' answers was calculated for each individual question. In addition, a mean difference (with a 95% confidence interval and p-value) was calculated for the overall quality score (OQAQ) of the updated and pre-updated versions.
Assessments of all individual reviews were conducted independently by at least two assessors. One of these assessors (CH) had been involved with the Cochrane Collaboration for six years. The other assessor (BS) had been involved with the Cochrane Collaboration for ten years, and had carried out a number of systematic reviews. A third assessor (DF) was available to assist in the resolution of assessment discrepancies.
In total, 53 reviews were included. The mean period between the publication of the original and updated versions of the reviews was 2.7 years (range four months to five years). Complete results for both instruments applied to the reviews are provided in Tables 1 and 2, but highlights are summarized below.
Table 1 presents the methodological quality assessments obtained using this scale. There was no significant difference in the global assessment (item 10 – How would you rate the scientific quality of the overview?) (Mean score original review 4.70, mean score updated review 4.81, difference in means 0.11 (95% CI -0.28; 0.70 p = 0.52)).
There were improvements on seven individual items, although only one item showed a significant improvement (item 9 – Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview?) (percent original reviews complying with item 76% (95% CI 4.47; 4.94) percentage updated reviews complying with item 94% (95% 88.1; 100.0) difference 18.9% (95% CI 7.2%; 30.6% p < 0.01)).
Table 2 presents the assessments of reporting quality obtained using the Quality of Reviews of Meta-analyses (QUOROM) checklist. Scores (awarded for 'yes' response) on the 18 items for original Cochrane reviews ranged from 0% (item 1) to 100% (items 2 and 18). Four of the 18 items revealed improvements relating to data sources reported in the abstract (item 4) with a significant difference of 17.0% (9.8; 28.7 p = 0.01), review methods reported in the abstract (item 5) with a difference of 35% (24.1; 49.1 p = 0.00), searching methods (item 9) with a difference of 18.9% (9.7; 31.6 p = 0.01), and data abstraction (item 12) with a difference of 18.9% (11.7; 30.9 p = 0.00).
Three questions, item (10) -3.7% (95% CI: -9.0; 3.8 p = 0.15), item (14) -3.7% (-14.8; 6.4 p = 0.62) and item (16), -5.7% (-15.8; 3.0 p = 0.40) had lower mean scores on updated reviews than on original reviews, but these differences were not statistically significant.
In assessing the quality of the sample of 53 systematic reviews of the Cochrane Collaboration, two assessment tools were used. The larger improvement on the QUOROM checklist than on the OQAQ suggests that although the quality of reporting has improved slightly, the quality of design and conduct has not changed. For example, the items in reporting of selection criteria and searching are improved on QUOROM, but the equivalent items on OQAQ, which relate to how well these were carried out have not changed.
The significant improvements for the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis reported in the overview is also worthy of note. This could suggest that authors are paying more attention to state their conclusion in relation to the data provided. This item has been reported by other methodologists in the context of asessing quality [16, 17].
Improvement in the quality of reporting of the abstract, as assessed by the QUOROM instrument, might also suggest that authors are paying more attention to the details. For example, authors might be more aware of better access to the Cochrane abstracts through electronic systems such as Medline. Also, reporting of literature searches improved. However, the literature searches themselves did not improve. This should be further explored.
Assessors expressed that they found the two instruments used in the study to have associated practical weaknesses. Their combined length (28 items) made their use somewhat cumbersome and inefficient. Another problem encountered was the lack of adequate published guidance on the application of the OQAQ question, assessing the overall methodological quality. The QUOROM checklist had the advantage of coming with fairly detailed user instructions. However, it proved very time-consuming to apply . Another problem noted was that several of the questions asked by these two instruments appeared to cover the same subject matter.
The methodological and reporting quality of Cochrane reviews was fair-to-good, but further improvement is obviously needed in both areas. This was also concluded in a recent study by Moja . In addition, the quality-improvement impact of updating was found to be relatively minor. On some assessed factors, particularly with respect to reporting quality, updated reviews actually scored lower than original reviews. Currently, Cochrane review updates are carried out primarily to incorporate new findings, rather than to improve quality. It would be beneficial for updates to also address reporting, such as the omission of methodological descriptions. And of course, updates should also try to improve methodological weaknesses of the reviews.
The Cochrane Collaboration endeavours to improve the quality of its systematic reviews through the application of a continuous peer review process during their development. The effectiveness of the peer-review process could probably be improved by increased attention to areas of reporting and/or methodological weaknesses. Reviewers should adhere more faithfully to the guidelines provided in the Cochrane handbook  in order to improve the design and conduct of reviews and to the QUOROM statement  to improve the quality of reporting of systematic reviews.
The overall quality of Cochrane reviews is fair-to-good. Although quality of reporting improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.
Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC: Meta-analyses of randomized controlled trials. N Engl J Med. 1987, 316: 450-5.
Sacks HS, Reitman D, Pagano D, Kupelnick B: Meta-analysis: an update. Mount Sinai Journal of Medicine. 1996, 63: 216-24.
Shea B: Assessing the quality of reporting meta-analyses of randomized controlled trials. MSc thesis. 1999, University of Ottawa, Department of Epidemiology and Community Medicine
Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM: The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA. 1995, 274: 1942-1948. 10.1001/jama.274.24.1942.
Jadad AR, McQuay HJ: Meta-analyses to evaluate analogies interventions: a systematic qualitative review of their methodology. J Clin Epidemiol. 1996, 49: 235-43. 10.1016/0895-4356(95)00062-3.
The Cochrane Collaboration. [http://www.cochrane.org/]
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998, 280: 278-280. 10.1001/jama.280.3.278.
Jadad AR, Moher M, Brownman GP, Booker L, Sigouin C, Fuentes M, Stevens R: Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ. 2000, 320: 537-540. 10.1136/bmj.320.7234.537.
Shea B, Moher D, Graham I, Pham B, Tugwell P: A comparison of the quality of reporting of Cochrane review and systematic reviews published in paper-based journals. Evaluations and the Health Professions. 2002, 25: 116-129. 10.1177/0163278702025001008.
The Cochrane Library. 2002, Chichester, UK: John Wiley & Sons, Ltd, 4
Clarke M, Oxman AD, editors: Cochrane Reviewers' Handbook. The Cochrane Library. 2002, UK: Update Softeware, 2
Shea B, Dube C, Moher D: Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools. Systematic reviews in health care: meta-analysis in context. Edited by: Egger M, Smith GD, Altman DG. 2001, London, UK: BMJ Publishing Group, 122-39.
Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-8. 10.1016/0895-4356(91)90160-B.
Oxman AD: Checklists for review articles. BMJ. 1994, 309: 648-651.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999, 354: 1896-900. 10.1016/S0140-6736(99)04149-5.
Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A, on behalf of the Metaquality Study Group: Assessment of methodological quality of primary studies by systematic review: results of the metaquality cross sectional study. BMJ. 2005, 330: 1053-1058. 10.1136/bmj.38414.515938.8F.
The Cochrane Library. 2005, Chichester, UK: John Wiley & Sons, Ltd, 4
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/6/27/prepub
The authors would like to acknowledge Daniel Francis for his assistance with the quality assessment, Ron Habinski, Ashley Porter and Crystal Huntly-Ball for their comments on earlier drafts of this paper; David Moher for his methodological recommendations; Tim Ramsey for his statistical advice, and our external reviewer Penny Whiting for her helpful suggestions.
The author(s) declare that they have no competing interests.
BS designed the protocol and the study, abstracted the data, conducted the analysis, and wrote the paper; MB designed the protocol and the study, and assisted with writing the paper; JMG provided intellectual input into the design and conduct of the study, and assisted with writing the paper; CH carried out the data abstraction, quality assessment and assisted with the analysis; LMB provided intellectual input into the design and conduct of the study, and assisted with writing the paper.
About this article
Cite this article
Shea, B., Boers, M., Grimshaw, J.M. et al. Does updating improve the methodological and reporting quality of systematic reviews?. BMC Med Res Methodol 6, 27 (2006) doi:10.1186/1471-2288-6-27
- Systematic Review
- Quality Score
- Methodological Quality
- Cochrane Review
- Cochrane Collaboration