- Research article
- Open Access
Compliance of systematic reviews in ophthalmology with the PRISMA statement
BMC Medical Research Methodology volume 17, Article number: 178 (2017)
Systematic reviews and meta-analyses are becoming increasingly important methods to summarize published research. Studies of ophthalmology may present additional challenges because of their potentially complex study designs. The aim of this study was to evaluate the reporting quality of systematic reviews and meta-analyses on topics in ophthalmology to determine compliance with the PRISMA guidelines. We assessed articles published between 2010 and 2015 in the five major relevant journals with the highest impact factors.
The MEDLINE and EMBASE databases were searched to identify systematic reviews published between January 2010 and December 2015 in the following 5 major ophthalmology journals: Progress in Retinal and Eye Research, Ophthalmology, Archives of Ophthalmology, American Journal of Ophthalmology, and Survey of Ophthalmology. The screening, identification, and scoring of articles were independently performed by two teams, and the results were submitted to statistical analysis to determine medians, ranges, and 95% CIs.
A total of 115 articles were included. The median compliance was 15 out of 27 items (56%), the range was 5–26 (26–96%), and the inter-quartile range was 10 (37%). Compliance was highest in items related to the ‘description of rationale’ (item 3, 100%) and sequentially lower in ‘the general interpretation of results’ (item 26, 96%) and ‘the inclusion of a structured summary in the abstract’ (item 2, 90%). Compliance was poorest in the items ‘indication of review protocol and registration’ (item 5, 9%), ‘specification of risk of biases that may affect the cumulative evidence’ (item 15, 24%), and ‘description of clear objectives in the introduction’ (item 4, 26%).
The reporting quality of systematic reviews and meta-analyses in ophthalmology should be significantly improved. While we recommend the use of the PRISMA criteria as a guideline before journal submission, additional research aimed at identifying potential barriers to compliance may be required to improve compliance with PRISMA guidelines.
Systematic reviews and meta-analyses are key study designs that are used to summarize published research findings that are replacing traditional reviews and expert commentaries. In these studies, the authors collect and critically assess all evidence that fits pre-specified criteria with the aim of answering a specific research question. They are therefore able to limit biases by including a wider range of populations than can be included in a single centre study. Moreover, they may quantify the heterogeneity in the available data and allow the results of multiple studies to be pooled into a meta-analysis, thereby increasing statistical power. In many cases, the goal is to identify the most appropriate treatment for a particular clinical paradigm [1,2,3]. Reliance on systematic reviews and meta-analysis by health care practitioners and policy makers, including those involved in ophthalmology research, in which approximately 1000 systematic review publications have been published, is increasing .
The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement was published in 2009 and consists of a 27-item checklist that covers each section of an article and includes a flow diagram. Its purpose is to help authors improve the quality of reporting in systematic reviews and meta-analyses . Similar studies have been reported in different fields, including orthorhinolaryngology , orthodontics , radiology , gastroenterology and hepatology , and orthopaedics . In a previous study, we also assessed the compliance of articles related to plastic surgery to the PRISMA statement. Our results showed that the reporting quality in plastic surgery was suboptimal .
Regardless of the field of research, poorly reported systematic reviews can be misleading and potentially dangerous in clinical practice. Studies in ophthalmology may present additional challenges to both researchers and readers because of the need for complexity in study designs. For example, there are two potential data points in each patient (i.e., two eyes), and accounting for this fact may require alternative designs and methods of analysis. Hence, the results and conclusions of such studies may be confusing if they do not provide an appropriate level of explanation . Systematic reviews play a role in enabling the accurate appraisal of the literature because they allow large pools of data to be simultaneously interpreted, which can resolve some of the complexity in this field. It is also important for the authors of these studies to provide complete, clear and transparent information by using good reporting methods [5, 13]. The PRISMA statement is an objective index that can be used to determine whether systematic reviews are adequately performing this role.
The aim of this study was to determine the reporting quality of systematic reviews and meta-analyses in ophthalmology according to the PRISMA statement. We evaluated articles published between 2010 and 2015 from five major journals with the highest impact factor.
This study is registered at Research Registry (unique identifying number: researchregistry952; http://www.researchregistry.com). Institutional review board approval was not required because the study did not involve human participants.
The MEDLINE and EMBASE databases were searched to identify systematic reviews published between January 1, 2010 and December 31, 2015 in the five major ophthalmology journals with the highest impact factor (IF) according to the Thomson Reuter Impact factor 2014 : The following journals were included: Progress in Retinal and Eye Research (IF 8.733), Ophthalmology (IF 6.135), Archives of Ophthalmology (IF 4.399), American Journal of Ophthalmology (IF 3.871), and Survey of Ophthalmology (IF 3.507), all of which represent a significant output for the field. The search strategies used in each database are described in Additional file 1.
The articles were manually and independently screened by three researchers, (H.S., K.W. and R.F.) who assessed their titles and abstracts. The following inclusion criteria were used: systematic reviews and/or meta-analyses published between January 2010 and December 2015. Any non-systematic reviews were excluded. Definition to identify systematic reviews has been adopted from PRISMA and the Cochrane Collaboration . Examples of excluded works included narrative reviews, expert reviews and randomized controlled trials. Any disagreements were forwarded to another member of the team (S.Y.L.) for a final decision.
Articles were scored against the 27 checklist items described in the PRISMA 2009 statement  by two independent researchers (H.S., and R.F.). Each item was weighted equally, and the same number of denominators was applied to each article. A score was computed for each article. Items received a mark only when the article fulfilled the criteria described within each item of the PRISMA statement. If the article did not explicitly mention the terms specified in the guidelines, efforts made to report the requirements per the guidelines were considered. For example, if the article did not specifically use the term ‘bias’, if an effort was made to assess the risk of bias, the relevant items were still given marks. Any description of a justification to not fulfil a criterion was also given a mark. For instance, we scored the article as fulfilled if it justified why meta-analysis was not performed. In cases where the PRISMA item included multiple elements, marks were given when all elements were reported. Discrepancies were initially solved by consensus, and any remaining disagreements were referred to another member of the team (S.Y.L.).
All statistical analyses were performed using Microsoft Excel 2011 (Microsoft Corporation). Descriptive statistics, including the median and range and their 95% Confidence Intervals (CIs), were calculated. The level of compliance with individual items was computed for each article as a percentage. Cohen’s Kappa statistic was used to calculate inter-observer agreement in scoring between two researchers. Cohen’s Kappa statistic measures the magnitude of agreement that goes beyond chance alone .
Among the 5 journals included in this study, only one journal (Ophthalmology) required the PRISMA checklist to be used to report a systematic review or meta-analysis.
The initial electronic search identified 253 articles. Of these, 115 were selected for data extraction (Fig. 1). Arbitration was required to determine whether to include 8 articles, and disagreement occurred regarding 375 points across the 115 articles. The Cohen’s Kappa statistic was 0.65, which indicates a good level of agreement between scorers.
The median compliance across the 115 included articles was 15 out of 27 items (56%) and had a range of 5–26 (19–96%) and an inter-quartile range of 10 (37%). The compliance of the articles with each PRISMA item showed wide variance (Table 1). More than half of the articles achieved more than 50% compliance with the PRISMA checklist. However, approximately 10% of the articles failed to achieve compliance higher than 30%, and no single article met all 27 criteria (Fig. 2).
Meanwhile, compliance with individual items was variable, with three items achieving a level of compliance equal to or greater than 90%, while compliance was less than or equal to 30% for 5 items. The compliance was highest for items related to ‘the description of the rationale for the study’ (item 3, 100%). This was followed by the ‘general interpretation of results’ (item 26, 96%) and the ‘inclusion of a structured summary in the abstract’ (item 2, 90%). Compliance was poorest for ‘indication of review protocol and registration’ (item 5, 9%), ‘specification of risk of bias that may have affected the cumulative evidence’ (item 15, 24%), and ‘description of clear objectives in the introduction’ (item 4, 26%).
The median compliance was 15 out of 27 (56%), which is relatively low in comparison with most previous studies that investigated the reporting quality of systematic reviews in different specialties [8,9,10,11, 18]. Half of the articles (62 articles) achieved a compliance of less than 60% (Fig. 2), and no single article met all 27 criteria of the PRISMA statement. Considering that only one journal required the use of a PRISMA checklist prior to the submission of a manuscript presenting a systematic review or meta-analysis, there is a significant amount of room in which improvement can be made in reporting the quality of systematic reviews in ophthalmology.
Five items with compliance less than or equal to 30% include items related to describing the review protocol and registration, describing the risk of bias, describing the objectives using PICOS, and describing additional analysis (items 5, 15, 4, 23, and 22, respectively). Interestingly, poor compliance with these items has also been noted in other previous studies that examined the reporting quality of systematic reviews in different specialties [6,7,8,9,10,11, 18]. Regardless of their overall compliance with the PRISMA checklist, previous studies in specialties including orthorhinolaryngology , orthodontics , and radiology  all achieved compliance lower than 30% in items relating to describing the review protocol and registration and the risk of bias. A study of journals in gastroenterology and hepatology  showed that there was generally high median compliance with the PRISMA checklist (above 80%) but poor compliance (less than 20%) in describing the protocol and registration. Similar studies that evaluated orthopaedic journals , our previous study that assessed the compliance of plastic surgery articles to PRISMA , and a study of nursing journals  produced overlapping results that showed there was low compliance regarding items related to the protocol and registration, the description of bias, and the description of additional analyses. Moreover, a review of 300 systematic reviews  found that just under a quarter (23.1%) of authors reported assessing possible publication bias and that no single article in the study mentioned the review protocol or review registration.
The generally low compliance observed in this study is an indication that there is a lack of awareness regarding the importance of review protocols and registration information among authors. The use of protocol and review registration is important because a protocol allows more accurate comparisons to be made between authors and leaders and supports better transparency during the review process [5, 20]. There are web database registration sites that authors can easily use to obtain information regarding protocol and registration. These include the PROSPERO  and Research registry  databases, which we recommend. It is also disappointing that most specialties achieve a low level of compliance in items related to the assessment of risk of bias. The attempt to eliminate bias is a very fundamental component of randomized controlled trials, as is the assessment of bias in systematic reviews and meta-analysis . Therefore, the poor compliance with items relating to assessing bias in most specialties is actually surprising. We suspect that this is caused by a lack of awareness regarding essential points of research reporting among authors because we noticed that many of the articles briefly mentioned ‘bias’ but then failed to describe the methods or results of their assessment of the risk of bias.
While improved awareness among stakeholders (i.e., journal reviewers and editors, funders, institutions, and readers) is required, we reiterate our previous call for journals to make the PRISMA checklist mandatory for the electronic submission of systematic reviews . Enforcing these guidelines at the time of journal submission may improve the quality of reporting, and their endorsement has been shown to improve adherence . Unfortunately, it is difficult to comment on this point with regard for the five ophthalmology journals included in this study because only one of them required the PRISMA checklist. However, it was recently demonstrated that journals increased their endorsement of reporting guidelines in their introduction to the authors between 2011 and 2014 (10% vs. 19%) . The number of systematic reviews and meta-analyses published in the field of ophthalmology has dramatically increased over the last decade [24, 25]. While this is a positive indication that there is adequate recognition of the increasing number of original high-quality articles published in this field, it also means that there is increased room for error due to the large amount of data. Increased importance should therefore be placed on accurate and detailed reporting . Clear and transparent information not only allows an adequate interpretation of results but also helps authors to correct errors during earlier stages of the study [5, 13].
Some limits need to be discussed regarding the methods used in our research. Because we focused on journals with high impact factors, our findings do not necessarily reflect adherence to PRISMA for systematic reviews published in journals with low impact factors. Additionally, the articles were electronically searched instead of handsearched, the latter of which would have ensured that we retrieved all articles that met the eligibility criteria. However, it was difficult to obtain access to paper copies of the journals, and using electronic search and screening procedures reduced the time that was consumed during the data search phase of this study. Additionally, we believe that MEDLINE and EMBASE cover a sufficient breath of articles (each contains over 20 million records) [26, 27].
We assigned equal weights to all items in the PRISMA checklist and used the same number of denominators in each article. However, this imposed a limitation on our study in that certain items on the checklist may have more impact than others on the reporting quality of systematic reviews. For example, it may be useful to weigh items related to the description of detailed methods more heavily than those related to the identification of the report as a systematic review in the title. Additionally, certain elements in the PRISMA checklist did not apply in some articles. For example, if the systematic review did not include a meta-analysis, items relating to meta-analyses, such as the presentation of the results of each meta-analysis (item 21), are not applicable to the review. Although we considered the article as fulfilled if it justified why meta-analysis was not performed, most articles failed to state such justification. Furthermore, the scoring of some items could vary according to the publishing style of each journal. For example, item 2 requires authors to provide a structured summary that includes many characteristics, such as the background, objective, data sources, and study eligibility criteria. However, compliance with this item can vary depending on how many words are used and what structure each journal requires the authors to provide. We attempted to be generous with such elements, but it remains possible that some articles may have lost a mark as a result of this type of limitation.
Systematic reviews in ophthalmology involve methodological issues that are specific to eyes and vision search, e.g. unit of analysis; we have not explicitly evaluated reporting of such items in this study. Because PRISMA was designed to cover a wide range of specialties, it has some limitations with regard for covering the specific details required in each specialty. In ophthalmology, PRISMA does not consider presentation of two sets of data, one from each eye (i.e., the right and left eye of each patient), a point that becomes even more complicated in studies wherein data can be obtained for only one eye in some patients .
Potential barriers to compliance with the PRISMA checklist may include a lack of awareness or understanding of the PRISMA guidelines, which we suspect is the most common cause of lack-of-compliance . However, further study is required to assess such barriers among key stakeholders in ophthalmology because no research into these barriers to compliance with PRISMA has been published.
The reporting quality of systematic reviews and meta-analysis in ophthalmology needs to be significantly improved, especially in describing the review protocol and registration, describing the risk of bias, describing the objectives, and describing additional analysis. We strongly suggest that journals require that all PRISMA criteria should be fulfilled before allowing electronic journal submission because we believe that this will eliminate ambiguity in the reporting of studies. Additional research aimed at identifying potential barriers to compliance may be required to improve compliance with the PRISMA guidelines.
Patient/Population/Problem, Intervention, Comparison, Outcome
the Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Hemingway P, What is a systematic review? http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/syst-review.pdf (accessed 2 Sep 2016).
Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29–37.
BMJ. 13. Study design and choosing a statistical test. http://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/13-study-design-and-choosing-statisti. Accessed 15 Dec 2017.
Li T. Publishing systematic reviews in ophthalmology: new guidance for authors. Ophthalmology. 2014;121(2):438–9.
Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi: 10.1371/journal.pmed.1000097.
Peters JPM, Hooft L, Grolman W, et al. Reporting quality of systematic reviews and meta-analyses of Otorhinolaryngologic articles based on the PRISMA statement. PLoS One. 2015;10:e0136540.
Fleming PS, Seehra J, Polychronopoulou A, et al. A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod. 2013;83(1):158–63.
Tunis AS, MD MI, Hanna R, et al. Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement? Radiology. 2013 Nov;269(2):413–26.
Panic N, Leoncini E, Belvis G, et al. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8(12):e83138. https://doi.org/10.1371/journal.pone.0083138. eCollection 2013
Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am. 2013;95(11):e771–7. doi: 10.2106/JBJS.L.00597.
Lee SY, Sagoo H, Whitehurst K, et al. Compliance of systematic reviews in plastic surgery with the PRISMA statement. JAMA Facial Past Surg. 2016;18(2):1–5. https://doi.org/10.1001/jamafacial.2015.1726. Published online December 17, 2015
Yao AC, Khajuria A, Camm CF, et al. The reporting quality of parallel randomised controlled trials in ophthalmic surgery in 2011: a systematic review. Eye. 2014;28:1341–9.
Agha RA, Lee SY, Lee-Jeong KJ, et al. Reporting quality of observational studies in plastic surgery needs improvement; a systematic review. Ann Plast Surg. 2015 Jan 30; [Epub ahead of print]
Cite Factor, Journal Impact Factor 2014 http://www.citefactor.org/journal-impact-factor-list-2014.html (accessed 26 Jun 2017).
PRISMA, The PRISMA Statement http://prisma-statement.org/PRISMAStatement/Default.aspx (accessed 22 Aug 2017).
Cochrane Consumer Network, What is systematic review? http://consumers.cochrane.org/what-systematic-review (accessed 22 Aug 2017).
Viera AJ, Garrett JM. Understanding Interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.
Tam WWS, Lo KKH, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7:e013905.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78. https://doi.org/10.1371/journal.pmed.0040078.
University of York, Centre for Reviews and Dessemination. About PROSPERO. http://www.crd.york.ac.uk/prospero/about.asp (accessed 5th June 2017).
Agha RA. About Research Registryhttp://www.researchregistry.com/about.html (accessed 5th June 2017).
Cochrane Methods, Assessing Risk of Bias in Included Studies http://methods.cochrane.org/bias/assessing-risk-bias-included-studies (accessed 26 Jun 2017).
Smith TS, Kulatilake P, Brown L, et al. Do surgery journals insist on reporting by CONSORT and PRISMA? A follow-up survey of ‘instructions to authors’. Ann Med Surg. 2015;4:17–21.
Cochrane Colloquium Abstracts, Systematic reviews and meta-analyses in eyes and vision: first steps in identifying gaps in ophthalmology research http://abstracts.cochrane.org/2013-qu%C3%A9bec-city/systematic-reviews-and-meta-analyses-eyes-and-vision-first-steps-identifying-gaps (accessed 26 Jun 2017).
Chen H, Jhanji V. Survey of systematic reviews and meta-analyses published in ophthalmology. BMJ. 2012;96:896–9.
U.S. National Library of Medicine. Fact Sheet, MEDLINE Journal Selection http://www.nlm.nih.gov/pubs/factsheets/jsel.html (accessed 3 Feb 2016).
Elsevier R&D Solutions. Embase Factsheets http://www.elsevier.com/__data/assets/pdf_file/0016/59011/R_D-Solutions_Embase-Fact-Sheet_Digital.pdf (accessed 3 Feb 2016).
Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1) doi: 10.1186/2046-4053-4-1.
none was received for this work.
Availability of data and materials
Please refer to our separate spreadsheet file (Additional file 2).
Ethics approval and consent to participate
Not applicable. This study does not involve human participants, human data or human tissue.
Consent for publication
Not applicable. This study does not involve individual data.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Search strategies. Search strategies used to identify systematic reviews and meta-analysis published from major ophthalmology journals between 2010 and 2015, via Medline and Embase. (PDF 13 kb)
Dataset of research. Raw data used for analysis and interpretation. (XLSX 48 kb)
About this article
Cite this article
Lee, SY., Sagoo, H., Farwana, R. et al. Compliance of systematic reviews in ophthalmology with the PRISMA statement. BMC Med Res Methodol 17, 178 (2017). https://doi.org/10.1186/s12874-017-0450-1
- Systematic reviews
- Research methodology
- Reporting quality