Research article | Open | Open Peer Review | Published:
Exploring reporting quality of systematic reviews and Meta-analyses on nursing interventions in patients with Alzheimer’s disease before and after PRISMA introduction
BMC Medical Research Methodologyvolume 18, Article number: 154 (2018)
Systematic reviews (SRs) and meta-analyses (MAs) are distillation of current best available evidence, but are potentially prone to bias. The bias of SRs and MAs comes from sampling bias, selection bias and within study bias. So, their reporting quality is especially important as it may directly influence their utility for clinicians, nurses, patients and policy makers. The SRs and MAs on nursing interventions in patients with Alzheimer’s disease (AD) have been increasingly published over the past decade, but the reporting quality of article has not been evaluated after the introduction of Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) Statement.
According to the inclusion and exclusion criteria, we searched the databases including PubMed, EMBASE and The Cochrane Library from inception through October 16th 2018. Two reviewers independently selected articles and extracted data. The PRISMA checklist was adopted to evaluate reporting quality. Comparisons were made between studies published before (2001–2009) and after (2011–2018) its introduction.
A total of 77 eligible articles, 18 (23.4%) were published before the PRISMA Statement and 59 (76.6%) were published afterwards. There was higher score after publication of the PRISMA Statement than before (20.83 ± 3.78 vs 17.11 ± 4.56, P < 0.05). There was an improvement in the following items after the PRISMA statement was released (P < 0.05): title (item 1, 50.0% vs 74.6%, OR = 3.10, 95CI%: 1.00–9.61), search (item8, 27.8% vs 57.6%,OR = 3.25, 95CI%: 1.14–9.28), study selection (item 9, 44.4% vs 81.4%,OR = 6.28, 95CI%: 1.93–20.37), Data collection process (item 10, 50.0% vs 76.3%,OR = 3.45, 95CI%:1.10–10.84), risk of bias in individual studies (item 12, 50.0% vs 83.1%, OR = 5.78, 95CI%:1.71–19.52), risk of bias across studies (item15, 5.6% vs 28.8%,OR = 3.60, 95CI%:1.04–12.43), study characteristics (item 18, 77.8% vs 98.3%, OR = 28.13, 95CI%:3.35-236.19), risk of bias with studies (item 19, 50.0% vs 83.1%, OR = 5.78, 95CI%:1.71-19.52), results in individual studies (item 20, 72.2% vs 94.9%, OR = 11.09, 95CI%:1.99–61.82), conclusions (item 26, 77.8% vs 98.3%, OR = 28.13, 95CI%:3.35–236.19). After controlling for the confounding factors, there were higher PRISMA score for systematic reviews including meta-analyses, protocol or registration, can’t answer of RCT, journal source of SCI (Science Citation Index), manuscript length > 13 page and funding support.
Since the publication of the PRISMA Statement, there has been an improvement in the quality of reporting of SRs and MAs on nursing interventions in patients with AD. More endorsement by journals of the report guideline for SRs/MAs may improve articles reporting quality, and the dissemination of reliable evidence to nurses. We recommend authors, readers, reviewers, and editors to become more acquainted with and to more strictly adhere to the PRISMA checklist.
Systematic reviews (SRs) and meta-analyses (MAs) keep up to date with developments in modern medicine. SRs and MAs can provide evidence of the value and feasibility on nursing interventions in patients with Alzheimer’s disease (AD), and to help clinicians, nurses, and policy makers to inform clinical decision-making. Despite their strengths, SRs have shown varying quality. As such, high quality of reporting is crucial to ensure reliable, transparent and accurate interpretation of evidence. Thus, some checklists have been published in an attempt to improve the reporting quality of systematic reviews and meta-analyses.
In an attempt to ensure validity of evidence and improve both the quality and completeness of reporting of SRs and MAs, a 27-item PRISMA statements have been published to assess the reporting quality of SRs in 2009, which was a successor to the original Quality Of Reporting Of Meta-analysis (QUOROM) guidelines [1, 2]. In recent years, a few of studies in various medical fields have been conducted for assessment of the qualities of SRs and MAs based on the fulfillments of PRISMA [3,4,5]. Cullis et al.  investigated that compliance with the PRISMA guidelines was poorer and the reporting quality of SRs and MAs in the published paediatric surgical literature needs to be improved. Panic et al.  found that the quality of reporting quality of SRs and MAs in journals in the field of gastroenterology and hepatology have significantly increased after PRISMA endorsement. Wasiak et al.  demonstrated that the reporting quality of SRs in burn care management is suboptimal and requires further improvement with stricter adherence by authors to the PRISMA checklist. Recently, more journals encourage the adoption of PRISMA include mandatory submission of reporting checklists and integration of PRISMA into the peer-review process . Hence, an assessment of compliance before and after introduction of the PRISMA Statement will play an increasingly important role.
Alzheimer’s disease (AD) is a disease that causes significant functional impairments even in its early stage . It is estimated that there will be 131.5 million people living with dementia globally by 2050. There are currently no treatments to reverse the course of AD, more by nursing intervention . However, certain treatments, both pharmacological and psychotherapeutic, do achieve a slowing of the impairment of AD, especially regarding to psychological treatments . So far, research on reporting quality evaluation of SRs and MAs on nursing interventions in patients with AD has not been available. Thus, this study evaluated the reporting quality of SRs and MAs on nursing interventions in patients with AD according to PRISMA checklists, and hoping to provide reference for authors, readers, reviewers, and journal editors. The secondary aim was to find that whether some predictive factors especially the publication of the PRISMA statement was associated with an improvement in reporting.
The article that met inclusion criteria was identified by searching in PubMed, EMBASE and The Cochrane Library up to October 16th 2018. The language was limited to English. The search strategy included the use of Title/Abstract related to: (“Alzheimer’s disease” or “Senile dementia” or “dementia” or “AD”) and (“systematic review” or “meta-analyses” or “overview” or “systematic literature review” or “meta-analysis” or “review” or “synthesize review” or “integrated review” or “comprehensive review”) and (“nursing” or “nursing intervention” or “nurse” or “care”). Combined with Google Scholar and Baidu Scholar, we also scanned the reference lists manually reviewed from included articles to identify additional relevant studies. The search strategy for PubMed is outlined in Fig. 1.
To be eligible for inclusion, the reports have the following inclusion criteria: (1) articles being identified as SRs or MAs; (2) published in English language; (3) an article of nursing interventions in patients with Alzheimer’s disease. The exclusion criteria were as follows: (1) scoping review, traditional literature review and evidence-based commentaries; (2) an article of non-nursing interventions in patients with Alzheimer’s disease. Two investigators independently screened the titles and abstracts of all the retrieved articles using the inclusion criteria. Full-text articles were retrieved and reviewed independently in duplicate by two authors for potentially eligible articles. Any discrepancy was resolved by discussion or by asking a third reviewer if consensus could not be reached.
Data extraction was performed independently on included articles by two investigators (S.X; L.H.H). Discrepancies and unobtainable data were resolved by group discussion between at least three investigators. General characteristics of SRs or MAs were extracted, including year of publication, author origin, the name and type of the journal, the authors’ affiliations, the number of affiliations, the article type, whether an RCT was identified, impact factor (IF), number of times cited, funding support, followed PRISMA guideline, international collaborative authorship, the number of included studies, topics of intervention, word count and protocol or registration, number of authors, journal source of SCI, manuscript length. Comparisons were made between manuscripts published before (2001–2009) and after (2011–2018) introduction of the PRISMA Statement to access the reporting quality. Since all three articles published in 2010 were submitted in 2009, we considered them as pre-published manuscripts.
According to the detailed per-item descriptions of the PRISMA Statement, appraisal of reporting was performed by two authors (S.X; L.H.H). The PRISMA statement covers seven modules with 27-items, ranging from 0 to 27. Each item was rated as yes, no or cannot answer, not applicable. A score of 1 was assigned to a “yes” answer, and a score of 0 was assigned to all other answers except “not applicable”. Items that were not applicable, such as those that only applied to MAs (ie, PRISMA items 14, 16 and 23), were scored 1. Therefore every study had an overall PRISMA score rated out of a maximum score of 27.
Initially, descriptive analysis on the characteristics of the included SRs/MAs published before (2001–2009) and after (2011–2018) was conducted. Differences in categorical variables and comparison overall compliance in each PRISMA item were explored using the χ2 test (≤ 2009 vs. ≥2011). Comparisons of the PRISMA score between manuscripts published before (2001–2009) and after (2011–2018) were conducted using the t-test and covariance of analysis for adjusting confounding factors. A p < 0.05 was considered significant on statistical analyses. All the statistics analyses were conducted by SPSS 19.0 and RevMan software version 5.3.
A total of 2413 articles were identified in initial search, whilst hand searching captured an additional 20 articles for potential inclusion. After removing duplicate articles, reviewing titles and abstracts, reviewing the full-text, there are a total of 77 articles to be included in this research finally. The process of literature retrieval was shown in a flow diagram (Fig. 2). The agreement between data extractors was moderate (kappa = 0.79, P < 0.01) for the PRISMA statement, which indicates a good level of agreement between scorers. All of the discrepancies were resolved by consensus between the two extractors.
These articles were published in different journals from 2001 to 2018. Of included manuscripts, 18 (23.4%) were published before the PRISMA Statement (2001–2009) and 59 (76.6%) were published afterwards (2011–2018). There was higher score after publication of the PRISMA Statement than before (20.83 ± 3.78 vs 17.11 ± 4.56, P < 0.05). The characteristics of the 77 articles are shown in Table 1.
Reporting quality of included reviews
To determine whether the publication of the PRISMA statement was associated with an improvement in the quality of reporting, the period ≤2009 was compared with ≥2011 in each of the PRISMA criteria (Fig. 3). The results showed that there was an improvement in the following items after the PRISMA statement was released, which was significant difference: title (item 1, 50.0% vs 74.6%,OR = 3.10, 95CI%:1.00–9.61), search (item 8, 27.8% vs 57.6%,OR = 3.25, 95CI%:1.14–9.28), study selection (item 9, 44.4% vs 81.4%, OR = 6.28, 95CI%:1.93–20.37),Data collection process (item 10, 50.0% vs 76.3%, OR = 3.45, 95CI%:1.10–10.84), risk of bias in individual studies (item 12, 50.0% vs 83.1%,OR = 5.78, 95CI%:1.71–19.52), risk of bias across studies (item 15, 5.6% vs 28.8%, OR = 3.60, 95CI%:1.04–12.43), study characteristics (item 18, 77.8% vs 98.3%, OR = 28.13, 95CI%:3.35-236.19), risk of bias with studies (item 19, 50.0% vs 83.1%,OR = 5.78, 95CI%: 1.71-19.52), results in individual studies (item 20, 72.2% vs 94.9%, OR = 11.09, 95CI%:1.99–61.82), conclusions (item 26, 77.8% vs 98.3%, OR = 28.13, 95CI%:3.35–236.19). Prevalence of reporting of these items was 3.10 to 28.13 times higher than before PRISMA Statement introduction. However, other items didn’t show significant difference or there was no improvement after the PRISMA statement was released. Per-item PRISMA analysis reveals six items that have less than 50% adherent studies (item 5, protocol and registration; item 13, summary measures; item 15, risk of bias across studies; item 21 and 22, synthesis of results and risk of bias across studies), while others have high compliance before and after PRISMA publication.
Univariable and covariance analysis on publication time
Univariable and covariance analysis of study demographics using PRISMA score as dependent variables identified several significant trends showed in Table 2. The following factors were significant in univariate analysis concerning PRISMA score: after PRISMA introduction (Mean difference (MD) = 3.72), SRs including MAs (MD = 5.00), number of authors > 4 (MD = 3.83), number of affiliations of author > 3 (MD = 2.47), affiliation of author is institute (MD = − 2.45), journal type is non nursing journal (MD = 2.58), only RCT (MD = 3.19), followed PRISMA guideline (MD = 2.91), having protocol or registration (MD = 5.03), journal source of SCI (MD = 7.44), manuscript length > 13 (MD = 2.68), funding support (MD = 3.49). In covariance analysis, those variables that were significant in univariate analysis were controlled to access reporting quality published before (2001–2009) and after (2011–2018). The result showed that PRISMA score improved after introduction of the PRISMA Statement (MD = 1.65). After controlling for the confounding factors, there were higher PRISMA score for systematic reviews including meta-analyses (increased by 2.57), protocol or registration (increased by 2.75), can't answer of RCT (reduced to 2.19), journal source of SCI (increased by 3.76), manuscript length > 13 page (increased by 1.63) and funding support (increased by 2.41).
The number of SRs/MAs about AD has been accumulating at an increasing rate in recent years. Although the quality of the reporting was suboptimal, the reporting quality of SRs and MAs of nursing interventions in patients with AD had been an improvement in the quality of reporting after the PRISMA statement was released. On average, PRISMA score with before and after publication of the PRISMA Statement was 17.11 and 20.83, respectively. Similarly, there was significant difference in compliance with some items of the PRISMA Statement before and after its introduction.
The articles [4, 10, 11] have evaluated compliance with the PRISMA Statement in other medical disciplines. Tunis et al. focused on SRs and MAs published from 2007 to 2011 in 11 high-impact radiology journals and showed an increase from an average of 20.90 of 27 items reported prior to PRISMA publication to an average of 22.60 of 27 items reported after publication of PRISMA . An assessment SRs and MAs of diagnostic tests published by in China revealed that there had been some statistically significant improvement in total compliance for 9 PRISMA items after the PRISMA was released . Like our study, the study demonstrated that several potential predictive factors for superior quality of urological meta-analyses were identified, including funding support, following PRISMA guideline, and “a priori” design . There are discrepancies in evaluation quality of SRs among various medical specialties.
The main strengths of this study include the focused search and selection of SRs/MAs, comprehensive assessment, and planned linear regression analyses. For selection of SRs/MAs, we not only included SRs but also MAs because our study has to be more comprehensive and objective. In addition, some of the assessment items only applied to meta-analyses, such as PRISMA items 14, 16, 23. To minimize bias against systematic review, we identified these items as not applicable and considered the item to be fulfilled when the total number of completed items was compiled. Thus, they have a more consistent score criterion. Since the PRISMA statement was published in 2009, we did not use this guidance to assess the quality of SRs/MAs published in 2010, which was sufficient to allow adequate uptake by authors and editors before assessment. Also, our results would have more credibility and our conclusions would have a more specific implication.
It is worth noting that a number of PRISMA items (item 1, item 8, item 9, item 10, item 12, item 15, item 18, item 19, item 20 and item 26) have increased in reporting after PRISMA Statement introduction. Prevalence of reporting of these items has gone up 3.10 to 28.13 times after PRISMA Statement introduction. The majority of included studies were described as SRs and/or MAs within the title (item 1). In order to identify instantly by the reader and to improve indexing, the title of the paper should include the PICOS approach (participants, intervention, comparison, outcome and study design), which will make readers know potential high level of evidence and help them decide whether the paper is worth reference . The study selection processes (item 9) perform that authors identify records from their search and sequentially exclude records according to eligibility criteria. It protects the rigor and science of the SRs and/or MAs. Of greater importance was the high rate of reporting of study characteristics (item 18). For readers to gauge the validity and applicability of a SR’s results, authors need to report information (such as PICOS, questions related to specific patients, interventions, comparisons,) about the included studies .
It is generally believed that PRISMA statement is significant for authors of SRs/MAs to follow to improve their reporting quality and transparency. But the poorly reported items need to further improvement. Whether it is before or after publication, five items with compliance less than 50% include items related to describing the review protocol and registration, describing the risk of bias, describing summary measures, and describing synthesis of results (items 5, 15, 22, 13, and 21, respectively). Based on our data, the generally lower compliance was reported and reflected a lack of awareness regarding protocols and registration (items 5) and risk of bias across studies (items 15 and 22) among authors as to their importance and potential implications. The use of protocol and review registration is important because a protocol makes more accurate for authors’ researches and supports better transparency during the review process. As for SRs, only Cochrane reviews require the authors to publish a peer-reviewed protocol before conducting the review. Previous studies have shown that Cochrane reviews appear to have higher reporting quality than SRs or MAs published in paper-based journals [3, 5, 15]. The registration resources such as PROSPERO  and Systematic Reviews  can make authors easily use to obtain information regarding protocol and registration. Risk of bias across studies is typically performed by testing for publication bias . Publication and outcome reporting biases are common in the literature and are likely to affect summary effect estimate [19,20,21]. To explore the publication biases, funnel plots and tests for funnel plot asymmetry should be provided in meta-analyses with more than 10 included studies . Therefore, authors should carefully consider whether their results are affected by these common biases by the examination of funnel plots or the performance of statistical tests .
Covariance analysis was conducted to explore characteristics associated with overall reporting quality. Overall, SRs including MAs score higher with regards PRISMA scores, than SRs alone. Compared with the traditional descriptive review, SRs including MAs can make validity and reliability of evidences. The reporting quality of SCI articles was better compared with non-SCI articles. For more than 50 years, the SCI database has been continuously developed and has become the most important large-scale database in the contemporary world. It is one of the important indicators to evaluate the academic level of a country, a scientific research institution, an institution of higher learning, a journal, and even a researcher. Manuscript length was also associated with higher compliance with the PRISMA statements. Manuscript length limits imposed by some journals are one of the major obstacles and these findings showed that reporting quality is compromised when inadequate space is provided in all sections . We also found evidence that funding support with better reporting quality were more likely to call for further research. The impact of funding on author conclusions has long been recognized in randomized trials and MAs of clinical trials [25, 26]. It is necessary to improve awareness among journal reviewers, editors, funders, institutions, and readers. We call for journals to make the PRISMA checklist mandatory for the electronic submission of SRs.
There are some limitations to this study. Firstly, this study was restricted to SRs/MAs examining nursing interventions in Alzheimer’s disease only, thus excluding a large number of reviews that may have produced a different response. Secondly, we were restricted to electronic databases, which may not have been representative of all indexed studies. And only articles written in English were enrolled for analysis.
Overall, some improvements in compliance with some items of the PRISMA Statement were found SRs and MAs articles of nursing interventions in patients with Alzheimer’s disease after the publication of the PRISMA statement, even the overall quality of reporting. After controlling for the confounding factors, there were higher PRISMA score for systematic reviews including meta-analyses, can’t answer of RCT, protocol or registration, can't answer of RCT, journal source of SCI, manuscript length > 13 page and funding support. But clinicians, nurses and investigators should critically appraise all reports of SRs/MAs before considering the results. More endorsement by journals of the report guideline for SRs/MAs may improve articles quality, and the dissemination of reliable evidence to nurses. We recommend readers, reviewers, and editors to become more acquainted with to the PRISMA checklist. In addition, authors should stricter adhere to the PRISMA checklist when conducting a SR/MA.
Preferred Reporting Items for Systematic Reviews and Meta Analyses
Liberati A, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. Bmj. 2009;339:b2700.
Moher D, et al. Improving the quality of reports of Meta-analyses of randomised controlled trials: the QUOROM statement. Onkologie. 1999;23(6):597–602.
Cullis PS, Gudlaugsdottir K, Andrews J. A systematic review of the quality of conduct and reporting of systematic reviews and meta-analyses in paediatric surgery. PLoS One. 2017;12(4):e0175213.
Panic N, et al. Evaluation of the endorsement of the preferred reporting items for systematic reviews and Meta-analysis (PRISMA) statement on the quality of published systematic review and Meta-analyses. PLoS One. 2013;8(12):e83138.
Wasiak J, et al. Poor methodological quality and reporting standards of systematic reviews in burn care management. Int Wound J. 2017;14(5):754-63.
Smith TA, et al. Do surgery journals insist on reporting by CONSORT and PRISMA? A follow-up survey of 'instructions to authors'. Ann Med Surg. 2015;4(1):17–21.
Mayeux R, Stern Y. Epidemiology of Alzheimer disease. Cold Spring Harb Perspect Med. 2012;2(8):137–52.
Carrion C, et al. Cognitive psychosocial intervention in dementia: a systematic review. Dement Geriatr Cogn Disord. 2013;36(5–6):363–75.
Orgeta V, et al. Psychological treatments for depression and anxiety in dementia and mild cognitive impairment: systematic review and meta-analysis. Br J Psychiatry. 2015;207(4):293–8.
Tunis AS, et al. Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement? Radiology. 2013;269(2):413–26.
Ge L, et al. The assessment of the quality of reporting of systematic reviews/Meta-analyses in diagnostic tests published by authors in China. PLoS One. 2014;9(1):e85908.
Xia L, Xu J, Guzzo TJ. Reporting and methodological quality of meta-analyses in urological literature. Peerj. 2017;5(4):e3129.
Liberati A, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. Epidemiol Biostatistics Public Health. 2009;6(4):e1–e34.
Richardson WS, et al. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995;123(3):A12.
Delaney A, et al. The quality of reports of critical care meta-analyses in the Cochrane database of systematic reviews: an independent appraisal. Crit Care Med. 2007;35(2):589–94.
PROSPERO - International prospective register of systematic reviews [cited 2018 25th May]; Available from: http://www.crd.york.ac.uk/prospero/
Systematic Reviews Journal [cited 2018 25th May]; Available from: http://www.systematicreviewsjournal/. com/
Sedgwick P. What is publication bias in a meta-analysis? Bmj. 2015;351:h4419.
Furukawa TA, et al. Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. Jama. 2007;297(5):468.
Kirkham JJ, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340(7747):637–40.
Dwan K, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2013;8(7):e66844.
Sterne JA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Bmj. 2011;343(5):d4002.
Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis : guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046–55.
Biondizoccai GG, et al. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. Bmj Br Med J. 2006;332(7535):202–6.
Bero L, et al. Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others. PLoS Med. 2007;4(6):e184.
Alsnielsen B, et al. Association of Funding and Conclusions in randomized drug trials: a reflection of treatment effect or adverse events? Jama. 2003;290(7):921–8.
The authors thank Guoyi Yu, director of the editorial board of Journal of Qingdao University Medical College and Jianqiang Li, vice director of the editorial board of Journal of Qingdao University Medical College for providing helpful comments on early draft of this article. The authors also thank the reviewers for their insightful comments and suggestions.
No current external funding sources for this study.
Availability of data and materials
Ethics approval and consent to participate
Not applicable. This study does not involve human participants, human dataor human tissue.
Consent for publication
Not applicable. This study does not involve individual data.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.