Skip to main content
Download PDF

A methodologic survey on use of the GRADE approach in evidence syntheses published in high-impact factor urology and nephrology journals

BMC Medical Research Methodology volume 22, Article number: 220 (2022) Cite this article

Abstract

Background

To identify and describe the use of the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach for rating the certainty of systematic reviews (SRs) evidence published in urology and nephrology journals.

Methods

SRs that were published in the top ten "urology and nephrology" journals with the highest impact factor according to the 2020 Journal Citation Reports (covering 2016–2020) were systematically searched and evaluated using the GRADE approach.

Results

A total of 445 SRs were researched. Sixty SRs of randomized control trials (RCTs) and/or non-randomized studies (NRSs) were evaluated using the GRADE approach. Forty-nine SRs (11%) rated the outcome-specific certainty of evidence (n = 29 in 2019–2020). We identified 811 certainty of evidence outcome ratings (n = 544 RCT ratings) as follows: very low (33.0%); low (32.1%); moderate (24.5%); and high (10.4%). Very low and high certainty of evidence ratings accounted for 55.0% and 0.4% of ratings in SRs of NRSs compared to 23.0% and 15.3% in SRs of RCTs. The certainty of evidence for RCTs and NRSs was downgraded most often for risk of bias and imprecision.

Conclusions

We recommend increased emphasis on acceptance of the GRADE approach, as well as optimal use of the GRADE approach, in the synthesis of urinary tract evidence.

Peer Review reports

Introduction

The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach is a system for rating the quality of a body of evidence in systematic reviews (SRs) and other evidence syntheses, such as health technology assessments, and guidelines and grading recommendations in health care [1]. GRADE approach offers a transparent and structured process for developing and presenting evidence summaries and for carrying out the steps involved in developing recommendations. It can be used to develop clinical practice guidelines and other health care recommendations (e.g. in public health, health policy and systems and coverage decisions), and is becoming increasingly popular among guideline developers and systematic reviewers [1]. Up to date, more than 110 organizations from 19 countries around the world have endorsed GRADE approach, including the World Health Organization, the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines, the World Allergy Organization, and the World Society of the Abdominal Compartment Syndrome and the Cochrane Collaboration [1]. In addition, it is also becoming an international standard for judging the evidence in SRs and clinical guidelines [2]. GRADE approach differs from other appraisal tools for three reasons: (i) because it separates quality of evidence and strength of recommendation, (ii) the quality of evidence is assessed for each outcome, and (iii) observational studies can be ‘upgraded’ if they meet certain criteria [3]. Thus, appropriate application of the GRADE approach can provide quality of evidence and strength of recommendations that is explicit, comprehensive, transparent, and pragmatic.

Systematic reviews (SRs) attempt to identify, select, synthesize, and appraise all high-quality research evidence relevant to a well-honed question. There have been many studies based on SRs, such as the GBD database [4], the World Cancer Research Fund/American Institute for Cancer Research report [5], as well as dietary guidelines [6]. As far as we know, untreated urological conditions are the major burden of patients all over the world. So far, many studies have summarized the factors affecting the urinary system and the prognosis of the urinary system disease using SRs.

However, no studies to date have assessed how SRs in urology and nephrology journals have used the GRADE approach to evaluate the certainty (or quality) of evidence. Therefore, the purpose of this study was to identify and describe all relevant SRs that use the GRADE approach to evaluate the outcome-specific certainty of evidence published from 2016–2020 in the top 10 urology and nephrology journals with the highest impact factor according to the 2020 Journal Citation Reports (JCR), and to summarize and present the GRADE-specific information, such as the outcomes rated, the number of primary studies, the exposure category, the use of summary of findings tables, the total number of down- and up-grading domains, while also considering the study design (SRs of randomized-controlled trials RCTs vs. non-randomized studies NRSs).

Methods

Search strategy

SRs published in the top 10 urology and nephrology journals with the highest impact factor according to the 2020 JCR between 1 January 2016 and 31December 2020 were identified by searching in the PubMed database (Supplementary Appendix 1).

Selection of documents

The SRs were included when the following criteria were met: (1) SRs utilizing the GRADE approach to assess the certainty of evidence. We excluded the following SRs: (1) a modified version of the GRADE approach was applied to assess the certainty of evidence; and (2) failure to provide details of the GRADE approach evaluation process and results.

First, two reviewers (SZ, S-X L.) independently screened identified articles by title and abstract. Only specific irrelevant articles were excluded at this stage. Second, we independently obtained and checked all potentially relevant articles for final inclusion by two reviewers (SZ, S-X L.). Selection disagreements were resolved by discussion or consultation with a third author (Q-J W.).

Data extraction

We extracted the following information for all identified SRs: year of publication; journal name; number of primary studies included; type of studies included RCTs vs. NRSs (including, non-randomized intervention, case–control, cohort, and cross-sectional studies) vs. combination RCTs/NRSs; number of participants; description of intervention(s)/exposure(s); number and types of outcome(s) and comparison(s) rated; category of certainty of evidence ratings (high, moderate, low, or very low); meta-analysis conducted (yes vs. no); summary of findings table reported (yes vs. no); number of down- and/or up-grading (count of the respective downgrading/upgrading domain used at the outcome level); and reasons for down- and/or up-grading. The one reviewer (SZ) extracted the study characteristics of the SRs, then one reviewer cross-checked all data (S-X L). We extracted all downgrading factors listed in the SRs of RCTs and NRSs according to the study design. There were five SRs with pooled RCTs and NRSs in which the combined evidence was rated [7,8,9,10,11].

We referred to minimum criteria proposed by GRADE working group when rating the certainty of evidence in this GRADE methodologic survey, which refers to the evidence that was assessed and the methods that were used to identify and appraise that evidence were not be clearly described for SRs that were assessed in our study.

Results

Search results and sample

A flow diagram of the literature search is shown in Fig. 1. The initial search yielded 767 entries, of which 322 were excluded that were not SRs. Of the 445 SRs identified, 60 met the eligibility criteria [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66]. Out of the 60 studies, 11 studies [56,57,58,59,60,61,62,63,64,65,66] (18.33%) did not report on outcome-specific rating, which refers to certainty of evidence of individual studies was assessed or overall certainty of the body of evidence was rated (Supplementary Table S1). Therefore, a total of 49 SRs (81.67%) that rated the outcome-specific certainty of evidence were included in our methodologic survey.

Fig. 1
figure 1

Flow diagram showing study selection process

Full size image

The distribution of the 49 SRs according to the journal and published year are presented in Table 1. The greatest and least number of SRs published in the top 10 urology and nephrology journals was in 2016 (n = 115) and 2018 (n = 67), respectively. In addition, there was an increase in the proportion of SRs that rated the certainty of evidence using the GRADE approach among all SRs published in 2019 and 2020 (17.7% and 14.4%, respectively) compared to 2016, 2017, and 2018 (6.1%, 10.0%, and 7.5%, respectively). Four journals (Journal of Urology, European Urology, Clinical Journal of the American Society of Nephrology, and American Journal of Kidney Diseases) accounted for 86% of all SRs, whereas 3 journals (Kidney International Supplements, Nature Reviews Urology, Nature Reviews Nephrology) did not publish SRs using the GRADE approach.

Table 1 The distribution of systematic reviews (n = 49) that rated the outcome-specific certainty of evidence with GRADE by year and journal
Full size table

Characteristics of the included SRs

SRs analyzing evidence from only RCTs (n = 20) [12, 14, 15, 18, 20, 22, 24, 26, 30, 32,33,34,35,36,37, 40, 41, 43, 45, 46], only NRSs (n = 15) [19, 21, 23, 31, 39, 42, 44, 47,48,49,50,51,52,53,54], or RCTs and NRSs (n = 14) [7,8,9,10,11, 13, 16, 17, 25, 27,28,29, 38, 55] were included into this methodological study. The main features of these 49 eligible SRs are summarized in Supplementary Table S2. Among the 14 SRs that combined RCTs and NRSs, 5 rated the outcome-specific certainty of evidence derived from RCTs and NRSs separately (in separate rows within a single Summary of Findings table or in separate Summary of Findings tables) [8, 17, 25, 27, 28], 5 pooled RCTs and NRSs and rated the combined evidence (in the same rows within a single Summary of Findings table) [7,8,9,10,11], and 6 only rated the NRSs [9,10,11, 13, 29, 38].

Forty-six SRs (94%) conducted at least one meta-analysis [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37, 39,40,41,42,43,44,45,46,47,48,49,50,51,52, 55], and 39 SRs (80%) presented the findings in a summary of findings table [8,9,10,11,12,13, 15, 17,18,19,20,21,22,23, 25,26,27,28,29,30,31,32,33,34, 36,37,38, 40,41,42,43, 45,46,47,48,49,50,51, 53]. The median number of primary studies included in the SRs was 19 (range = 5 to 104). The median number of participants included in the SRs based on RCTs was 4253 (range = 484 to 16,990), 14,550 (range = 1394 to 2,791,732) in the SRs based on NRSs, and 2958 (range = 708 to 8163) in the SRs based on RCTs and NRSs. Additionally, we classified the interventions and exposures in the SRs as clinical therapies (n = 22), drugs (n = 14), specific clinical diseases (n = 6), lifestyle factors (n = 3), supplements (n = 2), clinical approaches (n = 1), and clinical care (n = 1) (Table 2).

Table 2 Summary of the systematic reviews characteristics
Full size table

Certainty of evidence ratings

The median number of outcomes rated in a SR was 8 (range = 1 to 184) (Table 2). There were 811 individual outcome ratings (544 for RCTs and 256 for NRSs) The quality of evidence was assessed in 4 categories, as follows: very low (33%); low (32.1%); moderate (24.5%); and high (10.4%). Outcomes were rated in 23.0%, 34.0%, 27.7%, and 15.3% as a very low, low, moderate, and high certainty of evidence among SRs based on RCTs, respectively. The certainty of evidence was rated as very low (55.0%), low (27.0%), moderate (17.6%), and high (0.4%) among SRs based on NRSs. Among SRs that combined NRSs and RCTs, 11 outcomes were rated; the certainty of evidence was rated as very low (18.2%), low (54.5%), and moderate (27.3%) (Table 3).

Table 3 Details on the certainty of evidence ratings including down- and upgrading’s of the evidence according to study design
Full size table

Up- and down-grading domains

A total of 1012 instances of downgrading were identified due to a risk of bias (RoB; 52.2%), imprecision (27.5%), inconsistency (14.4%), indirectness (5.4%), and publication bias (0.5%). According to the authors of those SRs, rating down for publication bias was as a result of asymmetry of the funnel plot or it was strongly suspected the study design (patient reports published in toxicology report very severe poisoning either with or without impressive recovery with treatments attempted) [43, 53]. Twenty-one upgrades of the certainty of evidence were due to a large effect (14.3%), dose–response (47.6%), and plausible confounding (38.1%). The authors of those SRs considered that the all plausible residual confounding in the included studies would reduce the demonstrated effect [21, 29, 53]. Downgrading for RoB was more common in SRs of RCTs (53.8%) than SRs of NRSs (49.3%), whereas downgrading for publication bias was more common in SRs of NRSs (1.2%) than SRs of RCTs (0.1%). Additionally, upgrading for large effect, dose–response, and plausible confounding were all in SRs of NRSs (Table 3).

We calculated 1.2 mean downgrades per outcome in SRs of RCTs and 1.3 mean downgrades per outcome in SRs of NRSs. The downgrading frequency (the number of downgrades per number of rated outcomes) among SRs of RCTs for the RoB (65.8%) and imprecision (38.0%) domains was greater than the SRs of NRSs. In contrast, the downgrading frequency among SRs of NRSs for the inconsistency (23.4%), indirectness (14.5%), and publications bias (1.6%) was greater than the SRs of RCTs (Table 3). In addition, 7% of outcomes rated in SRs of NRSs were upgraded; 3.9% of outcomes were upgraded for dose–response, 3.1% for plausible confounding, and 0.4% for a large effect. The reasons for the downgrade or upgrade of the certainty of evidence for outcomes is shown in Supplementary Table S3.

Discussion

Summary of findings

This is the first study to evaluate the application of the GRADE approach in SRs published in the top 10 urology and nephrology journals. In general, there are relatively few urology and nephrology SRs using GRADE approach to rate the certainty of evidence, but an increasing trend in the level of implementation was noted in the past 2 years [9, 11, 17, 19,20,21,22,23,24,25,26,27,28,29, 31, 33, 40,41,42,43, 45,46,47,48,49,50,51, 53, 55]. We identified 49 SRs that rated the outcome-specific certainty of evidence. Overall, a low and very low certainty of evidence accounted for 32.1% and 33% of the individual outcomes, respectively. In addition, the certainty of evidence was downgraded most of for RoB and imprecision among SRs of RCTs and NRSs.

Strengths and limitations

There were several strengths in our study. First, we have searched all published SRs that utilized the GRADE approach among the top 10 urology and nephrology journals from 2016–2020. A wide range of relevant information was extracted, such as the classification of outcomes assessment, and the number and reasons for down- and up-grading domains. Second, this is the first study to summarize and present the GRADE-specific information on SRs published in urology and nephrology journals, which can provide essential information for follow-up research. There were also some limitations in our study. First, our results are limited due to the risk of selection bias. We only included the top 10 urology and nephrology journals with the highest impact factors within 5 years (n = 60). Therefore, we failed to obtain information from medical journals with lower impact factors. Nevertheless, the journals with the highest impact factor are persuasive, which can provide a basis for future clinical research. Second, because there was no clear and definite research protocol to follow, we adopted the PRISMA framework to report our findings. In fact, the PRISMA framework can provide comprehensive content and is often used in SRs and meta-analyses. Third, our study was based on a descriptive examination of the application of the GRADE approach in urology and nephrology SRs rather than to determine if the SR authors correctly followed guidance issued by the GRADE working group to rate the certainty of evidence. In the process of data extraction, we found that some authors did not follow minimum criteria of GRADE approach in the evaluation. Specifically, the certainty of evidence in some SRs was upgraded due to a low RoB, narrow confidence interval, a very low P value, and/or mild statistical heterogeneity rather than undergoing evaluation according to the GRADE approach domains for upgrading. Therefore, future research should have a priority for focusing on the optimal use of the GRADE approach, so that those applying the approach can have awareness brought to the main issues and difficulties faced when applying GRADE approach, to be able to correct them. Finally, this report does not address or assess any potential time trends. It is possible that the adoption of GRADE approach has increased and improved over time.

Findings from other studies

The GRADE approach has been used in the fields of urology and nephrology during the recent 5 years. KDIGO Working Group formulated the scope of guidelines and graded evidence according to the GRADE approach, which now serves as the practice standard for KDIGO [67]. Similar to the findings in our study, the level of evidence was very low in most cases in the study conducted by Zare and colleagues [68]. The most frequent limitation involved indirectness due to the limited number of studies for each pair. In addition, the evidence quality was mainly downgraded due to the RoB and inconsistency. Moreover, presenting comprehensive evidence from RCTs and NRSs, as in our study, was similar to the Cuello-Garcia et al. study [69], which indicated that most respondents presented integrated data separately from RCTs and NRSs, either in a single summary of findings table or a standalone table. Compared to our study, these studies or guidelines merely indicated the use of the GRADE approach when rating the certainty of evidence or the preferences of experts when integrating RCTs and NRSs.

Implications for broader research

Although the GRADE approach has been used for RCTs and NRSs in the fields of urology and nephrology, there are still some challenges in evaluating the certainty of evidence in NRSs with the GRADE approach. On the basis of the GRADE approach, the quality grade for an aggregate of RCTs would begin at the high entry level. A collection of observational studies would begin at the low entry level and evidence from other study designs, such as case–control studies, would begin at the very low level [70]. The GRADE approach, especially with respect to RoB assessment, is challenging and could lead to excessive downgrading. GRADE approach users may inappropriately double calculate the confounding and selection bias risk by downgrading the initial body of evidence to low, followed by further downgrading owing to unknown confounders in observational studies. In our study, in addition to the initial downgrading of two levels according to the study design, 53% of the SRs based on NRSs were further downgraded due to RoB.

The GRADE approach requires separate evaluations of RCTs or NRSs by any validated tool, but the specific method is not recommended because the selection of tools depends on the context [71]. With the emergence of tools that use the concept of target trials as a reference point, such as risk of bias in non-randomized intervention studies (ROBINS-I), the initial certainty of evidence is also considered high for bodies of evidence from NRSs [72, 73]. Several SRs have used the ROBINS-I tool in the field of nephrology thus far [74,75,76]. ROBINS-I can more systematically and accurately assess the RoB in NRSs to avoid extensive downgrading [73]. In general, the purpose of integrating rigorous RoB assessment into the GRADE approach, such as the application of the ROBINS-I tool in the SRs based on NRSs, is to improve the trustworthiness and credibility of the specific evidence [73]. To accurately rate the certainty of evidence, it is recommended to follow the current standards of RoB assessments, such as the ROBINS-I and the RoB tool by Cochrane [73, 77].

Conclusion

The GRADE approach provides a system for rating quality of evidence and strength of recommendations that is explicit, comprehensive, transparent, and pragmatic. The GRADE approach is increasingly adopted by professional organizations worldwide. Our study demonstrated that the GRADE approach is not widely used (only 13.5% (60/445) of SRs reported using GRADE approach), within urology and nephrology SRs in top 10 journals. Thus, future research should focus on the optimal use of the GRADE approach by following the criteria proposed by the GRADE working group.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

GRADE:

Grading of recommendations, assessment, development and evaluation

GBD:

Global burden of disease

JCR:

Journal citation reports

KDIGO:

Kidney disease: improving global outcomes

NRSs:

Non-randomized studies

RCTs:

Randomized control trials

RoB:

Risk of bias

SRs:

Systematic reviews

References

  1. GRADE home. https://www.gradeworkinggroup.org.

  2. Xie CX, Machado GC. Clinimetrics: Grading of recommendations, assessment, development and evaluation (GRADE). J Physiother. 2021;67:66. https://doi.org/10.1016/j.jphys.2020.07.003.

    Article  PubMed  Google Scholar 

  3. Goldet G, Howick J. Understanding GRADE: An introduction. J Evid Based Med. 2013;6:50–4. https://doi.org/10.1111/jebm.12018.

    Article  PubMed  Google Scholar 

  4. Naghavi M, Abajobir AA, Abbafati C, et al. Global, regional, and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the Global Burden of Disease Study 2016. The Lancet. 2017;390:1151-210. https://doi.org/10.1016/S0140-6736(17)32152-9.

  5. World Cancer Research Fund International. Diet, nutrition, physical activity and cancer: a global perspective: a summary of the Third Expert Report. World Cancer Research Fund International. 2018.

  6. You, Are. "Dietary guidelines for Americans." US Department of Health and Human Services and US Department of Agriculture 7 (2015).

  7. Sinha RA, Haikal A, Hammoud KA, et al. Vancomycin and the risk of AKI: a systematic review and meta-analysis. Clin J Am Soc Nephrol. 2016;11:2132–40. https://doi.org/10.2215/CJN.05920616.

    Article  Google Scholar 

  8. Koves B, Cai T, Veeratterapillay R, et al. Benefits and harms of treatment of asymptomatic bacteriuria: A systematic review and meta-analysis by the european association of urology urological infection guidelines panel. Eur Urol. 2017;72:865–8. https://doi.org/10.1016/j.eururo.2017.07.014.

    Article  PubMed  Google Scholar 

  9. Kim A, Kim MS, Park YJ, et al. Retropubic versus transobturator mid urethral slings in patients at high risk for recurrent stress incontinence: a systematic review and meta-analysis. J Urol. 2019;202:132–42. https://doi.org/10.1097/JU.0000000000000222.

    Article  PubMed  Google Scholar 

  10. Kowalewski KF, Tapking C, Hetjens S, et al. Interrupted versus continuous suturing for vesicourethral anastomosis during radical prostatectomy: a systematic review and meta-analysis. Eur Urol Focus. 2019;5:980–91. https://doi.org/10.1016/j.euf.2018.05.009.

    Article  PubMed  Google Scholar 

  11. Park KJ, Choi SH, Lee JS, et al. Risk stratification of prostate cancer according to PI-RADS(R) version 2 categories: meta-analysis for prospective studies. J Urol. 2020;204:1141–9. https://doi.org/10.1097/JU.0000000000001306.

    Article  PubMed  Google Scholar 

  12. Mustafa RA, Bdair F, Akl EA, et al. Effect of lowering the dialysate temperature in chronic hemodialysis: a systematic review and meta-analysis. Clin J Am Soc Nephrol. 2016;11:442–57. https://doi.org/10.2215/CJN.04580415.

    Article  PubMed  Google Scholar 

  13. Zumstein V, Betschart P, Vetterlein MW, et al. Prostatic artery embolization versus standard surgical treatment for lower urinary tract symptoms secondary to benign prostatic hyperplasia: a systematic review and meta-analysis. Eur Urol Focus. 2019;5:1091–100. https://doi.org/10.1016/j.euf.2018.09.005.

    Article  PubMed  Google Scholar 

  14. Quach K, Lvtvyn L, Baigent C, et al. The safety and efficacy of mineralocorticoid receptor antagonists in patients who require dialysis: a systematic review and meta-analysis. Am J Kidney Dis. 2016;68:591–8. https://doi.org/10.1053/j.ajkd.2016.04.011.

    Article  CAS  PubMed  Google Scholar 

  15. Viecelli AK, Irish AB, Polkinghorne KR, et al. Omega-3 polyunsaturated fatty acid supplementation to prevent arteriovenous fistula and graft failure: a systematic review and meta-analysis of randomized controlled trials. Am J Kidney Dis. 2018;72:50–61. https://doi.org/10.1053/j.ajkd.2017.11.017.

    Article  CAS  PubMed  Google Scholar 

  16. Shim SR, Kanhai KJ, Ko YM, et al. Efficacy and safety of prostatic arterial embolization: systematic review with meta-analysis and meta-regression. J Urol. 2017;197:465–79. https://doi.org/10.1016/j.juro.2016.08.100.

    Article  PubMed  Google Scholar 

  17. Teoh JY, MacLennan S, Chan VW, et al. An international collaborative consensus statement on en bloc resection of bladder tumour incorporating two systematic reviews, a two-round delphi survey, and a consensus meeting. Eur Urol. 2020;78:546–69. https://doi.org/10.1016/j.eururo.2020.04.059.

    Article  PubMed  Google Scholar 

  18. Ravani P, Quinn RR, Oliver MJ, et al. Preemptive correction of arteriovenous access stenosis: a systematic review and meta-analysis of randomized controlled trials. Am J Kidney Dis. 2016;67:446–60. https://doi.org/10.1053/j.ajkd.2015.11.013.

    Article  PubMed  Google Scholar 

  19. Navaneethan SD, Shao J, Buysse J, et al. Effects of treatment of metabolic acidosis in CKD: a systematic review and meta-analysis. Clin J Am Soc Nephrol. 2019;14:1011–20. https://doi.org/10.2215/CJN.13091118.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Goossen K, Becker M, Marshall MR, et al. Icodextrin versus glucose solutions for the once-daily long dwell in peritoneal dialysis: an enriched systematic review and meta-analysis of randomized controlled trials. Am J Kidney Dis. 2020;75:830–46. https://doi.org/10.1053/j.ajkd.2019.10.004.

    Article  CAS  PubMed  Google Scholar 

  21. Pyrgidis N, Mykoniatis I, Sokolakis I, et al. Renal transplantation improves erectile function in patients with End-Stage renal disease: a systematic review and meta-analysis. J Urol. 2021;205:1009–17. https://doi.org/10.1097/JU.0000000000001577.

    Article  PubMed  Google Scholar 

  22. Kallidonis P, Ntasiotis P, Somani B, et al. Systematic review and meta-analysis comparing percutaneous nephrolithotomy, retrograde intrarenal surgery and shock wave lithotripsy for lower pole renal stones less than 2 cm in maximum diameter. J Urol. 2020;204:427–33. https://doi.org/10.1097/JU.0000000000001013.

    Article  PubMed  Google Scholar 

  23. Abrar M, Pindoria N, Malde S, et al. Predictors of poor response and adverse events following botulinum toxin a for refractory idiopathic overactive bladder: a systematic review. Eur Urol Focus. 2020. https://doi.org/10.1016/j.euf.2020.06.013.

    Article  PubMed  Google Scholar 

  24. Cai T, Tamanini I, Tascini C, et al. Fosfomycin trometamol versus comparator antibiotics for the treatment of acute uncomplicated urinary tract infections in women: a systematic review and meta-analysis. J Urol. 2020;203:570–8. https://doi.org/10.1097/JU.0000000000000620.

    Article  PubMed  Google Scholar 

  25. Donaldson JF, Ruhayel Y, Skolarikos A, et al. Treatment of bladder stones in adults and children: a systematic review and meta-analysis on behalf of the european association of urology urolithiasis guideline panel. Eur Urol. 2019;76:352–67. https://doi.org/10.1016/j.eururo.2019.06.018.

    Article  PubMed  Google Scholar 

  26. Kallidonis P, Adamou C, Kotsiris D, et al. Combination therapy with alpha-blocker and phosphodiesterase-5 inhibitor for improving lower urinary tract symptoms and erectile dysfunction in comparison with monotherapy: a systematic review and meta-analysis. Eur Urol Focus. 2020;6:537–58. https://doi.org/10.1016/j.euf.2019.05.007.

    Article  PubMed  Google Scholar 

  27. Karavitakis M, Kyriazis I, Omar MI, et al. Management of urinary retention in patients with benign prostatic obstruction: a systematic review and meta-analysis. Eur Urol. 2019;75:788–98. https://doi.org/10.1016/j.eururo.2019.01.046.

    Article  PubMed  Google Scholar 

  28. St-Jules DE, Rozga MR, Handu D, et al. Effect of phosphate-specific diet therapy on phosphate levels in adults undergoing maintenance hemodialysis: a systematic review and meta-analysis. Clin J Am Soc Nephrol. 2020;16:107–20. https://doi.org/10.2215/CJN.09360620.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Sountoulides P, Pyrgidis N, Brookman-May S, et al. Does ureteral stenting increase the risk of metachronous upper tract urothelial carcinoma in patients with bladder tumors? a systematic review and meta-analysis. J Urol. 2021;205:956–66. https://doi.org/10.1097/JU.0000000000001548.

    Article  PubMed  Google Scholar 

  30. Sun M, Marconi L, Eisen T, et al. Adjuvant vascular endothelial growth factor-targeted therapy in renal cell carcinoma: a systematic review and pooled analysis. Eur Urol. 2018;74:611–20. https://doi.org/10.1016/j.eururo.2018.05.002.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Kim DK, Kim JW, Ro JY, et al. Plasmacytoid variant urothelial carcinoma of the bladder: a systematic review and meta-analysis of clinicopathological features and survival outcomes. J Urol. 2020;204:215–23. https://doi.org/10.1097/JU.0000000000000794.

    Article  PubMed  Google Scholar 

  32. Palmer SC, Tunnicliffe DJ, Singh-Grewal D, et al. Induction and maintenance immunosuppression treatment of proliferative lupus nephritis: a network meta-analysis of randomized trials. Am J Kidney Dis. 2017;70:324–36. https://doi.org/10.1053/j.ajkd.2016.12.008.

    Article  PubMed  Google Scholar 

  33. Shaman AM, Smyth B, Arnott C, et al. Comparative efficacy and safety of BP-Lowering pharmacotherapy in patients undergoing maintenance dialysis: a network meta-analysis of randomized, controlled trials. Clin J Am Soc Nephro. 2020;15:1129–38. https://doi.org/10.2215/CJN.12201019.

    Article  CAS  Google Scholar 

  34. Valentijn PP, Pereira FA, Ruospo M, et al. Person-Centered integrated care for chronic kidney disease: a systematic review and meta-analysis of randomized controlled trials. Clin J Am Soc Nephrol. 2018;13:375–86. https://doi.org/10.2215/CJN.09960917.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Pathan SA, Mitra B, Cameron PA. A systematic review and meta-analysis comparing the efficacy of nonsteroidal anti-inflammatory drugs, opioids, and paracetamol in the treatment of acute renal colic. Eur Urol. 2018;73:583–95. https://doi.org/10.1016/j.eururo.2017.11.001.

    Article  CAS  PubMed  Google Scholar 

  36. Mallat SG, Tanios BY, Itani HS, et al. CMV and BKPyV infections in renal transplant recipients receiving an mTOR Inhibitor-Based regimen versus a CNI-Based regimen: a systematic review and meta-analysis of randomized, controlled trials. Clin J Am Soc Nephrol. 2017;12:1321–36. https://doi.org/10.2215/CJN.13221216.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Badve SV, Palmer SC, Strippoli G, et al. The validity of left ventricular mass as a surrogate end point for All-Cause and cardiovascular mortality outcomes in people with CKD: a systematic review and meta-analysis. Am J Kidney Dis. 2016;68:554–63. https://doi.org/10.1053/j.ajkd.2016.03.418.

    Article  PubMed  Google Scholar 

  38. Drake T, Grivas N, Dabestani S, et al. What are the benefits and harms of ureteroscopy compared with shock-wave lithotripsy in the treatment of upper ureteral stones? A systematic review Eur Urol. 2017;72:772–86. https://doi.org/10.1016/j.eururo.2017.04.016.

    Article  PubMed  Google Scholar 

  39. Tikkinen K, Craigie S, Agarwal A, et al. Procedure-specific risks of thrombosis and bleeding in urological cancer surgery: systematic review and meta-analysis. Eur Urol. 2018;73:242–51. https://doi.org/10.1016/j.eururo.2017.03.008.

    Article  PubMed  Google Scholar 

  40. Marchioni M, Di Nicola M, Primiceri G, et al. New antiandrogen compounds compared to docetaxel for metastatic hormone sensitive prostate cancer: results from a network meta-analysis. J Urol. 2020;203:751–9. https://doi.org/10.1097/JU.0000000000000636.

    Article  PubMed  Google Scholar 

  41. Palmer SC, Mavridis D, Johnson DW, et al. Comparative effectiveness of calcimimetic agents for secondary hyperparathyroidism in adults: a systematic review and network meta-analysis. Am J Kidney Dis. 2020;76:321–30. https://doi.org/10.1053/j.ajkd.2020.02.439.

    Article  CAS  PubMed  Google Scholar 

  42. Drost FH, Osses D, Nieboer D, et al. Prostate magnetic resonance imaging, with or without magnetic resonance imaging-targeted biopsy, and systematic biopsy for detecting prostate cancer: a cochrane systematic review and meta-analysis. Eur Urol. 2020;77:78–94. https://doi.org/10.1016/j.eururo.2019.06.023.

    Article  PubMed  Google Scholar 

  43. Pilatz A, Dimitropoulos K, Veeratterapillay R, et al. Antibiotic prophylaxis for the prevention of infectious complications following prostate biopsy: a systematic review and meta-analysis. J Urol. 2020;204:224–30. https://doi.org/10.1097/JU.0000000000000814.

    Article  PubMed  Google Scholar 

  44. Kelly JT, Palmer SC, Wai SN, et al. Healthy dietary patterns and risk of mortality and ESRD in CKD: a meta-analysis of cohort studies. Clin J Am Soc Nephrol. 2017;12:272–9. https://doi.org/10.2215/CJN.06190616.

    Article  PubMed  Google Scholar 

  45. Pradere B, Veeratterapillay R, Dimitropoulos K, et al. Nonantibiotic strategies for the prevention of infectious complications following prostate biopsy: a systematic review and meta-analysis. J Urol. 2021;205:653–63. https://doi.org/10.1097/JU.0000000000001399.

    Article  PubMed  Google Scholar 

  46. Mavrakanas TA, Chatzizisis YS, Gariani K, et al. Duration of dual antiplatelet therapy in patients with CKD and Drug-Eluting stents: a meta-analysis. Clin J Am Soc Nephrol. 2019;14:810–22. https://doi.org/10.2215/CJN.12901018.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Pesonen JS, Vernooij R, Cartwright R, et al. The impact of nocturia on falls and fractures: a systematic review and meta-analysis. J Urol. 2020;203:674–83. https://doi.org/10.1097/JU.0000000000000459.

    Article  PubMed  Google Scholar 

  48. Pesonen JS, Cartwright R, Vernooij R, et al. The impact of nocturia on mortality: a systematic review and meta-analysis. J Urol. 2020;203:486–95. https://doi.org/10.1097/JU.0000000000000463.

    Article  PubMed  Google Scholar 

  49. Bach KE, Kelly JT, Palmer SC, et al. Healthy dietary patterns and incidence of CKD: a meta-analysis of cohort studies. Clin J Am Soc Nephrol. 2019;14:1441–9. https://doi.org/10.2215/CJN.00530119.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Kelly JT, Su G, Zhang L, et al. Modifiable lifestyle factors for primary prevention of CKD: A systematic review and Meta-Analysis. J Am Soc Nephrol. 2021;32:239–53. https://doi.org/10.1681/ASN.2020030384.

    Article  CAS  PubMed  Google Scholar 

  51. Covella B, Vinturache AE, Cabiddu G, et al. A systematic review and meta-analysis indicates long-term risk of chronic and end-stage kidney disease after preeclampsia. Kidney Int. 2019;96:711–27. https://doi.org/10.1016/j.kint.2019.03.033.

    Article  PubMed  Google Scholar 

  52. Tikkinen K, Craigie S, Agarwal A, et al. Procedure-specific risks of thrombosis and bleeding in urological non-cancer surgery: systematic review and meta-analysis. Eur Urol. 2018;73:236–41. https://doi.org/10.1016/j.eururo.2017.02.025.

    Article  PubMed  Google Scholar 

  53. Berling I, King JD, Shepherd G, et al. Extracorporeal treatment for chloroquine, hydroxychloroquine, and quinine poisoning: systematic review and recommendations from the EXTRIP workgroup. J Am Soc Nephrol. 2020;31:2475–89. https://doi.org/10.1681/ASN.2020050564.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Schroeck FR, Jacobs BL, Bhayani SB, et al. Cost of new technologies in prostate cancer treatment: Systematic review of costs and cost effectiveness of robotic-assisted laparoscopic prostatectomy, intensity-modulated radiotherapy, and proton beam therapy. Eur Urol. 2017;72:712–35. https://doi.org/10.1016/j.eururo.2017.03.028.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Omar MI, Pal RP, Kelly BD, et al. Benefits of empiric nutritional and medical therapy for semen parameters and pregnancy and live birth rates in couples with idiopathic infertility: a systematic review and meta-analysis. Eur Urol. 2019;75:615–25. https://doi.org/10.1016/j.eururo.2018.12.022.

    Article  PubMed  Google Scholar 

  56. Wagenaar S, Nederhoed JH, Hoksbergen A, et al. Minimally invasive, laparoscopic, and robotic-assisted techniques versus open techniques for kidney transplant recipients: a systematic review. Eur Urol. 2017;72:205–17. https://doi.org/10.1016/j.eururo.2017.02.020.

    Article  PubMed  Google Scholar 

  57. Zaid HB, Parker WP, Safdar NS, et al. Outcomes following complete surgical metastasectomy for patients with metastatic renal cell carcinoma: a systematic review and meta-analysis. J Urol. 2017;197:44–9. https://doi.org/10.1016/j.juro.2016.07.079.

    Article  PubMed  Google Scholar 

  58. Han S, Woo S, Kim YJ, et al. Impact of (68)Ga-PSMA PET on the management of patients with prostate cancer: a systematic review and meta-analysis. Eur Urol. 2018;74:179–90. https://doi.org/10.1016/j.eururo.2018.03.030.

    Article  CAS  PubMed  Google Scholar 

  59. Foroutan F, Friesen EL, Clark KE, et al. Risk factors for 1-Year graft loss after kidney transplantation: systematic review and meta-analysis. Clin J Am Soc Nephrol. 2019;14:1642–50. https://doi.org/10.2215/CJN.05560519.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Campi R, Brookman-May SD, Subiela HJ, et al. Impact of metabolic diseases, drugs, and dietary factors on prostate cancer risk, recurrence, and survival: a systematic review by the european association of urology section of oncological urology. Eur Urol Focus. 2019;5:1029–57. https://doi.org/10.1016/j.euf.2018.04.001.

    Article  PubMed  Google Scholar 

  61. Kotecha P, Sahai A, Malde S. Use of duloxetine for postprostatectomy stress urinary incontinence: a systematic review. Eur Urol Focus. 2021;7:618–28. https://doi.org/10.1016/j.euf.2020.06.007.

    Article  PubMed  Google Scholar 

  62. Reus CR, Phe V, Dechartres A, et al. Performance and safety of the artificial urinary sphincter (AMS 800) for non-neurogenic women with urinary incontinence secondary to intrinsic sphincter deficiency: a systematic review. Eur Urol Focus. 2020;6:327–38. https://doi.org/10.1016/j.euf.2018.10.009.

    Article  PubMed  Google Scholar 

  63. Cacciamani GE, Ghodoussipour S, Mari A, et al. Association between smoking exposure, neoadjuvant chemotherapy response and survival outcomes following radical cystectomy: systematic review and meta-analysis. J Urol. 2020;204:649–60. https://doi.org/10.1097/JU.0000000000000813.

    Article  CAS  PubMed  Google Scholar 

  64. Bedretdinova D, Ambuhl D, Omar MI, et al. What is the most effective treatment for nocturia or nocturnal incontinence in adult women? Eur Urol Focus. 2021;7:453–63. https://doi.org/10.1016/j.euf.2020.01.012.

    Article  PubMed  Google Scholar 

  65. Dunford C, Bell K, Rashid T. Genital reconstructive surgery in male to female transgender patients: a systematic review of primary surgical techniques, complication profiles, and functional outcomes from 1950 to present day. Eur Urol Focus. 2021;7:464–71. https://doi.org/10.1016/j.euf.2020.01.004.

    Article  PubMed  Google Scholar 

  66. Rahman SN, Cao DJ, Monaghan TF, et al. Phenotyping the association between nocturia and hypertension: a systematic review and meta-analysis. J Urol. 2021;205:1577–83. https://doi.org/10.1097/JU.0000000000001433.

    Article  PubMed  Google Scholar 

  67. Navaneethan SD, Zoungas S, Caramori ML, et al. Diabetes management in chronic kidney disease: synopsis of the 2020 KDIGO clinical practice guideline. Ann Intern Med. 2021;174:385–94. https://doi.org/10.7326/M20-5938.

    Article  PubMed  Google Scholar 

  68. Zare SM, Lequy E, Goldberg M, et al. Air pollution exposure and bladder, kidney and urinary tract cancer risk: a systematic review. Environ Pollut. 2020;267: 115328. https://doi.org/10.1016/j.envpol.2020.115328.

    Article  CAS  Google Scholar 

  69. Cuello-Garcia CA, Morgan RL, Brozek J, et al. A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments. J Clin Epidemiol. 2018;98:33–40. https://doi.org/10.1016/j.jclinepi.2018.01.010.

    Article  PubMed  Google Scholar 

  70. Uhlig K, Macleod A, Craig J, et al. Grading evidence and recommendations for clinical practice guidelines in nephrology a position statement from Kidney Disease: Improving Global Outcomes (KDIGO). Kidney Int. 2006;70:2058–65. https://doi.org/10.1038/sj.ki.5001875.

    Article  CAS  PubMed  Google Scholar 

  71. Schwingshackl L, Schunemann HJ, Meerpohl JJ. Improving the trustworthiness of findings from nutrition evidence syntheses: assessing risk of bias and rating the certainty of evidence. Eur J Nutr. 2021;60:2893–903. https://doi.org/10.1007/s00394-020-02464-1.

    Article  PubMed  Google Scholar 

  72. Schunemann HJ, Cuello C, Akl EA, et al. GRADE guidelines: 18 How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2019;111:105–14. https://doi.org/10.1016/j.jclinepi.2018.01.012.

    Article  PubMed  Google Scholar 

  73. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. https://doi.org/10.1136/bmj.i4919.

    Article  PubMed  PubMed Central  Google Scholar 

  74. Rodriguez RA, Trentin SM, Agharazii M, et al. Effects of living kidney donation on arterial stiffness: A systematic review protocol. BMJ Open. 2021;11:e45518. https://doi.org/10.1136/bmjopen-2020-045518.

    Article  Google Scholar 

  75. Hansrivijit P, Yarlagadda K, Puthenpura MM, et al. A meta-analysis of clinical predictors for renal recovery and overall mortality in acute kidney injury requiring continuous renal replacement therapy. J Crit Care. 2020;60:13–22. https://doi.org/10.1016/j.jcrc.2020.07.012.

    Article  PubMed  Google Scholar 

  76. Sidibe A, Moore L, Jean S, et al. Fracture risk in dialysis and kidney transplanted patients: a protocol for systematic review and meta-analysis. Syst Rev. 2017;6:37. https://doi.org/10.1186/s13643-017-0416-8.

    Article  PubMed  PubMed Central  Google Scholar 

  77. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. https://doi.org/10.1136/bmj.d5928.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank International Science Editing ( http://www.internationalscienceediting.com ) for editing this manuscript.

Funding

This study was supported by grants from the Dalian Key Medical Specialty Dengfeng Project (2022ZZ231 and 2022ZZ243 to Shu-Xin Liu; 2022ZZ236 to Shuang Zhang).

Author information

Authors and Affiliations

  1. Department of Nephrology, Dalian Municipal Central Hospital, No.826, Xinan Road, Dalian, Liaoning, 116033, China

    Shuang Zhang & Shu-Xin Liu

  2. Dalian Key Laboratory of Intelligent Blood Purification, Dalian Municipal Central Hospital, No.826, Xinan Road, Dalian, Liaoning, 116033, China

    Shuang Zhang & Shu-Xin Liu

  3. Department of Clinical Epidemiology, Shengjing Hospital of China Medical University, Shenyang, China

    Qi-Jun Wu

Authors
  1. Shuang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi-Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shu-Xin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Shu-Xin Liu: Study conceptualization and design; Shuang Zhang: Data collection; Shuang Zhang and Qi-Jun Wu: Analysis and interpretation of data; Shuang Zhang: Manuscript preparation; All authors read and approved the final manuscript.

Corresponding author

Correspondence to Shu-Xin Liu.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary file 1: Appendix 1.

Search strategy. Table S1. Overview of SRs that applied GRADE inappropriately a by rating the certainty of evidence for all studies or each individual study. Table S2. Summary of the study characteristics and the number and type of (un)rated outcomes of systematic reviews (n = 49) that rated the outcome-specific certainty of evidence with GRADE. Table S3. Overview of the GRADE domains of systematic reviews that rated the outcome-specific certainty of evidence.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Wu, QJ. & Liu, SX. A methodologic survey on use of the GRADE approach in evidence syntheses published in high-impact factor urology and nephrology journals. BMC Med Res Methodol 22, 220 (2022). https://doi.org/10.1186/s12874-022-01701-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-022-01701-x

Keywords

  • GRADE
  • Non-randomized studies
  • Randomized control trials
  • Systematic review
  • Urology and nephrology

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us