Skip to main content

Table 4 Descriptive characteristics of tools used to assess the quality of a peer review report

From: Tools used to assess the quality of peer review reports: a methodological systematic review

Journal or Company Name a First Author, Year Format Quality defined b Overall quality assessment Items (n) Items weights c Scoring range d Scoring system instruction e Scale/ Checklist Development f Validity g Reliability h Internal consistency RCTs i
Advances in Nursing Science; Issues in Mental Health Nursing; The Journal of Holistic Nursing Shattell 2010 [33] Scale N Summary Score 6 S 1–10 N NR NR NR NR 0
American Journal of Roentgenology Friedman 1995 [22] Scale N Single Score 1 NA 1–4 N NR NR NR NR 0
American Journal of Roentgenology Kliewer 2005 [49] Scale N Summary Score 4 NA 1–4 N NR NR NR NR 0
American Journal of Roentgenology Rajesh 2013 [32] Scale N Single Score 1 NA 1–4 P NR NR NR NR 0
American Journal of Roentgenology Berquist 2017 [50] Scale N Summary Score 4 NA 0–4 Y NR NR NR NR 0
Annals of Emergency Medicine Callaham 1998 [25] Scale N Single Score 1 NA 1–5 N NR NR Inter-Rater (ICC = 0.44, 0.24, 0.12) l NR 2 m
Annals of Emergency Medicine Callaham 2002 [26, 51] Scale N Summary Score 6 NA 1–5 N NR NR Inter-Rater (ICC = 0.44, 0.24, 0.12) l NR 1
Annals of Emergency Medicine; Annals of Internal Medicine; JAMA; Obstetrics & Gynecology and Ophthalmology Justice 1998 [35] Scale N Summary Score 4 S 1–5 N NR NR NR NR 0
British Journal of General Practice Moore 2014 [29] Scale N Single Score 1 NA A-E Y NR NR NR   0
British Medical Journal Black 1998 (RQI 3.2) [23, 39] Scale N Summary Score 7 S 1–5 N Y Face (N = 20) Test-Retest
(Kw = 1.00)
Internal Consistency (Cronbach’s alpha = 0.84) 5
Mean       Content (N = 20)
Construct
Inter-Rater
(Kw = 0.83)
British Medical Journal Van Rooyen 1999 (RQI 4) [27] Scale N Mean n 8 S 1–5 N NR NR Inter-Rater
(Kw = 0.38–0.67) o
  2
Chinese Journal of Tuberculosis and Respiratory Diseases Yang 2009 [52] Checklist N NA 5 NA NA N NR NR NR   0
Journal of Clinical Investigation Stossel 1985 [30] Scale N Single Score 1 NA Good-
Fair-
Poor
Y NR NR NR   0
Journal of General Internal Medicine McNutt 1990 [28, 40] Scale N Summary Score 9 S 1–5 N NR Construct NR   1
Journal of Vascular Interventional Radiology Feurer 1994 [41] Scale N Sum 7 D 0–14 N NR Content (N = 2)
Preliminary
Criterion (N = 2) (Kendall = 0.94)
Inter-Rater
(ICC = 0.84)
  0
NA Review quality collector (RQC) 2012 [53] Scale N Mean 4 User-defined weights 0–100 N NR NR NR   0
Nursing Research Henly 2009 [24] Scale N Mean (CAS, GAS scale) 15 S 1–5 P NR NR Inter-Rater (ICC = 0.79) p   0
Summary Score (OAS scale)    1–5
Summary Score (GRQ scale)    0–100
Nursing Research Henly 2010 [36] Scale N Mean (CAS, GAR, SARNR scale) 26 S 1–5 P NR NR Inter-Rater
(ICC = 0.75)p
  0
Summary Score (GRQ scale)    0–100
Obstetrics & Gynecology, Dutch Journal of Medicine Landkroon 2006 [42] Scale N Summary Score 5 NA 1–5 Y NR NR Test-Retest
(ICC =0.66–0.88)
Inter-Rater
(ICC = 0.62)
  0
Pakistan Journal of Medical Sciences Jawaid 2006 [34] Scale N NR q 5 S 1–5 N NR NR NR   0
Peerage of science Peerage Essay Quality (PEQ) 2011 [37] Scale N Mean 3 S 1–5 N NR NR NR   0
Publons Academy Review Rating and Feedback Form 2016 [38] Scale N Sum 4 S 0–3 (Full score: 0–12) N NR NR NR   0
The Journal of Bone and Joint Surgery Thompson 2016 [31] Scale N Single Score 1 NA 80–100 Y NR NR Inter-Rater
(ICC = -4.5 to 0.99) r
  0
The National Medical Journal of India Das Sinha 1999 [54] Scale N Sum 5 D 0–100 N NR NR NR   0
  1. aName of journal or company/organization where the tool was used to assess the quality of their peer review reports
  2. bThe quality of a peer review report is not clearly defined in any reports
  3. cNA Not applicable, S Same weight for each item, D Different weight for each item
  4. dNA Not applicable
  5. eY Yes defined, P Partially defined, N Not defined
  6. f, g, hNR Not reported
  7. iNumber of randomized controlled trials where the tool was used as outcome criteria
  8. lThe ICC was 0.44 for reviewers, 0.24 for editors, and 0.12 for manuscripts
  9. mOne article consists of two studies. First study is not a RCT while the second one is a RCT [55]
  10. nThe overall quality is based on the mean of the first seven items (the item about the tone of the review was not included)
  11. oThe inter-rater reliability was measured with weighted K for item from 1 to 7 for two editors’ independent assessments
  12. pThe tool includes more than one scale. We reported inter-rater reliability only for General Review Quality (GRQ) scale
  13. qNot reported. Although the authors reported that the reviewers were rated as excellent, good and average based on the quality of the reviews, it is not reported how they assessed the overall quality of peer review reports
  14. rICC range for 11 manuscripts. There was one outlier manuscript that if removed brought the range to 0.87–0.99