Skip to main content

Table 4 Descriptive characteristics of tools used to assess the quality of a peer review report

From: Tools used to assess the quality of peer review reports: a methodological systematic review

Journal or Company Name a

First Author, Year

Format

Quality defined b

Overall quality assessment

Items (n)

Items weights c

Scoring range d

Scoring system instruction e

Scale/ Checklist Development f

Validity g

Reliability h

Internal consistency

RCTs i

Advances in Nursing Science; Issues in Mental Health Nursing; The Journal of Holistic Nursing

Shattell 2010 [33]

Scale

N

Summary Score

6

S

1–10

N

NR

NR

NR

NR

0

American Journal of Roentgenology

Friedman 1995 [22]

Scale

N

Single Score

1

NA

1–4

N

NR

NR

NR

NR

0

American Journal of Roentgenology

Kliewer 2005 [49]

Scale

N

Summary Score

4

NA

1–4

N

NR

NR

NR

NR

0

American Journal of Roentgenology

Rajesh 2013 [32]

Scale

N

Single Score

1

NA

1–4

P

NR

NR

NR

NR

0

American Journal of Roentgenology

Berquist 2017 [50]

Scale

N

Summary Score

4

NA

0–4

Y

NR

NR

NR

NR

0

Annals of Emergency Medicine

Callaham 1998 [25]

Scale

N

Single Score

1

NA

1–5

N

NR

NR

Inter-Rater (ICC = 0.44, 0.24, 0.12) l

NR

2 m

Annals of Emergency Medicine

Callaham 2002 [26, 51]

Scale

N

Summary Score

6

NA

1–5

N

NR

NR

Inter-Rater (ICC = 0.44, 0.24, 0.12) l

NR

1

Annals of Emergency Medicine; Annals of Internal Medicine; JAMA; Obstetrics & Gynecology and Ophthalmology

Justice 1998 [35]

Scale

N

Summary Score

4

S

1–5

N

NR

NR

NR

NR

0

British Journal of General Practice

Moore 2014 [29]

Scale

N

Single Score

1

NA

A-E

Y

NR

NR

NR

 

0

British Medical Journal

Black 1998 (RQI 3.2) [23, 39]

Scale

N

Summary Score

7

S

1–5

N

Y

Face (N = 20)

Test-Retest

(Kw = 1.00)

Internal Consistency (Cronbach’s alpha = 0.84)

5

Mean

     

Content (N = 20)

Construct

Inter-Rater

(Kw = 0.83)

British Medical Journal

Van Rooyen 1999 (RQI 4) [27]

Scale

N

Mean n

8

S

1–5

N

NR

NR

Inter-Rater

(Kw = 0.38–0.67) o

 

2

Chinese Journal of Tuberculosis and Respiratory Diseases

Yang 2009 [52]

Checklist

N

NA

5

NA

NA

N

NR

NR

NR

 

0

Journal of Clinical Investigation

Stossel 1985 [30]

Scale

N

Single Score

1

NA

Good-

Fair-

Poor

Y

NR

NR

NR

 

0

Journal of General Internal Medicine

McNutt 1990 [28, 40]

Scale

N

Summary Score

9

S

1–5

N

NR

Construct

NR

 

1

Journal of Vascular Interventional Radiology

Feurer 1994 [41]

Scale

N

Sum

7

D

0–14

N

NR

Content (N = 2)

Preliminary

Criterion (N = 2) (Kendall = 0.94)

Inter-Rater

(ICC = 0.84)

 

0

NA

Review quality collector (RQC) 2012 [53]

Scale

N

Mean

4

User-defined weights

0–100

N

NR

NR

NR

 

0

Nursing Research

Henly 2009 [24]

Scale

N

Mean (CAS, GAS scale)

15

S

1–5

P

NR

NR

Inter-Rater (ICC = 0.79) p

 

0

Summary Score (OAS scale)

  

1–5

Summary Score (GRQ scale)

  

0–100

Nursing Research

Henly 2010 [36]

Scale

N

Mean (CAS, GAR, SARNR scale)

26

S

1–5

P

NR

NR

Inter-Rater

(ICC = 0.75)p

 

0

Summary Score (GRQ scale)

  

0–100

Obstetrics & Gynecology, Dutch Journal of Medicine

Landkroon 2006 [42]

Scale

N

Summary Score

5

NA

1–5

Y

NR

NR

Test-Retest

(ICC =0.66–0.88)

Inter-Rater

(ICC = 0.62)

 

0

Pakistan Journal of Medical Sciences

Jawaid 2006 [34]

Scale

N

NR q

5

S

1–5

N

NR

NR

NR

 

0

Peerage of science

Peerage Essay Quality (PEQ) 2011 [37]

Scale

N

Mean

3

S

1–5

N

NR

NR

NR

 

0

Publons Academy

Review Rating and Feedback Form 2016 [38]

Scale

N

Sum

4

S

0–3 (Full score: 0–12)

N

NR

NR

NR

 

0

The Journal of Bone and Joint Surgery

Thompson 2016 [31]

Scale

N

Single Score

1

NA

80–100

Y

NR

NR

Inter-Rater

(ICC = -4.5 to 0.99) r

 

0

The National Medical Journal of India

Das Sinha 1999 [54]

Scale

N

Sum

5

D

0–100

N

NR

NR

NR

 

0

  1. aName of journal or company/organization where the tool was used to assess the quality of their peer review reports
  2. bThe quality of a peer review report is not clearly defined in any reports
  3. cNA Not applicable, S Same weight for each item, D Different weight for each item
  4. dNA Not applicable
  5. eY Yes defined, P Partially defined, N Not defined
  6. f, g, hNR Not reported
  7. iNumber of randomized controlled trials where the tool was used as outcome criteria
  8. lThe ICC was 0.44 for reviewers, 0.24 for editors, and 0.12 for manuscripts
  9. mOne article consists of two studies. First study is not a RCT while the second one is a RCT [55]
  10. nThe overall quality is based on the mean of the first seven items (the item about the tone of the review was not included)
  11. oThe inter-rater reliability was measured with weighted K for item from 1 to 7 for two editors’ independent assessments
  12. pThe tool includes more than one scale. We reported inter-rater reliability only for General Review Quality (GRQ) scale
  13. qNot reported. Although the authors reported that the reviewers were rated as excellent, good and average based on the quality of the reviews, it is not reported how they assessed the overall quality of peer review reports
  14. rICC range for 11 manuscripts. There was one outlier manuscript that if removed brought the range to 0.87–0.99
Back to article page

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us