Inter-rater reliability of AMSTAR is dependent on the pair of reviewers

Table 5 Pairwise inter-rater reliability (Cohen’s kappa) for AMSTAR on item-level

Pair	1&2	1&3	1&4	1&5	2&3	2&4	2&5	3&4	3&5	4&5	Mean	Median
I1: Was an ‘a priori’ design provided?	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
I2: Was there duplicate study selection and data extraction?	0.33	0.71	0.50	0.33	0.61	0.88	0.16	0.75	0.20	0.13	0.46	0.42
I3: Was a comprehensive literature search performed?	0.45	0.45	0.45	0.20	0.18	1.00	0.29	0.18	0.29	0.29	0.38	0.29
I4: Was the status of publication (i.e. grey literature) used as an inclusion criterion?	0.30	0.19	0.38	0.14	0.48	0.88	0.33	0.63	0.54	0.50	0.44	0.43
I5: Was a list of studies (included and excluded) provided?	0.87	0.64	0.75	0.75	0.75	0.88	0.88	0.88	0.63	0.75	0.78	0.75
I6: Were the characteristics of the included studies provided? ^a
I7: Was the scientific quality of the included studies assessed and documented? ^a								0.33
I8: Was the scientific quality of the included studies used appropriately in formulating conclusions?	1.00	1.00	−0.07	0.33	1.00	−0.07	0.33	−0.07	0.33	−0.11	0.37	0.33
I9: Were the methods used to combine the findings of studies appropriate?	0.45	0.19	0.45	0.18	0.07	0.33	0.45	0.33	0.19	0.45	0.31	0.33
I10: Was the likelihood of publication bias assessed?	0.29	0.71	0.53	0.53	0.33	0.73	0.73	0.33	0.33	1.00	0.55	0.53
I11: Was the conflict of interest included?	0.24	0.16	0.24	0.20	0.29	0.59	0.35	0.76	0.48	0.67	0.40	0.32
Mean	0.55	0.56	0.47	0.41	0.52	0.69	0.50	0.53	0.43	0.52	0.52	0.52
Median	0.45	0.64	0.45	0.33	0.48	0.88	0.35	0.63	0.33	0.50	0.50

I Item
^aFor item 6 and 7 it was not possible to calculate pairwise inter-rater reliability, except for item 7 pair 3&5, because at least one reviewer of each pair scored “yes” for all 16 reviews (resulting in a constant variable)
Highest values per item are marked in bold, lowest values are marked in italics

ISSN: 1471-2288