Skip to main content

Table 4 Reasons for inter-rater disagreement

From: Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth

Information missed in extraction
a. Missed information in text
b. Missed information in in-text or summary tables in appendices
c. Missed information in a multi-dimensional measure
Information on target variables and processes being unclear
a. Lack of justification provided for target variables
b. In some instances it was unclear whether a target variable was defined on the basis of information present or absent in primary studies
c. Information provided in the review made it difficult to assign the target variable to a response category
d. In the absence of intervention theories or models linking intervention strategies to process, impacts and outcomes, it was difficult to interpret some variables
Limitations of the tool
a. Definitions in the tool did not adequately capture the heterogeneity in the target variable
b. Target variable has a two-part question which can lead to inconsistent ratings
c. Response category definitions
d. Definitions of the term is too narrow
e. Multiple indicators for a single measure
f. Reviews with one primary study
Limitations of the review
a. Inconsistency in the presentation of target variable in the review
b. Location of information in the review not in expected sections
c. Lack of sub-headings
d. Tables of summary characteristics