Skip to main content

Table 3 Inter-rater reliability results for 47 items in the checklist for implementation (Ch-IMP) (n = 27 reviews)

From: Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth

  Percentage agreement Kappa (95 % CI) AC1 statistic (95 % CI)
Action model  
Intervention and service delivery protocols    
 Intervention Heterogeneity 82 0.74 (0.56–0.93) 0.79 (0.62–0.96)
Target population
 Age 85 0.80 (0.63–0.98) 0.83 (0.68–0.98)
 Gender 89 0.85 (0.68–1.00) 0.87 (0.74–1.00)
 Grade 85 0.75 (0.57–0.92) 0.84 (0.69–0.99)
 Ethnicity 82 0.74 (0.54–0.94) 0.79 (0.62–0.96)
 SES 74 0.62 (0.41–0.84) 0.71 (0.52–0.90)
Implementers
 Implementer identified 100 1.00  
 Qualifications 74 0.59 (0.37–0.82) 0.70 (0.50–0.89)
 Ethnicity 96 0.84 (0.53–1.00) 0.96 (0.89–1.00)
 Age 96 0.82 (0.53–1.00) 0.96 (0.89–1.00)
 Gender 96 0.78 (0.37–1.00) 0.96 (0.89–1.00)
 Socio-economic status 100 1.00  
 Role of the evaluator 96 0.90 (0.70–1.00) 0.96 (0.88–1.00)
Implementing organization
 Leadership 89 0.46 (0.03–0.89) 0.88 (0.76–1.00)
 Resourcing 100 1.00  
 Intervention development 93 0.72 (0.34–1.00) 0.92 (0.82–1.00)
 Quality of materials 93 0.79 (0.53–1.00) 0.92 (0.82–1.00)
 Cultural sensitivity 93 0.71 (0.34–1.00) 0.92 (0.82–1.00)
 Training 82 0.64 (0.40–0.88) 0.80 (0.63–0.96)
 Program improvement processes 93 0.68 (0.33–1.00) 0.92 (0.82–1.00)
 Technical or supervisory guidance 78 0.54 (0.26–0.81) 0.76 (0.58–0.93)
Associate organizations and community partners
 Presence/absence of partnership 93 0.81 (0.4 8–1.00) 0.92 (0.81–1.00)
 Other partnership proc 96 0.79 (0.45–1.00) 0.96 (0.89–1.00)
Ecological context
 Settings considered 74 0.62 (0.42–0.82) 0.66 (0.46–0.86)
 # Settings 48 0.37 (0.15–0.59) 0.40 (0.18–0.62)
Process evaluation
 Recruitment 82 0.68 (0.46–0.91) 0.80 (0.63–0.96)
 Attrition 74 0.63 (0.41–0.85) 0.71 (0.52–0.90)
 Minimum attrition 74 0.47 (0.20–0.74) 0.72 (0.53–0.90)
 Reach 93 0.82 (0.59–1.00) 0.92 (0.81–1.00)
 Minimum reach 96 0.65 (0.02–1.00) 0.96 (0.89–1.00)
 Dose delivered 79 0.70 (0.50–0.90) 0.76 (0.58–0.93)
 Minimum dose delivered 85 0.45 (0.03–0.86) 0.85 (0.70–0.99)
 Dose received 89 0.54 (0.10–0.98) 0.88 (0.76–1.00)
 Minimum dose received 100 1.00  
 Fidelity 78 0.60 (0.33–0.87) 0.76 (0.58–0.93)
 Minimum fidelity 100 1.00  
 Adaptation 82 0.67 (0.42–0.92) 0.80 (0.63–0.96)
 Minimum adaptation 100 1.00  
 Participant engagement 89 0.54 (0.10–0.98) 0.88 (0.76–1.00)
 Provider engagement 93 0.64 (0.17–1.00) 0.92 (0.82–1.00)
 Co-intervention 78 0.38 (0.03–0.73) 0.76 (0.59–0.94)
 Contamination 74 0.42 (0.12–0.71) 0.72 (0.54–0.90)
Change model
 Apriori intervention model 78 0.65 (0.40–0.89) 0.72 (0.52–0.92)
 Logic diagram used 100 1.00  
Environmment
 Years 74 0.63 (0.41–0.86) 0.71 (0.52–0.89)
 Country 74 0.65 (0.45–0.85) 0.70 (0.51–0.90)
 Urbanicity 78 0.67 (0.45–0.90) 0.75 (0.57–0.93)