Skip to main content

Table 4 The percentage agreement (kappa statistic) in statistical significance (categorised as p \(\le\) 0.05 or p > 0.05) of level-change (upper triangle) and slope-change per month effect estimates (lower triangle, shaded grey) when ITS are analysed with OLS and REML, and meta-analysed by fixed, DL + WT and REML + HKSJ meta-analysis methods (n = 17)

From: Comparison of statistical methods used to meta-analyse results from interrupted time series studies: an empirical study

 

Level-change

Slope-change per month

OLS ITS

Fixed MA

70.6% (0.43)

64.7% (0.32)

70.6% (0.43)

64.7% (0.32)

82.4% (0.65)

82.4% (0.65)

70.6% (0.43)

76.5% (0.54)

64.7% (0.32)

76.5% (0.55)

OLS ITS

DL + WT MA

94.1% (0.82)

100.0% (1.00)

94.1% (0.82)

76.5% (0.51)

88.2% (0.72)

100.0% (1.00)

94.1% (0.85)

94.1% (0.82)

64.7% (0.35)

88.2% (0.72)

OLS ITS

DL + HKSJ MA

94.1% (0.82)

100.0% (1.00)

70.6% (0.39)

82.4% (0.56)

94.1% (0.82)

88.2% (0.68)

88.2% (0.60)

64.7% (0.35)

88.2% (0.72)

88.2% (0.67)

OLS ITS

REML + WT MA

94.1% (0.82)

76.5% (0.51)

88.2% (0.72)

100.0% (1.00)

94.1% (0.85)

94.1% (0.82)

58.8% (0.26)

82.4% (0.56)

94.1% (0.82)

94.1% (0.82)

OLS ITS

REML + HKSJ MA

70.6% (0.39)

82.4% (0.56)

94.1% (0.82)

88.2% (0.68)

88.2% (0.60)

82.4% (0.66)

82.4% (0.63)

70.6% (0.35)

70.6% (0.35)

64.7% (0.20)

REML ITS

Fixed MA

76.5% (0.52)

76.5% (0.51)

82.4% (0.64)

70.6% (0.39)

64.7% (0.35)

88.2% (0.72)

88.2% (0.67)

88.2% (0.67)

82.4% (0.46)

82.4% (0.61)

REML ITS

DL + WT MA

88.2% (0.72)

94.1% (0.87)

82.4% (0.56)

58.8% (0.26)

82.4% (0.56)

94.1% (0.82)

94.1% (0.82)

100.0% (1.00)

64.7% (0.20)

82.4% (0.46)

REML ITS

DL + HKSJ MA

94.1% (0.85)

94.1% (0.82)

58.8% (0.26)

82.4% (0.56)

82.4% (0.46)

94.1% (0.82)

88.2% (0.60)

76.5% (0.47)

94.1% (0.82)

88.2% (0.60)

REML ITS

REML + WT MA

88.2% (0.68)

58.8% (0.26)

82.4% (0.56)

94.1% (0.82)

94.1% (0.82)

100.0% (1.00)

64.7% (0.20)

82.4% (0.46)

100.0% (1.00)

88.2% (0.60)

REML ITS

REML + HKSJ MA

  1. The top row of the label indicates the ITS analysis methods, bottom row indicates the meta-analysis method, e.g., OLS Fixed is OLS ITS analysis and fixed effect meta-analysis. We used the following adjectives to describe agreement: moderate agreement as a kappa value of 0.41–0.6, substantial agreement as a value of 0.61–0.8, and almost perfect agreement as a value of 0.81–1.0
  2. DL DerSimonian and Laird, HKSJ Hartung-Knapp/Sidik-Jonkman, ITS interrupted time series, MA meta-analysis, OLS ordinary least squares, REML restricted maximum likelihood, WT Wald-type