Table 2 Concepts, limitations and applications of statistical approaches for the evaluation of natural policy experiments

Regression adjustment

Adjustment for confounders, i.e. factors related to both intervention allocation and health outcomes.

Cross-sectional

Observed confounders

Vulnerable to unobserved confounders

[50]

Propensity score matching

For a given propensity score, exposure to the intervention is random. The intervention effect is therefore the average difference in the outcomes between the exposed and the matched unexposed units with the same propensity scores.

Cross-sectional

Observed confounders

Vulnerable to unobserved confounders

[51]

Difference-in-differences

As long as the naturally occurring changes over time in the intervention and control group are the same, the difference in the change in the outcome between both groups can be interpreted as the intervention effect.

Repeated cross-sectional

Observed and time-invariant unobserved confounders

Vulnerable to violation of the common trend assumption

[22]

Fixed effects

Multiple observations within units are compared, such as repeated measurements over time within individuals. Effects of unobserved confounders that differ between units but remain constant over time are eliminated.

Longitudinal

Observed and time-invariant unobserved confounders

Vulnerable to unobserved time-variant confounders; Knocks out all cross-sectional variations between units; Susceptible to measurement errors over time;

[52, 53]

Instrumental variable approach

An instrument creates variation in exposure to the intervention, without being directly related to the outcome itself.

Cross-sectional

Observed and unobserved confounders

Difficult to find good instrumental variables; Exogeneity of instruments cannot be easily tested; Weak instruments and finite samples might result in bias; Local average treatment effect problem;

[54]

Regression discontinuity

As long as the association between a variable and an outcome is smooth, any discontinuity in the outcome after a cut-off point of this variable can be regarded as an intervention effect.

Cross-sectional

Observed and unobserved confounders

Low external validity; Local average treatment effect problem in a fuzzy design;

[23]

Interrupted time-series

Identification of a sudden change in level of the health outcome (a change of intercept) or a more sustained change in trend of the health outcome (a change of slope) around the time of the implementation of the intervention.

Repeated measures

Observed confounders

Difficult to evaluate the interventions implemented slowly, or need unpredictable time to be effective; Vulnerable to other external interventions or shocks within the period;

[55–57]