Skip to main content

Advertisement

Table 2 Concepts, limitations and applications of statistical approaches for the evaluation of natural policy experiments

From: Assessing the impact of natural policy experiments on socioeconomic inequalities in health: how to apply commonly used quantitative analytical methods?

Method Main concept Minimum data requirement Adjustment for confounders Main limitations Application to the evaluation of policies on health inequalities
Regression adjustment Adjustment for confounders, i.e. factors related to both intervention allocation and health outcomes. Cross-sectional Observed confounders Vulnerable to unobserved confounders [50]
Propensity score matching For a given propensity score, exposure to the intervention is random. The intervention effect is therefore the average difference in the outcomes between the exposed and the matched unexposed units with the same propensity scores. Cross-sectional Observed confounders Vulnerable to unobserved confounders [51]
Difference-in-differences As long as the naturally occurring changes over time in the intervention and control group are the same, the difference in the change in the outcome between both groups can be interpreted as the intervention effect. Repeated cross-sectional Observed and time-invariant unobserved confounders Vulnerable to violation of the common trend assumption [22]
Fixed effects Multiple observations within units are compared, such as repeated measurements over time within individuals. Effects of unobserved confounders that differ between units but remain constant over time are eliminated. Longitudinal Observed and time-invariant unobserved confounders Vulnerable to unobserved time-variant confounders; Knocks out all cross-sectional variations between units; Susceptible to measurement errors over time; [52, 53]
Instrumental variable approach An instrument creates variation in exposure to the intervention, without being directly related to the outcome itself. Cross-sectional Observed and unobserved confounders Difficult to find good instrumental variables; Exogeneity of instruments cannot be easily tested; Weak instruments and finite samples might result in bias; Local average treatment effect problem; [54]
Regression discontinuity As long as the association between a variable and an outcome is smooth, any discontinuity in the outcome after a cut-off point of this variable can be regarded as an intervention effect. Cross-sectional Observed and unobserved confounders Low external validity; Local average treatment effect problem in a fuzzy design; [23]
Interrupted time-series Identification of a sudden change in level of the health outcome (a change of intercept) or a more sustained change in trend of the health outcome (a change of slope) around the time of the implementation of the intervention. Repeated measures Observed confounders Difficult to evaluate the interventions implemented slowly, or need unpredictable time to be effective; Vulnerable to other external interventions or shocks within the period; [5557]