Fig. 2

Our approach to assess the transportability of machine learning models predicting cognitive impairment. Orange boxes mark the four general steps of our workflow. We first mapped knowledge about cognitive impairment into a Directed Acyclic Graph (DAG) and quantified those using Structural equation modelling (SEM) and data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The estimates were used in linear equations to generate datasets for training, internal validation and four external validation datasets with interventions on age, APOE ε4, and tau. The age variable was intervened two times (age and age2) by sampling from normal distributions with two different mean age values (35 and 65). We trained four machine learning algorithms (logistic regression, lasso regression, random forest and generalized boosted regression) to predict cognitive impairment using four sets of predictors. We measured transportability between internal and external settings using calibration differences, measured by Integrated Calibration Index (ICI) and Brier score, and differences in Area under the Receiver Operating Curve (AUC). Steps 3 to 4 (data synthesis and model training and validation) were repeated 10,000 times for each of the three imputed ADNI datasets