Table 2 Simulation results for the setting in which there are 10 useful predictors and no noise variables. For bootstrap imputation methods on incomplete data, we show results corresponding to the best threshold values of π based on F1. The sample size n=1000. Two missingness proportions were considered: 40% missingness in Y and 60% overall missingness; 20% missingness in Y and 40% overall missingness. The performance measures were computed across 250 data replications
| Â | AUC | Precision | Recall | F1 | Type I error |
|---|---|---|---|---|---|
Fully observed data | |||||
BART | 0.74 (0.68, 0.80) | 1.00 | 0.62 | 0.70 | NA |
XGBoost | 0.75 (0.69, 0.81) | 1.00 | 0.61 | 0.69 | NA |
Incomplete data: 40% missingness inY and 60% overall missingness | |||||
RR-BART | 0.73 (0.67, 0.79) | 1.00 | 0.36 | 0.48 | NA |
RR-BART (all selected) | 0.97 (0.95, 0.99) | 1.00 | 1.00 | 1.00 | NA |
BI-BART π=0.1 | 0.73 (0.67, 0.79) | 1.00 | 0.38 | 0.50 | NA |
BI-XGB π=0.2 | 0.79 (0.73, 0.85) | 1.00 | 0.54 | 0.64 | NA |
BART Complete cases | 0.50 (0.43, 0.57) | 1.00 | 0.15 | 0.35 | NA |
XGBoost Complete cases | 0.55 (0.48, 0.62) | 1.00 | 0.18 | 0.38 | NA |
MIA-BART (Impute missing Y) | 0.66 (0.60, 0.72) | 1.00 | 0.31 | 0.42 | NA |
MIA-BART (Exclude missing Y) | 0.63 (0.56, 0.70) | 1.00 | 0.25 | 0.40 | NA |
MIA-XGB (Impute missing Y) | 0.73 (0.67, 0.69) | 1.00 | 0.50 | 0.59 | NA |
MIA-XGB (Exclude missing Y) | 0.70 (0.62, 0.77) | 1.00 | 0.46 | 0.55 | NA |
Incomplete data: 20% missingness inY and 30% overall missingness | |||||
RR-BART | 0.77 (0.72, 0.82) | 1.00 | 0.51 | 0.67 | NA |
RR-BART (all selected) | 0.98 (0.96, 0.99) | 1.00 | 1.00 | 1.00 | NA |
BI-BART π=0.1 | 0.75 (0.70, 0.80) | 1.00 | 0.50 | 0.69 | NA |
BI-XGB π=0.2 | 0.80 (0.75, 0.85) | 1.00 | 0.52 | 0.70 | NA |
MIA-BART (Impute missing Y) | 0.70 (0.65, 0.75) | 1.00 | 0.46 | 0.60 | NA |
MIA-BART (Exclude missing Y) | 0.67 (0.61, 0.73) | 1.00 | 0.43 | 0.57 | NA |
MIA-XGB (Impute missing Y) | 0.70 (0.65, 0.75) | 1.00 | 0.46 | 0.64 | NA |
MIA-XGB (Exclude missing Y) | 0.67 (0.61, 0.73) | 1.00 | 0.42 | 0.61 | NA |
BART Complete cases | 0.54 (0.49, 0.59) | 1.00 | 0.16 | 0.39 | NA |
XGBoost Complete cases | 0.56 (0.51, 0.61) | 1.00 | 0.20 | 0.40 | NA |