Fig. 2
Combination of internal validation (Val), using the example of bootstrap (BS), and multiple imputation (MI). aVal: Visualization of BS in complete data. \(\hat \theta ^{{Dat}_{1},{Dat}_{2}}\) denotes performance when the model was fitted on Dat1 and evaluated on Dat2, where Orig denotes the original data set, BS(b) the bth BS set, OOB(b) the bth out-of-bag (OOB) set, b=1,…,B. Average performance values across the B sets are denoted by \(\hat \theta ^{{BS,BS}}\), \(\hat \theta ^{{BS, OOB}}\) and \(\hat \theta ^{{BS, Orig}}\). \(\hat \theta ^{{noinfo}}\) denotes the average performance in the absence of an effect (see text). Performance measures: \(\hat \theta ^{{opt.corr.}}\), ordinary optimism-corrected BS estimate [3]; \(\hat \theta ^{{OOB}}\), OOB performance estimate; \(\hat \theta ^{0.632+}\), BS 0.632+ estimate [23]. In the specific case of w=0.632, the BS 0.632 estimate (\(\hat \theta ^{0.632}\)) is obtained. bVal-MI: Combination of BS and MI by drawing BS samples followed by MI separately on the BS samples and on the OOB samples not contained in the respective BS draw. cMI-Val and MI(-y)-Val: Combination of MI and BS by conducting MI followed by drawing BS samples from the imputed data sets. For b and c, performance measures are derived similarly as for complete data (a), this time averaging across the B·M sets, and deriving apparent performance \(\hat \theta ^{{Orig,Orig}}\) as the average performance across the M imputed data sets