Skip to main content

Table 2 Performance measures for investigated simulation scenarios

From: Statistical model building: Background “knowledge” based on inappropriate preselection causes misspecification

Scenario S(*)_a: Univariable selection,   S(*)_b: Univariable selection,   S(*)_c: Backward elimination,  
  αin=0.05    αin=0.2    AIC
  MSF TPR FPR FNR MSF TPR FPR FNR MSF TPR FPR FNR
S1_(*) R1: 0 R1: 0.247 R1: 0.280 R1: 0.091 R1: 0 R1: 0.644 R1: 0.607 R1: 0.040 R1: 0.001 R1: 0.325 R1: 0.484 R1: 0.096
  R2: 0.001 R2: 0.014 R2: 0.092 R2: 0.182 R2: 0.001 R2: 0.189 R2: 0.232 R2: 0.101 R2: 0.006 R2: 0.021 R2: 0.102 R2: 0.218
  R3: 0 R3: 0 R3: 0.036 R3: 0.348 R3: 0.001 R3: 0.008 R3: 0.083 R3: 0.215 R3: 0 R3: 0 R3: 0.008 R3: 0.325
S2_(*) R1: 0 R1: 0.448 R1: 0.364 R1: 0.061 R1: 0 R1: 0.805 R1: 0.681 R1: 0.022 R1: 0.001 R1: 0.422 R1: 0.452 R1: 0.070
  R2: 0 R2: 0.087 R2: 0.137 R2: 0.102 R2: 0 R2: 0.382 R2: 0.309 R2: 0.069 R2: 0.017 R2: 0.052 R2: 0.089 R2: 0.152
  R3: 0 R3: 0.006 R3: 0.088 R3: 0.135 R3: 0.001 R3: 0.067 R3: 0.129 R3: 0.108 R3: 0.001 R3: 0.001 R3: 0.006 R3: 0.233
S3_(*) R1: 0 R1: 0.912 R1: 0.571 R1: 0.010 R1: 0 R1: 0.992 R1: 0.793 R1: 0.001 R1: 0.001 R1: 0.467 R1: 0.436 R1: 0.059
  R2: 0 R2: 0.581 R2: 0.331 R2: 0.047 R2: 0 R2: 0.889 R2: 0.531 R2: 0.012 R2: 0.034 R2: 0.092 R2: 0.081 R2: 0.102
  R3: 0 R3: 0.170 R3: 0.175 R3: 0.092 R3: 0 R3: 0.505 R3: 0.299 R3: 0.055 R3: 0.007 R3: 0.007 R3: 0.005 R3: 0.126
S4_(*) R1: 0 R1: 0.337 R1: 0.312 R1: 0.074 R1: 0 R1: 0.711 R1: 0.635 R1: 0.032 R1: 0.001 R1: 0.378 R1: 0.476 R1: 0.083
  R2: 0.001 R2: 0.031 R2: 0.108 R2: 0.142 R2: 0 R2: 0.244 R2: 0.256 R2: 0.088 R2: 0.010 R2: 0.029 R2: 0.098 R2: 0.193
  R3: 0 R3: 0 R3: 0.049 R3: 0.298 R3: 0.003 R3: 0.018 R3: 0.095 R3: 0.181 R3: 0 R3: 0 R3: 0.008 R3: 0.303
S5_(*) R1: 0 R1: 0.401 R1: 0.340 R1: 0.067 R1: 0 R1: 0.764 R1: 0.659 R1: 0.026 R1: 0.001 R1: 0.405 R1: 0.462 R1: 0.075
  R2: 0.001 R2: 0.061 R2: 0.125 R2: 0.110 R2: 0 R2: 0.322 R2: 0.285 R2: 0.076 R2: 0.014 R2: 0.044 R2: 0.093 R2: 0.168
  R3: 0 R3: 0.001 R3: 0.065 R3: 0.231 R3: 0.003 R3: 0.037 R3: 0.110 R3: 0.148 R3: 0 R3: 0 R3: 0.007 R3: 0.273
S6_(*) R1: 0 R1: 0.678 R1: 0.456 R1: 0.036 R1: 0 R1: 0.917 R1: 0.734 R1: 0.009 R1: 0.001 R1: 0.458 R1: 0.461 R1: 0.061
  R2: 0 R2: 0.157 R2: 0.166 R2: 0.096 R2: 0 R2: 0.488 R2: 0.359 R2: 0.057 R2: 0.023 R2: 0.063 R2: 0.092 R2: 0.141
  R3: 0.001 R3: 0.003 R3: 0.071 R3: 0.222 R3: 0.002 R3: 0.068 R3: 0.131 R3: 0.141 R3: 0.001 R3: 0.001 R3: 0.006 R3: 0.258
S7_(*) R1: 0 R1: 0.644 R1: 0.440 R1: 0.040 R1: 0 R1: 0.899 R1: 0.720 R1: 0.011 R1: 0.001 R1: 0.465 R1: 0.469 R1: 0.061
  R2: 0.003 R2: 0.080 R2: 0.139 R2: 0.130 R2: 0 R2: 0.403 R2: 0.322 R2: 0.069 R2: 0.015 R2: 0.051 R2: 0.096 R2: 0.167
  R3: 0 R3: 0.001 R3: 0.051 R3: 0.295 R3: 0.004 R3: 0.036 R3: 0.110 R3: 0.177 R3: 0 R3: 0.001 R3: 0.007 R3: 0.292
S8_(*) R1: 0 R1: 0.705 R1: 0.467 R1: 0.033 R1: 0 R1: 0.933 R1: 0.740 R1: 0.007 R1: 0.001 R1: 0.453 R1: 0.448 R1: 0.062
  R2: 0 R2: 0.199 R2: 0.188 R2: 0.089 R2: 0 R2: 0.557 R2: 0.387 R2: 0.049 R2: 0.023 R2: 0.070 R2: 0.087 R2: 0.129
  R3: 0.001 R3: 0.016 R3: 0.098 R3: 0.126 R3: 0.001 R3: 0.136 R3: 0.159 R3: 0.098 R3: 0.002 R3: 0.002 R3: 0.006 R3: 0.211
S9_(*) R1: 0 R1: 0.834 R1: 0.523 R1: 0.019 R1: 0 R1: 0.970 R1: 0.767 R1: 0.003 R1: 0.001 R1: 0.475 R1: 0.454 R1: 0.058
  R2: 0 R2: 0.362 R2: 0.252 R2: 0.071 R2: 0 R2: 0.729 R2: 0.455 R2: 0.030 R2: 0.031 R2: 0.090 R2: 0.088 R2: 0.108
  R3: 0.002 R3: 0.011 R3: 0.082 R3: 0.213 R3: 0.003 R3: 0.128 R3: 0.166 R3: 0.130 R3: 0.001 R3: 0.001 R3: 0.006 R3: 0.237
S10_(*) R1: 0 R1: 0.840 R1: 0.531 R1: 0.018 R1: 0 R1: 0.977 R1: 0.773 R1: 0.003 R1: 0.001 R1: 0.464 R1: 0.446 R1: 0.060
  R2: 0 R2: 0.403 R2: 0.267 R2: 0.066 R2: 0 R2: 0.774 R2: 0.472 R2: 0.025 R2: 0.031 R2: 0.091 R2: 0.085 R2: 0.106
  R3: 0.001 R3: 0.048 R3: 0.119 R3: 0.114 R3: 0.001 R3: 0.264 R3: 0.210 R3: 0.083 R3: 0.003 R3: 0.003 R3: 0.006 R3: 0.179
  1. MSF Model selection frequency; TPR True positive rate; FPR Average false positive rate; FNR Average false negative rate; R1 Rule 1 (one out of three preceding studies identifies predictor); R2 Rule 2 (Two out of three preceding studies identify predictor); R3 Rule 3 (all three preceding studies identify predictor)