Statistical model building: Background “knowledge” based on inappropriate preselection causes misspecification

Hafermann, Lorena; Becher, Heiko; Herrmann, Carolin; Klein, Nadja; Heinze, Georg; Rauch, Geraldine

doi:10.1186/s12874-021-01373-z

BMC Medical Research Methodology

Table 2 Performance measures for investigated simulation scenarios

From: Statistical model building: Background “knowledge” based on inappropriate preselection causes misspecification

Scenario	S(*)_a: Univariable selection,				S(*)_b: Univariable selection,				S(*)_c: Backward elimination,
	α_in=0.05				α_in=0.2				AIC
	MSF	TPR	FPR	FNR	MSF	TPR	FPR	FNR	MSF	TPR	FPR	FNR
S1_(*)	R1: 0	R1: 0.247	R1: 0.280	R1: 0.091	R1: 0	R1: 0.644	R1: 0.607	R1: 0.040	R1: 0.001	R1: 0.325	R1: 0.484	R1: 0.096
	R2: 0.001	R2: 0.014	R2: 0.092	R2: 0.182	R2: 0.001	R2: 0.189	R2: 0.232	R2: 0.101	R2: 0.006	R2: 0.021	R2: 0.102	R2: 0.218
	R3: 0	R3: 0	R3: 0.036	R3: 0.348	R3: 0.001	R3: 0.008	R3: 0.083	R3: 0.215	R3: 0	R3: 0	R3: 0.008	R3: 0.325
S2_(*)	R1: 0	R1: 0.448	R1: 0.364	R1: 0.061	R1: 0	R1: 0.805	R1: 0.681	R1: 0.022	R1: 0.001	R1: 0.422	R1: 0.452	R1: 0.070
	R2: 0	R2: 0.087	R2: 0.137	R2: 0.102	R2: 0	R2: 0.382	R2: 0.309	R2: 0.069	R2: 0.017	R2: 0.052	R2: 0.089	R2: 0.152
	R3: 0	R3: 0.006	R3: 0.088	R3: 0.135	R3: 0.001	R3: 0.067	R3: 0.129	R3: 0.108	R3: 0.001	R3: 0.001	R3: 0.006	R3: 0.233
S3_(*)	R1: 0	R1: 0.912	R1: 0.571	R1: 0.010	R1: 0	R1: 0.992	R1: 0.793	R1: 0.001	R1: 0.001	R1: 0.467	R1: 0.436	R1: 0.059
	R2: 0	R2: 0.581	R2: 0.331	R2: 0.047	R2: 0	R2: 0.889	R2: 0.531	R2: 0.012	R2: 0.034	R2: 0.092	R2: 0.081	R2: 0.102
	R3: 0	R3: 0.170	R3: 0.175	R3: 0.092	R3: 0	R3: 0.505	R3: 0.299	R3: 0.055	R3: 0.007	R3: 0.007	R3: 0.005	R3: 0.126
S4_(*)	R1: 0	R1: 0.337	R1: 0.312	R1: 0.074	R1: 0	R1: 0.711	R1: 0.635	R1: 0.032	R1: 0.001	R1: 0.378	R1: 0.476	R1: 0.083
	R2: 0.001	R2: 0.031	R2: 0.108	R2: 0.142	R2: 0	R2: 0.244	R2: 0.256	R2: 0.088	R2: 0.010	R2: 0.029	R2: 0.098	R2: 0.193
	R3: 0	R3: 0	R3: 0.049	R3: 0.298	R3: 0.003	R3: 0.018	R3: 0.095	R3: 0.181	R3: 0	R3: 0	R3: 0.008	R3: 0.303
S5_(*)	R1: 0	R1: 0.401	R1: 0.340	R1: 0.067	R1: 0	R1: 0.764	R1: 0.659	R1: 0.026	R1: 0.001	R1: 0.405	R1: 0.462	R1: 0.075
	R2: 0.001	R2: 0.061	R2: 0.125	R2: 0.110	R2: 0	R2: 0.322	R2: 0.285	R2: 0.076	R2: 0.014	R2: 0.044	R2: 0.093	R2: 0.168
	R3: 0	R3: 0.001	R3: 0.065	R3: 0.231	R3: 0.003	R3: 0.037	R3: 0.110	R3: 0.148	R3: 0	R3: 0	R3: 0.007	R3: 0.273
S6_(*)	R1: 0	R1: 0.678	R1: 0.456	R1: 0.036	R1: 0	R1: 0.917	R1: 0.734	R1: 0.009	R1: 0.001	R1: 0.458	R1: 0.461	R1: 0.061
	R2: 0	R2: 0.157	R2: 0.166	R2: 0.096	R2: 0	R2: 0.488	R2: 0.359	R2: 0.057	R2: 0.023	R2: 0.063	R2: 0.092	R2: 0.141
	R3: 0.001	R3: 0.003	R3: 0.071	R3: 0.222	R3: 0.002	R3: 0.068	R3: 0.131	R3: 0.141	R3: 0.001	R3: 0.001	R3: 0.006	R3: 0.258
S7_(*)	R1: 0	R1: 0.644	R1: 0.440	R1: 0.040	R1: 0	R1: 0.899	R1: 0.720	R1: 0.011	R1: 0.001	R1: 0.465	R1: 0.469	R1: 0.061
	R2: 0.003	R2: 0.080	R2: 0.139	R2: 0.130	R2: 0	R2: 0.403	R2: 0.322	R2: 0.069	R2: 0.015	R2: 0.051	R2: 0.096	R2: 0.167
	R3: 0	R3: 0.001	R3: 0.051	R3: 0.295	R3: 0.004	R3: 0.036	R3: 0.110	R3: 0.177	R3: 0	R3: 0.001	R3: 0.007	R3: 0.292
S8_(*)	R1: 0	R1: 0.705	R1: 0.467	R1: 0.033	R1: 0	R1: 0.933	R1: 0.740	R1: 0.007	R1: 0.001	R1: 0.453	R1: 0.448	R1: 0.062
	R2: 0	R2: 0.199	R2: 0.188	R2: 0.089	R2: 0	R2: 0.557	R2: 0.387	R2: 0.049	R2: 0.023	R2: 0.070	R2: 0.087	R2: 0.129
	R3: 0.001	R3: 0.016	R3: 0.098	R3: 0.126	R3: 0.001	R3: 0.136	R3: 0.159	R3: 0.098	R3: 0.002	R3: 0.002	R3: 0.006	R3: 0.211
S9_(*)	R1: 0	R1: 0.834	R1: 0.523	R1: 0.019	R1: 0	R1: 0.970	R1: 0.767	R1: 0.003	R1: 0.001	R1: 0.475	R1: 0.454	R1: 0.058
	R2: 0	R2: 0.362	R2: 0.252	R2: 0.071	R2: 0	R2: 0.729	R2: 0.455	R2: 0.030	R2: 0.031	R2: 0.090	R2: 0.088	R2: 0.108
	R3: 0.002	R3: 0.011	R3: 0.082	R3: 0.213	R3: 0.003	R3: 0.128	R3: 0.166	R3: 0.130	R3: 0.001	R3: 0.001	R3: 0.006	R3: 0.237
S10_(*)	R1: 0	R1: 0.840	R1: 0.531	R1: 0.018	R1: 0	R1: 0.977	R1: 0.773	R1: 0.003	R1: 0.001	R1: 0.464	R1: 0.446	R1: 0.060
	R2: 0	R2: 0.403	R2: 0.267	R2: 0.066	R2: 0	R2: 0.774	R2: 0.472	R2: 0.025	R2: 0.031	R2: 0.091	R2: 0.085	R2: 0.106
	R3: 0.001	R3: 0.048	R3: 0.119	R3: 0.114	R3: 0.001	R3: 0.264	R3: 0.210	R3: 0.083	R3: 0.003	R3: 0.003	R3: 0.006	R3: 0.179

MSF Model selection frequency; TPR True positive rate; FPR Average false positive rate; FNR Average false negative rate; R1 Rule 1 (one out of three preceding studies identifies predictor); R2 Rule 2 (Two out of three preceding studies identify predictor); R3 Rule 3 (all three preceding studies identify predictor)

Back to article page

ISSN: 1471-2288

Contact us

General enquiries: journalsubmissions@springernature.com