Comparison of two Bayesian methods to detect mode effects between paper-based and computerized adaptive assessments: a preliminary Monte Carlo study

Table 3 Univariate and multivariate multilevel logistic regression to predict correct detection of mode effects defined by Robust Z and Bayesian 95% credible interval as a function of study variables

Model/Predictor	Univariate		Multivariate
Model/Predictor	OR	AUC ^a	OR	95% CI
Robust Z (Model AUC = 0.95)
Size of DIF	1.49**	0.55	3.42**	(2.58-4.54)
Percentage of DIF	1.17	0.52	1.20	(0.89,1.61)
2PL IRT Model^b	0.76**	0.53	0.47**	(0.35,0.64)
Diff. Mean θ = 1.0	0.99	0.50	0.66**	(0.50,0.87)
CAT Item Usage^c	21133.86**	0.94	3111.68**	(1417.85,6829.03)
Absolute Item Difficulty^d	0.03**	0.85	0.10**	(0.07,0.14)
Item Discrimination^d	3.62**	0.60	3.12**	(2.34,4.17)
Bayesian 95% Credible Interval (Model AUC = 0.93)
Size of DIF	1.73**	0.56	3.52**	(2.73,4.53)
Percentage of DIF	1.17	0.52	1.16	(0.89,1.50)
2PL IRT Model^b	0.74**	0.53	0.50**	(0.39,0.65)
Diff. Mean θ = 1.0	0.91	0.49	0.60**	(0.47,0.77)
CAT Item Usage^c	2468.29**	0.92	505.64**	(264.29,967.37)
Absolute Item Difficulty^d	0.04**	0.83	0.15**	(0.11,0.20)
Item Discrimination^d	2.86**	0.58	1.99**	(1.54,2.56)

Correct Detection of Mode Effects = true positive detection of mode DIF among items simulated with mode DIF; AUC = area under the ROC curve; CI = 95% confidence interval; IRT = item response theory model used to generate response data and parameters used in CAT; CAT item usage = number of times a given item was administered by CAT divided by 100; * p < .05; ** p < .01.

ISSN: 1471-2288