Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Table 1 Analogy between clinical research and computational statistical research

	Clinical research	Computational statistical research
Trial type	In vitro/animal study	Simulation
	Clinical trial	Benchmark study
	Blinded	Neutral and blind analysis
	(Placebo) controlled	(Null-model) controlled
	Cross-over	Paired samples
	Multi-arm	Multiple methods
Investigators	Trialist	Researcher conducting benchmark experiment
	Medical researcher	Methodological researcher in computational statistics
	Sponsor	Methodological researcher in computational statistics
Observation unit	Clinical trial patients	Real datasets
Comparators	Therapies, interventions and controls	Statistical and machine learning methods
Problem	Treatment of medical condition	Answering a question using data, e.g. prediction problem
Context	Patient’s preference, social context	Substantive context
	Personalized medicine	Meta-learning
Objectives	Improving patient’s health	Yielding reliable answer, e.g. increasing prediction performance
	Selecting and applying therapy to patient	Selecting and applying methods to datasets
	Application by medical practitioner	Application by statistical practitioner/consultant
Endpoints	Relevant clinical endpoints	Error rate, AUC, computing time, etc.
	Missing value (e.g. dropout)	Failure to produce output

ISSN: 1471-2288