Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents

Table 4 “Out-of-the-box” individual PHI recall results for partial and fully-contained matches using the VHA evaluation corpus

RULE-BASED SYSTEMS
PHI type	#Inst.	PARTIAL MATCHES			FULLY-CONTAINED MATCHES
		HMS Scrubber	MeDS	MIT deid	HMS Scrubber	MeDS	MIT deid
Patient Name	206	0.83	0.99	0.98	0.57	0.69	0.69
Relative Name	30	0.76	0.95	1	0.57	0.67	0.77
Healthcare Provider Name	492	0.74	0.96	0.94	0.43	0.47	0.38
Other Person Name	20	0.66	0.81	0.74	0.30	0.25	0.35
Street City	137	0.90	0.96	0.81	0.70	0.78	0.78
State Country	161	0.45	0.49	0.85	0.43	0.45	0.84
Deployment	43	0.34	0.49	0.27	0.07	0.02	0.05
ZIP code	4	1	1	1	1	1	1
Healthcare Unit Name	1453	0.45	0.51	0.12	0.24	0.23	0.03
Other Org Name	86	0.33	0.50	0.27	0.03	0.20	0.03
Date	2547	0.74	0.87	0.80	0.34	0.27	0.46
Age > 89	4	0	0	1	0	0	1
Phone Number	90	0.73	0.79	0.80	0.42	0.5	0.48
Electronic Address	4	0	0.86	0.75	0	0	0.75
SSN	16	1	1	1	1	1	1
Other ID Number	123	0.66	0.82	0.41	0.43	0.61	0.27
MACHINE LEARNING-BASED SYSTEMS
PHI type	#Inst.	PARTIAL MATCHES			FULLY-CONTAINED MATCHES
		MIST	HIDE		MIST	HIDE
Patient Name	206	0.51	0.54		0.42	0.50
Relative Name	30	0.13	0.13		0.13	0.13
Healthcare Provider Name	492	0.53	0.59		0.44	0.53
Other Person Name	20	0	0.20		0	0.15
Street City	137	0.26	0.29		0.26	0.27
State Country	161	0.14	0.22		0.14	0.21
Deployment	43	0.07	0.05		0.07	0.02
ZIP code	4	0	0.75		0	0.75
Healthcare Unit Name	1453	0.09	0.09		0.06	0.05
Other Org Name	86	0.09	0.07		0.06	0.06
Date	2547	0.72	0.73		0.39	0.38
Age > 89	4	0	0		0	0
Phone Number	90	0.34	0.61		0.24	0.51
Electronic Address	4	0	0		0	0
SSN	16	0.56	0.87		0.56	0.87
Other ID Number	123	0.32	0.69		0.20	0.63

ISSN: 1471-2288