Validation of verbal autopsy methods using hospital medical records: a case study in Vietnam

Study setting and protocol

We designed this validation study based upon the information extracted from a recent VA study in 12 communes in Quang Ninh province. A detailed description of this VA study are available elsewhere [24]. Complete VA interviews for 670 deaths that occurred between 1/1/2014 to 31/12/2014 were successfully conducted, using a Vietnamese adapted version of international standard VA questionnaires. The adapted questionnaires have been used in a previous mortality surveillance project and some other studies in Vietnam [8, 25–27]. Interviewers were commune health workers and village health workers who received training in administering VA interviews. Multiple CODs and UCOD for each death were derived from completed VA questionnaires according to international standards and rules.

The VA questionnaire recorded the place of death, including the name of the hospital where death occurred. In case of deaths outside hospital, the questionnaire asked and recorded the last hospital attended during the terminal illness, and the interval (number of days and/or months) between hospital discharge and death.

Identification of MRs and MR abstraction

At first, the name, date of birth, sex and address were used to determine if the case found in the hospital database was the same as the case identified from the VA questionnaire. Subsequently, the complete MRs were retrieved from the archives, including documents from previous admissions in the same hospital. The MRs pertaining to the final admission were used as main source for MRs abstraction. The other records were reviewed to get additional information.

In each hospital, trained hospital nurses abstracted relevant details from the MRs onto a specific study data abstraction form (see Additional file 1). Accordingly, information about medical history of the patients, signs, symptoms, presenting illness and clinical events during hospitalization which culminated in either death or discharge of the patient, any relevant investigation results and diagnoses as well as laboratory tests performed were extracted onto the abstraction form.

UCOD identification and coding

In the next step, a senior trained physician with over 20 years of clinical experience reviewed the MRs abstracts and completed a Death Certificate (see Additional file 2) for each death, following the International Medical Certificate of cause of death format [28]. This form has two parts which allows documentation of the direct, antecedent and underlying causes of death to be recorded in part I and other conditions that may have contributed to the death to be recorded in part II. The completed death certificates then were coded by a senior trained coder using ICD-10 [29]. It is noted that all causes recorded in the death certificates were coded. Then the UCOD was assigned following the current international mortality coding rules [28]. According to the WHO standard definition, the UCOD is “the disease or injury which initiated the train of morbid events leading directly to death, or the circumstances of the accident or violence which produced the fatal injury” [28].

The accuracy of UCOD assignment was confirmed and reviewed by an independent expert coder using the most updated version of the Iris software (version 5) [30], using the “MUSE” function. The cause used for comparison for validating the VA was the UCOD. All people who participated in the data collection in hospitals, including the physician who reviewed the MRs abstraction and the ICD-10 coder, were blinded to the VA data to avoid information bias.

Data management and analysis

For efficiency in analyses, ICD codes for all COD were aggregated according to the WHO General Mortality Tabulation Condensed List I, comprising 103 categories of ICD-10 codes [31]. For assessing the validity of VA diagnoses, the UCOD identified from the MRs review were compared with the UCOD identified from the VA for each case. As a first step in the analysis, the percentage of agreement between underlying causes from MRs and VA were computed. Subsequently, we used the MRs diagnoses as the reference value to compute the sensitivity, specificity and positive predictive value (PPV) of the UCOD diagnosed from VA [19, 20, 22, 32–38]. Measures of validity were computed for conditions known to be among the leading causes of death in Vietnam [8]. We categorised validity as good if sensitivity was above 75%, average if sensitivity lay between 50 to 75%, and low if sensitivity is less than 50%, similar to ratings used in other studies [39, 40]. Standard errors were computed for these proportions, and used to develop 95% confidence intervals for each of these values.

Validity of VA methods was also analysed by age of the deceased broadly categorized into 25-69 years and 70 and above. It is generally perceived that VA interviews are less informative for deaths in the elderly (i.e above 70 years), hence this age group analysis was undertaken to observe any variations in levels of validity for specific causes by age. In a subset of cases where the UCOD from VA did not match that from the MRs, we provided a descriptive summary of all reported COD from the MR.

Description of the study population

The description of the selection of study population is shown in Fig. 1.

In our final study sample (116 cases), 70% of the deceased were male and 30% were female. Almost all were adult deaths with ages which ranged from 28 to 96 years old. About 45% of deaths were in the age group 70 years and older, 38% in the 50-69 year age group and 16% in the age group of 25-49 years. There was a single one-year-old child death which accounted for 0.9% of the sample. Only 5% of the deceased died in a hospital. 95% of them were discharged in a severe condition and then died within 30 days at their home.

Figure 2 describes the duration of the interval from the date of the patient’s discharge to the date of death. About 50% died on the same day as they were discharged from the hospital; 61% died within 24 hours (one day); over 70% of patients died within three days; 76% of them died within 10 days after the family took them home.

Validity of the UCOD from VA

VA performed very well in diagnosing almost all major site-specific cancers (such as lung cancer, liver cancer, colon and rectum cancer) and road traffic accidents with scores of 75 to 100% for sensitivity and PPV. In the cardiovascular diseases group, the validity of VA varied among different causes. For stroke, sensitivity indicated that VA could capture nearly 80% of patients whose cause of death was actually stroke. However, the PPV indicates that VA tended to over-diagnose deaths due to stroke, with only 61% of VA diagnoses of stroke being confirmed by the MRs review.

Several other causes were also found to have low sensitivity (below 50%). VA correctly diagnosed only 33% deaths due to ischemic heart disease (IHD), 40% deaths due to cirrhosis of liver, 40% deaths due to diabetes, 33% deaths due to pneumonia and only 14% deaths due to COPD. We conducted a similar analysis of validity of VA diagnoses for age groups 25-69 and 70+ years. However, there was no difference between the two age groups regarding sensitivity, specificity and PPV scores for leading CODs (see Additional file 3).

As mentioned in Table 1, we found 19% of cases where the UCOD from VA was one of causes listed on the MRs death certificate. We further analysed these 22 cases to better understand the misclassification patterns of underlying causes from VA, Table 4 presents information on multiple COD derived from available information in hospital MRs as well as from VA, according to the format of the international death certificate.

MR_UCOD is the underlying COD derived from information in MR. MR_Ia is the MR’s COD recorded in Part I(a) of death certificate. MR_Ib is the MR’s COD recorded in Part I(b). Similar to MR_Ic and MR_Id. MR_II.1, MR_II.2, MR_II.3, MR_II.4 are the MR’s COD recorded in the part II of death certificate. VA_UCOD is the underlying COD by VA. VA_Ia is the VA’s COD recorded in Part I (a). Similar explanation for VA_II.1 and VA_II.2. The “green data” indicated the UCOD which was assigned base on MRs while the “blue data” indicated the UCOD which was assigned base on VA methods. Some “blue data” was highlighted in other selected cells inside the table, which indicates that the cause/condition assigned as UCOD from VA is one of causes recorded in MRs (but not UCOD)

In about 32% (7/22) of cases, the UCOD from VA was either the direct cause or one of the antecedent causes identified from hospital MRs by the physician (cases 1, 2, 4, 12, 15, 18 and 22). The VA respondents for these cases might have only remembered the cause/conditions/signs which directly led to the death. For example, for case number 1 (Table 4), pneumonia was the direct cause which was a consequence of the UCOD, measles. However, the VA respondent might not have reported the skin rash which is the only differentiating symptom indicative of measles as the underlying condition that caused pneumonia. Hence, based on the reported signs and symptoms, a diagnosis of pneumonia was derived, and this is a reasonable expectation of cause of death assignment from VA. Eventually, the identification of pneumonia by the VA in this case underscores the public health utility of this method for ascertaining causes of death. Table 4 also shows that in 68% of cases (15/22) the UCOD from VA was a contributory cause recorded in Part II of the death certificate from the MRs. Most of these causes were non-communicable diseases such as hypertensive diseases, diabetes mellitus, or stroke. These results also indicate the usefulness of VA in correctly identifying these major co-morbidities, and hence justifying the value of VA in settings experiencing a growing epidemic of non-communicable diseases, as in Vietnam. Therefore, establishing the validity of VA in determining such comorbidity through ascertaining multiple causes of death also adds value to the overall public health utility of VA.

The reference diagnosis

Previous VA validation studies which have been conducted in other countries used the UCOD determined by physician review of hospital MRs of inpatient deaths as the reference diagnosis for validation [21, 22, 41, 42]. In Vietnam, VA methods are routinely used in three Demographic and Health Surveillance Systems and in a number of projects/studies to measure population level cause-specific mortality. However, non-availability of a “gold standard” reference diagnosis for the underlying cause has been described as a constraint for the implementation of a validation study [19]. This study demonstrated a potential solution for overcoming this constraint in the Vietnam context. In designing this study, we first noted that from the VA database of 670 deaths in Quang Ninh province, we found only 30 cases (about 5%) who died in a hospital/health facility. However, about 70% of the deceased had been hospitalised during their terminal illness [24]. Hence, we chose to use the MRs of patients who had been discharged from hospital within 30 days prior their death, as the source for reference diagnoses to validate the COD from VA.

Validity of VA and potential for its improvement

In this study we found that for the UCOD comparison across all cases, there was 57% agreement between VA and physician review of hospital MRs. This percentage of agreement on cause-specific codes is much higher than found in a study conducted in Malawi, which reported only 26.2% agreement [41]. When taking into account the comparison across multiple causes, the underlying cause from VA matched with one of the multiple causes in the MRs in 76% of cases. These overall findings confirmed the value of VA methods in identifying the COD for deaths which occur outside hospitals in Vietnam.

Despite these high levels of agreement as well as good sensitivity scores for some causes of death, the relatively lower sensitivity of VA UCOD diagnoses for several important conditions including ischaemic heart disease, COPD, diabetes and pneumonia is a cause for concern. Similar findings of low sensitivity scores (< 50%) for these specific causes of death have also been observed in other verbal autopsy validation studies in China [19, 40], Malaysia [43] and Thailand [42]. To some extent, the analysis of multiple causes of death as presented in Table 4 has improved our understanding of misclassification patterns that result in low sensitivity scores for these conditions as underlying causes. In view of the potential for several of these conditions to be co-existent in many instances, the tools and processes employed for VA data collection, diagnosis and coding should routinely facilitate the reporting and analysis of multiple causes. While this would enhance the utility of VA data for both descriptive and analytical epidemiology, multiple cause analysis will also strengthen the assessment of validity of VA diagnoses in future studies.

Obtaining medical records or at least discharge diagnoses for terminally ill patients could be an important source of information to strengthen cause of death ascertainment for such events. Such information is particularly relevant for conditions that need specific clinical diagnostic tests or imaging investigations. If every discharged patient is provided with a discharge summary relating to their hospitalisation, this would provide valuable information for ascertaining the COD, in the case of a VA interview. The interviewers could ask the family member to show those documents and therefore important information may be obtained.

Limitations

A key limitation of this study is the small sample size which resulted in the large ranges of 95%CI of sensitivity and PPV values for specific causes. The small sample size critically hampers more detailed analysis of validity for a wider range of causes, as well as for any variations in cause-specific validity across gender and rural/urban areas. To start with, the initial sampling frame for this study was limited to the sample of deaths associated with selected hospitals from one province, owing to restrictions in terms of costs, manpower, and other logistical factors. Further, as mentioned previously, there were losses to follow up during the study implementation. For instance, only 71.4% of the 213 eligible cases reported in the VA study to have accessed treatment from the three hospitals were actually traced in the hospital records. The remaining 28.6% cases could not be matched according to the full name, sex, date of birth and address of the deceased. There are probably two reasons for this phenomenon. The first may be the issue of misinformation or miscommunication of relevant identification details by the patient (or the patient’s relatives) with the hospital’s administration/reception at admission. In many cases in Vietnam, the elderly live with their children in another province, the hospital may record their current address, while the VA records their original home address [8], resulting in a mismatch. The second reason may due to incorrect information in the VA interview in response to the question on last treatment accessed before death. In some instances, individuals receive treatment as an outpatient in several different clinics or hospitals for the same illness, and at the VA interview, the deceased’s relative might have provided an incorrect health facility name as the site for last hospitalization.

There was an additional 24% (36 cases) of cases that were excluded since the actual date of hospitalization reported at the VA interview was incorrect. These individuals had actually been discharged more than 30 days prior to death, and hence had to be dropped from the study. Eventually, the final number of 116 study cases accounted for only 55% of the 213 cases in the initial extracted list. These losses to follow up further limited the study sample size but were considerably less than the 75% losses observed in a validation study which was conducted in Ethiopia with a similar approach to trace medical records of community reported deaths in hospital records [21]. Such potential losses to study samples and the reasons for the same should be anticipated in establishing the study design and sample size for future VA validation studies in Vietnam and other similar settings.

A potential limitation of the approach to use hospital discharge records as a data source for ascertaining causes of death is that there may be cause(s) other than the discharge diagnosis that could have resulted in the death. The potential for such instances would increase with the length of the interval between discharge and death. Further research evidence would be required to identify the most suitable cut-off time for the use of reference diagnoses from the medical records for discharged patients in VA validation studies.