Moderate agreement between self-reported stroke and hospital-recorded stroke in two cohorts of Australian women: a validation study
© Jackson et al.; licensee BioMed Central. 2015
Received: 8 April 2014
Accepted: 15 January 2015
Published: 23 January 2015
Conflicting findings on the validity of self-reported stroke from existing studies creates uncertainty about the appropriateness of using self-reported stroke in epidemiological research. We aimed to compare self-reported stroke against hospital-recorded stroke, and investigate reasons for disagreement.
We included participants from the Australian Longitudinal Study on Women’s Health born in 1921–26 (n = 1556) and 1946–51 (n = 2119), who were living in New South Wales and who returned all survey questionnaires over a defined period of time. We determined agreement between self-reported and hospitalised stroke by calculating sensitivity, specificity and kappa statistics. We investigated whether characteristics including age, education, area of residence, country of birth, language spoken at home, recent mental health at survey completion and proxy completion of questionnaire were associated with disagreement, using logistic regression analysis to obtain odds ratios (ORs) with 95% confidence intervals (CIs).
Agreement between self-report and hospital-recorded stroke was fair in older women (kappa 0.35, 95% CI 0.25 to 0.46) and moderate in mid-aged women (0.56, 95% CI 0.37 to 0.75). There was a high proportion with unverified self-reported stroke, partly due to: reporting of transient ischaemic attacks; strokes occurring outside the period of interest; and possible reporting of stroke-like conditions. In the older cohort, a large proportion with unverified stroke had hospital records of other cerebrovascular disease. In both cohorts, higher education was associated with agreement, whereas recent poor mental health was associated with disagreement.
Among women who returned survey questionnaires within the period of interest, validity of self-reported stroke was fair to moderate, but is probably underestimated. Agreement between self-report and hospital-recorded stroke was associated with individual characteristics. Where clinically verified stroke data are unavailable, self-report may be a reasonable alternative method of stroke ascertainment for some epidemiological studies.
KeywordsEpidemiology Stroke Cerebrovascular disease Validation studies Self-report Hospitalisation
Epidemiological studies often rely on self-report questionnaires to ascertain disease occurrence. This is a valuable method of ascertainment, especially in the absence of disease-specific population registers, since it is cost-efficient and feasible in large study populations. Much of our current knowledge on the incidence and aetiology of stroke generally derives from studies in clinical settings where strokes are carefully diagnosed and phenotyped. The collection of detailed clinical data allows thorough investigation of particular risk factors, stroke subtypes and outcome after stroke. However, whilst these studies are rich in clinical data, they collect far less information on other important aspects, including socioeconomic, lifestyle, psychosocial and environment/social context factors. In addition, these data are rarely collected prospectively in clinical studies, prior to stroke occurrence. Existing population-based studies that have the advantage of prospective data collection (and in the case of longitudinal studies, repeated data collection) may be used to study the contribution of these non-clinical factors to stroke aetiology and outcome. Similarly, such studies may be beneficial in studying trends in prevalence and incidence. However these studies may not always have access to clinically verified incident stroke, relying instead on self-report questionnaires. It is therefore crucial to establish the validity of self-reported stroke, especially given the complexity of diagnosing this disease.
Self-report of conditions that are well defined and/or easier to diagnose, such as cancer and diabetes, generally have a high positive predictive value (PPV)[1–3]. However, agreement is usually lower for diseases such as stroke that are more complex in their diagnosis. Stroke is a heterogeneous disease with symptoms ranging from mild to severe, and there is no definitive diagnostic test. Some or all of the symptoms may resolve prior to medical consultation, which can further complicate the diagnosis. Stroke largely affects older people and can impact on cognitive function, both of which may cause reduced recall capacity and accuracy of self-reporting. Furthermore, transient ischaemic attacks (TIA), where symptoms last less than 24 hours, are often misunderstood to be, and thus reported as, strokes. This may be partly due to TIAs being frequently referred to as ‘mini strokes’. Nevertheless, in some settings it is not possible to ascertain disease occurrence using alternative sources of information, such as health records or hospital discharge data. In addition, universal access to all health records of an individual is often impossible. For example, in Australia, linked admitted hospital patient data are not yet available nationally (only for some states), outpatient data are not included in routine hospital data, and there is no routine linkage to primary care records.
Reports of the validity of self-reported stroke vary considerably, from low/moderate[4–6], to good/very good[3, 7–9]. Some of this variation is most likely due to differences in settings, age groups, gender and the ‘gold standard’ against which self-reported stroke is verified. Conflicting recommendations arising from these studies[4, 5, 7, 9] have created uncertainty about the appropriateness of using self-reported stroke in epidemiological research. Whilst some authors suggest that self-reported stroke is a valid method of assessing stroke prevalence, others recommend that self-reported stroke should be used with considerable caution, or should only be used in combination with other ascertainment methods.
However, existing studies have generally compared self-reported stroke to other ascertainment methods without identifying the potential reasons for any observed discrepancy. A better understanding of the reasons for discrepancies will further inform the appropriateness and implications of using self-reported stroke data. In this study, we determined agreement between self-reported and hospital-recorded stroke in two age-groups of women. We identified individual-level factors influencing agreement and investigated reasons for disagreement.
We included participants from the Australian Longitudinal Study on Women’s Health (ALSWH), a national population-based study of women born in 1921–26, 1946–51 and 1973–78. Women were randomly selected from the Medicare database, which covers all citizens and permanent residents of Australia, including refugees and immigrants, with intentional oversampling of women living in rural and remote areas. Women were surveyed in 1996, followed up in 1998 (1946–51 cohort) and 1999 (1921–26 cohort), and subsequently followed up every three years. At baseline, the 1921–26 and 1946–51 cohorts included 12,432 and 13,715 women, respectively. Full details of the recruitment and response rates are reported elsewhere. The study participants are linked to the national death register. National linkage of the ALSWH cohorts to other routinely collected data, including hospital-admitted patient data, is underway, with New South Wales (NSW) being the first state in which ALSWH data are linked.
We included 3675 women in our analyses. This included 1556 women from the 1921–26 cohort who were alive between survey 3 (2002) and survey 5 (2008) and 2119 women from the 1946–51 cohort who were alive between survey 3 (2001) and survey 6 (2010). Women had to have resided in NSW and returned all survey questionnaires during this time. The study was approved by the Human Research Ethics Committee of the University of Newcastle, the Medical Research Ethics Committee of the University of Queensland and the Departmental Ethics Committee of the Australian Government Department of Health and Ageing.
The surveys collect data on demographic characteristics, health conditions and behaviours. A free-text section allows women to provide additional comments.
Self-reported stroke was defined as having occurred during the period of interest if the participant responded ‘yes’ to the question: “In the past three years have you been diagnosed with or treated for stroke?”
History of hypertension, diabetes and heart disease were self-reported and poor mental health in the four weeks prior to survey completion was determined using the SF36 mental health subscale, with a score of ≤ 52 indicating poor mental health.
Hospital data was available for 2000–2010 and included admission and discharge dates, principal and secondary diagnoses and procedure codes. Strokes were identified by the following International Statistical Classification of Diseases and Related Health Problems 10th revision (ICD-10) codes in principal or secondary diagnosis fields: I60-I60.9, I61.0-I61.9, I63.0-I63.9 and I64. Although women were asked about the occurrence of stroke within the last three years, we anticipated recall error in when an event occurred (particularly in the older cohort). Restriction of the hospital admission period to these three-year intervals would have led to over or under-estimated agreement due to recall error in when the stroke occurred and not whether it occurred at all. Stroke was therefore defined as having occurred if the admission date was between the dates of return of survey 3 and survey 6 for the 1946–51 cohort or between return of survey 3 and 5 for the 1921–26 cohort. ICD-10 procedure codes were examined to identify occurrence of brain imaging.
Analyses were performed using Stata version 11.0.
We compared characteristics of women who were eligible for inclusion versus those who were ineligible using Pearson’s chi-squared test or Student’s t-test for continuous variables.
Agreement between self-reported and hospital-recorded stroke
We calculated sensitivity, specificity, PPVs and negative predictive values (NPV). We calculated Cohen’s kappa with 95% confidence intervals (CI), using the following interpretation categories: poor kappa agreement, <0.2; fair, 0.21 – 0.40; moderate, 0.41-0.60; good, 0.61-0.80; and very good, 0.81-1.00. The magnitude of kappa is affected by imbalance between the positive and negative classifications, which occurs when the disease of interest has a low prevalence. To aid interpretation of kappa, we calculated the prevalence index, (a-d)/N, where a and d are concordant ‘ratings’ and N = total population. A greater prevalence effect leads to a higher prevalence index, greater chance agreement, and consequently, a reduced kappa.
Association between characteristics and disagreement
Agreement between self-reported and hospitalised stroke
Positive predictive value
Negative predictive value
Kappa (95% CI)
0.35 (0.25 to 0.46)
0.56 (0.37 to 0.75)
Investigation of disagreement
Among women with unverified self-reported stroke, we examined discharge codes of any hospital admissions to identify: transient ischaemic attacks (TIA); diagnoses of possible strokes (or possible ‘stroke mimics’); sequelae of cerebrovascular disease; and other cerebrovascular disease diagnoses (e.g. occlusion or stenosis of (pre)cerebral arteries). We examined: procedure diagnosis fields, especially for those women in whom a possible ‘stroke mimic’ may have occurred; residence postcodes to identify women living closer to a hospital outside NSW; and questionnaire comments for additional information concerning stroke occurrence.
Finally, we calculated the agreement statistics based on different definitions of self-report and hospital-recorded cerebrovascular disease.
Characteristics of study population
The mean age of women included from the older and mid-age cohorts was 78.2 (±1.5) and 52.5 (±1.5) respectively. Other characteristics are presented in Additional file1: Table S1 and Additional file2: Table S2.
Agreement in the 1921–26 cohort
Agreement in the 1946–51 cohort
Characteristics associated with disagreement
Characteristics associated with disagreement in self-report compared with hospital-recorded stroke
OR (95% CI)
OR (95% CI)
Agreement (N = 1472) n (%)
Disagreement (N = 84) n (%)
Agreement (N = 2102) n (%)
Disagreement (N = 17) n (%)
Age (mean ± SE)
1.11 (0.96 to 1.30)
1.13 (0.97 to 1.32)
0.87 (0.62 to 1.22)
0.87 (0.62 to 1.23)
High school education
No formal education/primary only*
1.03 (0.62 to 1.70)
0.97 (0.58 to 1.63)
4.71 (1.50 to 14.7)
4.21 (1.33 to 13.32) §
0.23 (0.08 to 0.65)
0.24 (0.09 to 0.67) ‡
1.00 (0.30 to 3.28)
0.80 (0.26 to 2.86)
Area of residence
Rural or remote
1.19 (0.76 to 1.86)
1.08 (0.68 to 1.72)
1.13 (0.41 to 3.06)
Country of birth
0.76 (0.34 to 1.69)
0.84 (0.11 to 6.36)
Language spoken at home
1.42 (0.50 to 4.04)
Recent poor mental†
2.80 (1.46 to 5.36)
2.41 (1.20 to 4.81) ‡
3.85 (1.33 to 11.18)
3.20 (1.08 to 9.46) §
Proxy completion of survey
3.82 (1.27 to 11.5)
2.36 (0.65 to 8.58)
Scenarios of agreement
Scenarios of agreement in the validation of self-reported stroke
Positive predictive value
Kappa (95% CI)
0.35 (0.25 to 0.46)
0.54 (0.44 to 0.63)
0.57 (0.48 to 0.67)
0.65 (0.57 to 0.74)
0.56 (0.37 to 0.75)
0.60 (0.42 to 0.78)
In our study, agreement between self-reported stroke and hospital-recorded stroke initially appears fair to moderate. Few women failed to report a stroke that was recorded in hospital records, but a substantial number reported a stroke which was unverified by hospital records. Almost a fifth of women in the older cohort who reported a stroke provided additional comments on the stroke occurrence (sometimes with detailed reference to symptoms and doctor consultations), suggesting that some women may have been diagnosed in a non-hospital setting. Furthermore, women who self-reported unverified strokes often misreported TIAs as strokes, had evidence of chronic stroke, or had evidence of other cerebrovascular disease from discharge diagnoses, suggesting that self-reported stroke may be a reasonable indicator of cerebrovascular disease in general. Our investigation of the individual factors associated with level of agreement suggests that validity may vary with education level and mental health status at time of self-report of stroke.
Comparisons with previous studies
Our findings are consistent with results of previous studies which generally found low or moderate levels of agreement, with low PPVs (22% to 55%)[4–6, 15]. In contrast, some studies found higher levels of agreement, with higher PPVs[2, 3, 7–9]. However, the ‘gold-standard’ against which self-reported stroke has been compared varies markedly between studies, ranging from hospital discharge or medical record review[2, 4–6, 8] to general practitioner or health centre record review[2, 16, 17] or a range of information including interview and/or clinical assessment of participants[3, 7, 9]; this explains some of the variation and highlights the potential limitations of relying on hospital data only to identify strokes. Interestingly, studies in which multiple sources of information (and not only hospital records) were used to verify self-report generally reported better validity of self-reported stroke than studies that include medical records only for example[3, 7, 9].
Few studies have investigated the influence of participant characteristics on the validity of self-reported stroke. In our study, higher educational level was associated with increased agreement amongst the older cohort, which is consistent with the findings of Okura et al., but not with a Norwegian study, which found no such association. We also found that recent poor mental health at time of survey completion was significantly associated with increased odds of disagreement. Engstad et al. similarly found that being happy or optimistic during the last two weeks was associated with higher agreement, but this was not statistically significant after adjusting for confounding.
Although we did not formally test it, disagreement was higher among the older cohort than the younger cohort in our study. Ageing was also associated with decreased agreement in two other studies[8, 17].
The prevalence of self-reported stroke in our cohort is in keeping with estimates of stroke prevalence in women of these ages from population-based studies in similar high-income countries during the 1990s[16, 18]. The prevalence of stroke as validated by hospital record is considerably lower, suggesting we may underestimate stroke prevalence if we rely solely on this ascertainment method. A recent Australian study found that verification of self-reported cardiovascular disease events (including stroke) by linkage to a state-wide hospital morbidity database gave similar estimates of validity to verification by adjudication of medical records. Although quality of hospital discharge data might vary across Australia, it is reassuring that these routinely available data are accurate in terms of reflecting medical record data and suggests that our findings may not have been significantly affected by errors in discharge data.
Our study has a number of strengths. First, we were able to assess and compare the validity of self-reported stroke in two age groups of women. Our examination of the validity of self-reported stroke in a middle-aged cohort is novel, since previous studies have generally only included an older study population. Second, our study population was large, resulting in a reasonably large number of stroke outcomes among the older cohort. Third, our analyses sought to identify reasons for disagreement between self-reported and hospital-recorded stroke, which extends the existing literature, with many of the existing studies on this topic only assessing agreement without further investigation. Fourth, the ALSWH surveys collect a wealth of demographic and health and well-being data, allowing us to investigate the characteristics associated with validity of stroke reporting, which few studies have explored.
Our study has some limitations. We included women only, and so the results may not be generalizable to men. However, agreement between self-reported and hospital-recorded stroke in our study is in keeping with that observed in a very similar population-based study of older Australian men. Studies reporting agreement by gender found some differences, with the PPV slightly lower in women[5, 6, 16], largely due to lower stroke prevalence. One study reported no association between gender and disagreement, whilst another found poorer agreement in women.
The limitations of the hospital data, in terms of the period for which data was available imposed some restrictions on our inclusion criteria, leading to the exclusion of women from our analyses. There were some differences between the included and excluded women, especially in the older cohort. Importantly, excluded women had a lower education level than included women. Our finding that a higher education level is associated with greater agreement suggests that excluded women may have been less reliable self-reporters, and thus we may have overestimated general population agreement.
Conversely, we probably underestimated the true validity of self-reported stroke. Firstly, we were unable to verify non-hospitalised strokes. In an Australian community-based study of stroke incidence, 86% of strokes were hospitalised. Given that a certain proportion of the remaining 14% had a non-hospitalised fatal stroke, and assuming a similar hospitalisation rate in NSW, these data suggest that a relatively small proportion (<10%) of women may have had a non-hospitalised stroke. We also could not verify stroke occurrence leading to hospitalisation in another Australian state. This is a particular problem for women who lived close to state borders. However we found that, based on residence postcode, this is likely to affect few participants. Finally, despite being asked about stroke events occurring in the past three years, some women may have reported strokes occurring before 2000, which we could not validate. However, we were able to account for some of these sources of error, by determining ‘best-case’ and ‘worst-case’ estimates of the validity of self-reported stroke.
Ideally, multiple methods of ascertainment should be used to identify stroke occurrence, but in questionnaire-based research as well as clinical practice, this is often not feasible, resulting in reliance on self-report or routinely available data such as hospital discharge information. Although the latter approach may not identify all strokes, the proportion of false positives will be low, and the direction of any bias more easily determined. This makes it a more attractive, and where the data are available, preferable method of stroke ascertainment compared to self-report. However, the limitations of this method (for example, selection bias) need to be acknowledged and the implications for interpretation considered. Where hospital data are unavailable, self-reported stroke may be a reasonable alternative method of ascertainment for some epidemiological studies, particularly where there is the opportunity to investigate the role of under-researched non-clinical factors that may play an important role in aetiology and outcome. Reassuringly, a recent study demonstrated similarities in stroke incidence and cardiovascular risk factor associations between a study using self-reported stroke and studies that included clinically-verified strokes. Also, recent findings from a population-based study of older men found that rates of recurrent stroke were similar regardless of whether stroke was ascertained by self-report or hospital diagnosis. However, researchers should be aware of the caveats to using self-reported stroke data, especially among studies of older people. Self-reported stroke will inevitably include the reporting of other stroke-like or stroke-related conditions (as well as some non-strokes) and findings should therefore be appropriately interpreted. If research questions relate to cerebrovascular disease in general, then using self-report to ascertain stroke may be less biased.
The Australian Longitudinal Study on Women’s Health is funded by the Australian Commonwealth Department of Health. CAJ and GDM were supported by the Australian National Health and Medical Research Council (Grant number: APP1000986).
The authors thank the NSW Ministry of Health, the NSW Central Cancer Registry and staff at the Centre for Health Record Linkage (CHeReL).
- Colditz G, Martin P, Stampfer M, Willett WC, Sampson L, Rosner B, et al.: Validation of questionnaire information on risk factors and disease outcomes in a prospective cohort study of women.Am J Epidemiol 1986, 123:894–900.View ArticlePubMedGoogle Scholar
- Haapanen N, Miilunpalo S, Pasanen M, Oja P, Vuori I: Agreement between questionnaire data and medical records of chronic diseaes in middle-aged and elderly Finnish men and women.Am J Epidemiol 1997, 145:762–9. 10.1093/aje/145.8.762View ArticlePubMedGoogle Scholar
- Simpson C, Boyd C, Carlson M, Griswold ME, Guralnik JM, Fried LP: Agreement between self-report of disease diagnoses and medical record validation in disabled older women: factors that modify agreement.JAGS 2004, 52:123–7. 10.1111/j.1532-5415.2004.52021.xView ArticleGoogle Scholar
- Carter K, Barber A, Shaw C: How does self-reported history of stroke compare to hospitalisation data in a population-based survey in New Zealand?Stroke 2010, 41:2678–1680. 10.1161/STROKEAHA.110.598268View ArticlePubMedGoogle Scholar
- Machon M, Arriola L, Larranaga N, Amiano P, Moreno-Iribas C, Agudo A, et al.: Validity of self-reported prevalent cases of stroke and acute myocardial infarction in the Spanish cohort of the EPIC study.JECH 2013, 67:71–5.Google Scholar
- Yamagishi K, Ikeda A, Iso H, Inoue M, Tsugane S, for the JPHC Study Group, et al.: Self-reported stroke and myocardial infarction had adequate sensitivity in a population-based prospective study JPHC (Japan Public Health Center)-based prospective study.J Clin Epidemiol 2009, 62:667–73. 10.1016/j.jclinepi.2008.07.016View ArticlePubMedGoogle Scholar
- Engstad T, Bonaa M, Viitanen M: Validity of self-reported stroke.Stroke 2000, 31:1602–7. 10.1161/01.STR.31.7.1602View ArticlePubMedGoogle Scholar
- Okura Y, Urban L, Mahoney D, Jacobsen S, Rodeheffer R: Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure.J Clin Epidemiol 2004, 57:1096–103. 10.1016/j.jclinepi.2004.04.005View ArticlePubMedGoogle Scholar
- O'Mahony P, Dobson R, Rodgers H, James O, Thomson R: Validation of a population screening questionnaire to assess prevalence of stroke.Stroke 1995, 26:1334–7. 10.1161/01.STR.26.8.1334View ArticlePubMedGoogle Scholar
- Lee C, Dobson A, Brown W, Bryson L, Byles J, Warner-Smith P, et al.: Cohort profile: the Australian longitudinal study on Women's health.Int J Epidemiol 2005, 34:987–91. 10.1093/ije/dyi098View ArticlePubMedGoogle Scholar
- Silveira E, Taft C, Sundh V, Waern M, Palsson S, Steen B: Performance of the SF-36 health survey in screening for depressive and anxiety disorders in an elderly female Swedish population.Qual Life Res 2005, 14:1263–74. 10.1007/s11136-004-7753-5View ArticlePubMedGoogle Scholar
- Altman D: Practical Statistics for Medical Research. London: Chapman & Hall; 1991.Google Scholar
- Sim J, Wright CC: The kappa statistic in reliability studies: use, interpretation, and sample size requirements.Phys Ther 2005, 85:257–68.PubMedGoogle Scholar
- Libman R, Wirowski E, Alvir J, Rao H: Conditions that mimic stroke in the emergency department.Arch Neurol 1995, 52:1119–22. 10.1001/archneur.1995.00540350113023View ArticlePubMedGoogle Scholar
- Jamrozik E, Hyde Z, Alfonso H, Flicker L, Almeida O, Yeap B, et al.: Validity of self-reported versus hospital-coded diagnosis of stroke: a cross-sectional and longitudinal study.Cerebrovasc Dis 2014, 37:256–62. 10.1159/000358583View ArticlePubMedGoogle Scholar
- Bots M, Looman S, Koudstaal P, Hofman A, Hoes A, Grobbee D: Prevalence of stroke in the general population.Stroke 1996, 27:1499–501. 10.1161/01.STR.27.9.1499View ArticlePubMedGoogle Scholar
- Kriegsman D, Penninx B, van Eijk J, Boeke A, Deeg D: Self-report and general practitioner information on the presence of chronic diseases in community dwelling elderly. A study on the accuracy of patients' self-reports and on determinants of inaccuracy.J Clin Epidemiol 1996, 49:1407–17. 10.1016/S0895-4356(96)00274-0View ArticlePubMedGoogle Scholar
- Feigin VL, Lawes CM, Bennett DA, Anderson CS: Stroke epidemiology: a review of population-based studies of incidence, prevalence, and case-fatality in the late 20th century.Lancet Neurol 2003, 2:43–53. 10.1016/S1474-4422(03)00266-7View ArticlePubMedGoogle Scholar
- Barr E, Tonkin A, Welborn T, Shaw J: Validity of self-reported cardiovascular disease events in comparison to medical record adjudication and a statewide hospital morbidity database: the AusDiab study.Int Med J 2009, 39:49–53. 10.1111/j.1445-5994.2008.01864.xView ArticleGoogle Scholar
- Thrift AG, Dewey HM, Macdonell RA, McNeil JJ, Donnan GA: Stroke incidence on the east coast of Australia: the north east Melbourne stroke incidence study (NEMESIS).Stroke 2000, 31:2087–92. 10.1161/01.STR.31.9.2087View ArticlePubMedGoogle Scholar
- Glymour M, Avendano M: Can self-reported strokes be used to study stroke incidence and risk factors?: evidence from the health and retirement study.Stroke 2009, 40:873–9. 10.1161/STROKEAHA.108.529479View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/15/7/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.