Selective attrition and bias in a longitudinal health survey among survivors of a disaster

Epidemiologic studies after disasters have shown elevated levels of health problems among survivors such as post-traumatic stress disorder (PTSD), depression and physical symptoms [1–3]. The majority of the disaster studies have been cross-sectional, and although cross-sectional studies are useful for assessing the public health burden of the disaster, they do not give insight into the course of the health consequences and the health needs of survivors at different times post-disaster.

Since relatively little is known about the course of health problems among survivors of disasters, more longitudinal studies are needed [1, 2, 4]. However, attrition is a main methodological problem in longitudinal studies. A common approach for handling attrition is to delete observations with missing values, but this complete case analysis can result in a substantial loss of power. In addition, if respondents systematically differ from non-respondents, deleting incomplete observations might introduce bias in the prevalence estimates of the health problems [5, 6]. A good way to deal with missing data and to overcome possible selection bias in the prevalence estimates is to conduct multiple imputation [6, 7]. This technique fills in various values for each missing data point based on a statistical model. Because the missing values are drawn from a distribution, there will be a range of values imputed for each missing value, with variation appropriately reflecting the uncertainty about that value. Using this technique, it can be estimated what the prevalence of the outcomes of interest would have been if there had been no (systematic) attrition in the longitudinal study.

Evidence concerning selective response among survivors of disasters is conflicting. Some studies have shown that non-respondents are more likely to be male, single and to have a low socioeconomic status [8–10], while other studies did not observe such an association [11–15]. Little is known about the association between the level of disaster exposure and non-response at follow-up. One may speculate that survivors who were highly affected by the disaster or who had high levels of post-disaster distress would be more motivated to participate at the follow-up of a health survey than survivors who were less affected. On the other hand, it can be hypothesized that highly exposed or distressed survivors would be less likely to respond because they do not want to be reminded of the stressful event. Several studies have found that depression, distress and symptoms of PTSD at baseline were associated with non-response at follow-up [8, 16, 17] while other studies found no association between baseline distress and non-response at follow-up [12–15]. In addition, the determinants of response at follow-up might also differ between groups of survivors. We recently observed that baseline health problems were associated with response among immigrant survivors and with non-response among native Dutch survivors at wave 2 of a study after a fireworks disaster in the Netherlands [18]. The information from these disaster studies is, however, not sufficient to understand the response mechanisms of survivors of disasters. Furthermore, none of the previous studies after disasters have examined whether selective response biased the prevalence estimates of the health problems among survivors.

Since attrition will most likely occur in future longitudinal studies after disasters, more insight into the response mechanisms among survivors and possible bias resulting from selective response is desirable. In the present longitudinal study after a fireworks disaster in the Netherlands, we examine the selective response among survivors at the follow-up surveys. In addition, we study whether possible selective response had biased the prevalence estimates of health problems among survivors at wave 3 by comparing the estimates resulting from multiple imputation with estimates resulting from complete case analysis.

In this study among survivors of a fireworks disaster, selective response occurred at the two follow-up surveys. We examined whether selective response had biased the prevalence estimates of the wave 3 health problems by comparing prevalence estimates resulting from complete case analysis and estimates resulting from multiple imputation. The complete case prevalence estimates of the wave 3 health problems were only somewhat biased, and the direction differed between the native Dutch survivors and immigrant survivors.

Similar to other studies after disasters, non-respondents at follow-up were more likely to be male [8–10]. In agreement with two longitudinal studies after disasters, we did not find an association between damaged or destroyed house or property loss and non-response at follow-up [10, 14]. In addition, we did not find clear associations between response at follow-up and other disaster-related experiences, such as the loss of a loved one and intense anxiety.

In this study, we found an association between health problems at wave 1 and response at waves 2 and 3. Some previous studies also showed an association between health problems at baseline and non-response at follow-up [8, 16, 17]. However, the response mechanisms in this study differed between native Dutch survivors and survivors of non-western origin. Among native Dutch survivors, health problems at wave 1 tended to be associated with non-response at follow-up (waves 2 and 3). In contrast, among immigrant survivors, health problems at wave 1 were associated with response at follow-up. These different response mechanisms between native Dutch and immigrant survivors were also found in a recent study among the survivors of the fireworks disaster in which determinants for response at wave 2 of the health survey were examined [18]. It can be speculated that immigrant survivors of the fireworks disaster were not accustomed to participation in a health survey and believed that completion of the questionnaire was not meaningful in the absence of health problems. Although the underlying reasons remain unclear, different response mechanisms among ethnic groups have also been found in a longitudinal general population study by Psaty et al [37]. In their study among whites and non-whites in the USA, poor health status was associated with non-response among whites and with response at follow-up among non-whites.

In this study we allowed survivors who participated at wave 1 but not at wave 2 to re-enter the study at wave 3. At wave 3, all eligible survivors were strongly motivated to participate; all survivors were stimulated to participate by means of telephone calls. In addition, eligible survivors for whom the telephone number was unknown and all immigrant survivors were visited at home. Survivors who re-entered the study at wave 3 differed from survivors who did not (wave 1 only). For example, among both native Dutch and immigrant survivors, those who re-entered the study at wave 3 were somewhat less likely to be male, were somewhat more likely to have a paid job and were somewhat less likely to be single. This group of respondents indirectly provides insight into non-response at wave 2 as well as additional information that is useful when performing MI.

The different response mechanisms among native Dutch and immigrant survivors were also confirmed after multiple imputation of the missing values. Among native Dutch, the imputed estimates of the wave 3 health problems tended to be higher than the complete case estimates. In contrast, the imputed prevalence estimates among immigrant survivors tended to be somewhat lower than the estimates of health problems at wave 3 resulting from complete case analysis. Additional analyses showed that the differences between imputed and complete case estimates of wave 2 health problems were similar to the differences between the complete case and imputed prevalence estimates of wave 3 health problems (data not shown). This confirms the robustness of our findings, since both native Dutch and immigrant survivors have similar response mechanisms from wave 1 to wave 2 and from wave 2 to 3.

We could not demonstrate very large differences between the imputed and complete case estimates of the health problems at wave 3. This result was unexpected since the selective response at the follow-up surveys would suggest prevalence estimates that were more strongly biased. While these results are reassuring, we can not exclude that some of the prevalence estimates were more biased than our results indicate. First, the lack of biased prevalence estimates at wave 3 might be due to weak associations between the predictor and outcome variables in the imputation model. Second, it is possible that other variables, not included in the model, were more important predictors of response and that missing data were not missing at random. However, we included all variables in the imputation model that were likely to be related to response in a health survey. Despite this, it is possible that the mechanism for missing data was non-ignorable; in other words, the missing data depended on variables not measured in this study or on the health status of non-respondents at follow-up. We believe that the method of multiple imputation was adequate given the strong correlations between the variables that were used in the multiple imputation model. In addition, a necessary condition for this method was fulfilled [5]; the existing correlations between all factors used in the imputation model were systematically in the same direction. In this study, multiple imputation gives insight into the magnitude of selection bias on the prevalence estimates. Multiple imputation has some additional advantages above other methods to handle missing data; with multiple imputation all available information is used, therefore avoiding the loss of power associated with complete case analysis [5, 6]. Furthermore, the fact that standard errors and confidence intervals resulting from multiple imputations are more appropriate than those resulting from other techniques such as single imputation is another important advantage of multiple imputation [5–7].

Besides non-response at follow-up, selective participation occurred at wave 1 of the longitudinal study in which 35.2% of all affected residents participated. Affected residents who participated at wave 1 were more likely to be women, to be between 25–44 or 45–64 years of age, to live with a partner, to be a single parent and to be of immigrant background. Analyses of multiple imputations showed that the selective participation did not affect the prevalence estimates of health problems at wave 1 [38]. In the present study we were, however, primarily interested in bias resulting from selective attrition at the follow-up surveys. Despite selective response, the prevalence estimates of health problems at wave 3 were not completely different when we corrected for the selective response by means of multiple imputation. This is important information, emphasizing the fact that selective response is only problematic when it biases the prevalence estimates of health problems.

To date, most studies that have examined response in longitudinal studies have focused on whether respondents were systematically different from non-respondents. Our results indicate that considerable attrition and selective response only somewhat biased that prevalence estimates of health problems among survivors. Therefore, future studies, both among survivors of disasters and among the general population, should not only examine selective response, but should also investigate whether selective response has biased the prevalence estimates of health problems by using statistical techniques such as multiple imputation. Although the present study focused on potential bias in the prevalence estimates of health problems, investigations into risk factors for health problems should also take into account possible bias due to selective response. This is especially important in longitudinal studies after disasters since these studies examine the health needs of survivors and provide information on which post-disaster health interventions are based.