Assessing and adjusting for non-response in the Millennium Cohort Family Study

Military families and communities serve as the cornerstone of support for US service members and may themselves be heavily impacted by military experiences including deployment [1–6]. The US Administration, the Department of Defense, and the Institute of Medicine have identified military family research as a national priority, encouraging studies that examine the unique risk and resilience factors among military families and assess the resources they need in order to promote familial adjustment and optimal health outcomes [7–9]. While the literature in this area has progressed substantially in the last decade [10–12], there is much to learn about the functioning of military families, the influence of family interactions on service member recovery and resiliency, and the relationship of military experiences, such as deployment with family well-being. Population-based survey research is uniquely positioned to address these critical research questions and inform policy decisions to support service members and their families; however, very few longitudinal studies have been conducted with representative samples of military families. In recent years, several large-scale studies have been implemented among active duty military families and spouses from all US service branches including the Deployment Life Study [13], the Millennium Cohort Family Study (Family Study) [14], the Intergenerational Impact of War Study [15], the Military Family Life Project [16, 17], and the Survey of Active-Duty Spouses [18]. However, several of these studies excluded important subgroups (e.g., dual military, reserve couples) which limits the generalizability of their findings, and only three (Deployment Life Study, Military Family Life Project, and Family Study) are positioned to address questions regarding the trajectory of experiences for military families due to their longitudinal design.

The Family Study, a dyadic longitudinal survey of married spouses and military members with two to five years of service, addresses several of these gaps. The study aims to examine the well-being of military spouses and children in a probability-based longitudinal sample of service members from all branches and components of the military. As a result, it provides a unique empirical resource for addressing critical scientific, operational, and policy questions, and for informing the development of interventions promoting resilience among service members and their families. In addition to its robust design as a prospective, longitudinal survey of a probability-based sample across military branches, the Family Study addresses important gaps in the literature by including under-studied subsamples, such as Reserve and National Guard families, dual military couples, and male spouses. The study further plans to follow families over 21 years, a much longer period of time than ever previously attempted. Only married couples of opposite sex were included in the study; thus results may not generalize to lesbian, gay, bisexual and transgendered spouses or to single-parent households. The Family Study is an adjunct to the Millennium Cohort Study, which is the first US population-based prospective study to investigate long-term health effects of military service among active duty military. The Millennium Cohort Study was launched in 2001 and has enrolled four panels of service members; the first wave of enrollment for military spouses in the Family Study occurred concurrently with recruitment for the fourth panel (Panel 4; 2011–2013). The Family Study baseline employed a nested design in which spouses of service members who completed the Millennium Cohort survey were subsequently invited to participate in the study.

Unit non-response is an inevitable feature of population-based surveys, and it is important to thoroughly examine and adjust for potential non-response bias to improve the representativeness of the sample prior to conducting analyses of the data and reporting findings [19, 20]. The literature on non-response in population-based surveys indicates the following sociodemographic characteristics are often associated with greater survey response in the civilian population: employment [21, 22], middle versus younger age [21, 23], female gender [23], higher socioeconomic status/income [24, 25], and higher levels of formal education [26, 27]. Surveys of the military population generally reveal a similar pattern of greater response associated with these sociodemographic variables [28–30], as well as military-specific variables, such as active versus reserve duty status, officer versus enlisted status, and senior versus junior pay grade/rank [13, 27]. However, very little is known about the response patterns/behaviors of military spouses and family members. Although several spousal survey studies adjust for non-response [13, 16, 31], very few have provided any detailed discussion on the factors associated with spousal non-response. Findings suggest that a higher percentage of response is likely from officer-headed households across services [13]. In longitudinal surveys, participation in early data collection waves and fewer missing items are also associated with future participation [22, 32]; therefore, study participation is a predictor of future response. Less is known regarding the influence of psychosocial factors on non-response in the military population, as researchers typically do not have such measures on non-responders or on a proxy for the non-responder (e.g., family member). Some authors, however, suggest that non-respondents may generally be less healthy [25, 33, 34], have more substance use disorders [25, 34], or a history of psychiatric conditions [27].

This current study examined factors contributing to second stage survey non-response during the baseline data collection for the Millennium Cohort Family Study conducted from 2011 to 2013. Due to its nested design within the larger Millennium Cohort Study, the Family Study offers a unique opportunity to thoroughly examine and adjust for non-response bias among military spouses by analysing extensive data collected from their service member partners on the Millennium Cohort survey, including sociodemographic and psychosocial characteristics. This study contributes to the interpretation and use of the Family Study data by describing the sample and examining and addressing systematic non-response in the baseline sample. The study provides insights to inform future study designs and recruitment practices involving military spouses.

This study examined correlates of second stage non-response at time of baseline data collection for the Millennium Cohort Family Study in order to provide important context for future study results and to establish a foundation for non-response adjustment. A total of 9,872 spouses participated in the Family Study for an overall response rate of 34.5% (34.3% using the MilCo weight). The probability sample of military spouses included spouses of Reserve/National Guard and dual-military couples, as well as an oversampling of male spouses of service members, all of whom are often under-represented subgroups in military family research. Consistent with the objectives of the Family Study, the majority of couples had experienced at least one combat-deployed separation.

The results of our initial unadjusted non-response analyses showed the majority of service member sociodemographic, military, and administrative variables were significantly associated with greater spousal response, along with various health behaviours, mental health indices, and financial and social issues. Similar to other survey research studies, we found that sociodemographic characteristics and survey administrative factors were most strongly associated with response propensity. In multivariate analyses, the most important correlates of spousal response were service member gender, age, race/ethnicity, education, income, number of dependent children, service branch, and Millennium Cohort completion status. The single strongest predictor of spousal response was if the spouse was referred by the service member for participation. Finally, as is commonly the case in health survey research studies, we found that Family Study respondents were slightly better adjusted, with fewer physical and mental health symptoms, than non-respondents. However, effects associated with health and well-being were small.

The Family Study developed weights to adjust for non-response using a propensity modeling and raking strategy. In support of this, one of the important objectives of this analysis was to choose the optimal logistic regression model from which to derive the response propensity adjustment. We made this selection by applying two alternative models to predict several key service member measures that had been withheld from the models for purposes of validation. For most of these dependent variables, there was a substantial improvement in fit between Model 1 (sociodemographic/administrative variables only) and Model 2 (sociodemographic/administrative, physical, mental health, and stressful life event variables); thus Model 2 was ultimately selected. This ensured variables in the response propensity model fit the two critical criteria for reducing non-response bias-association, both with the likelihood of response and study outcomes of greatest interest. It is also important to note that adding these reserved study outcomes to the response propensity model as a final check did not appreciably improve the fit.

Differential non-response occurs in a military survey when the response rate varies for sample subgroups, typically defined by demographic and service member characteristics. This can often be corrected by making non-response adjustments. However, non-response adjustments can only account for factors that have been measured in administrative records or in the course of the study, and most spousal surveys have limited information on the dyad to assess non-response. The Family Study is uniquely positioned to offer insights into spousal response because there is a wealth of information from the service member to utilize for both spousal responders and non-responders, including physical and mental health indices. It is important to note that despite the inclusion of a multitude of variables in the non-response models in the current study, sociodemographic factors alone performed nearly as well in predicting non-response. Further, although the breadth of variables examined in this study exceeds the typically narrow set of socio-demographic characteristics used to assess non-response in military studies, additional military-related variables could be of interest in future studies such as whether the family lives on base and time on duty station.

Our results suggest future studies should combine targeted oversampling and comprehensive response bias adjustment strategies to address the problem of non-response in population subgroups that are difficult to engage, and should assess sociodemographic factors to help examine and adjust for non-response. For example, since male spouses are a group known to be difficult to recruit, the Millennium Cohort Panel 4 study design oversampled married and female service members in order to better facilitate Family Study enrollment. Due to this oversampling, the final population of male spouses was almost the same as their proportion in the actual population of young spouses despite a substantially lower response rate among males. Ultimately, the combination of oversampling and the application of comprehensive non-response adjustment techniques used in the Family Study provided the opportunity to represent the subgroup of military couples involving a female service member and male spouse more comprehensively than any previous research effort. This type of combined approach would be helpful in future military family research, not only with male spouses, but also spouses of service members who are enlisted, younger (17–24 years old), and black or Hispanic personnel, all of whom are less likely to respond and may require additional or enhanced recruitment strategies.

In addition to oversampling, the Family Study designed a range of targeted recruitment materials to engage participants who may be difficult to enroll. A good match between survey topic and participant characteristics has been found to influence response in prior research [54]; therefore, our study team was concerned with the difficulty in recruiting spouses who did not identify strongly with the “military spouse” or “military family” community. Once again, we were particularly concerned that male spouses may not have a strong military spouse self-schema; dual-military partners also may not identify with this role. The team had further concerns that couples without children may not strongly identify with the “military family schema”. Despite inclusion of a broad definition of military families in study recruitment materials, results indicated some spouses in the sampling frame may have been influenced by a more narrowly defined military family schema in making the choice of whether or not to respond. Furthermore, an examination of the Family Study service member referral process suggested Millennium Cohort participants may have been influenced by similarly narrow military family schemas in deciding whether to encourage their marital partners to engage in the Family Study (McMaster H, Stander V, O’Malley C, Williams C, Woodall K, Bauer L; unpublished observations, manuscript in development). Unfortunately, Millennium Cohort participants were given minimal information regarding the breadth of targeted participation for the Family Study in their introduction to the project. Given that referral was a strong correlate of response likelihood in this study, future dyadic research investigations may improve subgroup representation and overall participant response by enhancing both partners’ understanding of the intended recruitment population. Future military family research should consider that some military spouses may not feel as well integrated into the military community as others, and this may influence response rates.

In summary, our results indicated the Family Study was significantly impacted by a number of non-response factors that commonly affect survey research. In particular, recruiting young, male, and minority populations as well as junior ranking personnel was challenging. Despite this, the Family Study successfully employed multiple strategies to minimize the impact of non-response bias, including a comprehensive propensity modeling approach to develop weights. Our results suggest the success of representative population sampling can be effectively augmented through targeted oversampling and recruitment, as well as more comprehensive survey weighting strategies. In military populations where more extensive information is available documenting characteristics of the total population, future studies could take advantage of available data in adjusting for non-response bias.

Ultimately, the Family Study enrolled a uniquely large dyadic cohort with extensive self-report and military archival data available for all 9872 couples. This is a substantial representation from subgroups that are difficult to engage and frequently excluded in military family research (e.g., male spouses, reservists, dual-military couples). A particular goal was to engage a study panel of junior military personnel and their families. These relatively new members of the military community are an at-risk group, and this young cohort could be followed over the course of their military careers and beyond, capturing critical life events such as divorce as well as separation from the military and associated outcomes. Further, the Family Study participants entered the military community at a time when they likely would be maximally impacted by operations in Afghanistan and Iraq. As such, this program of research presents a critical opportunity to understand the impact of deployment and military life stress on family well-being. This study demonstrates that by conducting comprehensive non-response bias analyses, accounting for a myriad of constructs potentially associated with non-response, and applying weights to adjust for those factors, the data are much more representative of the target population. Although the methodology is not novel in and of itself, this study clearly demonstrates the benefits of non-response modeling and weighting in bias minimization. Ultimately, these weighted data provide the opportunity to generalize to military spouses whose partners had two to five years of military experience as of 2011.

Currently, the study team is exploring deployment-related stressors, as well as mediating factors, influencing outcomes, such as spousal depression, substance use, and marital satisfaction. In future work, we plan to address a full spectrum of issues related to spouse and child physical and mental health, as well as marital and family adjustment. This study will be able to investigate areas that previous military family research has never examined, such as long-term outcomes for families impacted by deployment separations, and longitudinal trajectories of family adjustment over the course of career military service and beyond.