Survey response over 15 years of follow-up in the Millennium Cohort Study

Background Patterns of survey response and the characteristics associated with response over time in longitudinal studies are important to discern for the development of tailored retention efforts aimed at minimizing response bias. The Millennium Cohort Study, the largest and longest running cohort study of military personnel and veterans, is designed to examine the long-term health effects of military service and experiences and thus relies on continued participant survey responses over time. Here, we describe the response rates for follow-up survey data collected over 15 years and identify characteristics associated with follow-up survey response and mode of response (paper vs. web). Method Patterns of follow-up survey response and response mode (web, paper, none) were examined among eligible participants (n=198,833), who were initially recruited in four panels from 2001 to 2013 in the Millennium Cohort Study, for a follow-up period of 3–15 years (2004–2016). Military and sociodemographic factors (i.e., enrollment panel, sex, birth year, race and ethnicity, educational attainment, marital status, service component, service branch, pay grade, military occupation, length of service, and time deployed), life experiences and health-related factors (i.e., military deployment/combat experience, life stressors, mental health, physical health, and unhealthy behaviors) were used to examine follow-up response and survey mode over time in multivariable generalized estimating equation models. Results Overall, an average response rate of 60% was observed across all follow-up waves. Factors associated with follow-up survey response over time included increased educational attainment, married status, female sex, older age, military deployment (regardless of combat experience), and higher number of life stressors, mental health issues, and physical health diagnoses. Conclusion Despite the challenges associated with collecting multiple waves of follow-up survey data from members of the U.S. military during and after service, the Millennium Cohort Study has maintained a relatively robust response rate over time. The incorporation of tailored messages and outreach to those groups least likely to respond over time may improve retention and thereby increase the representativeness and generalizability of collected survey data. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-023-02018-z.


Background
Prospective cohort studies follow a group of participants and collect data on them over a period of time to answer important causal research questions related to emergence of disease in relation to known and unknown exposures [1].In the early 2000s, cohort studies began to observe a precipitous decline in responses to follow-up surveys [2][3][4][5][6].In addition, several characteristics were consistently associated with attrition, or loss to follow-up, in cohort studies, including male sex, younger age, single marital status, and lower socioeconomic status [7][8][9].Differential response rates among participants to follow-up surveys can negatively affect the validity and generalizability of study findings [10,11].
Prior to the wide availability of self-administered internet surveys, researchers often relied on paper-and-pencil surveys that were mailed back to the study operations center or administered in person.Reliance on paperbased surveys can be expensive and cost prohibitive in terms of materials, postage, and software and labor for data entry and cleaning.With the widespread increase in internet usage in the United States (from 55 to 93% between 2000 and 2021 [12]) survey data can now be collected remotely, which is often more cost effective, less labor intensive to process the data, and carries additional advantages including the ability to program complex skip logic [13][14][15].However, due to issues such as disparities in internet access, certain groups, such as those from racial or ethnic minority groups or who are older, of lower socioeconomic status, or reside in rural communities, may be underrepresented in studies that employ online surveys as their sole mode of data collection [12].To mitigate these concerns and boost response rates, some studies employ a multimodal survey design where participants can complete either a paper or a web-based survey [16].However, more research is needed to understand the characteristics that may be associated with mode of survey response in longitudinal cohort studies over time, particularly those focused on military personnel and other hard-to-reach populations.
The Millennium Cohort Study (hereafter the Study), the largest and longest running cohort study of service members and veterans, was initiated in 2001 with the objective of studying the long-term health impacts of military deployments.Detailed summaries of Study methodology are available elsewhere [17][18][19].Briefly, the Study employs a multi-panel, multi-wave design in which different cohorts (called panels) of service members are enrolled and invited to complete follow-up surveys in cycles called waves via a website or paper mode approximately every 3-5 years (Fig. 1) [20].
Participants from the first four panels (N=201,619) were invited to complete at least one follow-up survey as of 2016.Examinations of the representativeness and response patterns of the Study have been previously reported [18,[20][21][22], but they were limited to participants from the first enrollment panel.While findings suggested that Panel 1 was representative of the general military at baseline [18,21], there were observed differences in response rates at the first follow-up survey (called Wave 2, which was conducted between 2004 and 2006 for Panel 1).However, additional longitudinal patterns of response rates over multiple follow-up waves were not examined beyond Wave 2 for Panel 1.The aims of the current analysis are to (1) report the overall and by mode (paper vs. web) response rates between the second

Participants
Across the five panels, 260,228 participants enrolled in the Study between 2001 and 2021 (enrolled in 2001-2003, 2004-2006, 2007-2008, 2011-2013, and 2020-2021).While Panel 1 was invited from a random sample of service members from active duty and Reserve/National Guard rosters in October 2000, subsequent panels randomly sampled more recently accessed service members (i.e., with 1-5 years of military service) and oversampled certain subgroups (e.g., women, Marine Corps personnel) to enable sufficient sample size for between-group comparisons.At the start of each follow-up wave, participants were excluded if they withdrew from the study or were deceased.Panel 1 participants were eligible for five follow-up waves, but each subsequent panel was eligible for fewer follow-up waves due to the staggered multipanel and multi-wave study design (Fig. 1).

Methods to increase follow-up survey response
Historically, the Study employed multiple strategies to increase follow-up survey response such as targeting outreach (e.g., contacts intended for specific service branches or veterans), updating contact information from multiple sources, and using incentives.Some techniques, such as postcard and email reminders to complete the survey, were consistently utilized over time, but the frequency and timing of those techniques varied by survey cycle (Table 1).Reminder emails notified participants to start or complete the web survey and postcards were sent to nonresponders.During the 2007-2008 and 2011-2013 survey cycles, the Study used reminder voice messages and during the 2014-2016 survey cycle, one primer email was sent before the survey cycle began.The Study also sent out annual commemorative emails and postcards for Memorial Day and Veterans Day that served as reminders for participants to complete the survey if the holiday fell within a survey cycle.Endorsement letters from Department of Defense leadership were sent to participants during the 2011-2013 and 2014-2016 surveys.Type of incentives and timing of distribution (either before or after survey completion) changed over time; an experiment conducted during the 2014-2016 cycle found that monetary pre-incentives produced a modest increase in follow-up survey response [23].Finally, during the 2014-2016 survey cycle a short (2-page) paper survey was created and mailed to nonresponders 18 and 20 months into the cycle to increase response.

Military and demographic covariates
Demographic and military service factors such as birth year, sex, race and ethnicity, pay grade, service branch, military occupation, length of service (LOS), and service component were obtained from administrative records maintained by the Defense Manpower Data Center (DMDC).Military factors were measured concurrent with the follow-up response since they could be obtained from administrative records and did not rely on participant response.All other self-reported factors listed in the measures section were measured as ever occurring before each follow-up wave.Deployment dates were obtained from the Contingency Tracking System (CTS) and were used to calculate the total years deployed before each follow-up survey.Educational attainment and marital status were self-reported on the survey and backfilled with DMDC data if missing.

Stressful life events
Combat deployment experience was identified using a combination of administrative CTS records with a 5-item combat experience scale on the survey with responses categorized into one of four categories (i.e., deployed with combat, deployed without combat, deployed with unknown combat, not deployed) consistent with prior research [24].Life stressors were assessed from five modified items (i.e., divorce, financial issues, sexual assault, sexual harassment, and physical assault) from the Holmes and Rahe Social Readjustment Rating Scale, and participants were categorized as reporting 0, 1, or 2 + life stressors [25].

Mental and physical health and unhealthy behaviors
The Study survey includes a variety of validated instruments to screen for mental health conditions.The 17-item Posttraumatic Stress Disorder (PTSD) Checklist − Civilian Version was used to measure the severity of PTSD symptoms [26,27].The Patient Health Questionnaire was used to identify depression, panic, other anxiety, and binge eating disorder, respectively [27][28][29].Positive screens on these five validated mental health screeners were summed and categorized as 0, 1, or 2 + mental health conditions.
Physical health indicators were identified using a combination of self-reported provider diagnoses, height, and weight.A body mass index greater than 30 kg/m 2 and four specific diagnoses (hypertension, high cholesterol, migraine, and sleep apnea) were chosen from a list of 38 diagnoses because they were the most prevalent physical health conditions reported on the 2014-2016 survey and were always included on the survey.The sum of the five physical health indicators was categorized as 0, 1, or 2 + .
Unhealthy behaviors included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration.Current cigarette smoking was indicated if participants endorsed smoking 100 or more cigarettes in their lifetime and not successfully quitting smoking.Heavy weekly alcohol consumption was determined using the daily number of alcoholic beverages consumed in the past week with a threshold of 7 or 14 drinks per week for females and males, respectively [30].Unhealthy sleep duration was assessed from a single item coded such that 6 or fewer and 10 or more average hours in a 24-h period indicated unhealthy sleep duration based on National Sleep Foundation recommendations [31].The sum of the three unhealthy behaviors was categorized as 0, 1, or 2 + .

Statistical analyses
For each survey wave, we calculated the frequency and percentage of follow-up survey response (yes vs. no) and mode of response (web vs. paper) stratified by panel.Frequencies of military and demographic characteristics, stressful life events, mental and physical health, and unhealthy behaviors were also calculated for eligible participants.In addition, to examine the bivariate relationships between participant characteristics with followup survey response, Wave 2 response rate and consistent follow-up response (i.e., responded to all follow-up surveys) were reported for each category of participant-level characteristics.For survey mode, the percentages of web and paper surveys completed were reported among Wave 2 responders and those who responded by one mode more frequently across all follow-up surveys.Multicollinearity was tested among characteristics with a variance inflation factor threshold ≥ 4. Generalized estimating equations (GEEs) were used to estimate the associations between variables of interest and response to follow-up surveys over time.GEEs are a subclass of semiparametric models, which, unlike the parametric generalized linear mixed-effects models, impose no mathematical model for the distributions of the multivariate response due to repeated assessments and thereby provide valid inference for virtually all data distributions [32].GEEs are not only robust, but also most efficient, or powerful, in the sense that they have the largest power among all such semiparametric models.For both outcomes, survey response and response mode, effect estimates were generated from fully-adjusted models that included all variables of interest [33] (birth year, sex, race and ethnicity, pay grade, service branch, military occupation, LOS, service component, deployment experience, life stressors, mental and physical health, and unhealthy behaviors), panel, and wave were reported.All statistical analyses were conducted using SAS software version 9.4 (SAS Institute, Inc., Cary, North Carolina, USA).The study was approved by the Naval Health Research Center Institution Review Board (protocol number NHRC.2000.0007).

Follow-up response rate over time
Eligibility, response rate, and mode of response to surveys are reported in Table 2.Over 93% of participants were eligible (i.e., not deceased, withdrawn from the Study, or completed the 2014-16 short paper survey) at each follow-up wave.Among eligible participants, approximately 60% responded at Wave 2. Furthermore, approximately 70% of the cohort responded to at least one follow-up survey (which may or may not have been Wave 2) and 42% responded to every follow-up survey.However, the follow-up response rate decreased at each consecutive follow-up wave.Among Panels 1-4 participants who completed follow-up surveys between 2006 and 2016, 82% of surveys were completed online.
Additionally, approximately 60% responded at Wave 2 and 42% responded to all follow-up surveys (Table 3).These rates were at least 5% higher among those born before 1970 compared with those born in 1980 or later, those with a college degree compared with those with a high school equivalent or less, Reserve, National Guard personnel compared with those separated, Officers compared with junior enlisted, health care military occupation compared with administrative or supply.The Wave 2 response rate was also higher among Panel 1 compared with Panel 3 participants and active duty compared with separated personnel, while Panel 4 participants, compared with Panel 2, were more likely to be consistent responders.Wave 2 and consistent responders also had longer length of service, while consistent responders had also deployed for longer.
Finally, among 118,948 Wave 2 responders, 83% completed the web survey and 17% completed the paper survey (Table 4).Similar proportions were observed among the 130,134 participants who responded more often to one survey mode (86% web and 14% paper).Active duty participants, compared with those separated from the military, were more likely to complete web surveys while those separated from the military compared with active duty, junior enlisted compared with Officers, and those who screened positive for more than one mental health condition compared with those who did not screen positive were more likely to complete paper survey (either at Wave 2 or consistently).Those born before 1960 compared with those born between 1960-1969 were more likely to complete paper survey at Wave 2 while Officers, compared with junior enlisted, were more likely to respond using the web survey across all follow-up surveys.

Survey response over time
In the adjusted GEE model, all characteristics of interest were significantly associated with follow-up survey response over time except for cumulative years deployed (Table 5).Characteristics positively associated with follow-up survey response over time included being a participant in the initial panel (enrolled between 2001-2003), Wave 3 (compared with Wave 2), increased educational attainment, married, female sex, earlier birth years (i.e., before 1980), and non-Hispanic White race and ethnicity.In addition, serving in the Marine Corps, Navy or Coast Guard (compared with Army), military occupation of combat specialist, active duty (compared with separated participants), junior enlisted pay grade, and deployment experience (with or without combat) were associated with follow-up survey response.Lastly, reporting life stressors, screening positive for mental health conditions, reporting physical health conditions, and reporting no unhealthy behaviors were also associated with follow-up survey response.

Survey response and mode over time
In the adjusted GEE model, all characteristics were significantly associated with follow-up survey mode over time (Table 6).Certain characteristics, such as increased educational attainment, female sex, earlier birth year, Marine Corps service, deployment experience (with or without combat), screening positive for mental health conditions, and reporting physical health conditions, were associated with higher odds of web and paper survey completion compared with not responding to the follow-up surveys.Other characteristics, such as Wave 3 participation, currently married status, Navy or Coast Guard service, senior enlisted pay grade, and longer time deployed, were only associated with web survey completion.Conversely, those with a military occupation of health care or combat specialist (compared with other), and Reserve/National Guard service (compared with active duty service) were associated only with paper survey completion over time.

Discussion
Continued response to follow-up surveys is especially critical for prospective cohort studies, especially for the purpose of ascertaining rare outcomes or those with  long latency periods (e.g., cancers, neurodegenerative conditions).It is thus critical to adequately characterize response rates and potential sources of nonresponse bias.Over a follow-up period of 3-15 years, approximately 60% of enrolled participants responded to the first follow-up survey, approximately 70% responded to at least one follow-up survey, and 42% responded at every follow-up survey.Additionally, one notable finding was that Veterans were less likely to respond to follow-up surveys and more likely to complete the paper survey.
To our knowledge, no other longitudinal study of service members and veterans has reported factors associated with follow-up survey response, particularly over multiple waves.We were able to identify military factors associated with follow-up survey response in a longitudinal cohort, such as service in the Navy, Coast Guard or Marine Corps (compared with the Army) and active duty service (compared with Reserve, National Guard, separated from the military).These findings mirror results reported for Panel 1 at Wave 2 [20,22].The relatively low response observed among certain groups may be attributable to reasons identified in a previous cross-sectional survey of Air Force personnel: lack of time or interest, attitudes toward sponsoring  organization, survey length discouraging initiation or completion of survey, and internet access barriers [34].
Veterans may also feel a diminished connection with a military study.The observed response rate of 60% in this study was lower than the average response rate of 74% in a recent systematic review and meta-analysis of 141 longitudinal cohort studies [35].Other longitudinal cohort studies such as the Framingham Study [36], Multi-Ethnic Study of Atherosclerosis [37], Women's Health Initiative Observational Study [38], and the Nurses' Health Study [39] have reported higher follow-up response rates (80% or greater).Compared with the populations of interest in other cohort studies, service members are relatively young, highly mobile, extensively surveyed, and have rigorous work schedules and requirements [40] that create  challenges for repeated surveying over time.This, combined with differences in timing of follow-up and survey methods and operations, resulted in the lower follow-up survey response rate reported in this analysis.Historically, the Study did not track whether participants were not contacted during survey cycles (e.g., due to undeliverable email addresses or postal addresses that returned mail to sender), so we were unable to distinguish between those not contacted from other nonresponders.Panel 1 Study participants responded at a higher rate (Table 2; average of 70%), which is similar to higher response rates reported in other cohort studies.We conducted a sensitivity analysis with models examining response over time stratified by panel to examine whether there were differences in the associations of the characteristics with response (Supplemental Table ), which revealed similar associations across panels.Considering the challenges of retaining enrolled service members and veterans in a longitudinal study, the observed 60% response rate (range 43-71%) is remarkable given the large number of participants that the Study was unable to reach.Even with observed attrition, survey data collected from participants have been extensively analyzed and findings reported in over 150 peer-reviewed publications to date on a wide breadth of topics on the long-term health of service members and veterans [19].Similar to other cohort studies that have observed declining response rates over time, the Study reported a decline in response rates with each follow-up survey.For example, the Swedish Longitudinal Occupational Survey of Health reported a follow-up response rate of 65% in 2006 that declined to 51% in 2016 [41].The California Teachers Study reported a response rate of 43% at the sixth survey cycle and found that older participants were more likely to participate than younger participants, which is similar to the results for Panel 1 of the current Study [42].Demographic characteristics related to survey response among Study participants, such as increased educational attainment, female sex, older age, non-Hispanic White race and ethnicity, have also been reported in other cohort studies [7][8][9]43].The Study has modified its retention methods over time, such as adding endorsement letters, varying the incentives available, and modifying the type and timing of reminders (Table 1).Although  beyond the scope of the analyses presented here, it is important for longitudinal cohort studies to adapt their retention strategies and examine the effectiveness of strategies to reduce loss to follow-up.Finally, although survey attributes (e.g., number, order, sensitivity of survey questions) can impact response rates, it is beyond the aims of this paper to summarize the surveys instruments used during the study time-period.A detailed description of the measures and constructs included on the Study's surveys over time can be found elsewhere [44].Briefly, a core set of questions and constructs remained consistent over time were utilized in these analyses, but the Study was also able to add measures in response to research priorities, military policy, stakeholder input and scientific advances.
Our findings on the associations between life stressors, health status, and unhealthy behaviors on follow-up survey response align with findings from other cohorts.For example, we reported that self-reported physical health conditions were associated with survey response, which is similar to other studies.Baseline responders in Panel 1 were more likely to have certain outpatient diagnoses compared with nonresponders [45].Studies of other populations also observed higher rates of health care utilization among survey responders [46,47], which may indicate that survey responders have higher disease burden.In addition, we observed that reporting stressful life experiences was associated with later follow-up response, which was consistent with an analysis that observed higher levels of stress among a populationbased post-disaster follow-up questionnaire [48].Conversely, unhealthy behaviors were associated with lower follow-up survey response, which has been observed in other health surveys [49,50].
A notable limitation of this analysis is the fact that the Study historically did not track failed contacts (e.g., return to sender, unknown address, or email bounce backs) and we were thus unable to identify and exclude participants who did not receive survey notifications during the survey cycles.Thus, the response rates reported in this analysis may underestimate the true response rates, as the total eligible population would decrease if we were able to exclude those not contacted during follow-up waves.Moving forward, the Study will improve tracking of changes to participant contact information.Additionally, while these analyses allowed for intermittent responses (e.g., it was permissible to respond to Wave 3 even if they failed to respond to Wave 2), analyses did not examine differences in such patterns, which would be a useful avenue for future research.These analyses were not able to examine the impact of participant engagement methods utilized by the Study to increase response, such as incentives, type and frequency of reminders sent over the follow-up period and cannot speak to the influence these have on follow-up survey response.Future work should examine these methodological factors on survey response.Finally, it was beyond the scope of this paper to examine potential nonresponse bias among this cohort.Earlier work among Panel 1 participants observed that nonresponse had minimal impact on study findings [22] and ongoing analyses are updating these results among a contemporary cohort at more recent time points.Despite these limitations, the Study has many strengths, including enrolling a diverse cohort of service members from all service branches, components, pay grades, and occupations; being the longest running military cohort study with follow-up planned through 2068; and having the ability to merge with other data sources to supplement survey data.

Conclusions
Findings from this analysis indicate that the Study has maintained an adequate response rate over time despite the many challenges associated with surveying military and veteran populations.Additionally, documentation of the different characteristics associated with nonresponse and mode of response is informative for ensuring that appropriate conclusions are drawn over time.These findings can also be instructive for Study strategies for continued participant engagement and may be applicable to other longitudinal cohort studies.

Fig. 1
Fig. 1 Depiction of the multiple panel design used in the Millennium Cohort Study

Table 1
Millennium Cohort Study follow-up survey methods between 2004-2006 and 2014-2016 a Those

Table 2
Millennium Cohort Study participant eligibility, response, and mode of response between 2001-2003 and 2014-2016 surveys, n=201,619

Table 3
Participant characteristics by wave 2 response rate and consistent response rate, n=198,833

Table 3
(continued)All characteristics were significantly associated with Wave 2 follow-up survey response and consistent follow-up survey response (p < .05) a Mental health indicators included screens for PTSD, depression, panic, other anxiety, and binge eating disorder.Physical health indicators included obese body mass index and diagnoses of hypertension, high cholesterol, migraines, and sleep apnea.Unhealthy behavior indicators included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration

Table 4
Participant characteristics by wave 2 mode and preferred follow-up survey mode

Table 4
(continued)All characteristics were significantly associated with Wave 2 follow-up survey response and consistent follow-up survey response (p < .05) a Mental health indicators included screens for PTSD, depression, panic, other anxiety, and binge eating disorder.Physical health indicators included obese body mass index and diagnoses of hypertension, high cholesterol, migraines, and sleep apnea.Unhealthy behavior indicators included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration

Table 5
Adjusted GEE effect estimates for follow-up survey response over time

Table 5 (continued) Responded AOR (95% CI) Deployment experience (ref: not deployed)
All characteristics were significantly associated with follow-up survey response over time (p < .05)except for cumulative years deployed in the adjusted model (p = .34)AOR adjusted odds ratio, CI confidence interval, GEE generalized estimating equation a Mental health indicators included screens for PTSD, depression, panic, other anxiety, and binge eating disorder.Physical health indicators included obese body mass index and diagnoses of hypertension, high cholesterol, migraines, and sleep apnea.Unhealthy behavior indicators included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration

Table 6
Adjusted GEE effect estimates for follow-up survey response and mode over time

Table 6
(continued) All characteristics are significantly associated with follow-up web and paper survey response over time (p < .05)AOR adjusted odds ratio, CI confidence interval, GEE generalized estimating equation a Mental health indicators included screens for PTSD, depression, panic, other anxiety, and binge eating disorder.Physical health indicators included obese body mass index and diagnoses of hypertension, high cholesterol, migraines, and sleep apnea.Unhealthy behavior indicators included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration