Skip to main content
Download PDF

Survey response over 15 years of follow-up in the Millennium Cohort Study

BMC Medical Research Methodology volume 23, Article number: 205 (2023) Cite this article

Abstract

Background

Patterns of survey response and the characteristics associated with response over time in longitudinal studies are important to discern for the development of tailored retention efforts aimed at minimizing response bias. The Millennium Cohort Study, the largest and longest running cohort study of military personnel and veterans, is designed to examine the long-term health effects of military service and experiences and thus relies on continued participant survey responses over time. Here, we describe the response rates for follow-up survey data collected over 15 years and identify characteristics associated with follow-up survey response and mode of response (paper vs. web).

Method

Patterns of follow-up survey response and response mode (web, paper, none) were examined among eligible participants (n=198,833), who were initially recruited in four panels from 2001 to 2013 in the Millennium Cohort Study, for a follow-up period of 3–15 years (2004–2016). Military and sociodemographic factors (i.e., enrollment panel, sex, birth year, race and ethnicity, educational attainment, marital status, service component, service branch, pay grade, military occupation, length of service, and time deployed), life experiences and health-related factors (i.e., military deployment/combat experience, life stressors, mental health, physical health, and unhealthy behaviors) were used to examine follow-up response and survey mode over time in multivariable generalized estimating equation models.

Results

Overall, an average response rate of 60% was observed across all follow-up waves. Factors associated with follow-up survey response over time included increased educational attainment, married status, female sex, older age, military deployment (regardless of combat experience), and higher number of life stressors, mental health issues, and physical health diagnoses.

Conclusion

Despite the challenges associated with collecting multiple waves of follow-up survey data from members of the U.S. military during and after service, the Millennium Cohort Study has maintained a relatively robust response rate over time. The incorporation of tailored messages and outreach to those groups least likely to respond over time may improve retention and thereby increase the representativeness and generalizability of collected survey data.

Peer Review reports

Background

Prospective cohort studies follow a group of participants and collect data on them over a period of time to answer important causal research questions related to emergence of disease in relation to known and unknown exposures [1]. In the early 2000s, cohort studies began to observe a precipitous decline in responses to follow-up surveys [2,3,4,5,6]. In addition, several characteristics were consistently associated with attrition, or loss to follow-up, in cohort studies, including male sex, younger age, single marital status, and lower socioeconomic status [7,8,9]. Differential response rates among participants to follow-up surveys can negatively affect the validity and generalizability of study findings [10, 11].

Prior to the wide availability of self-administered internet surveys, researchers often relied on paper-and-pencil surveys that were mailed back to the study operations center or administered in person. Reliance on paper-based surveys can be expensive and cost prohibitive in terms of materials, postage, and software and labor for data entry and cleaning. With the widespread increase in internet usage in the United States (from 55 to 93% between 2000 and 2021 [12]) survey data can now be collected remotely, which is often more cost effective, less labor intensive to process the data, and carries additional advantages including the ability to program complex skip logic [13,14,15]. However, due to issues such as disparities in internet access, certain groups, such as those from racial or ethnic minority groups or who are older, of lower socioeconomic status, or reside in rural communities, may be underrepresented in studies that employ online surveys as their sole mode of data collection [12]. To mitigate these concerns and boost response rates, some studies employ a multimodal survey design where participants can complete either a paper or a web-based survey [16]. However, more research is needed to understand the characteristics that may be associated with mode of survey response in longitudinal cohort studies over time, particularly those focused on military personnel and other hard-to-reach populations.

The Millennium Cohort Study (hereafter the Study), the largest and longest running cohort study of service members and veterans, was initiated in 2001 with the objective of studying the long-term health impacts of military deployments. Detailed summaries of Study methodology are available elsewhere [17,18,19]. Briefly, the Study employs a multi-panel, multi-wave design in which different cohorts (called panels) of service members are enrolled and invited to complete follow-up surveys in cycles called waves via a website or paper mode approximately every 3–5 years (Fig. 1) [20].

Fig. 1
figure 1

Depiction of the multiple panel design used in the Millennium Cohort Study

Full size image

Participants from the first four panels (N=201,619) were invited to complete at least one follow-up survey as of 2016. Examinations of the representativeness and response patterns of the Study have been previously reported [18, 20,21,22], but they were limited to participants from the first enrollment panel. While findings suggested that Panel 1 was representative of the general military at baseline [18, 21], there were observed differences in response rates at the first follow-up survey (called Wave 2, which was conducted between 2004 and 2006 for Panel 1). However, additional longitudinal patterns of response rates over multiple follow-up waves were not examined beyond Wave 2 for Panel 1. The aims of the current analysis are to (1) report the overall and by mode (paper vs. web) response rates between the second and fifth waves spanning up to 15 years of follow-up, and (2) identify participant characteristics associated with response and mode of response across two to five waves of survey collection.

Methods

Participants

Across the five panels, 260,228 participants enrolled in the Study between 2001 and 2021 (enrolled in 2001–2003, 2004–2006, 2007–2008, 2011–2013, and 2020–2021). While Panel 1 was invited from a random sample of service members from active duty and Reserve/National Guard rosters in October 2000, subsequent panels randomly sampled more recently accessed service members (i.e., with 1–5 years of military service) and oversampled certain subgroups (e.g., women, Marine Corps personnel) to enable sufficient sample size for between-group comparisons. At the start of each follow-up wave, participants were excluded if they withdrew from the study or were deceased. Panel 1 participants were eligible for five follow-up waves, but each subsequent panel was eligible for fewer follow-up waves due to the staggered multi-panel and multi-wave study design (Fig. 1).

Methods to increase follow-up survey response

Historically, the Study employed multiple strategies to increase follow-up survey response such as targeting outreach (e.g., contacts intended for specific service branches or veterans), updating contact information from multiple sources, and using incentives. Some techniques, such as postcard and email reminders to complete the survey, were consistently utilized over time, but the frequency and timing of those techniques varied by survey cycle (Table 1). Reminder emails notified participants to start or complete the web survey and postcards were sent to nonresponders. During the 2007–2008 and 2011–2013 survey cycles, the Study used reminder voice messages and during the 2014–2016 survey cycle, one primer email was sent before the survey cycle began. The Study also sent out annual commemorative emails and postcards for Memorial Day and Veterans Day that served as reminders for participants to complete the survey if the holiday fell within a survey cycle. Endorsement letters from Department of Defense leadership were sent to participants during the 2011–2013 and 2014–2016 surveys. Type of incentives and timing of distribution (either before or after survey completion) changed over time; an experiment conducted during the 2014–2016 cycle found that monetary pre-incentives produced a modest increase in follow-up survey response [23]. Finally, during the 2014–2016 survey cycle a short (2-page) paper survey was created and mailed to nonresponders 18 and 20 months into the cycle to increase response.

Table 1 Millennium Cohort Study follow-up survey methods between 2004–2006 and 2014–2016
Full size table

Measures

Military and demographic covariates

Demographic and military service factors such as birth year, sex, race and ethnicity, pay grade, service branch, military occupation, length of service (LOS), and service component were obtained from administrative records maintained by the Defense Manpower Data Center (DMDC). Military factors were measured concurrent with the follow-up response since they could be obtained from administrative records and did not rely on participant response. All other self-reported factors listed in the measures section were measured as ever occurring before each follow-up wave. Deployment dates were obtained from the Contingency Tracking System (CTS) and were used to calculate the total years deployed before each follow-up survey. Educational attainment and marital status were self-reported on the survey and backfilled with DMDC data if missing.

Stressful life events

Combat deployment experience was identified using a combination of administrative CTS records with a 5-item combat experience scale on the survey with responses categorized into one of four categories (i.e., deployed with combat, deployed without combat, deployed with unknown combat, not deployed) consistent with prior research [24]. Life stressors were assessed from five modified items (i.e., divorce, financial issues, sexual assault, sexual harassment, and physical assault) from the Holmes and Rahe Social Readjustment Rating Scale, and participants were categorized as reporting 0, 1, or 2 + life stressors [25].

Mental and physical health and unhealthy behaviors

The Study survey includes a variety of validated instruments to screen for mental health conditions. The 17-item Posttraumatic Stress Disorder (PTSD) Checklist − Civilian Version was used to measure the severity of PTSD symptoms [26, 27]. The Patient Health Questionnaire was used to identify depression, panic, other anxiety, and binge eating disorder, respectively [27,28,29]. Positive screens on these five validated mental health screeners were summed and categorized as 0, 1, or 2 + mental health conditions.

Physical health indicators were identified using a combination of self-reported provider diagnoses, height, and weight. A body mass index greater than 30 kg/m2 and four specific diagnoses (hypertension, high cholesterol, migraine, and sleep apnea) were chosen from a list of 38 diagnoses because they were the most prevalent physical health conditions reported on the 2014–2016 survey and were always included on the survey. The sum of the five physical health indicators was categorized as 0, 1, or 2 + .

Unhealthy behaviors included current cigarette smoking, heavy weekly alcohol drinking, and unhealthy sleep duration. Current cigarette smoking was indicated if participants endorsed smoking 100 or more cigarettes in their lifetime and not successfully quitting smoking. Heavy weekly alcohol consumption was determined using the daily number of alcoholic beverages consumed in the past week with a threshold of 7 or 14 drinks per week for females and males, respectively [30]. Unhealthy sleep duration was assessed from a single item coded such that 6 or fewer and 10 or more average hours in a 24-h period indicated unhealthy sleep duration based on National Sleep Foundation recommendations [31]. The sum of the three unhealthy behaviors was categorized as 0, 1, or 2 + .

Statistical analyses

For each survey wave, we calculated the frequency and percentage of follow-up survey response (yes vs. no) and mode of response (web vs. paper) stratified by panel. Frequencies of military and demographic characteristics, stressful life events, mental and physical health, and unhealthy behaviors were also calculated for eligible participants. In addition, to examine the bivariate relationships between participant characteristics with follow-up survey response, Wave 2 response rate and consistent follow-up response (i.e., responded to all follow-up surveys) were reported for each category of participant-level characteristics. For survey mode, the percentages of web and paper surveys completed were reported among Wave 2 responders and those who responded by one mode more frequently across all follow-up surveys. Multicollinearity was tested among characteristics with a variance inflation factor threshold ≥ 4. Generalized estimating equations (GEEs) were used to estimate the associations between variables of interest and response to follow-up surveys over time. GEEs are a subclass of semiparametric models, which, unlike the parametric generalized linear mixed-effects models, impose no mathematical model for the distributions of the multivariate response due to repeated assessments and thereby provide valid inference for virtually all data distributions [32]. GEEs are not only robust, but also most efficient, or powerful, in the sense that they have the largest power among all such semiparametric models. For both outcomes, survey response and response mode, effect estimates were generated from fully-adjusted models that included all variables of interest [33] (birth year, sex, race and ethnicity, pay grade, service branch, military occupation, LOS, service component, deployment experience, life stressors, mental and physical health, and unhealthy behaviors), panel, and wave were reported. All statistical analyses were conducted using SAS software version 9.4 (SAS Institute, Inc., Cary, North Carolina, USA). The study was approved by the Naval Health Research Center Institution Review Board (protocol number NHRC.2000.0007).

Results

Follow-up response rate over time

Eligibility, response rate, and mode of response to surveys are reported in Table 2. Over 93% of participants were eligible (i.e., not deceased, withdrawn from the Study, or completed the 2014-16 short paper survey) at each follow-up wave. Among eligible participants, approximately 60% responded at Wave 2. Furthermore, approximately 70% of the cohort responded to at least one follow-up survey (which may or may not have been Wave 2) and 42% responded to every follow-up survey. However, the follow-up response rate decreased at each consecutive follow-up wave. Among Panels 1–4 participants who completed follow-up surveys between 2006 and 2016, 82% of surveys were completed online.

Table 2 Millennium Cohort Study participant eligibility, response, and mode of response between 2001-2003 and 2014-2016 surveys, n=201,619
Full size table

Descriptive characteristics of follow-up response

Among 198,833 eligible participants, most were male (69%), of non-Hispanic White race and ethnicity (75%), married (58%), senior enlisted personnel (56%), or did not have a college degree (57%). A plurality were born in 1980 or after (42%), served on active duty (42%), or served in the Army (45%) (Table 3).

Table 3 Participant characteristics by wave 2 response rate and consistent response rate, n=198,833
Full size table

Additionally, approximately 60% responded at Wave 2 and 42% responded to all follow-up surveys (Table 3). These rates were at least 5% higher among those born before 1970 compared with those born in 1980 or later, those with a college degree compared with those with a high school equivalent or less, Reserve, National Guard personnel compared with those separated, Officers compared with junior enlisted, health care military occupation compared with administrative or supply. The Wave 2 response rate was also higher among Panel 1 compared with Panel 3 participants and active duty compared with separated personnel, while Panel 4 participants, compared with Panel 2, were more likely to be consistent responders. Wave 2 and consistent responders also had longer length of service, while consistent responders had also deployed for longer.

Finally, among 118,948 Wave 2 responders, 83% completed the web survey and 17% completed the paper survey (Table 4). Similar proportions were observed among the 130,134 participants who responded more often to one survey mode (86% web and 14% paper). Active duty participants, compared with those separated from the military, were more likely to complete web surveys while those separated from the military compared with active duty, junior enlisted compared with Officers, and those who screened positive for more than one mental health condition compared with those who did not screen positive were more likely to complete paper survey (either at Wave 2 or consistently). Those born before 1960 compared with those born between 1960–1969 were more likely to complete paper survey at Wave 2 while Officers, compared with junior enlisted, were more likely to respond using the web survey across all follow-up surveys.

Table 4 Participant characteristics by wave 2 mode and preferred follow-up survey mode
Full size table

Survey response over time

In the adjusted GEE model, all characteristics of interest were significantly associated with follow-up survey response over time except for cumulative years deployed (Table 5). Characteristics positively associated with follow-up survey response over time included being a participant in the initial panel (enrolled between 2001–2003), Wave 3 (compared with Wave 2), increased educational attainment, married, female sex, earlier birth years (i.e., before 1980), and non-Hispanic White race and ethnicity. In addition, serving in the Marine Corps, Navy or Coast Guard (compared with Army), military occupation of combat specialist, active duty (compared with separated participants), junior enlisted pay grade, and deployment experience (with or without combat) were associated with follow-up survey response. Lastly, reporting life stressors, screening positive for mental health conditions, reporting physical health conditions, and reporting no unhealthy behaviors were also associated with follow-up survey response.

Table 5 Adjusted GEE effect estimates for follow-up survey response over time
Full size table

Survey response and mode over time

In the adjusted GEE model, all characteristics were significantly associated with follow-up survey mode over time (Table 6). Certain characteristics, such as increased educational attainment, female sex, earlier birth year, Marine Corps service, deployment experience (with or without combat), screening positive for mental health conditions, and reporting physical health conditions, were associated with higher odds of web and paper survey completion compared with not responding to the follow-up surveys. Other characteristics, such as Wave 3 participation, currently married status, Navy or Coast Guard service, senior enlisted pay grade, and longer time deployed, were only associated with web survey completion. Conversely, those with a military occupation of health care or combat specialist (compared with other), and Reserve/National Guard service (compared with active duty service) were associated only with paper survey completion over time.

Table 6 Adjusted GEE effect estimates for follow-up survey response and mode over time
Full size table

Discussion

Continued response to follow-up surveys is especially critical for prospective cohort studies, especially for the purpose of ascertaining rare outcomes or those with long latency periods (e.g., cancers, neurodegenerative conditions). It is thus critical to adequately characterize response rates and potential sources of nonresponse bias. Over a follow-up period of 3–15 years, approximately 60% of enrolled participants responded to the first follow-up survey, approximately 70% responded to at least one follow-up survey, and 42% responded at every follow-up survey. Additionally, one notable finding was that Veterans were less likely to respond to follow-up surveys and more likely to complete the paper survey.

To our knowledge, no other longitudinal study of service members and veterans has reported factors associated with follow-up survey response, particularly over multiple waves. We were able to identify military factors associated with follow-up survey response in a longitudinal cohort, such as service in the Navy, Coast Guard or Marine Corps (compared with the Army) and active duty service (compared with Reserve, National Guard, separated from the military). These findings mirror results reported for Panel 1 at Wave 2 [20, 22]. The relatively low response observed among certain groups may be attributable to reasons identified in a previous cross-sectional survey of Air Force personnel: lack of time or interest, attitudes toward sponsoring organization, survey length discouraging initiation or completion of survey, and internet access barriers [34]. Veterans may also feel a diminished connection with a military study.

The observed response rate of 60% in this study was lower than the average response rate of 74% in a recent systematic review and meta-analysis of 141 longitudinal cohort studies [35]. Other longitudinal cohort studies such as the Framingham Study [36], Multi-Ethnic Study of Atherosclerosis [37], Women’s Health Initiative Observational Study [38], and the Nurses’ Health Study [39] have reported higher follow-up response rates (80% or greater). Compared with the populations of interest in other cohort studies, service members are relatively young, highly mobile, extensively surveyed, and have rigorous work schedules and requirements [40] that create challenges for repeated surveying over time. This, combined with differences in timing of follow-up and survey methods and operations, resulted in the lower follow-up survey response rate reported in this analysis. Historically, the Study did not track whether participants were not contacted during survey cycles (e.g., due to undeliverable email addresses or postal addresses that returned mail to sender), so we were unable to distinguish between those not contacted from other nonresponders. Panel 1 Study participants responded at a higher rate (Table 2; average of 70%), which is similar to higher response rates reported in other cohort studies. We conducted a sensitivity analysis with models examining response over time stratified by panel to examine whether there were differences in the associations of the characteristics with response (Supplemental Table), which revealed similar associations across panels. Considering the challenges of retaining enrolled service members and veterans in a longitudinal study, the observed 60% response rate (range 43–71%) is remarkable given the large number of participants that the Study was unable to reach. Even with observed attrition, survey data collected from participants have been extensively analyzed and findings reported in over 150 peer-reviewed publications to date on a wide breadth of topics on the long-term health of service members and veterans [19].

Similar to other cohort studies that have observed declining response rates over time, the Study reported a decline in response rates with each follow-up survey. For example, the Swedish Longitudinal Occupational Survey of Health reported a follow-up response rate of 65% in 2006 that declined to 51% in 2016 [41]. The California Teachers Study reported a response rate of 43% at the sixth survey cycle and found that older participants were more likely to participate than younger participants, which is similar to the results for Panel 1 of the current Study [42]. Demographic characteristics related to survey response among Study participants, such as increased educational attainment, female sex, older age, non-Hispanic White race and ethnicity, have also been reported in other cohort studies [7,8,9, 43]. The Study has modified its retention methods over time, such as adding endorsement letters, varying the incentives available, and modifying the type and timing of reminders (Table 1). Although beyond the scope of the analyses presented here, it is important for longitudinal cohort studies to adapt their retention strategies and examine the effectiveness of strategies to reduce loss to follow-up. Finally, although survey attributes (e.g., number, order, sensitivity of survey questions) can impact response rates, it is beyond the aims of this paper to summarize the surveys instruments used during the study time-period. A detailed description of the measures and constructs included on the Study's surveys over time can be found elsewhere [44]. Briefly, a core set of questions and constructs remained consistent over time were utilized in these analyses, but the Study was also able to add measures in response to research priorities, military policy, stakeholder input and scientific advances.

Our findings on the associations between life stressors, health status, and unhealthy behaviors on follow-up survey response align with findings from other cohorts. For example, we reported that self-reported physical health conditions were associated with survey response, which is similar to other studies. Baseline responders in Panel 1 were more likely to have certain outpatient diagnoses compared with nonresponders [45]. Studies of other populations also observed higher rates of health care utilization among survey responders [46, 47], which may indicate that survey responders have higher disease burden. In addition, we observed that reporting stressful life experiences was associated with later follow-up response, which was consistent with an analysis that observed higher levels of stress among a population-based post-disaster follow-up questionnaire [48]. Conversely, unhealthy behaviors were associated with lower follow-up survey response, which has been observed in other health surveys [49, 50].

A notable limitation of this analysis is the fact that the Study historically did not track failed contacts (e.g., return to sender, unknown address, or email bounce backs) and we were thus unable to identify and exclude participants who did not receive survey notifications during the survey cycles. Thus, the response rates reported in this analysis may underestimate the true response rates, as the total eligible population would decrease if we were able to exclude those not contacted during follow-up waves. Moving forward, the Study will improve tracking of changes to participant contact information. Additionally, while these analyses allowed for intermittent responses (e.g., it was permissible to respond to Wave 3 even if they failed to respond to Wave 2), analyses did not examine differences in such patterns, which would be a useful avenue for future research. These analyses were not able to examine the impact of participant engagement methods utilized by the Study to increase response, such as incentives, type and frequency of reminders sent over the follow-up period and cannot speak to the influence these have on follow-up survey response. Future work should examine these methodological factors on survey response. Finally, it was beyond the scope of this paper to examine potential nonresponse bias among this cohort. Earlier work among Panel 1 participants observed that nonresponse had minimal impact on study findings [22] and ongoing analyses are updating these results among a contemporary cohort at more recent time points. Despite these limitations, the Study has many strengths, including enrolling a diverse cohort of service members from all service branches, components, pay grades, and occupations; being the longest running military cohort study with follow-up planned through 2068; and having the ability to merge with other data sources to supplement survey data.

Conclusions

Findings from this analysis indicate that the Study has maintained an adequate response rate over time despite the many challenges associated with surveying military and veteran populations. Additionally, documentation of the different characteristics associated with nonresponse and mode of response is informative for ensuring that appropriate conclusions are drawn over time. These findings can also be instructive for Study strategies for continued participant engagement and may be applicable to other longitudinal cohort studies.

Availability of data and materials

The data that support the findings of this study are not currently publicly available because of institutional regulations protecting service member survey responses, but they are available on reasonable request and will require data use agreements to be developed. The data use agreement would need to be approved by the NHRC HIPAA Privacy Officer and ensure that the dataset is truly de-identified based on Safe Harbor and expert determinations. Requests for data access may be sent to the Millennium Cohort Study Principal Investigator: usn.point-loma.navhlthrschcensan.mbx.nhrc-millennium-cohort-pi@health.mil.

Abbreviations

CTS:

Contingency Tracking System

DMDC:

Defense Manpower Data Center

GEE:

Generalized estimating equation

LOS:

Length of service

NHRC:

Naval Health Research Center

PTSD:

Posttraumatic stress disorder

References

  1. Rothman KJ, Greenland S, Lash TL. Modern epidemiology: Wolters Kluwer Health/Lippincott Williams & Wilkins Philadelphia. 2008.

  2. Jacobsen E, Ran X, Liu A, Chang C-CH, Ganguli M. Predictors of attrition in a longitudinal population-based study of aging. Int Psychogeriatr. 2021;33(8):767–78. https://doi.org/10.1017/S1041610220000447.

    Article  PubMed  Google Scholar 

  3. Mein G, Johal S, Grant RL, Seale C, Ashcroft R, Tinker A. Predictors of two forms of attrition in a longitudinal health study involving ageing participants: an analysis based on the Whitehall II study. BMC Med Res Methodol. 2012;12(1):164. https://doi.org/10.1186/1471-2288-12-164.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Mostafa T. Millennium Cohort Study: Technical report on response in sweep 5 (age 11). 2014. Report No.: 1906929785. Accessed 7 Oct 2022.

  5. Cacioppo JT, Cacioppo S. The population-based longitudinal Chicago Health, Aging, and Social Relations Study (CHASRS): Study description and predictors of attrition in older adults. Arch Sci Psychol. 2018;6(1):21–31. https://doi.org/10.1037/arc0000036.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Drivsholm T, Eplov LF, Davidsen M, Jørgensen T, Ibsen H, Hollnagel H, et al. Representativeness in population-based studies: a detailed description of non-response in a Danish cohort study. Scand J  Public Health. 2006;34(6):623–31. https://doi.org/10.1080/14034940600607616.

    Article  PubMed  Google Scholar 

  7. Young AF, Powers JR, Bell SL. Attrition in longitudinal studies: who do you lose? Aust N Z J Public Health. 2006;30(4):353–61. https://doi.org/10.1111/j.1467-842x.2006.tb00849.x.

    Article  PubMed  Google Scholar 

  8. Stafford M, Black S, Shah I, Hardy R, Pierce M, Richards M, et al. Using a birth cohort to study ageing: representativeness and response rates in the National Survey of Health and Development. Eur J Ageing. 2013;10(2):145–57. https://doi.org/10.1007/s10433-013-0258-8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Volken T. Second-stage non-response in the Swiss health survey: determinants and bias in outcomes. BMC Public Health. 2013;13(1):1–10. https://doi.org/10.1186/1471-2458-13-167.

    Article  Google Scholar 

  10. Kviz FJ. Nonresponse. Wiley StatsRef: Statistics Reference Online. 2014. https://doi.org/10.1002/9781118445112.stat05290.

    Article  Google Scholar 

  11. Peytchev A. Consequences of survey nonresponse. Ann Am Acad Pol Soc Sci. 2013;645(1):88–111. https://doi.org/10.1177/0002716212461748.

    Article  Google Scholar 

  12. Internet/Broadband Fact Sheet: Pew Research Center; 2021 [updated APRIL 7. 2021. Available from: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/?menuItem=6ba9316e-006c-482d-be4b-69feb64c4be8. Accessed: 19 May 2022.

  13. Couper MP. Web surveys: A review of issues and approaches. Public Opinion Q. 2000;64(4):464–94.

    Article  CAS  Google Scholar 

  14. De Leeuw ED. To mix or not to mix data collection modes in surveys. J Official Stat. 2005;21(5):233–55.

    Google Scholar 

  15. Smyth JD. Internet survey methods: a review of strengths, weaknesses, and innovations. Soc Behav Res Internet. 2018:11–44.

  16. Couper MP. The future of modes of data collection. Public Opin Q. 2011;75(5):889–908. https://doi.org/10.1093/poq/nfr046.

    Article  Google Scholar 

  17. Chesbrough KB, Ryan MA, Amoroso P, Boyko EJ, Gackstetter GD, Hooper TI, et al. The Millennium Cohort Study: a 21-year prospective cohort study of 140,000 military personnel. Mil Med. 2002;167(6):483–8.

    Article  PubMed  Google Scholar 

  18. Ryan MA, Smith TC, Smith B, Amoroso P, Boyko EJ, Gray GC, et al. Millennium Cohort: enrollment begins a 21-year contribution to understanding the impact of military service. J Clin Epidemiol. 2007;60(2):181–91. https://doi.org/10.1016/j.jclinepi.2006.05.009.

    Article  PubMed  Google Scholar 

  19. Belding JN, Castañeda SF, Jacobson IG, LeardMann CA, Porter B, Powell TM, et al. The Millennium Cohort Study: The first 20 years of research dedicated to understanding the long-term health of US service members and veterans. Ann Epidemiol. 2021. https://doi.org/10.1016/j.annepidem.2021.12.002.

    Article  PubMed  Google Scholar 

  20. Smith B, Smith TC, Gray GC, Ryan MA, Team MCS. When epidemiology meets the Internet: Web-based surveys in the Millennium Cohort Study. Am J Epidemiol. 2007;166(11):1345–54. https://doi.org/10.1093/aje/kwm212c.

    Article  Google Scholar 

  21. Chretien J-P, Chu LK, Smith TC, Smith B, Ryan MA. Demographic and occupational predictors of early response to a mailed invitation to enroll in a longitudinal health study. BMC Med Res Methodol. 2007;7(1):1–10. https://doi.org/10.1186/1471-2288-7-6.

    Article  Google Scholar 

  22. Littman AJ, Boyko EJ, Jacobson IG, Horton J, Gackstetter GD, Smith B, et al. Assessing nonresponse bias at follow-up in a large prospective cohort of relatively young and mobile military service members. BMC Med Res Methodol. 2010;10(1):99. https://doi.org/10.1186/1471-2288-10-99.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Powell TM. Pre-incentive Efficacy in Survey Response Rates in a Large Prospective Military Cohort. Field Methods. 2023. https://doi.org/10.1177/1525822X231163668.

  24. Porter B, Hoge CW, Tobin LE, Donoho CJ, Castro CA, Luxton DD, et al. Measuring aggregated and specific combat exposures: associations between combat exposure measures and posttraumatic stress disorder, depression, and alcohol-related problems. J Trauma Stress. 2018;31(2):296–306. https://doi.org/10.1002/jts.22273.

    Article  PubMed  Google Scholar 

  25. Holmes THRR. The Social Readjustment Rating Scale. J Psychosom Res. 1967;11:213–8. https://doi.org/10.1016/0022-3999(67)90010-4.

    Article  CAS  PubMed  Google Scholar 

  26. Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA. Psychometric properties of the PTSD checklist (PCL). Behav Res Ther. 1996;34(8):669–73. https://doi.org/10.1016/0005-7967(96)00033-2.

    Article  CAS  PubMed  Google Scholar 

  27. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: American Psychiatric Association; 1994. Accessed 7 Oct 2022.

  28. Spitzer RL, Kroenke K, Williams JB. Validation and Utility of a Self-report Version of PRIME-MD. JAMA. 1999;282(18):1737–44. https://doi.org/10.1001/jama.282.18.1737.

    Article  CAS  PubMed  Google Scholar 

  29. Kroenke K, Spitzer RL, Williams JBW, Löwe B. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiat. 2010;32(4):345–59. https://doi.org/10.1016/j.genhosppsych.2010.03.006.

    Article  Google Scholar 

  30. Dawson DA, Grant BF, Li T-K. Quantifying the Risks Associated With Exceeding Recommended Drinking Limits. Alcohol Clin Exp Res. 2005;29(5):902–8. https://doi.org/10.1097/01.ALC.0000164544.45746.A7.

    Article  PubMed  Google Scholar 

  31. Hirshkowitz M, Whiton K, Albert SM, Alessi C, Bruni O, DonCarlos L, Hazen N, Herman J, Katz ES, Kheirandish-Gozal L, Neubauer DN, O’Donnell AE, Ohayon M, Peever J, Rawding R, et al. National Sleep Foundation’s sleep time duration recommendations: methodology and results summary. Sleep Health. 2015;1(11):40–3. https://doi.org/10.1016/j.sleh.2014.12.010.

    Article  PubMed  Google Scholar 

  32. Tang W, He H, Tu XM. Applied categorical and count data analysis. CRC Press; 2012.

  33. Tsiatis AA. Semiparametric theory and missing data. Springer; 2006.

  34. Miller LL, Aharoni E. Understanding low survey response rates among young US military personnel. Santa Monica: Rand Corporation; 2015.

  35. Teague S, Youssef GJ, Macdonald JA, Sciberras E, Shatte A, Fuller-Tyszkiewicz M, et al. Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis. BMC Med Res Methodol. 2018;18(1):151. https://doi.org/10.1186/s12874-018-0586-7.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Tsao CW, Vasan RS. Cohort Profile: The Framingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology. Int J Epidemiol. 2015;44(6):1800–13. https://doi.org/10.1093/ije/dyv337.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Olson J, Bild D, Kronmal R. Burke GLJGh. Legacy of MESA. 2016;11(3):269–74. https://doi.org/10.1016/j.gheart.2016.08.004.

    Article  Google Scholar 

  38. Women's Health Initiative: Observational Study (OS). Available from: https://sp.whi.org/about/SitePages/Observational%20Study.aspx. Accessed: 13 Oct 2022.

  39. History, Nurse’s Health Study [History of the Nurse's Health Study]. Available from: https://nurseshealthstudy.org/about-nhs/history. Accessed: 13 Oct 2022.

  40. Rosenfeld P, and Office of the Under Secretary of Defense (Personnel and Readiness). Implementation of Department of Defense Survey Burden Action Plan-Reducing Survey Burden, Cost and Duplication. Washington DC, United States; 2016: 0022. Accessed 12 Dec 2022.

  41. Magnusson Hanson LL, Leineweber C, Persson V, Hyde M, Theorell T, Westerlund H. Cohort Profile: The Swedish Longitudinal Occupational Survey of Health (SLOSH). Int J Epidemiol. 2018;47(3):691–2. https://doi.org/10.1093/ije/dyx260.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Savage KE, Benbow JL, Duffy C, Spielfogel ES, Chung NT, Wang SS, et al. Using Marketing Automation to Modernize Data Collection in the California Teachers Study Cohort. Cancer Epidemiol Biomark Prev. 2020;29(4):714–23. https://doi.org/10.1158/1055-9965.EPI-19-0841.

    Article  Google Scholar 

  43. Galea S, Tracy M. Participation Rates in Epidemiologic Studies. Ann Epidemiol. 2007;17(9):643–53. https://doi.org/10.1016/j.annepidem.2007.03.013.

    Article  PubMed  Google Scholar 

  44. Castañeda SF, Belding JN, Kolaja CA, LeardMann CA, Jacobson IG, Rivera AC, et al. Cohort profile update: The US millennium cohort study—evaluating the impact of military experiences on service members and veteran health. Int J Epidemiol. 2023;52(4):e222–e231.

  45. Wells TS, IG Jacobson, TC Smith, CN Spooner, B Smith, RJ Reed, PJ Amoroso, MAK Ryan. Prior health care utilization as a potential determinant of enrollment in a 21-year prospective study, the Millennium Cohort Study. Eur J Epidemiol. 2008;23(2):79–87. https://doi.org/10.1007/s10654-007-9216-0.

  46. Lamers LM. Medical consumption of respondents and non-respondents to a mailed health survey. Eur J Pub Health. 1997;7(3):267–71. https://doi.org/10.1093/eurpub/7.3.267.

    Article  Google Scholar 

  47. Reijneveld SA, Stronks K. The impact of response bias on estimates of health care utilization in a metropolitan area: the use of administrative data. Int J Epidemiol. 1999;28(6):1134–40. https://doi.org/10.1093/ije/28.6.1134.

    Article  CAS  PubMed  Google Scholar 

  48. Hussain A, Weisaeth L, Heir T. Nonresponse to a Population-Based Postdisaster Postal Questionnaire Study. J Trauma Stress. 2009;22(4):324–8. https://doi.org/10.1002/jts.20431.

    Article  PubMed  Google Scholar 

  49. Van Loon AJM, Tijhuis M, Picavet HSJ, Surtees PG, Ormel J. Survey Non-response in the Netherlands: Effects on Prevalence Estimates and Associations. Ann Epidemiol. 2003;13(2):105–10. https://doi.org/10.1016/S1047-2797(02)00257-0.

    Article  PubMed  Google Scholar 

  50. Hill A, Roberts J, Ewings P, Gunnell D. Non-response bias in a lifestyle survey. J Public Health. 1997;19(2):203–7. https://doi.org/10.1093/oxfordjournals.pubmed.a024610.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors appreciate contributions from the Deployment Health Research Department, Millennium Cohort Family Study, Birth and Infant Health Research Team, and Leidos, Inc. We greatly appreciate the contributions of the Millennium Cohort Study participants. In addition to the authors, the Millennium Cohort Study Team includes Anna Baccetti, MPH; Ania Bukowinski, MPH; Felicia Carey, PhD; Clinton Hall, PhD; David Ignacio; Isabel G. Jacobson, MPH; Cynthia A. LeardMann, MPH; Vanessa Perez, MPH; Aprilyn Piega; Anna Rivera, MPH; Rosa Salvatier; Neika Sharifian, MS, PhD; Steven Speigle; Daniel Trone, PhD; Javier Villalobos, MS; and Katie Zhu, MPH.

Funding

The Millennium Cohort Study is funded through the Defense Health Program, U.S. Department of Veterans Affairs Office of Research and Development, and the U.S. Department of Veterans Affairs Office of Patient Care Services under work unit no. 60002. The funding agencies had no part in the study design, collection of the data, analysis of the data, or writing of the manuscript. No financial disclosures were reported by the authors of this paper.

Author information

Authors and Affiliations

  1. Deployment Health Research Department, Naval Health Research Center, San Diego, CA, USA

    Claire A. Kolaja, Jennifer N. Belding, Satbir K. Boparai, Sheila F. Castañeda, Toni Rose Geronimo-Hara, Jennifer L. Walstrom, Beverly D. Sheppard & Rudolph P. Rull

  2. Leidos, Inc, San Diego, CA, USA

    Claire A. Kolaja, Satbir K. Boparai, Toni Rose Geronimo-Hara, Teresa M. Powell, Jennifer L. Walstrom & Beverly D. Sheppard

  3. Army Resilience Directorate, Headquarters United States Department of the Army, Deputy Chief of Staff G-1, Arlington, VA, USA

    Teresa M. Powell

  4. Clinical and Translational Research Institute, University of California San Diego, San Diego, CA, USA

    Xin M. Tu

Authors
  1. Claire A. Kolaja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jennifer N. Belding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Satbir K. Boparai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheila F. Castañeda
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Toni Rose Geronimo-Hara
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Teresa M. Powell
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xin M. Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jennifer L. Walstrom
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Beverly D. Sheppard
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Rudolph P. Rull
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

for the Millennium Cohort Study Team

Contributions

CAK – Conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing – original draft, writing – reviewing and editing, visualization, supervision, project administration. JNB – Conceptualization, methodology, writing – reviewing and editing. SB – Validation, investigation, data curation, writing – reviewing and editing, visualization. SFC – Conceptualization, methodology, writing – reviewing and editing. TRGH – Validation, investigation, data curation, writing – reviewing and editing. TMP – Conceptualization, methodology, writing – reviewing and editing. XMT – Statistical methods for longitudinal regression analysis and editing of Statistical Analysis section. JLW – writing – reviewing and editing. BDS – writing – reviewing and editing. RPR – Conceptualization, methodology, writing – reviewing and editing.

Corresponding author

Correspondence to Claire A. Kolaja.

Ethics declarations

Ethical Approval and consent to participate

The study protocol was approved by the Naval Health Research Center Institutional Review Board in compliance with all applicable Federal regulations governing the protection of human subjects. Research data were derived from an approved Naval Health Research Center Institutional Review Board protocol, number NHRC.2000.0007. All participants provided informed consent at enrollment.

Consent to publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Supplemental Table. GEE effect estimates for follow-up survey response over time, stratified by panel.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kolaja, C.A., Belding, J.N., Boparai, S.K. et al. Survey response over 15 years of follow-up in the Millennium Cohort Study. BMC Med Res Methodol 23, 205 (2023). https://doi.org/10.1186/s12874-023-02018-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-023-02018-z

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us