Challenges to be overcome using population-based sampling methods to recruit veterans for a study of post-traumatic stress disorder and traumatic brain injury

Achieving patient enrollment targets in clinical research is one of the main challenges facing researchers today. The magnitude of the challenge is not well documented but according to one review that examined 41 clinical trials in the US, 34% of the trials recruited less than 75% of their planned sample [1]. Under-recruitment reduces the statistical power of the study and often leads to inconclusive results [2–4]. Remediation of under-recruitment to avoid study closure often leads to costly changes including a re-evaluation of the sample size required, extension of the patient intake period, modification of the inclusion/exclusion criteria, and addition of new study sites [5].

Clinical studies utilizing population-based sampling methods are particularly challenging in this regard as they rely upon the cooperation of a predetermined sample, members of which have no knowledge of the study prior to the first contact. The restricted pool of potential participants places additional importance on engaging each participant as a representative of the population relative to studies using convenience-based sampling methods. For example, a study of Gulf War Veterans [6] used a population-based sampling method to recruit a group of veterans who had been deployed to the Persian Gulf between 1990 and 1991. Twenty-one VA sites participated and reported that only 13.7% of potential participants were successfully contacted. However, population-based sampling methods offer the potential for high external validity such that the results can be applied directly to the defined population [7]. Use of techniques such as random selection of population members and invitation of greater proportions of high-interest subgroups, for example, increase the potential representativeness of such a sample. For these reasons, studies which estimate prevalence rates in the reference population and identify trends or risk factors frequently utilize this method. Population-based sampling methods can be highly effective for studies in which the sampling frame is clearly defined and all required data is recorded in a database or a medical record system. However, when human participation is required, the effectiveness of population sampling methods depends upon the willingness of the individuals to become involved in the study. These individuals may share some similarities which may introduce bias into the sample such as being the most severely impacted by the condition being investigated or having the least access to medical care. Other factors impacting the decision to participate include competing interests and time constraints, as well as concerns about risks and safety [8, 9], fear of additional financial costs [10, 11], concerns about randomization [12], loss of confidentiality [13], religious and cultural reservations [14, 15], and stigma associated with disease [16, 17]. Thus, although the initial random selection in a population-based sample may offer some preliminary protection from bias, responses may still be systematically biased, posing a threat to external validity with weaknesses that may rival convenience based sampling strategies.

The current pilot study entitled “Markers for the Identification, Norming, and Differentiation of TBI and PTSD” (MIND study) was designed as a clinical sub-study of a longitudinal study “National Health Study for a New Generation of US Veterans” (“New Generation Study”). The New Generation Study [18] was one of the first epidemiologic studies to examine the health outcomes of veterans returning from recent conflicts in Afghanistan and Iraq —Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF). The study began in 2009 and involved web-based, postal, or computer assisted telephone interviews to survey a stratified random sample of 60,000 veterans with documented military service between October 2001 and June 2008. The population sample included 30,000 veterans who had deployed and 30,000 non-deployed veterans who had served during the same era. Recruitment was achieved via a 72-item survey of veterans who had left combat theaters (reserve and National Guard) or separated from active duty by June 2008. The sample was stratified by deployment status, gender, unit component (active duty, reserve, National Guard), and branch of service (Army, Air Force, Navy, Marines). Women veterans were oversampled to comprise 20% of potential participants, reflecting an increased awareness of women’s unique health concerns [19]. The primary objective of the New Generation Study was prevalence estimation of health concerns specific to this cohort including traumatic brain injury (TBI) and posttraumatic stress disorder (PTSD). The findings revealed that those who deployed to Afghanistan, Iraq, and other countries had higher estimated prevalence of TBI, PTSD, and co-occurring TBI/PTSD than those who were not deployed (A. Schneiderman, personal communication). These results and other studies [20–22], reflect the now widely held view that TBI and PTSD are important consequences of the OEF/OIF/Operation New Dawn conflicts.

The MIND study was designed to be conducted at the War Related Illness and Injury Study Center (WRIISC). The main aim of the MIND study was to determine the prevalence of TBI and PTSD, to identify sensitive and specific measures for them, and to develop prediction models. The WRIISC, located in three sites (Washington, DC, Palo Alto, CA and East Orange, NJ) focuses on post-deployment health concerns of veterans for the Veterans Health Administration (VHA). The WRIISC provides in depth clinical assessment and diagnostics for unexplained symptoms, chronic multi-symptom illness, or treatment-resistant illness or injury that is putatively related to or exacerbated by their deployment.

The current study was conceived as a MIND pilot with a view to conducting a larger study involving 800 veterans. Participation in the MIND pilot study involved travelling to one of the three WRIISC study sites for a full day of assessment that included cognitive and sensorimotor testing, brain imaging and psychiatric evaluation. Since human studies using population-based sampling methods frequently have relatively low recruitment rates, one of the primary objectives of the current study was to demonstrate that adequate numbers of individuals from the New Generation Study could be contacted and successfully recruited. For the MIND pilot study, potential participants were recruited via the US Postal Service with subsequent direct telephone contact; the effectiveness of these methods was evaluated.

A secondary objective of the current study was to examine the association between recruitment, screening, and enrollment and the individuals’ characteristics from the New Generation Study. Some individuals may be less likely to volunteer for clinical research or participate in the health care system more generally either as a consequence of their medical symptoms or as a result of the stigma associated with illness (for discussions, see [16, 17]). Accordingly, we also examined recruitment rates from four self-reported diagnostic categories from the New Generation Study (probable TBI, probable PTSD, probable TBI and PTSD, and probable neither condition).

A total of 13,162 deployed Veterans responded to the New Generation survey. Of these respondents, 11,337 met the age criteria for the MIND pilot study and were considered for inclusion in the current study (see screening flowchart, Figure 1). Further, selecting only those who had a high probability of living within 60 miles from one of the study sites yielded a total of 445 potential participants. This sample (n = 445) was compared to the full sample of deployed veterans from the New Generation Study (n = 13,162). Despite the exclusion of veterans >50 years old, the two samples were not statistically different in age or gender (p’s > .05). A chi-squared test revealed a significant difference between the two samples in putative diagnostic groups (χ2 (3) = 47.8, p < .001). Examination of residuals showed the MIND pilot sample had significantly fewer individuals with a positive screening result for TBI (11% vs. 16%) or PTSD (9% vs. 16%) relative to the New Generation Sample (p’s < .05). Furthermore, the MIND pilot sample had significantly more individuals with a positive screening result for combined TBI and PTSD relative to the New Generation sample (15% vs. 7%) (p < .05). The proportions of individuals with neither TBI nor PTSD were statistically similar across the MIND pilot and New Generation studies (65% vs. 61%, respectively).

Of note, only two of the participants were enrolled had proactively contacted the study site after receiving the Advance Letter. In addition, the safety plan and response algorithm were not needed on any call at any of the three study sites.

Twenty-four veterans were recruited into a multi-site, multi-modal, observational pilot study of TBI and PTSD from a total pool of 11,337 and a likely pool of 445 participants. Between-sample comparisons revealed the two samples differed in putative diagnostic category taken from the New Generation Study. However, these statistical differences were relatively modest (the largest difference between the current study and New Generation Study was in the combined TBI and PTSD groups; 15% vs. 7%, respectively), and were driven by the large numbers involved. Challenges associated with locating, contacting, screening, and enrolling emerged. The selection of the population-based sample was based upon having previously participated in the “New Generation Study” and having been deployed to OEF/OIF conflicts. Of the 445 potential participants who were identified from records as living proximally (≤ 60 miles) to one of the three study sites, we successfully contacted 146 (contact rate = 33%). Of these, 24 were enrolled into the study, representing a recruitment yield (enrolled/total identified) of 5.4%. Factors restricting enrollment were potentially related to issues inherent to clinical research including difficulties achieving or maintaining contact with potential participants (possibly due to outdated contact information), challenges in recruiting individuals with TBI alone, the requirement to travel to the study site, and strict exclusionary criteria. Enrollment may also have been limited by the method of telephoning prospective participants who had not specifically asked to be contacted by a screener.

The recruitment yield (enrolled/total identified) of 5.4% achieved in this study is consistent with other published studies involving large civilian multi-site studies which have reported yields ranging from 0.5% [23], 1.1% [24], 2.6% [25], 5.5% [26], to 13% [27]. However, none of the aforementioned studies involved veterans or used population-based sampling methods and therefore generalization to the current study is not straightforward. Recruitment yields similar to ours have been reported for veteran studies using population-based sampling. For instance one study of Gulf War veterans [6] involving 21 VA study sites reported an overall recruitment yield of 3.5%. However, another larger study involving population-based sampling of Gulf War veterans by Eisen (2005) [28] reported a total sample of 2189 veterans enrolled from a population of 4879, representing a recruitment yield of 44.9%. The better recruitment yield in the latter study relative to ours is not likely due to demographic differences between the populations, which were quite similar. Rather, the difference probably reflects divergences in resources– for example, the Eisen (2005) study provided hotel accommodation for all participants. Of additional note, the Eisen (2005) study sample had a contact rate of 85.8%, compared to the 33% rate in the current study. This higher contact rate may reflect factors such as the relative accuracy of contact information, temporal shifts in awareness and willingness to participate in research studies, changes in telecommunication use between these studies (mobile phones vs. land lines), a lack of accurate mobile phone directories, differences in the duration and impact of the conflicts involved that may influence motivation to take part in subsequent research studies (i.e., months of engagement in the Eisen study versus years in the current study), media coverage of the issues related to Gulf War illnesses and the intense scrutiny of health issues that began shortly after the 1991 Gulf War.

In the present study, one of a number of factors associated with recruitment was our ability to contact potential participants. Of the 445 individuals identified as being eligible for the study, 67% of them were not contacted for a variety of reasons. An even larger proportion of potential participants (84%) were contacted but failed to enroll, suggesting the study population was either unable or unwilling to engage in what was offered. The largest group in this category stated they were “not interested” in the study (47%, n = 37). The reasons behind their unwillingness were not recorded but may have been due to perceived low compensation rate ($200), difficulty in committing to a full day of study, no medical treatment being offered, or a combination of factors. One solution would be to increase compensation rates. A recent review [29] notes the lack of agreement on standards of participant compensation by IRBs, who are primarily obligated to minimize coercion or undue influence. Even if recruitment was improved by increasing participant compensation, it should be noted that this may have biased the study sample. Assessing whether compensation significantly influenced epidemiological sample characteristics would be hard to determine. It was also notable that only two potential participants contacted the study sites to inquire about participation without first being called. This observation demonstrates the importance of follow-up telephone contact with potential participants.

Analysis revealed that fewer TBI cases were screened than expected. However, if these individuals were grouped together with those with TBI and PTSD, this effect disappeared. The reasons for this are not clear, but may be due to the small sample size of the “pure” TBI group (n = 4) rather than their diagnostic category. Individual characteristics such as age, gender, and distance from study site were not associated with a veteran being screened. Enhancing numbers of patients in specific subgroups who are less likely to engage would not undermine the value of population-based sampling, which should have as a primary concern recruitment of sufficient numbers of participants to enable group comparisons [7]. A recent review [30] considered the nature of barriers to participation in mental health research and identified many factors including transportation difficulties, distrust and suspicion of researchers, and the stigma attached to mental illness. The authors concluded that strategies to overcome these barriers included assistance with travel, avoiding the use of stigmatizing language in study materials and an emphasis on education about the disorder under investigation. Considering the recent increase in prevalence of PTSD and TBI in veterans [31, 32], more studies that systematically investigate recruitment strategies are needed in veteran populations with these disorders.

The observation that major factors in recruitment were the ability to contact participants and enroll them suggests a need for improved methods of communication with potential participants. Researchers are encouraged to consider novel approaches to establish contact, develop rapport, and secure a commitment to participate. Such approaches include communication technologies such as telehealth, internet, videoconferencing, email, and text messaging [33]. In this regard, it should also be noted that only 3.4% of potential participants lived within a 60-mile radius of one of the three study sites. Recruiting the remaining 96.6% of potential participants into the study would take considerable resources to bring them to the study center and to accommodate them. If these resources were not available, alternative study designs could be considered including the use of communication technologies instead of site visits. Communication technologies therefore offer an important alternative to an on-site evaluation, especially for participants who live remotely from study sites.

The current analysis represents a detailed report of recruitment metrics for a population-based sampling strategy. The VHA, Centers for Disease Control, National Center for Health Statistics, and Department of Defense all have several ongoing studies using population-based sampling methods but few published data exist describing the challenges of recruiting participants and the characteristics associated with contact, screening, and enrollment. One exception is the Millennium Cohort Study, which is a 22-year longitudinal study launched in 2001 to determine how military service affects long-term health [34]. The study uses several innovative procedures to contact, communicate with [35] and retain [36] large numbers of veteran participants. Our report takes advantage of clear criteria for each stage of the enrollment process from a well-described, representative sample allowing for more detailed assessment of bias in responses. One significant limitation of the current study is its small sample size. Despite this, we believe that our main findings can better inform the conduct of future clinical investigations using population-based sampling.

In conclusion, the screening methods, recruitment procedures, and enrollment numbers from this study using population-based sampling can help guide recruitment for future studies. Clearly, to achieve adequate sample sizes, a large pool of potential participants must be available, and adequate resources must be devoted to enhance participation. Exploration of solutions that overcome barriers to participation, including travel time, lack of interest, and failure to keep a study appointment, is warranted. Some of these issues may be universally relevant, while others (some unknown) may arise in particular studies.

The MIND Study Group: Julie C. Chapman, Aaron Schneiderman, Matthew J. Reinhard, J. Wesson Ashford, Drew A. Helmer, Gudrun Lange, Helena K. Chandler, Maheen M. Adamson, Peter J. Bayley, Jamie M. Zeitzer, Jerome Yesavage, Marc R. Blackman.