We collected all measures used in the field study (described above). In addition, we collected data on the following outcomes one year after Visit 1, emergency room (ER) use within the prior year, housing stability, change in employment status, change in marital status, HRQL, and new diagnosis of mental health condition. We hypothesized, based upon the literature, that these key 1 year outcomes might be associated with community reintegration scores.
ER use in the prior year was collected using two methods: self-report by the Veteran and abstraction of the VA medical records using the Austin Automated Database. Housing Stability was determined by asking respondents to indicate the number of times that they had moved in the past year (number of moves). For data analysis this variable was categorized into three classes: no moves, one move, and two or more moves.
Change in Employment Status was categorized into three groups: improved, worse, and no change. Improved Employment Status was defined as change from Visit 1 to Visit 2 from not working to working, i.e. unemployed or not working due to medical hold at baseline to working full or part-time at 1 year follow-up. Worse employment was defined as change from working to not working, i.e. working full or part time at baseline to being unemployed or not working due to medical hold at 1 year follow-up. We classified subjects as having no change in employment if their overall employment status (working or not working or retired) stayed the same between visits.
Change in Marital Status at Visit 2 was categorized into three groups: newly married, no longer married and unchanged. Newly married status was defined as change from any non-married group at baseline to married at 1 year follow-up. The no longer married status was defined as change from married at baseline to any non-married group at 1 year follow-up. We classified any subjects who remained married or remained in the non-married groups as having an unchanged marital status.
Health Related Quality of Life (HRQL) was assessed using the 12 question SF-12 (embedded in the SF-36 items at Visit 1, and asked by themselves at the follow-up visit) which evaluated two global health constructs: the physical component summary (PCS) and the mental component summary (MCS) [24].
New Diagnosis of Mental Illness was determining using mental health diagnoses codes abstracted from the subject’s medical records at two time points (Visit 1 and Visit 2). We categorized the ICD-9 codes into 21 diagnosis groups as follows: Presence of Adjustment Disorders, Affective Disorders, Alcohol Dependence Syndrome, Anxiety Disorders, Attention Deficit, Drug Dependence, Drug Psychoses, Eating Disorders, Gender Identity Disorders, Impulsive-Control Disorders, Mild Mental Retardation, Mood Disorders, Neurotic Disorders, Non-Dependent Abuse of Drugs, Personality Disorders, Post-concussion Syndrome, Psychotic Disorders, PTSD, Sexual Dysfunctions, Somatoform Disorders, and Unspecified Disorders. These diagnostic groups were based upon the classification system as defined by the Diagnostic Statistical Manual-IV (DSM-IV) [25]. We then modified this classification system so as to include categories comprised of diagnoses that we expected to be prevalent and/or important in this population (e.g. PTSD, alcohol abuse or dependence, substance use or dependence). If any of these disorders were present at Visit 2 but not present at Visit 1 we considered the subject to have a new mental health condition.
Administration study
Sample
The sample for the administration study was a convenience sample of 50 Veterans (ages 18–59) from the PVAMC who had not participated in the earlier studies.
Data collection
Subjects in the CRIS-CAT Administration Study participated in two data collection visits conducted within 1 week. Subjects completed the CRIS-CAT at both visits using newly develop CAT software. Demographic data were collected at the first visit. In order to assess respondent burden we tracked number of items administered in each scale within the software database.
Statistical analyses
Data from the field study were used to examine reliability of the CAT, and test concurrent and known groups validity. We examined descriptive characteristics of field study participants, and compared characteristics for each subsample.
Reliability represents the degree to which the differences across subject scores are due to real differences in scale (true variance) as opposed to measurement error. By assuming the latent trait under the partial credit model has a standard normal distribution, the conditional reliability of the CRIS-CAT scales was examined as 1/(1 + (standard error)^2 ), which was a function of latent trait level. The standard error of the person score estimates was derived from the information function. Any section of the reliability function that was greater than 0.70 served to indicate adequate reliability [26].
Concurrent validity of the CRIS-CAT was examined by exploring Pearson product correlations of the CRIS-CAT scales with existing measures that assess specific community reintegration dimensions. We expected to find similar correlations to those reported in our earlier research on the CRIS fixed form measure. (i.e. correlations between the CRIS fixed form measure and the 36-Item Short Form Health Survey scales of role physical, role emotional and social functioning of 0.25-0.80 [13, 14] and a correlation between the CRIS scores and QOL of 0.57-0.79.) [13].
We performed ANOVAs to examine whether the CRIS-CAT scores differed in Veterans from the 3 groups as expected: the homeless group, the working group and the OEF/OIF group.
Statistical methods - cohort
Data from the cohort study were used in the examination of the CRIS-CAT’s predictive validity. We examined the CRIS-CAT measure’s ability to predict key one year outcomes including change in marital status, employment status, housing stability, self-reported ER visit frequency, frequency of VA ER use, SF-12 scores, and new diagnoses of mental health condition. Likelihood of change in marital status was modeled using the three-level category of change in marital status. Similarly, likelihood of change in employment status between Visit 1 and Visit 2 was modeled using the three-level category of change in employment. We ran 3 separate multinomial regression models to predict change in employment status using three CRIS-CAT subscale scores at baseline as independent variables. Similarly, we ran 3 separate multinomial regression models to predict likelihood of housing stability.
We developed three separate logistic regression models to predict the likelihood of any self-reported ER visits for participants in the cohort. We also examined 1 year ER use for all field study participants as documented in the VA medical record. We ran three separate logistic regression models to predict the likelihood of any recorded ER visits based on the three CRIS subscale scores, using data abstracted from the Austin Automated Database. For the cohort participants, we ran three separate logistic regression models to predict the likelihood of the diagnosis of any new mental health condition based on the three CRIS-CAT subscale scores using the abstracted medical data. CRIS-CAT scores were used to predict SF-12 scores one year later while controlling for Visit 1 SF-12 scores in the linear regressions.
Statistical methods - administration study
Data from the 50-person CRIS-CAT administration study were used to assess respondent burden, test-retest reliability, estimate the minimal detectable change. We calculated CRIS-CAT scores for each visit and examined the number of items used by the CAT to estimate scores. Reliability of the 3 CRIS-CAT scales was evaluated using Shrout & Fleiss intraclass correlation coefficients which were calculated using SPSS (PASW Statistics 18). ICC(2,1) is a two-way mixed effects single measure of reliability, where the target is a random effect, the number of measurements on each target is a fixed effect, and the unit of analysis is the individual measurement instead of the mean of measurements [27].
Using a classical test theory approach, ICCs were then used to calculate the Standard Error of the Measurement (SEM) and Minimum Detectable Change (MDC). The SEM is the standard error in an observed score related to measuring with a particular test that obscures the true score. It is estimated by the standard deviation of the instrument multiplied by the square root of one minus its reliability coefficient [28]. In statistical terms, the MDC, also called smallest detectable change or smallest real change shows which changes fall outside the measurement error of the health status measurement (either based on internal or test-retest reliability in stable persons) [29, 30]. The Standard Error of the Measurement (SEM) and MDC were calculated at 95% confidence and 90% confidence levels [31].
To assess potential bias due to loss to follow-up we examined characteristics of participants in the cohort study and compared characteristics of eligible subjects lost to follow-up and those with complete data at one year.