Web-based computer adaptive assessment of individual perceptions of job satisfaction for hospital workplace employees

1. Standardized assessments of health status

The length and complexity of many fixed-form instruments are problematic and raise concerns about both the burden on respondents and the administration costs [14, 15]. Conversely, the shift to shorter fixed-form versions of patient-reported instruments has raised concern over possible resultant losses of precision and reliability [16] as well as insensitivity to clinically meaningful changes [17].

2. CAT reduces the burden on patients and diagnosticians

Studies have shown that computer adaptive testing (CAT) can save time and alleviate the burdens on both examinees (e.g., patients) and test administers (e.g., diagnosticians), as compared to traditional computer-based or pen-and-paper assessments [18–21]. CAT, which is based on item response theory (IRT)[21], is a test-administration method that tailors the assessment to the latent-trait level of the examinee. Only items that are neither too hard, nor too easy, are administered. IRT-based CAT has attracted much attention because of its better control of item exposure and lower cost of item development for medical and healthcare professionals [22, 23]. CAT can efficiently collect data from examinees and identify the degree of severity of each symptom of disorder. Thus, CAT overcomes the shortcomings of the two traditional forms of standardized assessments in clinical settings, both the burdens associated with lengthy assessments and the loss of precision and reliability of shorter fixed-form assessments.

3. Item-by-item questionnaire analyses

Although CAT and the aforementioned lengthy and short assessments are all used to obtain composite scores for measurement, item-by-item analyses are also common in research reports. In item-by-item analyses, perception changes between groups are compared across items. One item (or one composite score) is assessed at a time [22] by traditional one-way ANOVA, by a t-test, or even by Pearson's chi-square test [6]. Recently, item-by-item skewness analysis by a bootstrapping procedure has been reported as effective for identifying quality-of-life concerns of patients [24]. The problem we face when using CAT is how to obtain the specific responses interacted by item and person because only individual measures were stored in the CAT module.

4. Study Objectives

This study aimed to answer two questions: (1) Can a CAT be used via a website to facilitate more efficient response collection for the self-evaluation of job satisfaction by workers? and (2) Is it possible to generate data using the Rasch model (1960) to assess achievement through item-by-item analysis?

1. Study participants and research instrument

The study was conducted in a 1,200-bed hospital in Taiwan. One-tenth of hospital employees were randomly enrolled for surveys of job satisfaction in September of 2008 and 2009. The self-administered 37-item Job Content Questionnaire (JCQ-37) was designed for use on a website via NAT (non-adaptive testing) in 2008 and CAT assessments with 24 items in 2009 was provided to workers. The response rates were 92.6% and 91.1% for 2008 and 2009, respectively. This study was approved and monitored by the administration units of the hospital.

2. Instrument selection

3. Web-CAT assessment

We designed a CAT questionnaire that complies with rules and criteria for CAT-based testing on the internet http://www.healthup.org.tw/irt_test4/irt_start.htm.

Based on person-separation reliability (e.g., Rasch_rel, similar to Cronbach's alpha) calculated from the job-satisfaction survey conducted in 2008, the CAT termination rule for measurement of standardized error (MSE) is determined by formula (1)[43].

(1)

where, SD _x represents the standard deviation of person measures estimated in 2008. We also defined another termination rule for CAT so that the minimum number of items required for completion of the CAT questionnaire was 10. The initial item was selected according to the overall job-satisfaction level designated by the examinee's response at the beginning of the CAT questionnaire. When an examinee rated the CAT questionnaire after completing three items on the web, the computer could update the estimate of the examinee's satisfaction level (ability) after each subsequent item's answer was complete. The provisional-person measures was estimated by the iterative Newton-Raphson procedure [18, 44], a brief algorism was presented in Additional file 1. The next item selected was that with the most information about the provisional-person measures in the remaining unanswered items.

4. Generation of person responses across items

Only individual measures were stored in the CAT module. We should thus generate appropriate responses for each person and each item so that item-by-item comparisons can be made over several years. A standard item-response generation method, as used in previously published papers [24, 45–48], was conducted using the Rasch rating scale model. An Excel routine was demonstrated in Additional file 1.

1. Descriptive Statistics

2. Unidimensional validity and the identification of concerns

The principal components analysis of the residuals demonstrated that the 24-item scale accounted for 52.2% of the raw variance explained by the measures. The first contrast had an eigenvalue of 1.8 (less than 3 [41]) and accounted for 4.2% (less than 5% [40]) of the total variance, suggesting that the 24-item scale can be regarded as substantially unidimensional. A parallel analysis also indicated that the 24-item questionnaire regarding job satisfaction measures a common entity. These findings indicate that these 24 items measured a single construct for job satisfaction. The three intersection parameters (also called the step calibrations [48]) under the Rasch rating scale model for the 24-item questionnaire were set at -4.16, -1.50 and 2.66 logits. These thresholds are congruent with the guidelines proposed by Linacre [49] as follows: (1) average measures advance monotonically within each category, (2) step calibrations advance, (3) step difficulties advance by at least 1.4 logits and (4) step difficulties advance by less than 5.0 logits.

3. Web-CAT performance

Based on the finding of a unidimensional construct in Table 2, we embedded the stop rules of SEM = 0.68 and the minimal corresponding item length = 10 into the CAT questionnaire. The Web-CAT is at http://www.healthup.org.tw/irt_test4/irt_start.htm.

(2)

where P _nij and P _ni(j-1) are the probabilities of being scoring j and j - 1 in item i for person n, θ _n is the ability of person n, δ _i is the difficulty of item i, and τ _j is the j-th step difficulty. (see Additional file 1). (3) Outfit MNSQ for CAT was determined by the average squared residuals (i.e., squared observation minus the expected score and then divided by the variance, see Additional file 1) across all items. The outfit MNSQ terminated the CAT procedures once the item length was longer than 10 or the MNSQ was greater than 10. An outfit MNSQ of greater than 2.0 was referred to the aberrant responses given by the person [50](Figure 1). We assumed that aberrant respondents, participants' guessing, inattentiveness, carelessness and coaxing could be caused by fatigue, misunderstanding, or a poor fit of the examinee for evaluation based on item-response theory [51, 52]. Z-scores beyond +/- 1.96 were marked on observation with a symbol * to designate that an unexpected response was given to a specified item (p < .05).

4. Item difference between years

1. Features

2. Study strengths

3. Study limitations

Because many studies have shown that CAT can save time and alleviate burdens on examinees compared to traditional non-adaptive computer-based or pen-and-paper assessments [18–21], we thus did not demonstrate the efficiency and precision of CAT as compared to non-adaptive assessments. Obtaining high quality examinee feedback from CAT assessments is essential to produce accurate results, and adequate training is required to facilitate an efficient health-promotion system. Without such results and training, it will be extremely difficult for readers to understand the computation of outfit and infit statistics with regard to probability and outfit MNSQ disclosed in Table 3. In this study, the job-satisfaction questionnaire was used as a tool to collect information about workers' perceptions using the CAT feedback system. Accordingly, diagnosticians may need training to interpret the results of the data adequately.

4. Problems in application and daily use