Agreement between pre-post measures of change and transition ratings as well as then-tests

A valid measurement of change is a prerequisite for evaluating health outcomes. From a clinical perspective, observing change over the course of a patient’s disease is a crucial part of the treatment process. From a research perspective, it is important to know whether, to what extent and how any observed changes are causally related to medical interventions. On the level of the healthcare system, the effectiveness of healthcare measures has to be demonstrated or continually monitored, e.g. for quality assurance programmes.

Different approaches for measuring change have been developed. A) Clinicians often rely on ‘direct’ measures of change, also referred to as ‘transition ratings’. To assess change directly, clinicians either form an impression of how much the patient’s complaints or symptoms have changed, or they solicit the patient’s judgment of this change directly (e.g. “Has your leg pain improved, stayed the same, or worsened?”). B) In clinical studies, however, ‘indirect’ measures of change are preferred. To determine change indirectly, researchers assess a patient’s status at different points in time and obtain measures of observed change by calculating the respective differences (deltas) between measurement points. C) Another important approach for measuring change has emerged in quality-of-life research. It has been shown that patients’ response shifts may bias results of clinical studies if the internal criteria or metric they base their responses on change in the time interval between the two responses used to calculate change [1–3]. The ‘then-test’ method has been developed to take this response-shift phenomenon into account. Patients are asked at t1 not only to rate their current status at t1 but also retrospectively to rate their status at t0 (hence the ‘then-test’ designation). The assumption is that response shift is eliminated because the patient will have used the same metric for both the t0 and t1 ratings since they were assessed at the same time. The researcher then calculates the delta between the t1 rating and the retrospective t0 rating. We will refer to this type of change measurement as ‘quasi-indirect’ [4]. Figure 1 depicts these three common approaches to measuring change and provides examples for each of them.

From a naïve perspective, one might assume that all three approaches to measuring change should yield similar results because they should all measure the same change process. However, a number of studies have only been able to detect low to moderate correlations between indirect and direct (transition ratings) measures of change. Although studies exist reporting correlation coefficients in the r = 0 to r = 0.40 range [5], there are also studies reporting correlation coefficients well above r = 0.60 [6] and even above r = 0.80 [7]. There are very few studies evaluating the correlation between indirect and quasi-indirect measures of change, despite a considerable number of studies on the response-shift phenomenon and the then-test approach [3]. In one sample of frail, elderly patients accessing community-based rehabilitation services, the correlation coefficients between indirect and quasi-indirect measures of change were moderate to low: intra-class-correlations of ICC = 0.41 and ICC = 0.21 were reported for the EQ-5D utility score and general health perception (visual analogue scale), respectively [8].

To our knowledge, there are no studies available that would explain this lack of agreement^a. However, there are a number of theoretical reasons why different biases might affect the different approaches to measure change.

Recall bias: With direct measures of change, patients have to recall a specified prior state and compare it with their present state in order to come up with a transition rating. With quasi-indirect measures of change, patients have to recall a specified prior state and give it an explicit rating. These memories of past states are known to be biased [9, 10]. That being said, there are empirical studies that have found substantial associations between pre-status reports and “then-test” measures [6, 7].

Present state effect: It has been postulated that patients use their present state to judge whether or how much they have changed, i.e. patients’ assessments of change would be unduly influenced by their present states. For example, if a person feels well at the time of measurement, he might infer that his status has improved, or vice versa, without actually having taken his prior state into account. In fact, transition ratings have been shown to be highly correlated to post-treatment ratings [11, 12]. Guyatt et al. have argued that if an assessment of change using transition ratings is unbiased, then post scores and pre scores should correlate with direct measures of change, with equal magnitude and opposite direction [7]. Empirical studies have repeatedly shown transition ratings to be more strongly correlated to post-status scores than to pre-status scores [9, 13]. That said, pre-status scores have usually been able to account for additional variance in transition ratings when used second to post-status scores [7].

There is a major drawback in the way that current studies are interpreted. Transition ratings are usually elicited on an aggregate level, i.e. single items are used to cover a whole domain of a construct of interest. For example, general transition ratings (also called ‘global perceived-effect scales’; cf. [8]) are used routinely to measure change directly (e.g. [9–11]), while multi-item scales are routinely used when measuring change indirectly. There are different theoretical reasons for why change measurements based on multi-item scales could differ from those based on general or aggregate transition ratings. We assume that there could be a substantial difference between the constructs the multi-item scales are intended to represent and those constructs that are evoked in a patient confronted with a single general term in a transition rating, i.e. these multi-item and general item assessments may refer to different aspects of the construct because of their differing levels of abstraction [5, 14]. For example, a multi-item scale measuring functional disability and a single general question on functional disability might not evoke the same associations in the patient being questioned. From a psychometric perspective, multi-item scales should be more reliable than single-item measures [15].

The aim of the present study was to analyse the level of agreement between indirect and direct as well as indirect and quasi-indirect measures of change by using multi-item scales for all three approaches (including direct measures). Specifically, we aim to analyse 1) the level of agreement between direct and indirect as well as quasi-indirect and indirect measures of change, 2) how recall bias might account for differences between the performance of direct and quasi-indirect measures of change, and 3) how much the present-state effect affects direct measures of change.

We re-analysed a data set originally collected to investigate the prognostic validity of different approaches for measuring change [16]. The original study had been motivated by the decision to use direct measurements of change in the quality-assurance programme for medical rehabilitation clinics under the purview of German statutory pension funds [17].

Sample

Five rehabilitation clinics located in the German federal state of Schleswig-Holstein recruited study participants in 1999 (August to November) using the following inclusion criteria: (a) between 18–60 years old, (b) German speaking, (c) participating in a rehabilitation programme for either a musculoskeletal (ICD-9 710 to 739.9) or cardiovascular disease (ICD-9 393 to 429.9) at one of the five cooperating clinics.

Four hundred and twenty-six patients gave written, informed consent to participating in the study. They filled out a self-administered questionnaire both pre (before) treatment (t0; responding: n = 426, 100%) and post treatment (t1; responding: n = 397, 93.2%). At t1, all participants were randomised and asked to fill out one of two additional questionnaires, which were either designed to measure change directly (transition ratings) or quasi-indirectly (the “then-test” approach). In each clinic, participants were randomly allocated 1:1 either to group 1 (reporting change directly; responding: n = 194) or group 2 (reporting their pre status retrospectively; responding: n = 201). The standard duration of rehabilitation was three weeks, which represents the difference between t0 and t1. Figure 2 illustrates the study design.

The original study also included additional measurement points at follow-ups 6 and 12 months after t0 for the purpose of analysing predictive validity of the three different approaches of change measurements. These results are not part of the present analysis.

We re-analysed a data set that had originally been collected to investigate the prognostic validity of different approaches to measuring change in the context of rehabilitation treatment^b. We focused on three self-reported outcome domains (sleep, physical functioning, somatisation) for which the three different approaches to measuring change of interest to us were based on scales with equal numbers of items and equivalent content. To our knowledge, there has only been one other study to analyse the use of a multi-item approach in transition ratings in direct comparison to indirect change measures [15]. Indirect and quasi-indirect change measurements both yielded comparable results measurements, indicating that recall bias does not necessarily affect quasi-indirect change measurements and that the quasi-indirect method has the potential to serve as a substitute for the indirect method (pre-post measurements). Direct change measurement reflects different aspects of change compared to indirect and quasi-indirect change measurements but is not necessarily biased by patients’ present states.

Previous studies have indicated that effect sizes as found using direct change measures are systematically larger than those found using indirect measures of change [13]. This was not the case in our study, however. Therefore, it remains to be shown, in a future head-to-head comparison of general transition items with multi-item transition scales, whether or not the effect reported in previous studies – of direct change measurements overestimating effect sizes – is attributable to the general nature of direct change measures.

Indirect and quasi-indirect measurement of change yielded comparable results in our study. The agreement between pre status and retrospective pre status (“then-test”) was notably high. Thus, for multi-item scales the retrospective pre test might have the potential to measure the same construct as the pre status. Recall bias did not appear to play a major role in this regard. In fact, quasi-indirect assessments are superior to indirect measurements of change in predicting change in physiological indicators in AIDS patients [26]. Quasi-indirect change assessments are not only a feasible approach for estimating the amount of response-shift in quality-of-life studies, but may also come to play an interesting role in clinical studies and quality-assurance programmes. Quasi-indirect measurements are made by asking patients two questions at one time point (after an intervention). In contrast, indirect measurements require contact to be made with the patient two separate times, therefore requiring more resources, and perhaps also causing more patients to drop out. Quasi-indirect and direct measures of change are thus more economical to obtain than indirect measures of change. However, response-shift literature warns us not to be overly optimistic, as no variables have been identified that consistently moderate different degrees of response-shift [5]. Before starting to substitute the quasi-indirect approach of measuring change for the indirect approach in different applications, it is essential to understand these moderating factors. The pre–post interval should be short enough that the patient is able to remember the pre state. In the case of our study, this time interval was about three weeks. Also, a patient should relate his or her responses to a specified point in time which is meaningful to him, e.g. events that are salient to the disease trajectory of a patient. In our study it was the admission to a rehabilitation clinic. To this end, it may prove valuable to use multi-item scales and to avoid single general assessments. A future challenge to research would be to test these moderator variables and also identify additional conditional factors that would allow both indirect and quasi-indirect change measurements to produce equivalent results.

In comparison to other studies [4, 27], we found the correlation of direct to indirect measurement of change to be substantive. The correlation coefficients were not as high as those reported by Middle et al. [6] (canonical correlation R _c = .63) or even Guyatt et al. [7] (correlation coefficients from r = .56 to r = .82), but higher than the values reported by Kohlmann and Raspe [4] (correlation coefficients from r = .10 to r = .37). The strength of this relationship might be interpreted as an indication that both direct and indirect measurement approaches capture the same change process, albeit different aspects of it.

Various studies have reported direct ratings to be more strongly related to a patient’s status at the time of measurement than to change as assessed using indirect measures, a phenomenon which is referred to as present-state bias [8, 15]. These findings, if true, imply that direct measurements of change are highly influenced by a patient’s state at the time of measurement and that these direct measurements are only minimally influenced by those aspects of change that are reflected in indirect measurements. Applying the analytical approach used by Guyatt et al. [7], we were able to show, as expected, that performing a regression of transition ratings on post status and pre status yielded beta parameters of the post status and pre status of inverse signs and similar magnitudes – although the betas for the pre-status variables were slightly lower than the post-status variables, as has been reported in other studies [7]. The amount of variance accounted for by the post scores was below 10% in all three outcome domains, while adding the pre score increased the amount of variance accounted for from 10% to 15%. While these results indicate that transition ratings are not necessarily dominated by the present state of the respondent, it has to be acknowledged that pre- and post-status scores were unable to account for a substantial amount of variance in transition ratings. It is therefore necessary to identify additional explanatory variables to further our understanding of transition ratings.

A major limitation of our study design is that it did not allow for a head-to-head comparison between quasi-indirect and direct measurements of change, nor did it allow for a head-to-head comparison between multi-item approaches and general approaches for measuring change directly or (quasi-) indirectly. Also, the study design led to there being fewer data points for the direct and quasi-indirect approaches than for the indirect approach (cf. Table 2).

Two biases might have unduly elevated the (unusually strong) correlation between indirect and quasi-indirect measures of change. First, our then-test might be prone to a memory bias if patients were able to remember how they rated the different items at t0 and if they had tried to present themselves accordingly, i.e. they might have attempted to make their current (t1) ratings of their to condition correspond to how they rated them before. Omitting the status assessment at t0 would eliminate any pre testing effects, such as patients’ deliberately making their answers on the retrospective pre test correspond to those they had given on the actual pre-test, or perhaps patients’ being sensitized to particular changes. However, omitting the pre assessment would still not allow for a head-to-head comparison between indirect and quasi-indirect measures of change. Nevertheless, we do not believe this kind of recall to be a major threat to validity because in our experience most patients have substantial difficulties remembering even important activities and interactions from the admission phase, which is likely due in part to how overwhelming rehabilitation appears to be for patients at the start of rehabilitation. This does not preclude that the patients are able to remember their health status at the time around their submission. It is reasonable to assume that this information is more general in nature compared to marks on a questionnaire, and it is inextricably linked to a patient’s reason for applying to medical rehabilitation or to his or her perception of the course of illness. A second bias is due to the fact that both indirect and quasi-indirect measures of change rely on the same post-status score, which systematically increases the association between these two variables. However, it did not seem reasonable to carry out two independent post-status assessments. Therefore, this second bias cannot be eliminated. Furthermore, estimating its magnitude is not possible given the current study design.

As Table 2 shows, the sleep-function scale yielded considerably more missing values than the other two scales did. There is no obvious reason for why this would be the case. One possible reason could be the ‘checklist misconception effect’ [28]: subjects might have misunderstood the items of the sleep-function scale as a checklist, in which ‘true’ items were to be checked (thereby confirming that they experienced these symptoms) whereas items that did not apply were to be left blank. Controlling for the checklist misconception effect in our analysis did not however substantially affect the results (data not shown).

We chose not to classify the results of the transition-rating scales into clinically meaningful categories. We avoided making such decisions for two reasons: First, we would need to know how large to make the ‘region of indifference’ – i.e. how small empirical differences would have to be for them to be regarded as too minor to merit classifying the patients as having changed – or which thresholds for change to use in general. Second, further research is needed on the question of whether these thresholds should be symmetrical around the point of indifference, i.e. whether they should be equidistant from zero in both positive and negative directions.

Quasi-indirect measurement of change has the potential to serve as a substitute for indirect measurement of change. It appears to be a suitable assessment method in situations where no baseline assessments are possible, especially non-elective care situations. However, further exploration is needed into potential moderating factors and their implications. Also, the correlation between quasi-indirect and indirect change scores might be spuriously strong due to the fact that the post-test measurement is used to compute both indirect and quasi-indirect change scores.

Transition ratings measure different aspects of change than indirect measurement of change do. We still need a comprehensive model of what transition ratings actually measure. Research making use of qualitative methodology or cognitive interviewing techniques may prove to be valuable in identifying important factors of such a model, as has been suggested by Nieuwkerk et al. [26].

^aIt should be noted that we are not using the term ‘agreement’ here in the strict sense of perfect equivalence, since direct and indirect approaches to measure change are both based on different scales.

^bThis study was not able to find any advantage to one of these approaches to measuring change over the other; this result has not been published.