Randomised controlled comparison of the Health Survey Short Form (SF-12) and the Graded Chronic Pain Scale (GCPS) in telephone interviews versus self-administered questionnaires. Are the results equivalent?

The survey methods most commonly used in clinical trials are self-administered questionnaires (SAQ), telephone interviews (TI), and a mixture of both ("mixed-mode method") that consists of mailing the self-administered questionnaires and following up by telephoning non-respondents. The highest response rates are generally achieved either with telephone interviews or with the mixed-mode method, both of which tend to minimise complete drop-outs and missing values for individual items [1–5]. It is known that patients who do not respond to mailed questionnaires, on average report greater dissatisfaction with treatment when contacted by telephone than do those who mail back their questionnaires [5, 6]. A recently published study comparing the telephone-administration mode of the SF-36 with the self-administered mode concluded that the telephone-administration mode is equivalent to and as valid as the self-administered mode[7].

By contrast, in designing the present study we hypothesized that patients respond differently to questions about psychological states than to those about physical symptoms. The latter will probably be answered more honestly, because physical problems are more socially accepted[8, 9]. For our study we therefore chose to compare patient responses to the 12-Item Short-Form Health Survey (SF-12) [10] and the Graded Chronic Pain Scale (GCPS) questionnaire[11] – two widely used survey instruments that collect data on both mental and physical aspects of pain disorders – in the telephone interview mode and the self-administration mode. A test-retest design was selected to examine whether the order of administration and/or the preliminary information of half the respondents had any effect on patient response behaviour in the comparison of SAQ and TI.

The test-retest design was chosen in order to test memory effects. It is conceivable, for example, that subjects would be better able to memorize their answers in one of the survey modes, thus resulting in greater similarity in responses between the first and second measurements. The point in time at which subjects are informed that they would be asked to answer a second questionnaire could affect results in a similar way. For example, subjects concerned about social acceptance and wishing to give very precise answers might, if told ahead of time that they would be asked to respond to more than one questionnaire, use memory aids such as making notes before the questionnaires were administered.

The purpose of this study was to assess the agreement between results of telephone interviews and self-administered mailed questionnaires for the SF-12 and the GCPS, two important instruments in clinical research and practice, with the help of a two-period change-over design. To our knowledge no such data have ever been obtained for the GCPS. The study should provide insight as to whether patients' response behaviour is influenced by motivational aspects, information that would be useful for planning clinical and epidemiological trials.

The results of equivalency testing show that the response behaviour of chronic pain patients is subject to different motivational mechanisms, depending on whether the questions concern mental or physical health. Patients gave a more positive estimation of their mental health in telephone interviews than in the self-administered questionnaires. The same was not true for the SF-12 physical scales or the GCPS subscales. The most likely explanation is that the taboo that society still places on mental disability seems to cause patients speaking with another person in a telephone interview to minimise mental problems that accompany physical illness. This tendency is less likely to affect responses to the more anonymous self-administered questionnaires.

Another result of this study is that computer-assisted telephone interviews have clear advantages over mailed self-administered questionnaires when it comes to completeness of data. In addition to this previously recognized advantage of telephone interviews, however, we found that the number of missing responses was closely related to question content. There tended to be more missing responses for items concerning mental status than for those relating to physical condition. These findings support the hypothesis that patients suffering from chronic pain often view their illness as purely physical and therefore shy away from answering questions about their mental state. In the telephone interview, on the other hand, the trained interviewer is able to obtain substantially more complete responses.

A third result of this study is that, in contrast to previous findings [2, 15, 16], the level of agreement between SF-12 scores in TI and SAQ mode for the mental health subscale was dependent on the survey mode sequence. The differences were more marked if the patient had first given a more positive assessment of mental status in the telephone interview. These results can be explained with the help of findings from the psychology of memory (e.g. [17]): a more positive assessment of mental health status in the telephone interview is associated in the respondent's memory with more positive emotions, which facilitate retrospective recall of memory content when the patient subsequently fills out the paper questionnaire. A more negative assessment of physical health would have the reverse effect: negative emotions block recall of memory content, making it more likely that the respondent will describe his or her current physical state in the subsequent survey. The point in time when patients were informed about the second survey had no effect on response behaviour. Evidently the announcement to participants that they will be asked to complete a second survey is understood as information at a formal level only. Cognitive and emotional processing related to estimating one's own state of health is not likely to be influenced by when this information is received.

The strengths of our study are the comparison of GCPS values in the two survey modes, and the new control variable "Point in time when patients were informed of second survey." To our knowledge neither of these has done before.

One limitation of our study is that we cannot rule out a real remission of symptoms in the interval between the administration of the two surveys, which was at least 14 days, and therefore cannot rule out the possibility that response behaviours were influenced by real symptom improvements.

Another limitation is the non-response rate of close to 25%, as the response behaviour of this group could well differ from that of the rest of the study population. However, a systematic bias resulting from a high non-response rate is more likely to occur when the purpose of the survey is to measure treatment success, which was not the case in the present study. A more generous time frame for the return of questionnaires might increase the response rate. In this study, 662 patients did not return the questionnaire until after the four-week period allotted. The average time taken by those patients to return the questionnaire was seven weeks.

There are also some practical disadvantages to using the CATI-System for data collection: depending on the population size and the technical equipment available, this mode is more time consuming and more costly than mailing out paper questionnaires. Administration of surveys via the Internet might be a cost-effective alternative. A factor to be considered, however, is that proportionately fewer people have access to this means of communication than to the telephone.

Chronic pain research and therapy has traditionally been an interdisciplinary undertaking. Besides quality of life questionnaires, other important data gathering instruments are questionnaires that measure fear or depression. Using instruments such as the CES-D in telephone interviews[18] could, by analogy to results obtained for the mental health subscale of the SF-12 in our study, lead to a systematic underestimation of depression.

Since telephone interviews offer significant advantages over self-administered questionnaires, further mode-comparing studies in this area, particularly with chronic pain patients, are clearly needed.

The most commonly used method of collecting data from patients is still the self-administration of a paper questionnaire. But telephone interviews are being more widely used because of the markedly better data quality obtained by this means. Until now RCT evidence as to whether patient responses differ depending on the survey mode has been lacking. We strongly recommend that mixing of questionnaire modes should be avoided when gathering data with respect to mental health criteria. When a homogeneous questionnaire mode is used, the reliability of responses should theoretically not be affected, since deviations will always be in the same direction. However, outcomes may not be directly comparable to those of other studies if the data were gathered by means of a different mode. Normative data for standardized telephone questionnaires could contribute to a better comparability with the results of the corresponding standardized paper questionnaires.