Assessment of the usefulness of a diagnostic test: A survey of patient preference for diagnostic techniques in the evaluation of intestinal inflammation

Abstract

Background

In order to assess the usefulness of radiolabeled white cell scanning in the diagnosis of intestinal inflammation, subjects were asked to rank several dimensions of preference for white cell scanning in relation to other diagnostic tests. Two groups were surveyed: one known to have inflammatory bowel disease and the second not familiar in most cases with the tests. Subjects were asked to rank preference for each of seven tests: radiolabeled white cell scan, colonoscopy, barium enema, sigmoidoscopy, enteroclysis, stool analysis and laparotomy for the diagnosis of IBD and impressions of discomfort, embarrassment, inconvenience and danger related to each test. Mean rank scores were calculated, test ranks compared within groups and significance determined by the Wilcoxon rank test.

Results

Significant differences were seen in overall preference for white cell scan over barium enema and colonoscopy (p < 0.01) in both survey groups. Perceived discomfort and embarrassment demonstrated similar rankings.

Conclusion

This patient preference combined with the reported accuracy of white cell scanning further establishes the usefulness of this means of IBD diagnosis.

Background

The radio labeled white cell scan has been shown to be an accurate means of detecting intestinal inflammation [1]. Our initial report of the use of Indium¹¹¹ radiolabeled white cell scanning in the diagnosis of inflammatory bowel disease described a sensitivity and specificity of 97% and 100% respectively in 49 scans in 39 patients in all of whom confirmatory studies were done. Since that time, 1130 scans have been performed at our institution. Clinical follow-up of these patients has revealed only three cases of confirmed false negative scans. No cases of false positive scans have been encountered. This accuracy, it should be stressed, is not in the grading of disease activity, but in the dichotomous determination of disease presence and its anatomic location. The lack of confirmatory testing of all subsequent patients certainly results in the overestimation of accuracy [2], but not greatly so, since all patients have been available for clinical follow-up and diagnostic errors would become apparent with the passage of time. Even if it is a ten fold overestimate, the white cell scan still compares well with other diagnostic modalities. In 1993 Indium¹¹¹ became unavailable in Chicago and we switched to the use of Technetium⁹⁹. We found that the higher radiation dose of Technetium injected (20 mCI versus 0.5 mCI Indium) resulted in clearer images with less motion artifact, yet its shorter half life resulted in the total radiation dose to the patient being similar to Indium¹¹¹. First images are available within 5 hours from the patient's arrival using Technetium, compared with 30 hours for Indium, making this applicable to emergent situations. In routine clinical follow-up, no measurable change was noted in the number of false positive or negative scans since the change to Technetium⁹⁹ (unpublished data). These advantages of Technetium over Indium have resulted in our continued use of Technetium for white cell scanning (WCS).

The further establishment of the usefulness of this diagnostic clinical intervention was our next goal. Usefulness depends in the final analysis upon the likelihood that a patient will decide to undergo the intervention. Patient preference may therefore be an important part of the process of determining clinical usefulness. In order to establish further its usefulness, a survey was conducted in which subjects were asked to rank their preference for white cell scanning in relation to other means of diagnosing intestinal inflammation.

Methods

Questionnaire

The questionnaire designed for the study is presented in the Appendix. It begins with a description of seven diagnostic tests of intestinal inflammation. These are colonoscopy, barium enema, radio-labeled white cell scanning, flexible sigmoidoscopy, enteroclysis, exploratory surgery, and stool collection and analysis. Tests are described both in terms of the preparation required for the test and the test procedure itself.

Participants were then asked to rank order the seven tests by overall preference, by perceived or anticipated physical discomfort, by inconvenience, by embarrassment, and by danger. Finally, they were asked to indicate which of the tests, if any, they had previously experienced, and their age and gender.

Participants and sampling

Two groups of participants were invited to respond to the questionnaire: an experienced group and a naive group:

The experienced group were 25 patients of the University of Illinois colorectal surgery clinic known to have inflammatory bowel disease (IBD), either ulcerative colitis or Crohn's disease, and likely to have experienced most, if not all, of the procedures described in the questionnaire. Eighty-five such patients were mailed questionnaires (response rate 30% - a low rate, typical of mailed questionnaires). Respondents ranged from 14-66 years old with a median age of 38. Fifteen (60%) were women.

The naive group of participants were a convenience sample of 39 adults, consisting of parents or grandparents of children visiting a general pediatrician's office and personnel working in that office. These participants had experienced few or none of the diagnostic tests.

93% of those requested by office staff to participate returned the questionnaire. One participant failed to complete the majority of the questionnaire and was dropped from the study. Remaining participants ranged from 18-81 years old with a median age of 44. Twenty six (68%) were women.

Analyses

Data were analyzed using SPSS 9.0. Analyses reported here focus on comparisons within the two groups only and not across groups, between ranks assigned white cell scanning, barium enema, and colonoscopy on preference and the other assessed attributes. We also consider whether these rank orders may differ between participants who have and have not received these tests. The other tests in the questionnaire, fiberoptic sigmoidoscopy, enteroclysis, laparotomy and stool analysis, though capable of detecting inflammation, were included in the survey principally as distractors. This was done in order to avoid keying the subjects on white cell scanning and the tests which are used most prevalently as the principal modes of IBD diagnosis and evaluation: colonoscopy and barium enema.

Results

Participant experience

As expected, the experienced group were more likely than the naive group to have experienced the three focal tests. Of the 25 experienced participants, 17 had experienced all three focus tests. Three had not had a barium enema, 2 had not had a white-cell scan, 1 had not had a colonoscopy, 1 had experienced only a white cell scan, and 1 had experienced only a colonoscopy.

Of the 38 naive participants, 26 had never experienced any of the focal tests, 9 had experienced a barium enema in the past, 1 had experienced a white cell scan, 1 had experienced a colonoscopy, a 1 had experienced both a barium enema and a colonoscopy.

Preference

The Table reports the mean preference ranks assigned to the three focal tests by each group of participants. Lower ranks indicate greater preference. Because of the distractors, ranks could range from 1 (most preferred) to 7 (least preferred).

Both groups indicated significantly greater preference for white cell scan than for either colonoscopy or barium enema (Wilcoxon signed ranks tests, experienced group WCS vs. colonoscopy, Z = 2.78, naive group WCS vs. barium enema, Z = 4.05, p < 0.01 for all tests). In the experienced group, preference order for colonoscopy and barium enema did not differ significantly (Wilcoxon signed ranks Z = 0.60), but in the naive group, barium enema was significantly preferred to colonoscopy (Wilcoxon signed ranks Z = 4.64).

Dimensions of test experience

The figure presents the mean ranks for the three focal tests on the dimensions or attributes of physical discomfort, inconvenience, embarrassment, and danger for each group of participants. Lower ranks indicate better perceptions (less discomfort, inconvenience, embarrassment, or danger).

Physical discomfort and embarrassment ranks essentially duplicated the preference ranks. Both groups considered white cell scan significantly less uncomfortable and less embarrassing than either colonoscopy or barium enema (discomfort: Wilcoxon signed ranks tests, experienced group WCS vs. colonoscopy Z = 3.08, naive group WCS vs. colonoscopy Z = 3.39, p < 0.01; embarrassment: experienced group WCS vs. barium enema Z = 3.01, naive group WCS vs. barium enema Z = 4.71). The naive group also expected colonoscopy to be significantly more uncomfortable than barium enema (Wilcoxon signed ranks Z = 4.11, p < 0.01) and significantly more embarrassing (Wilcoxon signed ranks Z = 2.80, p < 0.01).

Inconvenience ranks were less consistent. In the experienced group, white cell scan was considered significantly less inconvenient than colonoscopy (Z = -2.39, p < .05), but not significantly less inconvenient than barium enema (Z = 1.86). In the naive group, white cell scan was not considered significantly less inconvenient than either barium enema (Z = 0.29) or colonoscopy (Z = 1.80). The naive group did consider colonoscopy to be more inconvenient than barium enema (Z = 3.33, p < .01).

Danger ranks in each group reflected only the perceived danger of colonoscopy. In both groups, colonoscopy was perceived as significantly more dangerous than white cell scan (experienced group Z = 2.23, naive group Z = 2.49, p < 0.01) and barium enema (experienced group Z = 3.01, naive group Z = 4.03, p < 0.01). White cell scan was not considered significantly more or less dangerous than barium enema by either group.

Excluding naive group members who had experienced any of the tests and experienced group members who had not experienced all of the tests did not change the overall pattern of the results, with two exceptions. After the exclusion, the naive group ceased to find colonoscopy significantly more dangerous than white cell scan (Z = 1.60) and ceased to find colonoscopy significantly more embarrassing than barium enema (Z = 1.59).

Discussion

Clinical usefulness

The clinical usefulness of a diagnostic test is an elusive paradigm to measure. The test should be reliable (i.e. repeatable), but also accurately represent the actual event or state that must be known (i.e. valid). An assessment of the radio labeled white cell scan's accuracy has already satisfied these criteria. There are limits to this accuracy. Quantitative assessment of disease activity has for the most part been unsuccessful. But as a dichotomous assessment of inflammatory bowel disease presence, and anatomic localizer of disease activity, both the initial published assessment [1] and subsequent clinical experience have shown the WCS to be an extremely sensitive diagnostic tool. However usefulness encompasses many other parameters besides reliability and validity, some quite qualitative, such as convenience, perceived or feared pain, compliance or preference, and others more quantitative such as cost or timely retrieval of results. A test that performs well in all these areas should allocate an individual patient to a course that improves clinical outcome, the quinticential measure of usefulness. In a chronic illness such as those causing intestinal inflammation, where quantifiable absolute survival (as in cancer) is not relevant, issues such as accuracy, preference and cost become paramount in measuring usefulness. The clear preference for white cell scanning in two very different populations (Table 1, Figure 1) further establishes its usefulness in the evaluation of intestinal inflammation.

Figure 1

Mean rank scores for dimensions of test preference. Mean Preference Ranks for the perceived dimensions of discomfort, embarrassment, inconvenience and danger in the diagnosis of inflammatory bowel disease comparing colonoscopy, barium enema and radiolabled white cell scanning in both individuals who have experienced these test (experienced) and those who have not (naive).

Table 1 Mean Rank Order (lower is preferred)

No other surveys could be found of patient preference in diagnostic evaluation of inflammatory bowel disease, either of IBD patients or of healthy individuals. The studies that have been reported comparing diagnostic tools in the colorectum have been performed in the context of colorectal cancer screening [3,4,5,6,7,8], and for the most part compared barium enema to colonoscopy [5,6,7,8]. The patient preference ranks often reflected the source of the study, barium enema preference in studies reported by radiologists [5] and colonoscopy preference reported by gastroenterologists [6]. The amount of sedation administered during the exam varied widely in these reports and clearly effected impressions. The sedation administered to the experienced group in this report might also have been the factor that differentiated impressions of discomfort of colonoscopy between the experienced and naive groups. There was also, in a study that presented scripted descriptions of the diagnostic tests, what appeared to be prompting that favored colonoscopy over other choices [3]. In this report we attempted to avoid this by not involving the subject in controversies having to do with test accuracy. It was stated in the preamble that the subject was to assume that all tests gave roughly equivalent diagnostic information. In the context of detection of intestinal inflammation, for all seven tests listed, this is far closer to being true than in the context of screening and diagnosis of neoplasia. In addition several tests were used principally as distractors (sigmoidoscopy, stool analysis, enteroclysis and laparotomy) in order to avoid keying the subject on the comparisons of primary interest which were white cell scan compared to barium enema and colonoscopy. In addition subjects were not told that one test might inevitably lead to another. This is of importance regarding WCS as it was not our practice to follow a positive or negative scan with a colonoscopy or barium enema unless for other clinical indications. A WCS combined with sigmoidoscopy and stool culture (often done at the first visit) was considered adequate to establish the presence of inflammatory bowel disease and initiate treatment.

Conclusion

Two different groups were chosen for the survey to obtain different perspectives of these exams: individuals who had experienced all of the tests and were burdened with a chronic illness, and individuals who had experience or prior impressions of none or few of the tests and were representative of a healthy population. The former group is often faced with repeated testing in the course of their disease and must decide along with their physician when and how often to undergo evaluation of disease activity. The latter naive group may at some time face the decision how much intervention to undergo in the evaluation of symptoms that may only be functional in etiology such as diarrhea or pain. The impressions of these two groups were remarkably similar. Both showed a clear preference for WCS over barium enema and colonoscopy, except for some concern about the time it takes to perform the radiolabeled white cell scan (Figure 1). If we are to take these preferences seriously, they could then be incorporated, along with other outcomes and risks, into a decision analysis of testing options in inflammatory bowel disease that seeks to maximize the patient's expected utility [9].