Table 2 Comparison of QUADAS items⁸ and validity issues raised by Guyatt.²*

1. Was the spectrum of patients representative of the patients who will receive the test in practice?

The specific clinical indication for performing the test should be clear.

2. Were selection criteria clearly described?

Bias may be introduced if the sample does not comprise consecutive patients presenting with a specific clinical problem

3. Is the reference standard likely to correctly classify the target condition?

Not all before after studies evaluate a reference standard as the after test. Where the after test is not a recognised reference standard for the target disease full delineation of the impact of test errors requires evaluation of all study subjects with a reference standard.

4. Is the time period between reference standard and index test short enough to be reasonably sure that the target condition did not change between the two tests?

Not addressed by Guyatt

5. Did the whole sample or a random selection of the sample, receive verification using a reference standard of diagnosis?

Not all before after studies evaluate a reference standard as the after test. Where the after test is not a recognised reference standard for the target disease full delineation of the impact of test errors requires evaluation of all study subjects with a reference standard.

6. Did patients receive the same reference standard regardless of the index test result?

Not addressed by Guyatt

7. Was the reference standard independent of the index test (the index test did not form part of the reference standard)?

Before after studies are concerned with evaluating the value of adding a test. Consequently the results of the before test are usually known at the time of interpreting the after test results. This item is therefore redundant for diagnostic before-after studies.

8. Was the execution of the index test described in sufficient detail to permit replication of the test?

Whether a diagnostic technology has therapeutic impact might depend on the physician using the test.

9. Was the execution of the reference standard described in sufficient detail to permit replication of the test?

10. Were the index test results interpreted without knowledge of the results of the reference standard?

To evaluate the technology's contribution to change in therapy therapeutic plans based on the 'before test' must be elicited before knowledge of the 'after test' results.

11. Were the reference standard results interpreted without knowledge of the results of the index test?

Before after studies are concerned with evaluating the value of adding a test. Consequently the results of the before test are usually known at the time of interpreting the after test results. However therapeutic plans based on the before test results should not be known at the time of making therapeutic plans based on a combination of the before and after test results. Before-after studies can be strengthened by an independent review of the after test's contribution to therapeutic decisions relative to the before test.

12. Were the same clinical data available when test results were interpreted as would be available when the test is used in practice?

The addition of a test may have appeared to have had an impact because of an incomplete pre-test work-up.

13. Were uninterpretatable/intermediate test results reported?

Not addressed by Guyatt

14. Were withdrawals from the study explained?

Bias may be introduced if the sample does not comprise consecutive patients presenting with a specific clinical problem

Additional methodological issues raised, specific to the outcome of 'therapeutic impact'

• Criteria for establishing therapeutic impact should be formulated a priori.

• Sufficient clinical information and/or evidence of the effectiveness of available therapies are required to judge whether therapeutic changes will alter patient outcomes