Meta-analysis and meta-modelling for diagnostic problems

The proportional hazards measure

Numerous summary measures for a pair of specificity and sensitivity have been suggested: we mention here the Youden index, J _i =p _i +q _i −1 [10], and the squared Euclidean distance to the upper left corner in the SROC diagram, E _i =(1−p _i )²+(1−q _i )². [A review of summary measures is given in Liu [20].] Using an average over any of these measures might be problematic: not only might sensitivities and specificities be heterogeneous, this might also be true for the associated summary measures such as the Youden index or the Euclidean distance (as demonstrated by Figure 2 using the data of the meta-analysis of BNP and heart failure).

We suggest using the measure $\theta =\frac{logp}{log(1-q)}$, which relates the log-sensitivity to the log-false positive rate; we call it the proportional hazards (PH) measure. In Figure 3 we see that this measure shows a reduced variability for the meta-analysis of BNP and heart failure, making it more suitable as an overall measure in the meta-analysis of diagnostic studies or diagnostic problems. While the measure appears to be like any other summary measure of the pair sensitivity and specificity, it has a specific SROC-modelling background and motivation. We have mentioned previously the cut-off value problem: observed heterogeneity might be induced by cut-off value variation which could lead to different sensitivities and specificities – despite the accuracy of the diagnostic test itself not having changed – and might also lead to an induced heterogeneity in the summary measure. Hence, it is unclear whether the observed heterogeneity is due to heterogeneity in the diagnostic accuracy (authentic heterogeneity) or whether it has occurred due to cut-off value variation (artificial heterogeneity). This second form of heterogeneity can also occur when the background population changes with the study.

This model was suggested by Le [21] for the ROC curve. It is an appropriate model since, for feasible q, (1−q) ^θ is also feasible as long as θ is positive. Note that (1) is defined for all values of p∈[0,1] and q∈ [ 0,1] whereas $\theta =\frac{logp}{log(1-q)}$ is only defined for p∈(0,1) and q∈(0,1). Population values of sensitivity and specificity of 1 are rarely realistic, although observed values of 1 for sensitivity and specificity do occur in samples. This can be coped with by using an appropriate smoothing constant such as estimating specificity as (n _i −1)/n _i when x _i =n _i and sensitivity as (m _i −1)/m _i if y _i =m _i .

In Figure 4 we see a number of examples of the proportional hazards family. It becomes clear now why θ is called the proportional hazards measure. By taking logarithms on both sides of (1) we achieve

$\theta =logp\left(t\right)/log[1-q(t\left)\right],$

(2)

meaning if model (1) holds, the ratio of log-sensitivity to log-false positive rate is constant across the range of possible cut-off value choices t. Hence the name proportional hazards model, which was suggested in a paper by Le [21] and used again in Gönen and Heller [22]. The idea of representing an entire ROC curve in a single measure is illustrated in Figure 5. While sensitivity and specificity vary over the entire interval (0,1), the value of θ remains constant. Hence, log-sensitivity is proportional to the log-false positive rate. This assumption is similar to an assumption used for a model in survival analysis, where it is assumed that the hazard rate of interest is proportional to the baseline hazard rate; this might have motivated the choice of name used by Le [21] and Gönen and Heller [22] in this context.

However, it is not our intention to make the assumption that an entire SROC curve can be represented by model (1); the explanations above are instead meant as a motivation that the PH-measure is not just another summary measure, but can be derived from a ROC modelling perspective. We envisage that each study, with associated pair of sensitivity and specificity, can be represented by a specific PH-model, as illustrated in Figure 6.

where ${\widehat{\theta }}_i=log{\widehat{p}}_i/log[1-{\widehat{q}}_i]$, so that the curve (3) passes exactly through the point $(1-{\widehat{q}}_i,{\widehat{p}}_i)$.

which can be written as $\alpha +\theta [log(1-\widehat{q}\left)\right]$ where α= log(p)−θ log(1−q) and $\theta =\rho \frac{{\sigma }_p}{{\sigma }_q}$. This is an important result since it means that, in the log-space, sensitivity and false–positive rate are linearly related. Furthermore, if α is zero, the proportional hazards model arises.

The answer is that such a model is not identifiable since we have only one pair of sensitivity and specificity observed in each study and it is not possible to uniquely determine a straight line by just one pair of observations since there are infinitely many possible lines passing through a given point in the logp – log(1−q) space. However, the proportional hazards model as a slope-only model is identifiable and it is more plausible than other identifiable models such as the intercept–only model. Clearly, a logistic-transformation would be more consistent with the existing literature [14, 15] than the log-transformation. However, both models would give a perfect fit (within each study) since there are no degrees of freedom left for testing the model fit. The situation changes when there are repeated observations of sensitivity and specificity per study available. However, these meta-analyses with repeated observations of sensitivity and specificity according to cut-off value variation are extremely rare.

A mixed model approach

where x _i is a known covariate vector in study i, δ _i is a normally distributed random effect δ _i ∼N(0,τ ²) with τ ² being an unknown variance parameter, and ${\epsilon }_i\sim N(0,{\sigma }_i^2)$ is a normally distributed random error with variance ${\sigma }_i^2$ known from the i−th study.

There are several noteworthy points about the mixed model (7). The response is measured on the log-scale, where the transformation improves the normal approximation and also brings the diagnostic accuracy into a well-known link function family: the complementary log-log function. The difference of the probability for a positive test in the groups with and without the condition is measured on the complementary log-log scale. The fixed effect part involves a covariate vector x which could contain information on study level such as gold standard variation, diagnostic test variation, or sample size information. It should be noted that there are two variance components, τ ² and ${\sigma }_i^2$. It is important to have information on the second variance component. If the second component is unknown, even under the assumption of homogeneity ${\sigma }_1^2=\cdots ={\sigma }_k^2$, the variance component model would not be identifiable. Hence, we need to devote some effort to derive expressions for the within study variances; this can be accomplished using the δ−method as discussed in the next section.

We acknowledge that the above are estimates of the variances of the diagnostic accuracy estimates, but are used as if they were the true variances.

which have gained some popularity in the meta-analytic literature.

Case study on MMSE and dementia

We illustrate the approach with an example and revisit a meta–analysis by Mitchell [26] on the diagnostic accuracy of the mini-mental state examination (MMSE) as a diagnostic test for the detection of dementia and, more recently, mild cognitive impairment (MCI). In this meta–analysis 38 studies were included and the entire data are reproduced in Table 2. We are interested in the question: is there a difference in diagnostic accuracy of the MMSE in the detection of dementia and MCI, as Figure 7 suggests.

Study	Condition	TP	FN	FP	TN
1	Dementia	65	3	240	870
2	Dementia	117	12	10	110
3	Dementia	48	19	63	989
4	Dementia	134	8	28	152
5	Dementia	24	5	44	292
6	Dementia	67	15	48	153
7	Dementia	64	17	1	71
8	Dementia	281	64	20	286
9	Dementia	13	1	44	286
10	Dementia	262	20	29	177
11	Dementia	143	18	29	123
12	Dementia	183	33	33	51
13	Dementia	22	1	152	140
14	Dementia	112	1	590	2091
15	Dementia	152	81	126	1009
16	Dementia	29	26	26	236
17	Dementia	31	6	3	247
18	Dementia	10	3	12	333
19	Dementia	707	88	1438	10447
20	Dementia	181	108	17	184
21	Dementia	59	29	23	74
22	Dementia	74	23	16	143
23	Dementia	27	12	26	209
24	Dementia	40	6	75	528
25	Dementia	317	52	173	578
26	Dementia	387	116	16	54
27	Dementia	118	65	1	44
28	Dementia	44	7	34	396
29	Dementia	123	46	98	309
30	Dementia	25	43	3	171
31	Dementia	73	32	2	225
32	Dementia	37	45	1	440
33	Dementia	78	34	45	376
34	MCI	72	12	53	214
35	MCI	106	23	410	379
36	MCI	37	36	22	118
37	MCI	67	30	22	75
38	MCI	17	77	1	90

The inference is based here on a procedure called the Wald test. The estimated parameter value is divided by its estimated standard error, and the result is given in column four in Table 4. The likelihood ratio test may be considered as an alternative. It is defined as two times the difference of the log-likelihood including the effect of interest and the log-likelihood not including the effect of interest. For the effect of condition in Table 4, we find a value of 6.8 for the likelihood-ratio test. The Wald test is asymptotically standard normal under the null-hypothesis of absence of effect, whereas the likelihood ratio test statistic is asymptotically chi-squared distributed with degrees of freedom equal to the number of parameters associated with the effect considered (in this case one). It is well-known that the likelihood ratio test is more powerful. Here, both tests provide similar p-values, with 0.0091 for the likelihood ratio test and 0.0069 for the Wald test; this confirms the significance of the effect (dementia/MCI) on the diagnostic accuracy.

Note that the likelihood ratio test as well as the Wald test need modification in situations where the null hypothesis is part of the boundary of the alternative such as when testing H ₀:τ ²=0. In this case, the asymptotic null distribution of the likelihood ratio test statistic is no longer χ ² with 1 df but rather a mixture of a two-mass distribution giving equal weights 0.5 to the one-point mass distribution at 0 and a χ ² with 1 df [28]. Practically, this means that standard 2-sided p-values have to be divided by 2.

Case study on MOOD and depressive disorders

The MOOD module of the Patient Health Questionnaire (PHQ-9) has been developed to screen and to diagnose patients in primary care with depressive disorders. The instrument consists of 9 questions, each scored from 0 to 3 points with a total score ranging from 0 to 27. In a meta–analysis of the diagnostic accuracy of MOOD, Wittkampf et al.[29] included 12 studies. These studies used either a cut-off of 10 (referred to here as “summary score”) or a more complex evaluation algorithm (“algorithm”). The complete data are listed in Table 5 and the associated SROC diagram is given in Figure 8. The impression from the graph is that the cut-off of 10 used by the summary score has a higher diagnostic accuracy than the alternative.

Study	Cut-off	TP	FN	FP	TN
1	algorithm	65	26	104	1192
2	algorithm	70	13	74	846
3	sum score	62	10	27	429
4	sum score	36	5	65	474
5	sum score	55	11	43	392
6	algorithm	6	8	12	144
7	sum score	121	103	80	720
8	algorithm	11	5	5	76
9	algorithm	6	5	0	3
10	algorithm	85	31	9	460
11	sum score	15	1	4	42
12	sum score	96	10	23	187

Global versus local criteria

We have focussed on the PH measure so far, as it provides an appropriate measure for comparing SROC curves globally, in the sense that cut-off value variation will not necessarily effect the estimate of the SROC curve. The situation is illustrated in Figure 9.

where ${\widehat{p}}_i=y_i/m_i$ and ${\widehat{q}}_i=x_i/n_i$. Each point in the SROC diagram has a unique circle with center (0,1) that passes through this point. In Figure 9, one such circle is illustrated which also has the smallest Euclidean distance among the three available points (since it has smallest radius among the three possible points with associated circles). In the following, we will consider the criterion (14) as an alternative criterion to choose the cut-off value.

Evidently, both criteria lead to the same conclusion, namely that using the summary score with a cut-off value of 10 leads to the higher diagnostic accuracy (although the effect is not significant). It might also be worthwhile looking at the results of the likelihood ratio test: for the PH-measure as the outcome variable, the likelihood ratio test provides a value of 1.5; for the Euclidean distance, the value of the likelihood ratio test is 1.7, confirming the non-significance of the effect. Nevertheless, the analysis shows that the cut-off value of 10 provides the higher diagnsotic accuracy.

Meta–analysis of magnetic resonance spectroscopy and prostate cancer

This case study provides an example where the use of a global or local criterion leads to a different conclusion. Magnetic resonance spectroscopy has the ability to discriminate between prostate cancer and benign prostatic hyperplasia, based on reduced citrate and elevated choline in the cancerous region. The diagnostic test works on a voxel of signal intensity ratios of (choline+creatine)/citrate. Two cut-off points are in use: <0.75 and <0.86. The results collected in a meta–analysis by Wang et al.[30] include 12 studies, as presented in Table 8; the associated SROC diagram is presented in Figure 10. From the graph, there is no obvious choice for the “best” cut-off value.

Study	Cut-off	TP	FN	FP	TN
1	0.75	122	30	35	55
2	0.75	73	8	80	219
3	0.75	75	6	92	207
4	0.75	123	39	38	50
5	0.75	134	21	40	39
6	0.75	12	12	7	75
7	0.86	81	71	24	59
8	0.86	56	25	32	267
9	0.86	52	29	20	59
10	0.86	98	57	20	59
11	0.86	6	9	15	266
12	0.86	44	8	32	264

PH measure and positive likelihood ratio

Another frequently used diagnostic measure is the positive likelihood ratio, defined as the ratio of sensitivity to false positive rate P(T=1|D=1)/P(T=1|D=0) or p/(1−q). It is different to the PH measure in that the ratio is taken on the log-scale: θ= logp/ log(1−q). Furthermore, if re-expressed as models, the positive-likelihood ratio corresponds to p=θ ^′(1−q), a straight line with no intercept, whereas the the PH measure corresponds to p=(1−q) ^θ , a straight line on the log-scale with no intercept. The positive likelihood ratio is a natural measure since it transfers the concept of relative risk (risk of a positive test in the diseased group to the risk of a positive test in the non-diseased group) to the diagnostic setting. However, it is less suitable as an (S)ROC model since it does not provide a function which connects the lower left vertex with the upper right vertex in the ROC diagram (which, in contrast, the PH-model does provide).