Incorporating natural variation into IVF clinic league tables: The Expected Rank

There is an increasing interest in measuring quality of health care. Data on quality of care often include outcome measures, such as mortality. The idea behind collecting and publishing information on quality of care is that the performance of providers should be measured, as stakeholders such as government, patients, and insurance companies, have a right to know what the services are achieving (an external purpose). Another reason for the continuing demand to broadcast health figures is that publication is thought to act as an incentive for low performers to adopt best practices from the top of the league in pursuit of improvement (an internal purpose). Specifically, rankings may be made according to provider performance. Such ranking was already done in 1995 for physician-specific mortality after coronary-artery bypass grafting surgery in New York State.[1] Currently, ranking health care providers is very popular in the lay press.[2]

In this paper, we use data from IVF clinic rankings based on pregnancy rates. In IVF, the competition to improve results and ranking is even stronger because a large proportion of IVF treatments take place in the private sector.

The pregnancy rate is however influenced not only by quality differences, but also by case mix and natural variation. Natural variation means variation caused by chance, and can also referred to as random variation or unavoidable variation. Case mix and natural variation obscure assessment of the true performance of a clinic and may make standard rankings meaningless. To do justice to the clinics, and to allow patients to make evidence based choices between clinics it is necessary to correct for case mix (such as age of the woman or number of previous cycles of IVF) and to take into account the natural variation that is caused just by chance.[3, 4]

We here focus on the natural variation. Different approaches have been suggested. Lemmers et al recently proposed a best-case and worst-case scenario ranking and illustrated this approach on the dataset we also use in this paper.[5] Marshall et al. calculated 95% confidence intervals to the ranks.[6] The aim of this study is to present another method: the Expected Rank (ER), which prevents over interpretation of the rankings. We present our method using IVF data from 13 clinics in the Netherlands.

The Expected Ranks differ from the simple ranks, in that they are shrunken towards the median rank of 7. Clinic A has an ER of 1.4, clinic M has an ER of 11.9. The Percentiles based on Expected Rank, that can have a range from 0% to 100%, vary from 7% for clinic A to 88% for clinic M. This means that clinic A has only a 7% probability to perform worse than another randomly selected clinic. Clinic M has a 88% probability to perform worse than another randomly selected clinic.

Figure 1 shows the ranks based on fixed and random analyses and the ER. The dot size indicates the number of treatment cycles. As one could expect, large clinics change less than smaller clinics.

The heterogeneity between the clinics τ², obtained from the random effect models, was 0.08. The corresponding 95% range of odds ratios was 0.57 to 1.74, meaning that clinics at the higher end of the pregnancy rate distribution have a 1.74 times higher chance of pregnancy than the average. Similar, clinics at the lower end have a 0.57 times smaller chance. The uncertainty measured by the median s_i ² from the fixed effect model was 0.008. This resulted in a substantial rankability (ρ = 0.9). So 90% of the observed differences between IVF clinics was actually due to 'true' differences.

We made rankings of 13 IVF clinics in the Netherlands, based on pregnancy rates. We calculated the Expected Rank and the Percentiles based on Expected Ranks, to incorporate both the magnitude and the uncertainty of the differences in pregnancy rates between the clinics.

When we want to measure provider performance with outcome measures, in this case pregnancy rates of IVF clinics, two issues are important: case-mix adjustment and natural variation.[3, 4] Regarding the first issue, the chance of pregnancy is not only determined by the performance of the clinic but also by characteristics of the mother. When different clinics treat different patients this can already cause variation in pregnancy rate that the clinics can not prevent. Therefore adjustment for case-mix is very important, but was not in the scope of this study. It is however technically readily possible to calculated the case-mix adjusted ER. The only difference is that we obtain the pregnancy rate coefficients from a (random effect) logistic regression model that includes the relevant patient characteristics as explaining variables.

The second issue is natural variation that exists just by chance. Random effect models allow imprecisely estimated outcomes from smaller clinics to 'borrow' information from other clinics, causing their estimates to be shrunk toward the overall mean. Each estimate reflects a compromise between the clinic-specific mean and the overall mean based on the relative magnitude of the variance within a clinic to the total variance (between and within the clinics). Random effect estimates can be considered as the 'true' pregnancy rate coefficients, beyond natural variation.[8] In our study sample size was large. Hence there were no large differences between fixed and random effect analyses, and the ranking did not change.

We used R to calculate the random effect estimates. It is however not always straightforward to derive these and to select the correct information from the output of the various statistical programs. SPSS for example does not fit random effect logistic regression models. Other packages such as SAS and Stata do.

Simple rankings, based both on fixed and random estimates of the pregnancy rate coefficients, disregard both the magnitude and the uncertainty of the differences between clinics. With the expected rank (ER) however, both are incorporated in the ranking. We see that the best clinic has an ER of 1.4 instead of 1, the worst clinic has an ER of 11.9 instead of 13. We also see that some clinics perform in a very similar fashion, such as clinic D and E. The magnitude of the difference is disregarded in the simple ranking (clinic D rank 4, clinic E rank 5), but shown in the ER (clinic D ER 5.0, clinic E ER 5.1). We also see that there is more uncertainty about the performance of the smaller clinics like H and J. This uncertainty is disregarded in the simple rankings (clinic H rank 8, clinic J rank 10) but included in the ER (clinic H ER 8.5, clinic J rank 9.4) So the ER is much more subtle than the simple ranking. For ease of interpretation we calculated the percentile based on expected rank (PCER), which is independent from the number of centers in the sample and indicates the probability that a hospital is worse than a randomly selected other hospital.

Approaches similar to the ER have been proposed by others. Lemmers et al. proposed a best and worst case ranking and Spiegelhalter et al proposed a 95% confidence interval around the ranks.[5, 6] Both methods provide 3 numbers: a 'point estimate' and the uncertainty around this estimate. The ER consists of only one number, which is an advantage in our perspective because it is generally easier to process one number than three numbers. On the other hand, the ER does not show the amount of uncertainty, although it is included in the number. The ER also incorporates the magnitude of the difference between the clinics, this is not directly included in the approaches of Lemmers et al. and Spiegelhalter et al.

Another advantage of the ER is the intermediate step of calculation of the probability that a certain clinic performs worse than another one, for all the clinics. These individual comparisons can be very useful for couples when they want to choose an IVF clinic. For example, we saw that the probability that clinic G performs worse than clinic F is 54%. In decision making this probability can be weighed against possible advantages of clinic G, e.g. distance or familiarity with the hospital. The presentation in a table like table 2 however, is only feasible when the number of clinics to compare is small.

The rankability can be used as an indication to see whether it make sense to rank the clinics at all. In this study rankability was 0.9, which is substantial. 90% of the differences between the clinics can be attributed to true differences, and 10% to random variation. It is however a value judgment what can be considered as a high rankability. We would suggest that a rankability above 0.7 is reasonable. The rankability is a function of the heterogeneity τ² and the uncertainty of the individual center effects s_i ². In this case the heterogeneity was considerable (τ² = 0.08) and the uncertainty was small because of the large numbers per clinic (si² = 0.008).

Because of the relatively small uncertainty in this example, the natural variation is limited and there were no large differences between the standard rankings and the Expected Ranks. This is a result of the data, not of the proposed method. Therefore there might have been examples that would have better demonstrated the features of the method. We nevertheless used this dataset since it was used before to present novel ranking methods.

For comparison, in a study on 10 centers in the Netherlands treating stroke patients, the unadjusted τ2 was 0.38, but rankibility was only 0.55, due to small numbers. The ERs ranged from 1.7 to 8.6 instead of the original ranks 1–10 and six of the ten centers had en ER close to the median rank of 5.5. (Lingsma et al, How to compare center based on outcome: results from the Netherlands Stroke Survey, submitted).

Regarding the communication of performance measures, rankings may be not the ideal way to present the data in all its subtleties, they are however attractive to the press and the public. Our proposed measure, the Expected Rank, provides a way to combine the attractiveness of a ranking, a single number and easy interpretation with reliable analyses that does justice to the providers, and also allows individual comparisons.

In comparing health care providers, the Expected Rank combines the attractiveness of a ranking, a single number and easy interpretation, with reliable analyses that does justice to the providers, and also allows individual comparisons.