Calls for robust systems for monitoring healthcare outcomes have been made [1]. The increasing use of electronic patient records in healthcare presents an opportunity for the development and application of real-time monitoring systems that can lead to the rapid detection of adverse trends in healthcare. Statistical process control (SPC) methods, developed and long used in quality control systems in the manufacturing industry [2], could become central to such efforts. We describe the design and retrospective application of a surveillance system in the continuous monitoring of clinical outcomes using an SPC tool known as the CUmulative SUM (CUSUM) chart, using routinely collected data. We used a neonatal clinical outcome, the low Apgar score (5 minute Apgar score < 7), as an example outcome.
The CUmulative SUM (CUSUM) chart
The CUSUM chart method, first described by Page in 1954[3], is based on sequential monitoring of a cumulative performance measure over time. With several developments and adaptations, it has emerged as a suitable method for monitoring healthcare outcomes [4–8]. We selected and used the "Observed minus Expected" (O-E) and the Log-likelihood CUSUM chart methods in designing our surveillance tool.
Observed – Expected (O-E) CUSUM chart
This form of the CUSUM chart is a graphical representation of a running total of the difference between the number of observed adverse events and that expected according to a specified baseline reference rate. It is mathematically defined as:
C
t
= C
t-1 + (X
t
- X
0),
where X
t
is the outcome measurement for subject t (Observed), and X
0 is the baseline reference rate (Expected), hence O – E. In binary outcome measures, X
t
equals 0 for a successful or desired outcome, and 1 for an adverse outcome. The chart is a plot of the cumulative sum, C
t
, against t. Desired outcomes result in downward steps, and upward steps are produced when adverse outcomes are encountered. When the outcome rate is consistent with the baseline reference rate, the plot runs randomly about the horizontal line at zero. Although in its simple form, the O – E chart has no limits and therefore does not give a signal if and when the rate changes have become statistically significant, it serves to illustrate an overall general trend in the rates of the adverse outcome monitored.
The Log Likelihood CUSUM chart
The Log Likelihood CUSUM chart is a probability testing procedure that sequentially assesses whether the observed adverse outcome rate is consistent with a specified baseline reference rate. Each subject is given a weight W
t
, which is obtained as follows [7];
where P
t
is the baseline reference rate for subject t. OR
A
is a pre-specified odds ratio under the alternative hypotheses.
If desired, a two-sided CUSUM chart can be designed, where the upward chart is designed to detect an increase in the adverse outcome rate (OR
A
> 1), while the downward chart is designed to detect a reduction in the adverse outcome rate (OR
A
< 1). In the upward chart, is plotted against t, where . The downward CUSUM plots against t, where . These CUSUM charts are thus restricted to always stay above or below zero, respectively.
Limits are determined and placed, with the CUSUM chart considered to signal when such a limit is crossed. The "signal" is an indication of sufficient evidence that the adverse outcome rate is no longer consistent with the baseline level. When this happens, the monitoring process is stopped to allow for appropriate previously agreed action to be taken. This response begins with checking and confirming the accuracy of the data before further investigations and subsequent changes are introduced. After this, monitoring is continued, either with the settings unchanged or with changes made as appropriate.
Limits have an inter dependence with the Average Run Length (ARL). The out-of control average run length (OC-ARL) is the average number of subjects required before the CUSUM chart signals when the level of performance is unacceptable, and the in-control average run length (IC-ARL) is the average number of consecutive subjects required for the CUSUM chart to signal despite the true rate being at an acceptable level.
The clinical outcome: The low Apgar score
We selected a typical neonatal clinical outcome, the low Apgar score (5 minute Apgar score < 7), as an example outcome. The Apgar score is a convenient shorthand for reporting the status of newborn babies as well as the effect of resuscitation[9]. It is a zero to ten (0 – 10) aggregate score based on 5 parameters assessed in nearly all babies born in UK hospitals as well as the rest of the world. The low Apgar score (5 minute Apgar score < 7), has been identified as a "key outcome" to be used in assessing the role and impact of Electronic Fetal Monitoring (EFM) guideline [10], produced by the National Institute for Clinical Excellence (NICE). The Royal College of Obstetricians and Gynaecologists (RCOG) have also included it in a "trigger list" of outcomes to be monitored using the adverse incident reporting system [11]. It is also one of the components of the Adverse Outcome Index (AOI), a ten outcome system recently proposed for use in the assessment of quality of care in delivery units in the US [12].
Southmead Hospital, a District General Hospital (DGH) in Bristol with around 5000 deliveries a year, reported a 50% reduction in the rate of low Apgar scores in a subset of term (>37 weeks gestation) deliveries, following the introduction of regular training of all labour ward staff in the management of obstetric emergencies[13]. Their low Apgar score rate, which had been 0.86% in the 1998 to 1999 period, decreased to 0.44% after the introduction of the training programme in 2000. With new standards achieved, Southmead Hospital's data were ideal for testing a surveillance system that could be used in prospective monitoring of such clinical outcome in maternity units in the UK. Such a system would provide early warnings when rates of adverse outcomes are seen to be rising.