Ongoing monitoring of data clustering in multicenter studies

Quantifying clustering--The ICC

The ICC is a statistic that can be used to quantify the degree to which observations within a cluster differ from those between clusters. A summary of the theory of and methodology for ICC calculation can be found in a paper by Donner [6]. We calculated confidence limits for the ICC's using the multivariate delta method [7].

When employed in a multicenter study, the ICC compares the variance between centers (${\text{σ}}_{\mathsf{\text{b}}}^{\mathsf{\text{2}}}$) to the total variance (i.e. the variance within centers (${\text{σ}}_{\mathsf{\text{w}}}^{\mathsf{\text{2}}}$) plus the variance between centers (${\text{σ}}_{\mathsf{\text{b}}}^{\mathsf{\text{2}}}$)), where the $\mathsf{\text{ICC}}={\sigma }_{\mathsf{\text{b}}}^{\mathsf{\text{2}}}/\left({\sigma }_{\mathsf{\text{b}}}^{\mathsf{\text{2}}}+{\sigma }_{\mathsf{\text{w}}}^{\mathsf{\text{2}}}\right)$[8]. If all the variation is between centers (i.e. ${\text{σ}}_{\mathsf{\text{w}}}^{\mathsf{\text{2}}}=0$), then the ICC = 1. If all the variation occurs within centers (${\text{σ}}_{\mathsf{\text{b}}}^{\mathsf{\text{2}}}=0$), then the ICC = 0.

The ICC is frequently used to assess the degree of clustering in multicenter studies during data analysis, after data collection is complete. However, we believe that it can also be used as a component of periodic monitoring during data collection. Suppose the centers are in the process of measuring a specific physical characteristic of a homogenous population. A high ICC would indicate higher than expected variation due to clustered measurement between centers. If detected early, the reasons for the clustered measurement may be corrected.

The promotion of breastfeeding intervention trial (PROBIT)

To illustrate how the ongoing calculation of ICCs can be used to monitor and reduce measurement error in multicenter studies, we review our recent experience with the Promotion of Breastfeeding Intervention Trial (PROBIT). Thirty-one maternity hospitals in the Republic of Belarus, and one associated outpatient polyclinic for each hospital, were randomly assigned to implement a breastfeeding promotion intervention or to continue the procedures in place at the time of randomization; 17 046 mother-infant pairs were recruited in 1996/1997 [9, 10]. We include here data from the most recent follow-up in 2008-2010, when the children were aged 11.5 years. Children were seen by 39 pediatricians at the 31 polyclinics, herein labeled alphabetically from A to AE. PROBIT was registered with Current Controlled Trials as ISRCTN37687716.

In September 2007, we held a weeklong workshop to train pediatricians to perform the measurements to be collected in PROBIT children at age 11.5 years. Training included lectures and practical sessions. We also gave each pediatrician a manual of procedures and instructive DVD in Russian. Study visits commenced in January 2008. We held a re-training workshop in March 2008 and biannually thereafter. At the visits, pediatricians measured child height (using stadiometers with moveable headboards); circumferences (using non-stretchable measuring tapes); skinfold thicknesses (using Lange spring-loaded calipers); weight and bioelectrical impedance (using the TANITA TBF 300 GS body fat analyser); blood pressure (using OMRON 705IT devices and appropriately sized cuffs); and fasting glucose (using Roche Advantage Accu-Chek Glucometers). At the biannual re- training workshops we monitored follow-up rates, clarified procedures, and re-standardized measurements. Lastly, independent study monitors visited each clinic early in the data collection process for on-site monitoring visits, to ensure early correction of faulty/inconsistent techniques.

The study received ethical approval from McGill University Health Center Research Ethics Board; the Human Subjects Committees at Harvard Pilgrim Health Care; and the Avon Longitudinal Study of Parents and Children Law and Ethics Committee. Prior to any measurements being taken, a parent or legal guardian provided written informed consent and all children provided written assent. PROBIT has been registered as ISRCTN-37687716 and conforms to the CONSORT recommendations for the design, analysis, and reporting of cluster-randomized trials [11].

We received periodic data downloads from the data coordinating center in Minsk, approximately every three months. Upon receipt, we checked for implausible values or extreme outliers that might represent transcription or data entry errors. We reported all outliers to the data entry technicians, who corrected the database by comparing against the paper records, or in consultation with the pediatricians. We calculated ICCs on all continuous measurements using SAS version 9.2 (Cary, NC).

Addressing and correcting procedural inconsistencies

Because we recognized most of these procedural discrepancies among clinics early in the data collection process, we were able to address them early enough to have a beneficial effect on data quality. Our first step was to provide written clarification of any ambiguity in the manual of procedures, including a description of which stool and ruler to use for sitting height, the correct placement and tension of the measuring tape for waist circumference, and the importance of five minutes rest before measuring blood pressure.

At the first refresher training workshop, we reviewed measurement procedures for sitting height, waist circumference, and blood pressure. We discussed the protocol and reinforced the correct procedures with the pediatricians in a large group session, followed by small-group practice sessions. At subsequent refresher training sessions, we revisited these topics to ensure that there was no lingering confusion.

Because some clinics had been using non-standard sitting stools, we also decided to measure the height of the stool used at each polyclinic. In cases where we confirmed that a clinic had used an incorrect ruler or study stool, we adjusted the sitting height measurements accordingly.

While the large difference in sitting height is easy to detect visually with box plots, a similar equipment error might result in a smaller absolute difference that would not be so apparent. Our experience with sitting height measurement provides a rather extreme example of differences in equipment or technique across centers that are likely very common, and may not be so readily detected if smaller in magnitude.

Following our retraining and clarification efforts at subsequent workshops, we also noticed a decrease in the ICCs for waist circumference and systolic blood pressure (Table 1). The ICC for waist circumference displayed a steady, gradual decline from a peak of 0.11 to 0.07, while the ICC for systolic blood pressure dropped from 0.18 to 0.12. However, the remaining variability was higher than we would expect, especially as we measured blood pressure in triplicate using a well calibrated machine. Our results contrast with those of Vierron et al., who found that about 20% of the variability SBP measurement with mercury sphygmomanometer and Doppler probe was attributable to differences between centers, but identified no center effect for measurements with a semiautomatic device [12]. We hypothesize that differences in the demeanor of the pediatricians and the setting of the polyclinics (e.g. the presence or absence of central heat) may explain the residual clustering we observed, but further evaluation including centralized measurement would be necessary to confirm this hypothesis.

Although we did identify the cause of the head circumference ICC increase, we were unable to clarify the procedure in time to result in a substantial decrease in the ICC value. By the time we discussed this measurement technique with the pediatricians, the two clinics that had been incorrectly measuring head circumference had completed the majority of their study visits, and we could not reliably adjust the incorrect measurements.

For measurements that are highly operator dependent, such as measurement of skinfold thicknesses and circumferences, it is likely that the reproducibility of each individual's measurements increase with experience. However, it is uncertain whether this increased precision would improve the accuracy of measurement and consistency across centers. Therefore, ongoing evaluation using ICC's, followed by retraining, is essential to ensure that measurement techniques are optimally consistent across sites.

We suggest that investigators leading similar studies do their best to ensure that data entry and reporting to study investigators occur as quickly as possible, and that auditing and retraining workshops occur as soon as possible after data collection commences, optimally within about a month. While we recognized these precepts from the start of our study, the logistical challenges of leading a large study distributed across a country with transportation challenges and limited computer accessibility outside of the central data center in Minsk were such that we were unable to entirely eliminate measurement clustering. Centralized measurement would limit the influence of measurement clustering, but it was not feasible for participants to travel to a central site for research measurements in our study, and we suspect this would be true in most other studies as well.

While an ICC of 0 might be ideal for a characteristic that truly does not vary by cluster, it is likely that many characteristics vary between centers for reasons in addition to clustered variation in measurement technique [13]. For example, there may be geographic differences in the racial or ethnic compositions of populations served in different centers, or there may be regional differences in diet or other behaviors that lead to expected variation in characteristics such as weight or BMI. These differences are a particular concern in cluster-randomized trials, in which a limited number of often heterogeneous groups often makes it difficult for randomization to distribute potential sources of confounding evenly [14]. In Belarus, the population is relatively homogenous, which is reflected by the low ICC's (< = 0.03) for many measurements such as height, weight, upper arm circumference, and bioimpedance. Although some guidelines are available for expected values of ICC, especially among studies of adults [14], little information is available for anthropometric values among children. Given the lack of regional variation in our study population, we considered an ICC to be higher than expected if it was above 0.05. However, in other settings in which regional variation is more likely, investigators will need to the appropriate threshold for investigation, perhaps based on instrument-measured characteristics or centralized measurement of selected variables.

Even after correcting errors in measurement techniques and achieving low ICC's, some clustering remains, as we found in our analysis of the intervention upon sitting height, unadjusted and accounting for clustering. Therefore, even with excellent consistency of measurement resulting in low ICC's, it is still vital to account for clustering within center in statistical analyses.