Cohort ascertainment and data collection processes have been described elsewhere.[9] In brief, we identified potentially eligible subjects for inclusion in a cohort study from electronic databases at six of the Cancer Research Network (CRN) health care delivery systems: Group Health, Washington; Kaiser Permanente Southern California; Lovelace, New Mexico; Henry Ford Health System, Michigan; Health Partners, Minnesota; and Fallon Community Health Plan, Massachusetts. The CRN is a consortium of healthcare delivery systems, funded by the National Cancer Institute to increase the effectiveness of preventive, curative and supportive interventions that span the natural history of major cancers.
Women aged 65 years or older and diagnosed with primary, histologically confirmed, early stage unilateral breast cancer between January 1, 1990 and December 31, 1994 were eligible for inclusion in the study. Before the start of medical record-based data collection, we collected electronically demographic, tumor, treatment, and surgeon data from cancer registry, administrative, and clinical databases at each site. Medical record abstractors verified (at sites with cancer registries) or determined (at sites without cancer registries) eligibility for each identified subject and collected any information that had not been available from electronic sources. We abstracted data from the date of initial diagnosis through death, disenrollment from the health plan or 10 years post-diagnosis. There were no control or intervention strategies in this study. We describe the variability in electronically available data across six sites and report efficiencies gained from implementing an electronic data collection (EDC) instead of the originally proposed paper-based data collection.
The EDC system
We developed a computer-based automated menu-driven EDC system using Microsoft® Access 2000. The "back end" of the system consisted of six tables that stored 658 exported data variables after completion of data abstraction in the "front end" of the system. The "front end" of the system was organized into five forms for collecting information on (1) eligibility, patient characteristics, tumor characteristics, and treatment, (2) diagnosis and treatment of recurrence and second primary cancer, (3) comorbidity at three time points, (4) surveillance testing for recurrent breast cancer after completing primary therapy, and (5) mammography screening. All forms were menu-driven, in a tabular format (as shown in Figure 1), and linked by a unique study subject identification (ID) number. Each site maintained a Microsoft® Excel file that linked each subject's study number in the EDC to the subject's original medical record number to allow local access to electronic data and medical records as needed. Study specific queries and macros were programmed to allow for toggling between the forms, verification of input data, final checking for completeness of the data collected, and for export into the "back-end" database. Consistent with data use agreements between Boston Medical Center and the data collection sites, and per HIPAA agreements, personal identifiers such as surgeons' names and patients' day of birth were deleted before exporting into the "back-end" database.
For each site, we preloaded into the EDC all study ID numbers for potentially eligible subjects along with all electronically available data. Each site-specific EDC system was configured for use on individual computers. To ensure cross-site consistency, one person trained medical record abstractors at each participating site using local, de-identified medical records as sample records. For enhanced efficiency a 10% sampling scheme was implemented to capture a sub-sample of the stage I, non-Hispanic White, less than 80 years age group at Kaiser Permanente Southern California, the largest site.
To begin the abstraction, the abstractor chose the subject ID from a pull down list on the Demographic, Breast Cancer and Treatment Form and all electronic data available for this ID would fill the corresponding data fields in the electronic abstraction forms. Each abstraction began with verification of the eligibility criteria, allowing in one step the abstractor to continue if further record abstraction was indicated or to stop immediately and move on to the next case. Upon confirmation of eligibility, stage, age and race, if the sampling quota had been met for the specified group, the abstractor was prompted to move on to the next subject. The EDC operated on the premise that all cases were in the process of completion, thereby allowing editing of all pre-filled data elements. Extensive quality control procedures were included in the EDC to minimize abstraction error. These procedures included (1) range checks for dates and value responses, (2) logic checks prompting the abstractor to verify the answer if it was either an out-of-range value or the response was not feasible given other clinical information, and (3) command buttons with embedded coding information to assist in data abstraction. The abstractor was able to set the abstraction to complete electronically only when all data and logic checks had been satisfied. Once the data abstraction was complete and passed the pre-programmed check for completeness and logical consistency (Final Check), the record was exported to the "back-end" database. At the beginning of each month, all data were sent via a secure internet transfer site to the data coordinating site at Boston Medical Center in electronic format. Data from each electronic source within and across participating sites were merged and cleaned at the data coordinating site in preparation for analyses, using SAS statistical software.[10]
The inter-/intra-rater reliability (IRR) system
We used a completely automated triple abstraction process to incorporate quality controls to reduce inter- and intra-site variability and to improve accuracy of data collection. We developed a Microsoft® Access-based IRR system using a modified version of the automated EDC system. The IRR system contained a subset of 54 key data elements (of 658 collected by the EDC), and was organized on a single form with tabular format to differentiate five areas of interest for evaluating data reliability and consistency: tumor characteristics, tumor treatment, development of recurrence or second primary, comorbidity, and surveillance mammography. The same range checks, logic checks, and a final check for completion as in the EDC system were programmed for these key variables. The IRR system contained in its "back end" tables the data from original abstractions on five records for each abstractor, randomly selected from records completed three months prior to the IRR exercise. For re-abstractions, record for each subject ID in the IRR system was pre-populated with the same data that was preloaded into the EDC system for the original medical record abstraction.
The key data elements were re-abstracted and entered directly into the IRR system by the original abstractor (for intra-rater reliability comparison) and by the site project manager (for inter-rater reliability comparison). Upon completion of all five records by an abstractor, the IRR system compared re-abstracted data with the original abstraction and the pre-programmed reporting function in the IRR system provided both the number of mismatches and percent agreement for each record re-abstracted and by sub-areas in the abstraction instrument. The IRR data were sent by each site to the Boston Medical Center and pooled to determine a study-wide reliability rate. Reports were generated for each abstractor by subject ID, comparing the re-abstractions to the original abstraction, aggregated by content sections of compared data (tumor characteristics, treatment, recurrence/second primary, comorbidity, and surveillance mammography), and disaggregated into each data element. The number of mismatches and percent agreement were electronically calculated for all items by subject ID and automatically displayed in three categories: total, preloaded variables, and non-preloaded variables. Data were shared with the abstractors to develop strategies to reduce errors. IRR exercises were conducted once each during the first and second halves of the data collection period.