Differential response effects of data collection mode in a cancer screening study of unmarried women ages 40–75 years: A randomized trial

Over 18 million women aged 40–75 in the United States are currently unmarried [1]. The designation of "unmarried" refers to women who are legally separated, divorced, widowed, or never legally married. Sexual minorities (e.g., lesbians and bisexuals) are an important segment of the unmarried female population. Sexual minority women may be living in committed relationships comparable to married heterosexual couples but are unable to legally marry in any state except Massachusetts. Although few studies include sufficient samples of unmarried women for analysis, data suggest that the risk for breast and cervical cancer may be higher for subgroups of unmarried women than women in general [2–5]. Therefore, the burden of disease and risk of adverse health outcomes may be greater for unmarried women if detection is delayed or forgone.

Determining effective modes of obtaining sensitive personal information is one important aspect to improving rates of cancer detection and control among unmarried women. Prior studies have documented that the methods used to elicit information can influence both individuals' willingness to disclose personal information and the quality of data that are obtained. The advantages of anonymity in self-administered questionnaires (SAQ) relative to telephone and face-to-face interviews have been demonstrated [6–8]. However, the use of paper-and-pencil SAQ is disadvantageous in terms of: lower unit response rates, higher levels of missing data, higher numbers of inconsistent or illogical responses across related questions, and limitations on questionnaire complexity such as skip patterns [9]. Conversely, significant advantages in data quality and flexibility of questionnaire format with telephone and face-to-face interviews have been well-documented, including quality of recorded answers, control of response order, and use of complicated skip patterns [9, 10].

Computer assisted self-interviewing (CASI) has advantages in data quality and questionnaire design flexibility comparable to telephone and face-to-face interviews [11–13]. However, studies comparing CASI to other modes of data collection for reporting sensitive behaviours have shown varying results. Some investigators have found mixed [12] or limited main effect differences [13–15] for CASI versus face-to-face interviews and SAQ. Others have found that CASI may be as good as, or even better than, face-to-face interviews at fostering a sense of privacy and increasing the willingness of respondents to report sensitive information [11, 16–19].

Respondent age, general trust in others, attitudes about privacy and confidentiality, and attitudes towards computers may influence reactions towards CASI [15, 20, 21]. These issues may be particularly relevant for women 40–75 years who are age-eligible for breast and cervical cancer screening, and who may have less experience with computers than younger women. In addition, sexual minority women may be particularly concerned about privacy and confidentiality.

There is limited information about the effect of interview mode on the willingness of unmarried middle-aged and older women to reveal personal information about cancer-related attitudes and practices. Without specific information about differential effects of data collection mode, it is impossible to determine the extent to which women are under-represented in surveillance and interventions because they are less likely to participate and/or are afraid to acknowledge potentially sensitive information. In addition, we must have methods that provide optimally valid and reliable self-reported behavioural and attitudinal data. Previous studies have found that women self-report cancer screenings at rates higher than indicated in clinical records [22–26]. These issues are particularly important as researchers and health care organizations seek valid, cost-effective forms of data collection for interventions to improve quality of care indicators. Therefore, the objectives of this study were to:

1. Describe the utility and feasibility of different modes for collecting data from middle-aged and older unmarried women;

2. Examine the effects of randomized interview mode on responses to indicators of mammography and Pap test screening; and

3. Determine whether the effects of interview mode on responses to indicators of mammography and Pap test screening differ by partner gender.

The study design included a three-step process: recruitment, allocation to eligibility strata, and randomization to data collection mode.

Our findings contribute preliminary evidence of the effect of interview mode on responses to indicators of cancer screening behaviours among middle-aged and older heterosexual and sexual minority women. These findings add to the body of research about methods that can be used to best identify subgroups of the population most at risk for not receiving recommended cancer screenings. Women were randomly assigned to one of three data collection methods: computer assisted telephone interview (CATI), self-administered mailed questionnaire (SAMQ) and computer-assisted self-interview (CASI). Women assigned to CASI could choose to complete the assessment during an in-person CASI (CASI-I) or by receiving the questionnaire on disk (CASI-D).

We examined the effects of randomized interview mode on responses to items associated with mammography and Pap test screening. Overall, we found few meaningful differences by mode of data collection for indicators of cancer screening. Surprisingly, among the few significant mode differences, we found that women who were interviewed by research staff (CATI) were more likely than those not interviewed (CASI, SAMQ) to have an unfavourable status on the indicators. Women in the CATI mode were more likely to report being off-schedule for recent Pap testing than women in CASI and the trend was similar, but non-significant, for mammography. The other significant findings associated with cancer screening behaviours were between CASI conditions. Because we did not randomize women into the different computer-assisted methods, we cannot rule out selection bias as a threat to the validity of the findings. Furthermore, given the lower response rate in the CASI condition (Figure 1), apparently higher rates of recent screening among women in CASI may be due to the fact that those who completed the assessment were also those most knowledgeable about cancer screening recommendations. Therefore, we cannot conclude that any mode of data collection has a consistent effect on rates of reporting screening behaviours.

There are potential reasons why we did not find consistent mode differences in our sample. First, items about cancer screenings may not be considered sensitive or associated with social rejection since questions about mammography and Pap testing are routinely asked of women 40–75 years in clinical settings. Second, many of the studies that showed differences between CASI and other modes of data collection were conducted and published in the early and late 1990s [11, 14–16, 36, 37]. At that time, CASI was a novel interview mode. The increased access to, and use of, computers may explain why we did not find more significant differences between CASI and the other data collection modes. Finally, due to the relatively high percentages of women reporting mammography and Pap testing at recommended intervals (more than 80% for both behaviours), we may not have had sufficient power to detect statistically significant differences. With a sample size of 364 for comparisons between CATI and CASI, we only had statistical power of 0.78 to detect differences in means of 0.10 or higher with a standard deviation of 0.35. Similarly, we only had statistical power of 0.80 to detect comparable differences in means of 0.10 or higher between CATI and SAMQ with a sample size of 387. However, because the percentages of endorsement were remarkably consistent across mode for several items, it is not likely that increased sample sizes would change the conclusions substantially.

Another study objective was to determine whether the effects of interview mode differed by partner gender. We found only two significant mode differences for items related to self-reported mammography and Pap test screening by partner gender. There are several potential reasons that may explain the lack of more significant findings. First, Rhode Island is one of only a few states in the United States to have non-discriminatory policies towards sexual minorities. Therefore, within the political and social context, women in Rhode Island may be more willing than women in other parts of the country to disclose potentially unfavourable information. Second, all women interested in study participation were required to answer screening questions about marital status and partner gender prior to study enrollment. Asking these screening items provided women with examples of the types of questions that would be asked in the study. Women who considered these items too personal may have declined study participation. Finally, sample size, particularly for sexual minority women, may have limited our ability to detect important mode differences.

In the CASI condition, WPW were significantly more likely to select the mailed computer disk than WPM/NPP when given a choice of completing the assessment by a laptop provided by the research team or by a disk mailed to the participant's home. WPW were also more likely to have a college education, be employed full or part-time, have higher incomes and identify as White than WPM/NPP. Therefore, it is likely that WPW had greater access to, and experience with, computers than WPM/NPP and were able to complete the assessment independent of assistance from a research assistant with a laptop computer. Given our findings, we encourage future studies to further explore women's preferences for data collection methods and whether mode of data collection influences the responses of middle-aged and older sexual minorities.

Our findings also provide information about the feasibility of different methods for collecting data from a traditionally under-represented group of women. Of the 630 women who were eligible and enrolled in the study, 95% agreed to be randomized to one of three modes of data collection. Not surprisingly, women who were more likely to have access to a computer (e.g., more education, employed, white race) chose CASI-D. Women who refused randomization (self-choice) were more likely to have less than a college degree, to identify as Hispanic, and to choose SAMQ. Despite the informed consent process, women in the self-choice option may not have completely understood the concept of randomization and been concerned about the implications of agreeing to randomization. They may have chosen the mode that was most familiar to them, offered the most perceived anonymity, and provided the greatest degree of flexibility in completing the assessment (e.g., time and availability of assistance with question understanding).

We obtained an overall response rate of 93%. This response rate is generally higher than for most other studies, particularly SAMQs, and is a strength of our study because of the low potential for non-response bias. The high response rate is likely a result of the initial contact we had with women during recruitment and screening for eligibility. Unfortunately, we do not have data to inform other studies of comparable populations that do not employ similar pre-survey contact with participants.

Despite the high overall response rate, we found noteworthy differences in response rate by mode. The response rates for CATI and SAMQ were over 95%, while only 86% for CASI. The lower response rate by computer was not unexpected given other mode experiments [19] and the age of the participants. There were likely some women with less experience using computers who, despite initially agreeing to participate, worried about their ability to correctly use the software or feared unknown potential consequences of responding to a computer program. Additionally, women may have had technical difficulties with the computer that we were unaware of because they indicated that they were no longer interested in study participation rather than acknowledging problems with computer software.

We also found that more contact attempts with participants were required for CASI relative to SAMQ and CATI to achieve comparable response rates (Figure 2). Furthermore, the estimated costs per randomized participant were approximately $60 for CASI compared to $30 for SAMQ and $20 for CATI. Within the CASI condition, the cost per participant was about $115 for CASI-I and $20 for CASI-D. Had we used Internet-based data collection, the costs associated with CASI would have been substantially lower. However, the sample would have been biased towards women with higher socioeconomic positions who had access to a computer. Women in our sample who chose in-person CASI were more likely to identify as a racial minority, to be less educated and not employed compared to those who chose to complete the questionnaire on a disk that was mailed to them.

In addition to sample size, there are a number of other study limitations. First, to include sufficient numbers of sexual minorities, we used non-probability based sampling methods. Our sample was highly educated, predominantly white, and employed, with relatively higher incomes. Unfortunately, because sexual orientation is not asked of all individuals in the Census or on any large state-wide population-based survey, we do not have data to compare our sample to the eligible Rhode Island population. Therefore, care should be taken when generalizing our findings. We also did not use methods to verify self-reported data and cannot confirm whether there was substantial over- or under-reporting where differences were observed across modes. Finally, we cannot discern which mode provided the most accurate estimates of true behaviour, nor can we distinguish the extent to which differences across modes indicate differences in accuracy of reports as opposed to mode artefacts. However, given the few statistically significant differences, it appears that the incidence of mode artefacts is low.

Using computer-assisted self-interviewing for surveillance and intervention studies may result in lower response rates than telephone interviewing or self-administered mailed questionnaires. However, there does not appear to be consistent differences by mode of data collection for responses to indicators of mammography and Pap test screening among middle-aged and older women who complete the assessment. Therefore, other measures such as efficiency and cost-effectiveness of the mode should also be considered when determining the most appropriate form of data collection for use in monitoring indicators of cancer detection and control.