A comparison of small monetary incentives to convert survey non-respondents: a randomized control trial

Abstract

Background

Maximizing response rates is critically important in order to provide the most generalizable and unbiased research results. High response rates reduce the chance of respondents being systematically different from non-respondents, and thus, reduce the risk of results not truly reflecting the study population. Monetary incentives are often used to improve response rates, but little is known about whether larger incentives improve response rates in those who previously have been unenthusiastic about participating in research. In this study we compared the response rates and cost-effectiveness of a $5 versus $2 monetary incentive accompanying a short survey mailed to patients who did not respond or refused to participate in research study with a face-to-face survey.

Methods

1,328 non-responders were randomly assigned to receive $5 or $2 and a short, 10-question survey by mail. Reminder postcards were sent to everyone; those not returning the survey were sent a second survey without incentive. Overall response rates, response rates by incentive condition, and odds of responding to the larger incentive were calculated. Total costs (materials, postage, and labor) and incremental cost-effectiveness ratios were also calculated and compared by incentive condition.

Results

After the first mailing, the response rate within the $5 group was significantly higher (57.8% vs. 47.7%, p < .001); after the second mailing, the difference narrowed by 80%, resulting in a non-significant difference in cumulative rates between the $5 and $2 groups (67.3% vs. 65.4%, respectively, p = .47). Regardless of incentive or number of contacts, respondents were significantly more likely to be male, white, married, and 50-75 years old. Total costs were higher with the larger versus smaller incentive ($13.77 versus $9.95 per completed survey).

Conclusions

A $5 incentive provides a significantly higher response rate than a $2 incentive if only one survey mailing is used but not if two survey mailings are used.

Background

In survey research maximizing response rates is critically important in order to provide the most generalizable and unbiased results. High response rates reduce the chance of respondents being systematically different from non-respondents, and thus, reduce the risk of results not truly reflecting the study population. Even small proportions of non-response have been shown to bias study findings and lead to spurious conclusions[1]. Offering nominal financial incentives for participating in survey research is a common practice and often a cost-effective method to improve response rates[2–4]. While some, including institutional review boards, have questioned whether monetary incentives provide an inappropriate influence on potential participant's decisions to participate or not[5, 6], evidence suggests that nominal incentives are a harmless approach for improving response rates[7]. Systematic reviews of randomized trials of monetary incentives have found that response rates increase when: 1) any incentive is offered versus no incentive[2]; 2) incentives are unconditional (incentives are pre-paid with survey mailing and not dependent on survey completion)[2]; and, 3) larger versus smaller monetary incentives are sent[8]. Studies have also shown that incentives versus no incentives increase response rates for certain populations that typically have high rates of refusals in research studies, and therefore are underrepresented in research, including those with lower levels of education, income[9, 10] and non-whites[9, 11]. Less is known, however, about whether larger incentives improve response rates among people who have refused requests to participate in previous research[12–15] or whether incentives in such groups are cost-effective [10, 16–19].

In this study we examine how different cash incentives ($5 versus $2) attached to an ancillary mailed survey affect response rates in a group of patients who either passively or actively declined participation in a face-to-face survey with a $25 incentive. The primary study hypothesis was that those randomly chosen to receive a $5 incentive would be more likely to respond than those who received a $2 incentive and that the $5 incentive would be more cost effective than the $2 incentive. We also hypothesized that, compared to $2, a $5 incentive would improve response rates for groups with traditionally lower response rates (i.e., non-whites, lower educational attainment, low income and in poor health).

Methods

Parent study population

This incentive study was an ancillary study of non-respondents from a larger face-to-face survey of veterans. The study population for the larger main study included all primary care patients at four Veterans Health Administration (VHA) medical centers (Minneapolis, MN; West Los Angeles, CA; Portland, OR; Durham, NC) who were scheduled to have at least one primary care visit during the study recruitment period (June, 2004 through May, 2005) and who did not suffer from a severe cognitive disorder (i.e., Alzheimer's disease, severe dementia, schizophrenia) or blindness, as determined from an initial review of medical records. Invitations to participate in the main study, which included a face-to-face interview at the medical center to assess the patient's health literacy skills and an offer of a $25 cash incentive for completing the interview, were mailed to randomly selected patients at each site and then participants were recruited by phone. Study recruiters telephoned each potential participant to determine their willingness to participate approximately 10 days after the mailed invitations were sent. Six attempts were made to reach participants at different times of day. Patients were classified into willing participants, hard refusers (e.g., did not want anything to do with research), soft refusers (e.g., could not participate because of logistical reasons), and those whom we could not reach by phone or mail. Those who were reached and willing to participate booked a one-hour research appointment, usually on the same day as their scheduled primary care appointment[20]. Institutional Review Boards from the each study site (Minneapolis, MN; West Los Angeles, CA; Portland, OR; Durham, NC) approved the study protocol.

Ancillary study population

In order to assess the effect of non-participation on prevalence estimates of poor health literacy, eligible patients from the parent study who could not be reached, did not attend their scheduled research appointment, or refused because of transportation, scheduling difficulties, or other conflicts were mailed a one page, plain language, ancillary survey designed to characterize non-responders. This group of parent study non-respondents, as outlined in Figure 1, comprises the sample for this study.

Figure 1

Participant flow chart for parent and ancillary studies.

Data and measurement

Data from the ancillary survey and medical record and administrative sources were used as independent variables in this investigation. Sex, age (< 50, 50-75, > 75), urban/rural residence (determined from U.S. census data), comorbidity history and mental health diagnoses were extracted from VHA administrative and medical record data; therefore, data were available for ancillary survey respondents and non-respondents. Comorbidities were summarized using the Charlson Comorbidity Index score[21] and a measure of mental health diagnoses which categorized individuals into one of three groups: (1) no mental health diagnoses, (2) at least one psychiatric (ICD-9 codes 290-302 and 306-311) or substance abuse related (ICD-9 codes 303-305) diagnosis, or (3) dual diagnosis (psychiatric and substance abuse). Measures of mental health diagnoses were included because they could conceivably affect the accuracy of survey response and are not captured in the Charlson Comorbidity Index. Ancillary survey questions included 4 self-reported health literacy questions[22], marital status (recoded into married versus unmarried), race (white, African American, other), education (≤ high school, some college, ≥ college graduate), employment status (employed versus not employed) and income (recoded into ≥ $20,000, $20-40,000, > $40,000).

Ancillary survey administration

The ancillary survey was administered by a university-affiliated survey center not associated with the parent study or its investigators. In three of the medical center sites, patients were randomly assigned to either a $2 or $5 prepaid cash incentive condition, and the data from these three sites were used for this paper. The fourth site did not participate in the incentive experiment. Survey packets were mailed using first-class postage to 1,328 patients. Participants were assigned random numbers between 0 and 1 and then based on the number (< 0.5 or > 0.5), participants split into two equal groups. Randomization was performed blindly, with no information about individual cases to affect the distribution. The first survey packet included the cash incentive, a cover letter, a pre-addressed postage-paid business envelope and the 10-item questionnaire. A reminder postcard was mailed approximately one week after the first packet. A second questionnaire packet without an incentive was mailed to any subject who did not return a blank or completed survey within 3-4 weeks of the first mailing. Surveys returned before the mailing of the second questionnaire were categorized "first mailing respondents." Surveys received after this period were categorized "second mailing respondents."

Analysis

All analysis was completed using SAS version 9.1. To determine the success of the randomization, contrasts of the respondent's demographic and health characteristics by their incentive condition were first compared using chi-square tests. Significant differences found from these results were later controlled for in logistic regressions. Response rates were calculated using American Association of Public Opinion Research (AAPOR) RR1 criteria[23]. Differences in response rates by incentive group were compared by demographic and health factors. Cumulative response rates by incentive group ($2 or $5) were then compared for the "first mailing participants" and then for the "second mailing participants." Logistic regression was then used to calculate the odds of responding to the survey with a $5 versus a $2 incentive, controlling for variables that were not successfully randomized (sex, rural/urban status, and site), as well as incentive. Because other demographic and health characteristics were not significantly different in bivariate analyses, these variables were not included in the regression analyses used to calculate odds ratios.

The cost of each survey packet included materials (1 page survey, outgoing and return envelopes), postage (outgoing and return) and labor. Labor costs per survey were calculated by multiplying the hourly rate, including benefits, in the year the survey was fielded (2005) by the time that employees and supervisors spent stamping, sealing and stuffing packets as well as logging returns and data entry of survey results, divided by the number of surveys sent. Time spent processing returned surveys was calculated for the total response to each mailing, without regard to incentive status. The cost of the reminder postcard includes printing, labor and outgoing postage. The reminder postcard was sent to all recipients of the first mailing and is included in the total cost of that mailing. The cost per completed survey was determined for each round of mailing by dividing the total cost of the mailings by the number of completed surveys returned. Indirect costs were not included.

In order to assess whether ancillary survey respondents were representative of all those to whom the survey was sent, demographic and health status of respondents for each incentive condition and mailing were compared to the entire ancillary study sample.

Results

Sample characteristics

Demographic and health characteristics are shown in Table 1. There were no significant differences between the incentive groups in age, race, marital status, Charlson comorbidity score or mental health diagnoses. The $5 incentive group had a higher proportion of men and those with a rural address than the $2 incentive group. Among survey responders, there were no differences in either level of education completed, or self-reported annual income (these data were not available for non-responders).

Table 1 Demographic and Health Characteristics by Incentive Condition

Response rates

A total of 881 (66%) of the 1,328 patients completed and returned the questionnaire. As shown in Table 2, unadjusted response rates were significantly higher in the $5 incentive group than the $2 incentive group after the first mailing and reminder postcard (58% and 48%, p = 0.0002). (Table 2). There were no significant differences in response rates after the second survey mailing. The final response rates for the $5 and $2 incentives were 67% and 65%, respectively (p = 0.47). While response rates differed significantly across all demographic and health categories after the first mailing, with older, white, married and male participants, and those with two or more physical comorbidities and no mental health conditions having higher response rates, cumulatively, after both mailings, response rates showed no significant differences across demographic categories.

Table 2 Unadjusted and Adjusted odds of response by patient characteristics and incentive condition

These same patterns persisted after adjusting for urban/rural status and sex. After the first mailing, adjusted odds of recipients of the $5 incentive responding versus recipients of the $2 incentive responding show that, overall, recipients of the $5 incentive were 50% more likely to respond than those who received the $2 incentive (Table 2). Odds of the $5 recipients responding were significantly higher within most sub-groups, except for women, those aged 50-75, not married, those with few physical comorbidities, substance use or substance use/psychiatric diagnoses, and those living in a rural area. After the final mailing, however, there were no significant differences in odds of responding across any of the demographic categories.

Table 3 shows the results of the cost analysis. The total cost per survey for the first mailing, which included the survey packet, the $2 or $5 incentive and the reminder postcard was $5.25 for the $2 incentive group and $8.25 for the $5 group; the cost of the second mailing was $2.40 per survey. The cost per completed survey (cost of the mailings\number of surveys returned) overall was lower in the $2 incentive group than the $5 group after both the first and second mailing.

Table 3 Total costs comparing $2 and $5 conditions and one versus two mailings

Table 4 displays the representativeness of respondents to the total ancillary study population. These data show that the two incentive conditions provide equally representative samples with the exception of urban/rural status, where the $5 incentive condition, but not the $2 condition, produced a respondent population more heavily weighted toward those living in rural areas than the population sampled.

Table 4 Representativeness of Responders on Demographic Characteristics

Discussion

Reasons for not participating in research are numerous, but typically are demarcated by researchers not being able to reach participants (e.g., incorrect or unavailable address or phone number) or participants not interested in (e.g., refusals) or not able to complete (e.g., poor literacy, limited capacity to understand study protocol) a study[24, 25]. Gathering as much information as possible about non-responders in order to assess potential bias of study results is often endorsed by survey methodologists[26]. One option is to re-contact non-responders and ask to gather a small set of critical data that will allow for a basic description of the non-responders. Re-contacting initial non-responders, however, is potentially costly, especially because it is unclear what the likelihood is that non-responders will convert to responders; therefore, maximizing the effectiveness of incentives for the greatest response is important. In this study we evaluate if monetary incentives and multiple mailings are effective methods for increasing response rates in a sample of participants that had been difficult to reach or unavailable or unwilling to participate in a longer, more complex study that required face-to-face interviews.

Survey methodologists recommend the use of incentives and multiple mailings to increase response rates[8, 27, 28] and our findings support this approach. Like previous studies of patient populations, our results show that a $5 incentive produces higher overall response rates than a $2 incentive[19, 21]. However, our study also showed that, while the response rate in the $5 incentive group was significantly higher after the first mailing, the response rates were relatively equal after a second mailing. We also found that the respondents across incentive conditions, with the exception of urban/rural status, represented the overall ancillary study population. Parkes et al, in a case-control study, randomized 2561 controls to receive no incentive, $2 or $5 and had comparable findings to ours: those receiving a $5 incentive, even after multiple mailing and reminder phone calls, had comparable response rates to those receiving a $2 incentive, although the non-significant difference in response rates (4.2%) was slightly higher than we observed[21]. Shaw et al., in their community-based survey of 1800 health plan enrollees, tested differences in survey response rates with either $2 or $5 cash incentives and found that although a $5 incentive yielded a higher response rate after one mailing, with multiple mailings the response rate from the $2 incentive was reasonable and adequate[19]. The consistency of our findings with others suggests that regardless of whether participants are being contacted for the first time, as was the case in Parkes' and Shaw's studies, or have been difficult to reach previously or hesitant to participate in prior research, as was the case in our study, the conclusions are relatively similar: with a restricted timeline, a $5 incentive will provide a higher response rates for most groups, yet if time permits, a $2 incentive with multiple contacts may be sufficient to yield acceptable response rates.

Our cost analysis suggests that it may be more cost effective to have multiple contacts than to provide an increased incentive in order to maximize response rates. We found the best "bang for the buck" was a smaller incentive with two mailings when costs were compared relative to response rates. With an initial higher response rate for the $5 group after the first mailing, one might conclude that the extra incentive was well spent. However, the total cost per returned survey shows that response rate was not high enough to decrease the overall costs. Moreover, the cost of the second survey packet ($2.40) was less than the $3 difference in incentives, making the $2 incentive with multiple mailings the more economical method. What we are not able to distinguish from our analysis, however, is why this is the case. While it is likely that some proportion of respondents will respond quickly regardless of the incentive amount, it is not clear if the $5 incentive also entices some of those who otherwise would hesitate to respond more promptly. If the probability of response does increase within this hesitant group, then we would expect the respondents to the first mailing in the $2 incentive condition to include mostly those who would respond regardless of the incentive amount. Therefore, the effect of a second mailing may be stronger than the effect of increasing the incentive by $3 in the first mailing for the $2 incentive group because the remaining participants include more hesitant non-responders.

While several studies have shown that larger incentives significantly improve the odds of responding for groups with low-income, low educational attainment, gender and non-white race[8, 29], fewer have investigated differences in other sub-groups, such as marital status or poor physical or mental health, factors that may, in addition to the incentive, affect either external support or personal capacity for responding to a survey. Like the cumulative response rates, our findings show higher odds of responding with a $5 incentive within every demographic and health sub-group after the first mailing, but no significant differences overall after both mailings, suggesting that with greater response time and multiple mailings, the effect of an increased incentive is negligible. It should be noted, however, that although there are no significant differences across incentive groups after both mailings, the absolute response rate for those less than 50 years, women, non-whites, those unmarried or with chronic physical or mental illness is low. Our data suggest that, perhaps with the exception of those with chronic physical or mental illness, these groups are underrepresented compared to the overall population and it is possible that neither incentives nor multiple mailings may entice them to participate. These results suggest that additional studies need to be designed in order understand reasons for high non-participation rates in hard-to-reach groups and that more innovative strategies, including but not limited to incentives, need to be developed and tested to encourage adequate representation of these populations in research.

This study is tempered by a number of limitations. First, with the design of our study we assume that response is based on incentive condition, but other unmeasured factors, could have varied by condition and accounted for some of the differences in response rates or timing of response. Second, it is unclear why randomization was not successful, with more women and fewer rural residents in the $2 incentive group. Because randomization does not guarantee balance in any variable among treatment groups, there is always some amount of chance that imbalance between groups will occur. In order to account for this limitation in our randomization strategy, we adjusted for these variables in our analyses. Third, while we have some demographic and health data on ancillary survey non-respondents, we do not have income or education for the full sample, limiting our ability to assess the effect of these variables on non-response bias.

Conclusions

We conclude that for studies with a limited budget, increasing the incentive from $2 to $5 is less effective than a subsequent mailing. If time is limited or only one mailing is possible, a $5 incentive leads to higher initial response. However, regardless of incentives and number of mailings, some demographic groups may not respond and innovative strategies are needed assure adequate representation of these groups.