A comparison of small monetary incentives to convert survey non-respondents: a randomized control trial

In survey research maximizing response rates is critically important in order to provide the most generalizable and unbiased results. High response rates reduce the chance of respondents being systematically different from non-respondents, and thus, reduce the risk of results not truly reflecting the study population. Even small proportions of non-response have been shown to bias study findings and lead to spurious conclusions[1]. Offering nominal financial incentives for participating in survey research is a common practice and often a cost-effective method to improve response rates[2–4]. While some, including institutional review boards, have questioned whether monetary incentives provide an inappropriate influence on potential participant's decisions to participate or not[5, 6], evidence suggests that nominal incentives are a harmless approach for improving response rates[7]. Systematic reviews of randomized trials of monetary incentives have found that response rates increase when: 1) any incentive is offered versus no incentive[2]; 2) incentives are unconditional (incentives are pre-paid with survey mailing and not dependent on survey completion)[2]; and, 3) larger versus smaller monetary incentives are sent[8]. Studies have also shown that incentives versus no incentives increase response rates for certain populations that typically have high rates of refusals in research studies, and therefore are underrepresented in research, including those with lower levels of education, income[9, 10] and non-whites[9, 11]. Less is known, however, about whether larger incentives improve response rates among people who have refused requests to participate in previous research[12–15] or whether incentives in such groups are cost-effective [10, 16–19].

In this study we examine how different cash incentives ($5 versus $2) attached to an ancillary mailed survey affect response rates in a group of patients who either passively or actively declined participation in a face-to-face survey with a $25 incentive. The primary study hypothesis was that those randomly chosen to receive a $5 incentive would be more likely to respond than those who received a $2 incentive and that the $5 incentive would be more cost effective than the $2 incentive. We also hypothesized that, compared to $2, a $5 incentive would improve response rates for groups with traditionally lower response rates (i.e., non-whites, lower educational attainment, low income and in poor health).

Reasons for not participating in research are numerous, but typically are demarcated by researchers not being able to reach participants (e.g., incorrect or unavailable address or phone number) or participants not interested in (e.g., refusals) or not able to complete (e.g., poor literacy, limited capacity to understand study protocol) a study[24, 25]. Gathering as much information as possible about non-responders in order to assess potential bias of study results is often endorsed by survey methodologists[26]. One option is to re-contact non-responders and ask to gather a small set of critical data that will allow for a basic description of the non-responders. Re-contacting initial non-responders, however, is potentially costly, especially because it is unclear what the likelihood is that non-responders will convert to responders; therefore, maximizing the effectiveness of incentives for the greatest response is important. In this study we evaluate if monetary incentives and multiple mailings are effective methods for increasing response rates in a sample of participants that had been difficult to reach or unavailable or unwilling to participate in a longer, more complex study that required face-to-face interviews.

Survey methodologists recommend the use of incentives and multiple mailings to increase response rates[8, 27, 28] and our findings support this approach. Like previous studies of patient populations, our results show that a $5 incentive produces higher overall response rates than a $2 incentive[19, 21]. However, our study also showed that, while the response rate in the $5 incentive group was significantly higher after the first mailing, the response rates were relatively equal after a second mailing. We also found that the respondents across incentive conditions, with the exception of urban/rural status, represented the overall ancillary study population. Parkes et al, in a case-control study, randomized 2561 controls to receive no incentive, $2 or $5 and had comparable findings to ours: those receiving a $5 incentive, even after multiple mailing and reminder phone calls, had comparable response rates to those receiving a $2 incentive, although the non-significant difference in response rates (4.2%) was slightly higher than we observed[21]. Shaw et al., in their community-based survey of 1800 health plan enrollees, tested differences in survey response rates with either $2 or $5 cash incentives and found that although a $5 incentive yielded a higher response rate after one mailing, with multiple mailings the response rate from the $2 incentive was reasonable and adequate[19]. The consistency of our findings with others suggests that regardless of whether participants are being contacted for the first time, as was the case in Parkes' and Shaw's studies, or have been difficult to reach previously or hesitant to participate in prior research, as was the case in our study, the conclusions are relatively similar: with a restricted timeline, a $5 incentive will provide a higher response rates for most groups, yet if time permits, a $2 incentive with multiple contacts may be sufficient to yield acceptable response rates.

Our cost analysis suggests that it may be more cost effective to have multiple contacts than to provide an increased incentive in order to maximize response rates. We found the best "bang for the buck" was a smaller incentive with two mailings when costs were compared relative to response rates. With an initial higher response rate for the $5 group after the first mailing, one might conclude that the extra incentive was well spent. However, the total cost per returned survey shows that response rate was not high enough to decrease the overall costs. Moreover, the cost of the second survey packet ($2.40) was less than the $3 difference in incentives, making the $2 incentive with multiple mailings the more economical method. What we are not able to distinguish from our analysis, however, is why this is the case. While it is likely that some proportion of respondents will respond quickly regardless of the incentive amount, it is not clear if the $5 incentive also entices some of those who otherwise would hesitate to respond more promptly. If the probability of response does increase within this hesitant group, then we would expect the respondents to the first mailing in the $2 incentive condition to include mostly those who would respond regardless of the incentive amount. Therefore, the effect of a second mailing may be stronger than the effect of increasing the incentive by $3 in the first mailing for the $2 incentive group because the remaining participants include more hesitant non-responders.

While several studies have shown that larger incentives significantly improve the odds of responding for groups with low-income, low educational attainment, gender and non-white race[8, 29], fewer have investigated differences in other sub-groups, such as marital status or poor physical or mental health, factors that may, in addition to the incentive, affect either external support or personal capacity for responding to a survey. Like the cumulative response rates, our findings show higher odds of responding with a $5 incentive within every demographic and health sub-group after the first mailing, but no significant differences overall after both mailings, suggesting that with greater response time and multiple mailings, the effect of an increased incentive is negligible. It should be noted, however, that although there are no significant differences across incentive groups after both mailings, the absolute response rate for those less than 50 years, women, non-whites, those unmarried or with chronic physical or mental illness is low. Our data suggest that, perhaps with the exception of those with chronic physical or mental illness, these groups are underrepresented compared to the overall population and it is possible that neither incentives nor multiple mailings may entice them to participate. These results suggest that additional studies need to be designed in order understand reasons for high non-participation rates in hard-to-reach groups and that more innovative strategies, including but not limited to incentives, need to be developed and tested to encourage adequate representation of these populations in research.

This study is tempered by a number of limitations. First, with the design of our study we assume that response is based on incentive condition, but other unmeasured factors, could have varied by condition and accounted for some of the differences in response rates or timing of response. Second, it is unclear why randomization was not successful, with more women and fewer rural residents in the $2 incentive group. Because randomization does not guarantee balance in any variable among treatment groups, there is always some amount of chance that imbalance between groups will occur. In order to account for this limitation in our randomization strategy, we adjusted for these variables in our analyses. Third, while we have some demographic and health data on ancillary survey non-respondents, we do not have income or education for the full sample, limiting our ability to assess the effect of these variables on non-response bias.

We conclude that for studies with a limited budget, increasing the incentive from $2 to $5 is less effective than a subsequent mailing. If time is limited or only one mailing is possible, a $5 incentive leads to higher initial response. However, regardless of incentives and number of mailings, some demographic groups may not respond and innovative strategies are needed assure adequate representation of these groups.