Getting physicians to open the survey: little evidence that an envelope teaser increases response rates

Health services researchers are dependent on physicians' participation in surveys in order to assess provider attitudes, beliefs and self-reported behaviors such as guideline adherence. There is some evidence that physician response rates have been falling in more recent years [1, 2]. While low response rates are known to reduce statistical power and result in higher costs per completed survey, the impact of low response rates on nonresponse bias may not be as strong as previously thought [3]. Nevertheless, exploring methods to increase response rates in surveys of physicians is an important area of study, one that was recently called for in a review of the physician response literature by VanGeest et al. [4] In November of 2010, the National Cancer Institute convened a Provider Survey Methods Workshop where a research agenda for the field was developed, underscoring the broad attention to surveying this population with specific attention to methods designed to enhance participation. Due to the growing evidence that response rate is not necessarily correlated with response bias [3] these calls simultaneously call for systematic examination of nonresponse bias as is undertaken herein.

For an individual to respond to a mailed survey a series of steps must occur - not the least of which is opening the envelope in which the survey is sent. While this may seem trivial, for physicians it may be less so. There is evidence that physicians receive a large amount of materials (including surveys) in the mail [5] thus taking this important first step is far from guaranteed. In a qualitative study asking nonresponding general practitioners why they did not respond, fully 34% said that the questionnaire "got lost in a pile of paper work." [6] An additional barrier to obtaining survey responses from physicians is the potential presence of gatekeepers, or individuals who screen physicians' mail and pass on only those materials which, in the absence of any clarifying information about their contents, are deemed "important" by a third party. While this extent of gatekeeping has not been empirically documented, it has been hypothesized to put a downward pressure on response rates [4, 7–9]. In preparation for our own earlier work, we have anecdotal evidence that this is the case. We conducted informal focus groups in a physician population about potential issues in responding to surveys; not opening one's mail was a cited barrier [9].

One way to motivate the person who would regularly be responsible for opening mail (either the potential responders or the gatekeeper if indeed they do exist as hypothesized) to open an envelope is with a "teaser" on the outside of the envelope itself. This teaser would entice or suggest to the recipient that it is worth their time to open the letter or package, lessening the chance that the envelope is "lost in a pile of paperwork". A recent meta-analysis of methods to increase response rate suggested that including an envelope teaser increased response rates more than three-fold [10]. This conclusion, however, was based on only one study with a sample size of 190 in which recipients were assigned to an envelope teaser condition (i.e. a stamp on the envelope reading "Did you know you were entitled to more money?") or a control group with no such stamp [11]. While this finding is suggestive, the small sample size does not support a strong inference, nor can the findings be generalized to a physician population. Moreover, the current context of mailed surveys is likely quite different more than a decade after this initial publication.

More generally, however, there is evidence that other elements of packaging can significantly affect response rates in physician surveys (for review see: Kellerman & Herold [1]). For example, Asch and Christakis [12] manipulated the envelope so that the letter appeared from the University Medical School or the Veterans Affairs Hospital. Response rates were significantly higher for the latter envelope, suggesting that aspects of the envelope itself can matter and that respondents were more likely to open an envelope from the Veteran Hospital than from the University Hospital as conveyed on the outside of the envelope itself. Moreover, in the executive summary from the National Cancer Institute's provider methods workshop the importance of engaging physicians with the envelope itself was cited due to the observation that the decision about whether to open envelopes may be made "in seconds"[13].

Here we report results of a randomized study testing the assertion that an envelope teaser could increase response rates among a nationally representative physician population receiving a mailed questionnaire. This manipulation occurred in the second wave mailing to initial survey nonrespondents.

Adding a teaser sticker to the outside of an envelope in a mailed survey did not significantly increase response rates among a nationally representative sample of physicians in the United States. However, because the relative cost of adding an envelope teaser is minimal, the evidence that it introduces nonresponse bias small, and given the sparse existing literature on envelope teasers, future work in this area is warranted. Specifically, wording variations on the teaser sticker itself, the incentive amount and type (monetary vs. nonmonetary and guaranteed vs. lottery) should be further tested in a physician population. It is possible that the teaser for a $25 incentive offer was insufficient to change behaviors in this population of physicians, as the findings of Keating et al. [15], who found significantly higher response rates in a survey of physicians when a prepaid check for $50 was issued versus a prepaid check for $20, would suggest.

This study has several limitations. One of the mechanisms through which the envelope teaser is hypothesized to work is through increasing the chance that a physician gatekeeper would pass the survey along to the potential respondent, thereby increasing the likelihood of response. The present study cannot determine if there was a differential response to the envelope teaser by the presence or absence of a gatekeeper as it did not explicitly measure the presence or absence of a gatekeeper. Future research should work toward moving beyond the proposed existence and impact of gatekeepers to a documentation of this extent that gatekeeping actually impacts survey response. Once this is determined, the mechanism by which something like a teaser functions with or without gatekeeping can be elucidated. Because the present study provided a manipulation in the follow-up to a survey, its findings cannot be generalized to an initial mailing as all of the responders in this experiment had already not responded to an initial request. In order to better understand the true potential of envelope teasers, and given their low cost, their value should be tested in initial mailings as well as later mailings. Finally, the extent that these findings can be generalized beyond physicians in the United States is unknown.

In conclusion, even though neither response rates nor nonresponse bias were impacted by the use of an envelope teaser in a survey of physicians in the United States, it is important that health survey methodologists continue to respond to the calls of Cull et al. [16], Kellerman and Harold [1], Klabunde et al. [13], McMahon et al. [17], and VanGeest and Johnson [4] by continuing to further test methods of enhancing survey participation among elite populations such as physicians while simultaneously examining nonresponse bias. While there is mounting evidence that there is no linear relationship between response rate and nonresponse bias it is important to consider the potential for nonresponse bias in each survey context [3]. Otherwise, the physician perspective may not be adequately represented in debates and issues germane to the practice of medicine or to the realm of health care reform.