Effect of numbering of return envelopes on participation, explicit refusals, and bias: experiment and meta-analysis

The protection of the identity of participants is an important issue in health surveys, particularly when the topic of the survey is perceived to be sensitive or intrusive by potential respondents. The reasons are both ethical and pragmatic. The risk of breaching participants’ privacy should be minimized in all research as a matter of principle. In addition, people who fear that answers may be traced back to them may choose not to participate at all, may skip the sensitive questions, or may alter their responses, consciously or unconsciously, to make them more inconspicuous or acceptable [1]. Non-response may cause selection bias, and altered answers may cause information bias (similar to social desirability bias). For these reasons health surveys are typically anonymous. However, many surveys employ a study identifier, such as a study number, that is used to track responses and manage reminders. This identifier may raise concerns among potential participants about the traceability of the respondent.

Some (but not all) older studies suggest that respondent identification may decrease participation in surveys [2–9]. Several [3–5] were marketing studies in which respondent motivations and concerns about confidentiality may differ from those of health surveys. Studies in the health or health care domain have reported small and non-significant differences in response rates associated with respondent identifiability [6–9].

We report here on a potentially sensitive study where the numbering of return envelopes may have increased non-response and possibly caused bias. The context was a mail survey of a random sample of hospital doctors and nurses regarding their experience of regrets encountered during patient care [10]. A previous qualitative study established that this was indeed a sensitive topic for most health care professionals [11]. The survey cover letter informed the potential participants that this was a research project in which participation was voluntary, that the data would not be linked with personal identifiers, and that the number on the return envelope would only be used to track responses and avoid sending unnecessary reminders. The front page instructed the recipient to check the appropriate box and send back the questionnaire without filling it in if a) the respondent had not practiced patient care in the past 5 years, b) the respondent did not wish to participate, or c) the respondent had retired. Upon receipt of the first returned questionnaires it appeared that an unusually large proportion explicitly declined to participate. To compensate for the anticipated low sample size, and to test the hypothesis that numbering of the return envelopes contributed to this problem, we selected an additional random sample of doctors and nurses. For this additional sample the return envelopes were not numbered, so that everyone received the initial mailing and the subsequent reminders.

In this report, we compare the original sample (numbered surveys) and the additional sample (non-numbered surveys) in terms of overall response rate, the proportion of explicit refusals, and the distributions of survey variables among the respondents. We also performed a meta-analysis of randomised studies of health surveys that compared participation rates for traceable and untraceable responses.

We found that numbering return envelopes in a survey about health care provider regrets has significantly increased the proportion of explicit refusals to participate and reduced the overall survey response rate. The reduction in the proportion of participants of 4.7% did not reach the pre-specified level of statistical significance of 0.05 (p = 0.073). This constitutes weak evidence against the hypothesis of no effect. Pooling this result with similar observations made in previous studies suggests that numbering questionnaires reduces participation by approximately two to three percentage points.

Whether this decrease is important enough to affect survey data collection procedures is debatable. Numbering questionnaires or return envelopes allows the researcher to send reminders only to those who have not yet responded. This reduces costs, and avoids the irritation some respondents may feel when they receive an unnecessary reminder. A small reduction in the response rate may be an acceptable price to pay for these benefits. On the other hand, response rates in mail surveys are typically lower than desirable, and may be declining [16, 17]. Any measure that increases the response rate of a survey and its representativeness of the target population is welcome. Optimal response rates can be obtained only through an accumulation of design features that yield marginal gains, including a clear explanation of the utility of the survey, an attractive layout, incentives, and reminders [2]. It appears that the assurance of confidentiality may be another useful measure.

It is unclear whether numbering survey returns plays a role in all surveys or only in those instances where confidentiality is particularly important. We tested the effect of numbering in a sensitive survey. Regrets felt by doctors and nurses may be related to errors or to communication issues when providing healthcare. A potential respondent may well decide that it may not be prudent to participate and run the risk of being identified. Among previous studies of the effect of numbering, one concerned medical residency programs, which is not a particularly sensitive topic, and others pertained to drug use, AIDS knowledge, and patient safety climate. The meta-analysis did not show any amount of heterogeneity. There is thus far no evidence that the degree of sensitivity of the survey topic modifies the effect of questionnaire numbering.

Regardless of the effect of numbering, this survey was notable by its low response rate of 31.7%. While such response rates are not uncommon (e.g., [18]), participation was considerably lower than in several surveys of doctors and nurses conducted at the same hospital in recent years, which typically yielded response rates between 50% and 60% [19–21]. We believe that the topic of the survey – regret related to health care – was the chief cause of this reluctance to participate. By definition a regretted situation is unpleasant to recall, and revisiting it in detail and sharing these details with others may be too much to ask for, whether confidentiality is guaranteed or not. As evidence of an unusual and possibly emotional response pattern, we have not previously seen explicit refusals approaching the proportion of 21.4% observed in this study; in our experience less than one tenth of the eligible sample will check the “I do not wish to participate” box in a typical survey – e.g., about 5% in a survey of health insurance plan members [22]. Limited evidence suggests that people who explicitly refuse to participate in a survey may differ from those who fail to respond [23]. Our results suggest that regret related to providing health-care may have been a too sensitive topic for many potential respondents.

A low response rate may cause selection bias, but the numbering of return envelopes may also produce information bias, by causing the respondents to answer differently than they would have done in the absence of numbering. These 2 types of bias cannot be distinguished in our study. In comparisons of item responses, 12 of 105 differences between the numbered and non-numbered groups were statistically significant. However, the various survey items were mutually correlated, and a permutation test showed that this proportion (11.4%) did not significantly differ from the type 1 error rate of 5% that characterizes the null hypothesis of no bias. It is tempting to focus on the elevated proportion of significant differences observed for the regret intensity items (7 out of 19) and to conclude that response bias concerns these particularly sensitive items. However, any post-hoc selection of a subset of items increases the likelihood of misinterpreting type 1 errors as true differences. Of note, a low response rate does not always imply a large amount of bias [23]; e.g., in a patient opinion survey, the mean problem scores obtained after the first mailing, when only about 30% of questionnaires hada been returned, differed by less than one tenth of a standard deviation from the final results (difference 1.9 points, SD 23.8), when a response rate of 70% had been achieved [24].

This study has several limitations. First, the study was set up in the middle of an ongoing survey, prompted by an unusual response pattern, and not planned in advance. Therefore randomization was non-concurrent (numbered envelopes were sent first and non-numbered were sent two months later, even though both samples were drawn from the same randomly ordered lists). The delay could alter the outcomes if it changed exposure to external events that might influence survey response, such as a summer holiday period; however, this was not the case to our knowledge. Second, the sample size was driven by the primary purpose of the survey (data collection on regret), and not by a power analysis related to the comparison of numbered and non-numbered envelopes; as a result the size of the second (non-numbered) group was comparatively low. A post-hoc power estimation suggests that the power to detect a difference between 30% and 35% with the actual sample size was below 50%. Perhaps most importantly, we were not able to establish with certainty the reasons for the negative effect of envelope numbering, as we did not discuss the motivations of the respondents and non-respondents. The meta-analysis included only a small number of eligible studies, and had limited power to examine differences between survey identifiability methods (e.g., different effect of questionnaire numbering versus envelope numbering).

As our study employed paper questionnaires and postal delivery, the results are not directly applicable to electronic surveys. However, concerns about anonymity and traceability may be particularly important for electronic surveys, because online privacy management remains opaque and potentially untrustworthy for many users. This may contribute to the low participation rates regularly encountered in internet surveys e.g., [25].

Our results, along with the meta-analysis of similar studies, indicate that the identifiability of completed questionnaires will lower the response rate of postal surveys by two to three percentage points. However, there was no strong evidence that identifiability causes bias. Further research should ascertain whether this effect depends on the surveyed population, the contents of the questionnaire, or other characteristics of the data-collection process.