We found that numbering return envelopes in a survey about health care provider regrets has significantly increased the proportion of explicit refusals to participate and reduced the overall survey response rate. The reduction in the proportion of participants of 4.7% did not reach the pre-specified level of statistical significance of 0.05 (p = 0.073). This constitutes weak evidence against the hypothesis of no effect. Pooling this result with similar observations made in previous studies suggests that numbering questionnaires reduces participation by approximately two to three percentage points.
Whether this decrease is important enough to affect survey data collection procedures is debatable. Numbering questionnaires or return envelopes allows the researcher to send reminders only to those who have not yet responded. This reduces costs, and avoids the irritation some respondents may feel when they receive an unnecessary reminder. A small reduction in the response rate may be an acceptable price to pay for these benefits. On the other hand, response rates in mail surveys are typically lower than desirable, and may be declining [16, 17]. Any measure that increases the response rate of a survey and its representativeness of the target population is welcome. Optimal response rates can be obtained only through an accumulation of design features that yield marginal gains, including a clear explanation of the utility of the survey, an attractive layout, incentives, and reminders [2]. It appears that the assurance of confidentiality may be another useful measure.
It is unclear whether numbering survey returns plays a role in all surveys or only in those instances where confidentiality is particularly important. We tested the effect of numbering in a sensitive survey. Regrets felt by doctors and nurses may be related to errors or to communication issues when providing healthcare. A potential respondent may well decide that it may not be prudent to participate and run the risk of being identified. Among previous studies of the effect of numbering, one concerned medical residency programs, which is not a particularly sensitive topic, and others pertained to drug use, AIDS knowledge, and patient safety climate. The meta-analysis did not show any amount of heterogeneity. There is thus far no evidence that the degree of sensitivity of the survey topic modifies the effect of questionnaire numbering.
Regardless of the effect of numbering, this survey was notable by its low response rate of 31.7%. While such response rates are not uncommon (e.g., [18]), participation was considerably lower than in several surveys of doctors and nurses conducted at the same hospital in recent years, which typically yielded response rates between 50% and 60% [19–21]. We believe that the topic of the survey – regret related to health care – was the chief cause of this reluctance to participate. By definition a regretted situation is unpleasant to recall, and revisiting it in detail and sharing these details with others may be too much to ask for, whether confidentiality is guaranteed or not. As evidence of an unusual and possibly emotional response pattern, we have not previously seen explicit refusals approaching the proportion of 21.4% observed in this study; in our experience less than one tenth of the eligible sample will check the “I do not wish to participate” box in a typical survey – e.g., about 5% in a survey of health insurance plan members [22]. Limited evidence suggests that people who explicitly refuse to participate in a survey may differ from those who fail to respond [23]. Our results suggest that regret related to providing health-care may have been a too sensitive topic for many potential respondents.
A low response rate may cause selection bias, but the numbering of return envelopes may also produce information bias, by causing the respondents to answer differently than they would have done in the absence of numbering. These 2 types of bias cannot be distinguished in our study. In comparisons of item responses, 12 of 105 differences between the numbered and non-numbered groups were statistically significant. However, the various survey items were mutually correlated, and a permutation test showed that this proportion (11.4%) did not significantly differ from the type 1 error rate of 5% that characterizes the null hypothesis of no bias. It is tempting to focus on the elevated proportion of significant differences observed for the regret intensity items (7 out of 19) and to conclude that response bias concerns these particularly sensitive items. However, any post-hoc selection of a subset of items increases the likelihood of misinterpreting type 1 errors as true differences. Of note, a low response rate does not always imply a large amount of bias [23]; e.g., in a patient opinion survey, the mean problem scores obtained after the first mailing, when only about 30% of questionnaires hada been returned, differed by less than one tenth of a standard deviation from the final results (difference 1.9 points, SD 23.8), when a response rate of 70% had been achieved [24].
This study has several limitations. First, the study was set up in the middle of an ongoing survey, prompted by an unusual response pattern, and not planned in advance. Therefore randomization was non-concurrent (numbered envelopes were sent first and non-numbered were sent two months later, even though both samples were drawn from the same randomly ordered lists). The delay could alter the outcomes if it changed exposure to external events that might influence survey response, such as a summer holiday period; however, this was not the case to our knowledge. Second, the sample size was driven by the primary purpose of the survey (data collection on regret), and not by a power analysis related to the comparison of numbered and non-numbered envelopes; as a result the size of the second (non-numbered) group was comparatively low. A post-hoc power estimation suggests that the power to detect a difference between 30% and 35% with the actual sample size was below 50%. Perhaps most importantly, we were not able to establish with certainty the reasons for the negative effect of envelope numbering, as we did not discuss the motivations of the respondents and non-respondents. The meta-analysis included only a small number of eligible studies, and had limited power to examine differences between survey identifiability methods (e.g., different effect of questionnaire numbering versus envelope numbering).
As our study employed paper questionnaires and postal delivery, the results are not directly applicable to electronic surveys. However, concerns about anonymity and traceability may be particularly important for electronic surveys, because online privacy management remains opaque and potentially untrustworthy for many users. This may contribute to the low participation rates regularly encountered in internet surveys e.g., [25].
