Effect of paper quality on the response rate to a postal survey: A randomised controlled trial. [ISRCTN32032031]

Abstract

Background

Response rates to surveys are declining and this threatens the validity and generalisability of their findings. We wanted to determine whether paper quality influences the response rate to postal surveys

Methods

A postal questionnaire was sent to all members of the British Society of Gynaecological Endoscopy (BSGE). Recipients were randomised to receiving the questionnaire printed on standard quality paper or high quality paper.

Results

The response rate for the recipients of high quality paper was 43/195 (22%) and 57/194 (29%) for standard quality paper (relative rate of response 0.75, 95% CI 0.33–1.05, p = 0.1

Conclusion

The use of high quality paper did not increase response rates to a questionnaire survey of gynaecologists affiliated to an endoscopic society.

Introduction

Postal surveys are commonly used in medical research. Response rates to surveys are declining [1] and this threatens the validity and generalisability of their findings. It is therefore important that strategies are developed in order to reverse this trend. [2, 3] We hypothesized that the paper quality on which the postal questionnaire was printed, may effect response rates. This is because the recipient may be inclined to look upon the questionnaire more approvingly, if the quality of paper used is high, thereby increasing the chance of a response. In order to test the effectiveness of this strategy, we conducted a randomised controlled trial, as part of a survey of gynaecologists, to determine if high paper quality increases the response rate to questionnaires.

Methods

All gynaecologists identified from the British Society of Gynaecological Endoscopy (BSGE) database of members were sent a questionnaire with a covering letter and prepaid response envelope in April 2000. The questionnaire sought views about current and future research priorities in gynaecological endoscopy. Recipients were randomised to receiving the questionnaire with a covering letter printed on standard quality white paper or high quality white paper. High quality paper was defined as a weight of 100 grams and standard quality paper as a weight of 80 grams. The participants were not informed of the randomisation to paper quality. The randomisation sequence was computer generated and group allocation was concealed from the participants throughout the study. No reminders were sent. Based on the response rate from a recent gynaecological survey, [4] we assumed that provision of high quality paper would increase the proportion of responders by 15%, from 45% to 60%. This meant that the sample size had 80% power to detect a statistically significant difference at the level of alpha= 0.05. Relative response rates were determined and statistical significance tested for a difference in proportions. The trial results were reported according to the CONSORT guidelines. [5]

Results

Of the 389 gynaecologists surveyed, 195 were randomised to receive the questionnaire on high quality paper and 194 were to receive the questionnaire on standard quality paper (figure 1). The overall response rate was 100 (26%). There were no differences between the intervention and control groups responding as regards postgraduate certification (e.g. Membership of the Royal College of Obstetricians and Gynaecologists (95% versus 93%) and consultant grade (79% versus 75%). The response rate for the recipients of high quality paper was 43/195 (22%) and 57/194 (29%) for standard quality paper. The relative rate of response to the questionnaire printed on high quality paper compared to standard quality was therefore 0.75 (95% CI 0.53–1.05, p = 0.1).

Figure 1

Flow diagram showing randomisation and response rates of the survey

Discussion

The use of high quality paper did not increase response rates to a questionnaire survey of gynaecologists affiliated to an endoscopic society. The low response rate in our survey may have resulted from the content our questionnaire as it enquired about research issues and so it is more likely that those with an active interest were likely to respond. However, any such selection biases should be minimised by the randomisation process and does not therefore affect the internal validity of our findings. The low response to our survey does limit the external validity or generalisability of our findings. Our power assumptions were not borne out by the response rates and some may argue that the apparent lack of an effect may be due to an inadequate sample. However, this would not explain a trend towards a lower response rate in the group allocated high quality paper. It may be that the 20 gram difference between paper weight in the two groups was too small so that recipients of 'high' quality paper did not readily distinguish it from there general day to day paperwork. It is also possible that our definition of paper quality using weight alone was inadequate and other features of stationary quality should have been used, such as colour intensity, laid paper and watermarking.

Postal surveys are widely used because they represent a cost effective method of obtaining information from large numbers of geographically disparate medical professionals about their attributes, behaviours, attitudes and beliefs. It is of concern that response rates are declining [1] and therefore there is a need to develop effective strategies, in addition to the content of questionnaire itself, [6] to counter this trend. Data from primary and secondary research have indicated that prenotifying recipients, personalising questionnaires and providing follow up letters improves response rates. [7–9] Other potentially useful techniques include the colour of questionnaires, sponsorship from academic institutions, inclusion of return envelopes and utilising monetary and non-monetary incentives. [7–9] In contrast, provision of pens, [3] the use of covering letters, assurances of anonymity and stating deadlines do not increase rates of return. [8] Studies have reported conflicting findings regarding the effect of "help the researcher" type appeals in covering letters [8, 9] and the provision of return postage, [9, 10] although the type of return postage provided appears to influence response. [10] To our knowledge the effect of paper quality on response rates to postal surveys has not been previously tested in a randomised controlled trial.

Given the lack of effectiveness shown in our study and the costs associated with higher quality paper in a questionnaire (approximately 35% increase in costs for higher quality paper – €66 versus €43 for 5 reams (2500 sheets) of A4 size (local National Health Service suppliers)), investigators should carefully consider the use of this particular strategy to improve response rates. If quality differences are marginal, there may not be a substantial improvement in response rates.