A randomised controlled trial to assess the effectiveness of offering study results as an incentive to increase response rates to postal questionnaires [ISRCTN26118436]

  • Sarah Cockayne1Email author and

    Affiliated with

    • David J Torgerson1

      Affiliated with

      BMC Medical Research Methodology20055:34

      DOI: 10.1186/1471-2288-5-34

      Received: 04 May 2005

      Accepted: 26 October 2005

      Published: 26 October 2005

      Abstract

      Background

      Postal questionnaires are widely used to collect outcome data on participants. However, a poor response to questionnaires will reduce the statistical power of the study and may introduce bias. A meta analysis of ten trials offering study results, largely in the fields of education and marketing, was shown to be ineffective, with the odds ratio for response with offering research findings is 0.92 (95% CI 0.75 to 1.11). However uncertainty still exists as it is uncertain whether results from such trials can be extrapolated to that of a health care setting. The aim of this study was to assess whether offering participants study results increases the response rates to postal questionnaires.

      Methods

      1038 women aged over 70 years were remotely randomised by computer in a 3:1 ratio. 250 participants did not receive the offer of knowing the results of the trial and 788 participants were offered the results of the trial in a postal questionnaire. The main outcome measure was response rate. Chi square test was used to evaluate the overall differences in response rate between the two groups. An adjusted analysis, adjusting for whether the participant was taking calcium and age was also undertaken.

      Results

      The response rates were not significantly different Odds Ratio 0.88 (95% confidence intervals 0.48 to 1.63) p = 0.69.

      Conclusion

      Offering study results to women living in the community aged over 70 does not increase response rates to postal questionnaires. Although researchers have an ethical obligation to offer participants study results, since 10% of women did not wish to receive the results, investigators should give participants the option to opt out of receiving the study's results.

      Background

      Postal questionnaires are widely used to collect data on participants in health research. They are an attractive means of collecting data, because they are easy to administer and may be the only economically viable method of collecting data on large numbers of participants who may be geographically dispersed. However, if there is a poor response rate to these questionnaires the validity of the study may be at risk for two main reasons. First, the statistical power of the study will be reduced. Second, bias may be introduced if non-responders differ significantly from those who do respond. In other words low response can threaten the internal validity of randomised trials as bias can be introduced and the external validity of surveys are undermined as non-response will prevent random sampling. Identifying effective strategies to increase response rates to postal questionnaires is therefore highly relevant to researchers since it could improve the quality of their research.

      Offering participants the chance of being informed of the results of the study in which they are participating could improve response rates for three reasons. First it demonstrates the researcher's goodwill and sincerity [1, 2]. Second, it adds greater credibility and importance to the respondent's efforts and third, it could act as a non-monetary incentive. In 2002 Edwards [3] identified 75 different strategies for increasing response rates including non-monetary incentives versus no incentive, which were shown to be effective (odds ratio of 1.19 95% confidence interval 1.11 to 1.28). In 2004 the same review group reported the results of a meta analysis of ten trials offering study results as an incentive which included 13 642 participants [4]. When the results of these trials were pooled in a random effects meta analysis the odds ratio for response with research findings was 0.92 (95% CI 0.75 to 1.11). However, uncertainty still exists as to the effectiveness of this strategy in a health care setting. This is because only one of these trials [5] was undertaken in a health care setting. The results of this study showed no evidence to suggest that promising study results to participants increases response rates. However, participants were hospital chief executives who were sent questionnaires regarding managerial issues such as expenditure and not items directly relating to their health. The effectiveness of this strategy has yet to be tested in a setting related directly to the population's health and it is uncertain whether trials conducted in other areas such as marketing and education can be extrapolated to a health care setting. The aim of this study therefore was to assess whether offering to tell participants the results of the study would increase the response rates to postal questionnaires in a health care setting

      Methods

      Study population

      The subjects in this study were community dwelling women aged over 70 living in the York and Cumbria area. These women were due to be sent a final follow-up questionnaire as part of a multi-centred randomised controlled trial of calcium and vitamin D supplementation for fracture prevention. Participants had been originally assigned to one of two groups. The intervention group received daily oral supplementation of 1,000 mg of calcium with 800 IU vitamin D3 with a patient information leaflet on dietary calcium intake and falls prevention. The control group received the patient information leaflet only. Further details of this trial have been reported elsewhere [6].

      Inclusion/exclusion criteria and randomisation

      Women were eligible for this trial if they had been recruited from the York centre and were due to receive their final follow-up questionnaire in March 2004. Women were excluded if they had withdrawn from the trial or if notification of the participant's death had been received. An independent researcher from the York Trials Unit randomised eligible women in a 3:1 ratio in favour of offering the results of the trial, by computer. An unequal randomisation ratio was used for two reasons. First, there was an issue of administrative convenience. The 'standard' letter to trial participants was to offer them the study results and this letter was going out to all centres including those not participating in this study. Therefore, it was less costly with lower administrative input to produce a smaller separate batch of letters to send out than if we had used equal allocation. Second, although we had some uncertainty with respect to the direction of the effect (hence the trial) on balance we expected that the response rates would be lower in the group assigned to 'not offered the results.' If this had been the case and more participants were assigned to this group, then the overall response rate to the final follow-up would have been reduced.

      Control

      250 women assigned to the group 'not offered the results' of the trial received a one-page questionnaire asking whether the participant had had a fracture in the past three years, how much calcium and vitamin D supplement they were taking and if applicable the reason for stopping the supplement, along with a request to give permission to send details of future research. However, all participants returning their questionnaire would be informed of the trial's results.

      Intervention

      788 women assigned to 'offered the results' received the same questionnaire but with an additional question asking whether they would like to be notified of the results of the trial.

      Both groups received a personalised cover letter showing university sponsorship, along with a business reply envelope. Those participants not returning a questionnaire within three weeks were sent up to two reminder letters, questionnaires and business reply envelopes, three and six weeks after the initial mailing. Administration of the questionnaire was not blind to group allocation.

      Ethical approval for the study was obtained from the Northern and Yorkshire multicentre research ethics committee and relevant local research ethics committees.

      Outcome

      The primary outcome was return of final follow-up questionnaire or reminder by the participant.

      Sample size and statistical analysis

      The sample size for this study was calculated to give 95% power (5% two sided significance) to detect a 10% difference in response rates. We assumed that the baseline response rate would be approximately 80%, a figure based on work from a previous trial conducted by the authors on women of a similar age and risk of fracture [7]. Results were analysed using SPSS 11. All participants were included in the analysis (intention to treat). We used Chi square test to evaluate the overall differences in response rate between the two groups. We also undertook an adjusted analysis, adjusting for whether the participant was taking calcium and age. Statistical analysis was not undertaken blind to group allocation.

      Results

      The baseline characteristics for the participants are shown in table 1 whilst figure 1 describes the participant recruitment profile. The overall response rates are shown in table 2. A total of 153 participants were sent reminder questionnaires,112 of which were in the intervention group and 41 in the control group. 63 participants (56%) in the intervention group and 27 participants (66%) in the control group returned a reminder. These response rates were not significantly different. Chi square = 0.16 df = 1 p = 0.69 Odds Ratio (OR) 0.88 (95% Confidence Intervals [CI] 0.48 to 1.63). As the study was a factorial design, the data were tested to ensure there was no interaction between taking calcium and being allocated to being either offered/not offered the results of the trial. Adjusted OR for returning the postal questionnaire showed no association between offering the results of the trial and the return of the questionnaire (OR 0.81 [CI 0.44 to 1.51] p = 0.51) whilst taking calcium significantly predicted (OR 5.10 [CI 2.02 to 12.84] p = 0.001) the return of the questionnaire as did increasing age, but to a lesser degree (OR 1.05 [CI 1.00 to 1.11] p = 0.05).
      Table 1

      Baseline characteristics of participants

      Baseline Variable

      Intervention

      Control

      Age (mean, SD)

      76.2 (4.41)

      77.09 (5.34)

      Weight (Kilos, mean SD)

      65.23 (11.88)

      65.65 (12.45)

      Percent (n) < 58 kilos

      31.2% (231/740)

      30.8% (72/234)

      Prior any fracture

      59.0% (454/770)

      61.5% (152/247)

      Smoker

      5.8% (32/551)

      5.4% (9/166)

      Poor/Fair Health

      33.8% (260/769)

      28.7% (71/247)

      Maternal hip fracture

      13.6% (105/770)

      15.0% (37/247)

      Fall in previous 12 months

      31.6% (243/770)

      36.8% (91/247)

      Taking anti-fracture treatment

      5.5% (42/770)

      3.2% (8/247)

      SF12

        

      MCS (mean; SD)

      52.20 (9.05)

      53.00 (8.66)

      PCS (mean; SD)

      41.33 (11.88)

      40.28 (11.60)

      Euroqol

      0.73 (0.218)

      0.69(0.26)

      http://static-content.springer.com/image/art%3A10.1186%2F1471-2288-5-34/MediaObjects/12874_2005_Article_122_Fig1_HTML.jpg
      Figure 1

      Origin and flow of participants in trial.

      Table 2

      Response rate to final follow up questionnaire

       

      Intervention

      Control

      Replied

      721

      233

      Not replied

      49

      14

      Total number of questionnaires analysed

      770

      247

      Overall response rate

      93.6%

      94.3%

      Of the 721 participants returning questionnaires in the group offered the results of the trial, 647 (89.7.%) requested the results of the study and three women in the group 'not offered the results' pro-actively asked to be sent a copy of the results. The baseline characteristics of participants who did and did not request the study's results are reported in table 3. Adjusted OR showed taking calcium (OR 4.00 [CI 1.88 to 8.51]) p < 0.05) significantly predicted whether participants requested trial results, however age (OR 0.96 [CI 0.91 to 1.01]) p = 0.12) and fair or poor health (OR 0.66 [CI 0.40 to 1.10]) p = 0.11) did not predict whether participants requested trial results.
      Table 3

      Baseline characteristics of participants who did and did not request the trial's results

      Baseline Variable

      Requested trial results

      Did not request trial results

      Age (mean, SD)

      76.00 (4.30) n = 647

      76.80 (4.68) n = 74

      Weight (Kilos, mean SD)

      65.59 (11.94) n = 618

      64.28 (11.77) (n = 71)

      Percent (n) < 58 kilos

      30.7% (190/618)

      32.4% (23/71)

      Prior any fracture

      59.2% (383/647)

      53.4% (39/73)

      Smoker

      4.5% (21/463)

      3.8% (2/52)

      Poor/Fair Health

      31.9% (206/646)

      41.7% (30/72)

      Maternal hip fracture

      13.3% (86/647)

      16.7% (12/72)

      Fall in previous 12 months

      31.1% (201/647)

      31.9% (23/72)

      Taking anti-fracture treatment

      5.9% (38/647)

      4.2% (3/73)

      SF12

        

      MCS (mean; SD)

      52.61 (8.49) n = 629

      50.98 (11.5) n = 70

      PCS (mean; SD)

      41.73 (11.8) n = 629

      39.65 (11.20) n = 70

      Euroqol

      0.74 (0.21) n = 640

      0.75 (0.19) n = 71

      The characteristics of those who did and did not respond to the questionnaire are reported in table 4.
      Table 4

      Baseline characteristics of responders and non-responders

      Variable

      Responders

      Non-responders

      p value

      Age (mean, SD)

      76.32 (4.62) n = 954

      77.43 (5.22) n = 63

      0.07

      Weight (Kilos, mean SD)

      65.54 (12.06) n = 912

      62.29 (10.93)

      0.04

      Percent (n) < 58 kilos

      30.8% (281/912)

      35.5% (22/62)

      0.02

      Prior any fracture to randomisation

      59.4% (567/954)

      61.9% (39/63)

      0.80

      Smoker

      4.5% (30/671)

      23.9% (11/46)

      < 0.05

      Poor/Fair Health

      31.7% (302/953)

      46.0% (29/63)

      0.03

      Maternal hip fracture

      14.0% (134/954)

      12.7% (8/63)

      0.91

      Falls in 12 month prior to randomisation

      32.3% (308/954)

      41.3% (26/63)

      0.18

      Taking anti-fracture treatment

      5.0% (48/954)

      3.2% (2/63)

      0.72

      SF12

         

      MCS (mean; SD)

      52.63 (8.78) n = 929

      48.53 (10.6) n = 60

      0.01

      PCS (mean; SD)

      41.29 (11.68) n = 929

      37.66 (13.36) n = 60

      0.02

      Euroqol

      0.73 (0.22) n = 938

      0.62 (0.30) n = 62

      0.01

      Discussion

      This study found no evidence that offering study results to participants increased the response rate to a postal questionnaire. There are two possible reasons for this strategy being ineffective. First, offering results of the trial may not act as an incentive to participants to respond because it appeals only to those interested in the trial and who were likely to respond anyway. If this were the case then offering the results of the trial was insufficient incentive to motivate non-responders. Second, the following strategies were adopted which are known to increase response rate; a short, one-sided, straightforward questionnaire designed to be of interest to participants but not containing sensitive questions, reminders with a second copy of the questionnaire and university sponsorship. This combination of strategies produced a high response rate of over 93% and it may be that any strategy would struggle to significantly increase the response rate further (i.e., a ceiling effect).

      This study was undertaken among women over 70 living independently in the community therefore it is not possible to generalise these results to either men, women younger than 70 or to those living in nursing homes or other forms of residential care. Also because the participants had already agreed to take part in a randomised controlled trial the results may not apply to the wider population of women over 70 years. The incentive of 'offering trial results' was tested in combination with other strategies known to increase response rates, which together produced an excellent response rate, which any strategy would find difficult to improve upon. It may be that this incentive is effective in a different combination of strategies.

      Conclusion

      Although offering the results of the study did not increase the response rates, researchers do have an ethical obligation to offer participants the results of study if they wish to receive them. Our results show that 90% of women did want to know the results of our trial however, 10% did not wish to receive the results, which emphasises the need for investigators to give participants the option to 'opt out of' receiving the trial's results. Within the UK, for new studies the issue of disseminating study results will have to be addressed since the introduction of the new Research Ethics Committee (REC) form in 2004. The investigator is asked whether the study's results will be disseminated to participants and details of what will happen to the results of the research, when the results are likely to be published and how participants can obtain a copy of the results needs to be stated on the patient information leaflet.

      Declarations

      Acknowledgements

      We thank Ben Cross for undertaking the randomisation of participants onto this trial.

      Authors’ Affiliations

      (1)
      Department of Health Sciences, University of York, York Trials Unit

      References

      1. Fernandez CV, Kodish K, Weijer C: Informing study participants of research results: an ethical imperative. Ethics and Human Research 2003, 3:12–19.View Article
      2. Partridge AH: Informing Clinical Trial Participants About Study Results. JAMA 2002, 288:363–365.View ArticlePubMed
      3. Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I: Increasing response rates to postal questionnaires: systematic review. BMJ 2002, 324:1183–1185.View ArticlePubMed
      4. Edwards P, Cooper R: Previous trials of dissemination were identified. J Epidemiol Community Health 2004, 58:354–355.a.View ArticlePubMed
      5. Mullner RM, Levy PS, Byre CS, Matthews D: Effects on characteristics of the survey instrument on response rates to a mail survey of community hospitals. Public Health Rep 1982, 97:465–9.PubMed
      6. Porthouse J, Cockayne S, King C, Saxon L, Steele E, Aspray T, Baverstock M, Birks Y, Dumville J, Francis R, Iglesias C, Puffer S, Sutcliffe A, Watt I, Torgerson DJ: Randomised Controlled Trial of Calcium and Vitamin D Supplementation for Fracture Prevention in Primary Care. BMJ 2005, 330:1003–1008.View ArticlePubMed
      7. Birks YF, Porthouse J, Addie C, Loughney K, Saxon L, Baverstock M, Francis RM, Reid DM, Watt I, Torgerson DJ, Primary Care Hip Protector Trial Group: Randomized controlled trial of hip protectors among women living in the community. Osteoporosis International 2004, 9:701–6.
      8. Pre-publication history

        1. The pre-publication history for this paper can be accessed here:http://​www.​biomedcentral.​com/​1471-2288/​5/​34/​prepub

      Copyright

      © Cockayne and Torgerson. 2005

      This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.