Evaluation of the effects of an offer of a monetary incentive on the rate of questionnaire return during follow-up of a clinical trial: a randomised study within a trial
BMC Medical Research Methodology volume 16, Article number: 82 (2016)
A systematic review on the use of incentives to promote questionnaire return in clinical trials suggest they are effective, but not all studies have sufficient funds to use them. Promising an incentive once data are returned can reduce the cost-burden of this approach, with possible further cost-savings if the offer were restricted to reminder letters only. This study aimed to evaluate the effect of promising a monetary incentive at first mailout versus a promise on reminder letters only.
This was a randomised Study Within A Trial (SWAT) nested within BUMPES, a multicentre randomised controlled trial of maternal position in the late stage of labour in women with an epidural. The follow-up questionnaire asked for information on the women’s health, wellbeing and health service use one year following the birth of their baby. Women who consented to be contacted were randomised to a promise of a monetary incentive at first mailout or a promise on reminder letters only. Women were given an option of completing the questionnaire on paper or on online. The incentive was posted out on receipt of a completed questionnaire. The primary outcome was the overall return rate, and secondary outcomes were the return rate without any chasing from the study office, and the total cost of the vouchers.
A total of 1,029 women were randomised, 508 to the first mailout group and 518 to the reminder group. There was no evidence to suggest a difference between groups in the overall return rate (adjusted RR 1.03 (95 % CI 0.96 to 1.11), however the proportion returned without chasing was higher in the first mailout group (adjusted RR 1.22, 95 % CI 1.07 to 1.39). The total cost of the vouchers per participant was higher in the first mailout group (mean difference £4.56, 95 % CI £4.02 to £5.11).
Offering a monetary incentive when a reminder is required could be cost-effective depending on the sample size of the study and the resources available to administer the reminder letters.
The BUMPES Trial is registered with Current Controlled Trials: ISRCTN35706297, 26th August 2009.
Maximising follow-up rates for postal questionnaires in randomised controlled trials is an important aspect of a well-designed and well conducted study. Loss to follow-up can lead to bias and compromise the internal and external validity of the results.
Use of monetary incentives to promote questionnaire return in clinical trials has been researched. Existing systematic reviews suggest they are effective [1, 2], but not all studies have sufficient funds to use them. Promising an incentive once data are returned can reduce the cost burden of this approach. In a systematic review Brueton et al.  showed evidence that an offer of a monetary incentive was comparable to the addition of a monetary incentive with the questionnaire in 2 studies with a total of 297 participants (pooled risk ratio 1.04, 95 % confidence interval 0.91 to 1.19). However, it may be possible to provide further cost-savings if the offer was restricted to the reminder letters only.
This randomised study within a trial (SWAT) was nested within the BUMPES trial, a multicentre randomised controlled trial investigating the effect of maternal position during the late stages of labour in women with an epidural. The SWAT was carried out on a population of women in the UK one year after the birth of their first child and was developed because the return rate of the follow-up questionnaire for BUMPES was lower than expected in the early stages of the trial. Since current evidence on the use of incentives includes a variety of populations, providing an evidence base on the use of incentives for postnatal women will enhance future research methodology in this population.
This SWAT aimed to evaluate the effect on the return rate of a 1-year follow-up postal questionnaire, comparing the promise of a monetary incentive at first mailout with a promise on reminder letters only.
This parallel group, randomised controlled SWAT nested within BUMPES was conducted on women randomised into the BUMPES study who had not already been sent their 1-year follow-up questionnaire. The questionnaire asked for information on the women’s health, wellbeing and health service use one year following the birth of their baby. Women had the option of completing the paper questionnaire and returning it using a freepost envelope, or completing it on-line using a secure web-interface.
Women were included if they were recruited to BUMPES, consented at recruitment to receive the 1-year follow-up questionnaire, and the questionnaire had not already been sent. Women were excluded if they had had a stillbirth, their infant had died by the time of follow-up, their address details were unknown or if they were not living at the same address as their infant.
Women were randomly allocated to one of the following two groups: (1) an incentive cover letter sent with the first mailout of the questionnaire containing details of a promise of a £10 gift voucher (redeemable at high street shops) on return of a completed questionnaire. The covering letter included a sentence explaining that the voucher was to thank participants for their time and effort. All reminder letters included a sentence about the incentive; (2) an incentive reminder letter. For this group the cover letter sent at first mailout did not mention the incentive. If the questionnaire was not returned, all reminder letters detailed the promise of a £10 gift voucher on return of a completed questionnaire.
For both groups women were additionally contacted electronically and via text message if the contact details were available. The content of the emails and texts sent reflected the group to which the woman was randomised.
Allocation was by computer random number generation and stratified by BUMPES allocation and by centre. The randomisation schedule was generated by the National Perinatal Epidemiology Unit Clinical Trials Unit and sent to the BUMPES trial office at the Comprehensive Clinical Trials Unit at University College London via a secure web-link. Randomisation to incentive cover letter at first mailout or incentive reminder letter occurred at each woman’s next follow-up point during the conduct of the BUMPES study. Each BUMPES participant was randomised once only. BUMPES trial staff were aware of the allocation due to the nature of the interventions, and the practicalities involved in sending the letters and the vouchers.
The pre-specified primary outcome measure was questionnaire return, defined as receipt of a completed or partially completed questionnaire at the BUMPES office. Pre-specified secondary outcome measures were the number of questionnaires returned without chasing by the study team and the total cost of the vouchers sent out by study arm.
Recording of questionnaire receipt, date received and voucher sent was made using internal trial administration systems. Postal versus online receipt was also recorded.
The sample size was predetermined by the numbers of questionnaires that remained to be sent at the estimated start time of the SWAT. BUMPES started recruiting in October 2010 and finished in January 2014. A total of 3,236 women were randomised. It was estimated that approximately 1,150 women would remain to be followed up at the start date of this study (beginning August 2014). Assuming that approximately 15 % of these women would be excluded from receiving the questionnaire due to stillbirth, infant death, or address details unknown or different to the infant, 980 women would be eligible to be randomised in the nested study (approximately 490 per group).
In order to assess the detectable effect size possible with the given sample size, we estimated the control group risk based on current literature. Khadjesari et al.  investigated the use of an offer of an incentive (a £10 Amazon gift voucher) versus no offer of an incentive on follow-up rates in an online trial. They found an increase of 9 % (95 % CI 5 % to 12 %) when using the offer of an incentive. Kenyon et al.  investigated the use of a monetary incentive included in reminder letters versus no incentive and found an improvement in the response rate between the two groups of 11.7 % (95 % confidence interval 4.7 to 18.6 %).
The follow-up questionnaire return rate for BUMPES up to June 2014 was 59 %. Assuming that this could increase by at least 5 % with the use of an offer of an incentive either with an incentive cover letter at first mailout or an incentive reminder letter only, a sample size of 980 would be sufficient to demonstrate an increase in questionnaire return of 8 % from 64 % in the reminder group to 72 % in the first mailout group at a two-sided 5 % significance level with 80 % power. A detectable difference of between 8 and 8.5 % would be possible for return rate estimates in the reminder group of between 60 and 70 %.
For all analyses, an intention to treat approach was taken and participants were analysed in the groups into which they were randomly allocated, i.e. comparing outcomes for women allocated to the first mailout group with outcomes for women allocated to the reminder group, regardless of allocation received.
All analyses were based on all women randomised for whom we had data available.
Participants in the two randomised groups are described separately with respect to baseline demographics and clinical characteristics, including the primary outcome for the main BUMPES study, and recorded on the BUMPES Woman and Infant Data Collection Booklet.
The return rate and chase rate before the introduction of the randomised interventions (i.e. before the SWAT started) and at the end of the study (with both SWAT trial arms combined) is presented using numbers and percentages. The return rate and chase rate by method of completion (online versus postal) is described by trial arm using numbers and percentages.
An adjusted analysis was performed on the two return rate outcomes adjusting for centre (the stratification factor at randomisation)  as a random effect. The analysis was carried out using log binomial regression models, and results presented as adjusted risk ratios with 95 % confidence intervals (CI). Differences in means of the total cost averaged over the total number of participants is presented with 95 % confidence intervals.
To examine whether the effect of when vouchers are sent is consistent across specific subgroups of women, a subgroup analysis by IMD (Index of Multiple Deprivation) quintile was pre-specified. Results are presented as risk ratios plus 95 % CI for each subgroup, by intervention group, with the p value for the statistical test of interaction.
Stata/SE for Windows (version 13.1) was used for all analyses.
Randomisation to the incentive nested study started on 31st July 2014 and continued until all questionnaires and reminders had been sent (last letter sent 6th March 2015). The total number of women in the SWAT was 1026. Eight women were excluded from all analyses as it was discovered after they had been randomised to the SWAT that they had moved address (see Fig. 1).
Balance between the SWAT trial arms in baseline characteristics was good. There were only small imbalances in onset of labour (spontaneous or induced), diagnosis of pre-eclampsia, and spontaneous vaginal birth (the BUMPES primary outcome) (see Table 1).
The overall percentage of questionnaires returned before the SWAT started was considerably lower compared to that for participants included in the SWAT (1149/2067, 55.6 % vs. 743/1018, 73.0 %). This trend is also seen in the percentage returned without any reminder letters being sent (729/2067, 35.3 % vs. 476/1018, 46.8 %).
For the primary outcome, the percentage of questionnaires returned overall for those in the first mailout group was slightly higher than those in the reminder group (74.2 vs. 71.8 %), but this was not statistically significant at the 5 % level (adjusted risk ratio (RR) 1.03 and 95 % CI 0.96 to 1.11). Women who receive a cover letter promising an incentive at first mailout are more likely to return their questionnaire without a reminder letter being required compared to those receiving a standard cover letter (adjusted RR 1.22, 95 % CI 1.07 to 1.39). One in five of all questionnaires returned (152/743, 20.5 %) were completed online, with slightly fewer being returned online in the first mailout group compared to the reminder group (18.0 vs. 23.0 %). The mean difference in the total cost of the vouchers per participant was £4.56 (95 % CI £4.02 to £5.11), with the cost being higher in the group receiving the incentive cover letter at first mailout (see Table 2).
Figure 2 presents the percentage of questionnaires returned according to how many times a reminder letter was sent, and broken down by postal versus online completion. If a reminder was sent fewer women returned the questionnaire in the group receiving the promise of an incentive in the first mailout compared to those receiving a promise in the reminder letter (11.5 vs. 14.6 % for the first reminder, and 8.9 vs 11.5 % for the second reminder), with consistently more women completing the questionnaire online in the reminder group.
The pre-specified subgroup analysis is presented as a forest plot in Fig. 3. There is no evidence of heterogeneity for IMD subgroups for the primary outcome of overall response rate (p = 0.43), suggesting no differential intervention effect across deprivation quintiles.
In this study within a trial, there is no evidence to suggest that an offer of a monetary incentive at first mailout compared to only when a reminder letter is sent makes a substantial difference to the overall return rate of a 1-year follow-up questionnaire. Although there were slightly more questionnaires returned in the group receiving the offer at first mailout (an absolute difference of 3.4 %) the corresponding adjusted risk ratio of 1.03 (95 % CI 0.96 to 1.11) was not statistically significant at the 5 % level.
The return rate for women included in the SWAT compared to that before the SWAT was introduced demonstrated a marked improvement (absolute difference 17 %). Although this is not a randomised comparison, it is consistent with that found by Kenyon et al.  in a randomised study within a trial which showed an increase in the return rate of 11.7 % (95 % CI 4.7 % to 18.6 %) with the inclusion of a high-street voucher vs no voucher sent with a reminder letter to parents of seven year old children. In addition, a systematic review  including the study by Kenyon et al., showed that the addition of a monetary incentive was more effective than no incentive at increasing response rates to postal questionnaires (RR 1.18, 95 % CI 1.09 to 1.28).
This SWAT used a £10 high-street gift voucher as a monetary incentive. The mean cost of vouchers per participant was greater in the group receiving the offer at first mailout (£7.50 vs £3.00). Coupled with the lack of evidence of a difference in the overall return rate, this would indicate that sending the offer of an incentive with a reminder letter only is a cost-effective approach to improving return rates. However, there is evidence to suggest that the return rate without requiring reminders is higher in the group for whom the incentive is offered in the first mailout (absolute difference 9.4 %). The cost of administering the additional reminder letters was not calculated, but is a serious consideration that would need to be offset against the expected cost of the vouchers and could depend on the administration resources available in the trial team as well as the sample size of the study.
There are ethical issues to consider with the approach of only sending an offer of an incentive to those participants who do not return their questionnaire promptly. Consideration should be given to the chance that participants in a study may communicate with each other, and share their experiences regarding whether or not they received an incentive.
This is the first known SWAT to investigate the use of incentives for improving questionnaire return rates in a population of first time mothers with infants around 1 year old. This study suggests that offering a monetary incentive when a reminder is required could be cost-effective depending on the sample size of the study and the resources available to administer the reminder letters.
CI, confidence interval; IMD, index of multiple deprivation; IQR, interquartile range; RR, risk ratio; SD, standard deviation; SWAT, study within a trial
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires (Review). Cochrane Database of Systematic Reviews; 2009. p. 1-527. Issue 3. Art. No.: MR000008.
Brueton VC, Tierney J, Stenning S, Harding S, Meredith S, Nazareth I, Rait G. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev; 2013. p. 1-126. Issue 12. Art. No.: MR000032.
Khadjesari Z, Murray E, Kalaitzaki E, White IR, McCambridge J, Thompson SG, Wallace P, Godfrey C. Impact and Costs of Incentives to Reduce Attrition in Online Trials: Two Randomized Controlled Trials. J Med Internet Res. 2011;13(1), e26.
Kenyon S, Pike K, Jones D, Taylor D, Salt A, Marlow N, Brocklehurst P. The effect of a monetary incentive on return of a postal health and development questionnaire: a randomised trial. BMC Health Serv Res. 2005;5:55. ISRCTN53994660.
Kahan BC, Morris TP. Analysis of multicentre trials with continuous outcomes: when and how should we account for centre effects? Stat Med. 2013;32:1136–49.
We thank all the women who participated in this study.
The Epidural and Position Trial Collaborative Group:
Professor Debra Bick, Professor of Evidence Based Midwifery Practice, Kings College London
Dr Annette Briley, Consultant Midwife, Guys and St Thomas’s NHS Foundation Trust (replaced Geraldine O'Sullivan, Lead Clinician in Obstetric Anaesthesia in 2012)
Associate Professor Edmund Juszczak, Director, National Perinatal Epidemiology Unit, Oxford
Lynn Lynch, Senior Research Midwife
Professor Christine MacArthur, Professor of Maternal and Child Epidemiology, University of Birmingham
Dr Phillip Moore, Consultant Anaesthetist, University Hospital Birmingham NHS Trust
Professor Mary Nolan, Professor of Perinatal Education, University of Worcester
Dr Felicity Plaat, Lead Clinician & Consultant Anaesthetist, Queen Charlotte’s and the Hammersmith Hospital/Senior Lecturer, Imperial College London
Dr Julia Sanders, Consultant Midwife/Reader in Midwifery, Cardiff University
Professor Andrew Shennan, Professor of Obstetrics, Kings College London
Dr Matt Wilson, Consultant in Obstetric Anaesthesia/Senior Lecturer in Anaesthesia, Sheffield Teaching Hospital/University of Sheffield
Steve Hibbert, IT Manager, Comprehensive Clinical Trials Unit, University College London
Elizabeth Howden, Trial Manager, Comprehensive Clinical Trials Unit, University College London (until December 2013)
Sawretse Leslie, Trial Manager, Comprehensive Clinical Trials Unit, University College London (December 2013 until September 2015)
Amber Gibney, Data Manager, Comprehensive Clinical Trials Unit, University College London (until October 2013)
Claire Nelis, Statistician, National Perinatal Epidemiology Unit Clinical Trials Unit, University of Oxford (until June 2011)
The BUMPES Trial was funded by the UK National Institute for Health Research Health Technology Assessment programme (NIHR HTA) (project number 08.22.02). No additional funding was required to carry out this study. The NIHR HTA had no role in the design of the study and collection, analysis, and interpretation of data or in writing this paper.
Availability of data and materials
The dataset supporting the conclusions of this article is available from the last author upon request.
PH and PB developed the protocol and managed the trial. JB did the analyses. PH wrote the first draft of the manuscript and revised it with input from PB and JB. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Research ethics committee approval for the incentives study was granted by the National Research Ethics Service - Oxfordshire REC B on the 30th July 2014 (reference number 09/H0605/114).
For the BUMPES Trial women were provided with written information during pregnancy and again in labour. Women were able to provide written informed consent during the first stage of labour but were not eligible to be randomised until the second stage of labour had been diagnosed. No consent from participants was sought for the incentive study.
About this article
Cite this article
Hardy, P., Bell, J.L., Brocklehurst, P. et al. Evaluation of the effects of an offer of a monetary incentive on the rate of questionnaire return during follow-up of a clinical trial: a randomised study within a trial. BMC Med Res Methodol 16, 82 (2016). https://doi.org/10.1186/s12874-016-0180-9