Skip to content

Advertisement

  • Research article
  • Open Access
  • Open Peer Review

Effects of pre-notification, invitation length, questionnaire length and reminder on participation rate: a quasi-randomised controlled trial

BMC Medical Research MethodologyBMC series – open, inclusive and trusted201818:3

https://doi.org/10.1186/s12874-017-0467-5

  • Received: 22 May 2017
  • Accepted: 28 December 2017
  • Published:
Open Peer Review reports

Abstract

Background

Improving participation rates in epidemiologic studies using questionnaires and biological sampling is important for the generalizability of the outcome. The aim of this study was to examine the effects of pre-notification, invitation length, questionnaire length, and reminder on participation rate and to investigate whether some factors contributed to participants doing both the questionnaire and blood sampling as oppose to only one part.

Methods

Our study was embedded within the pilot testing of a large population-based study about prostate cancer screening. Our study sample consisted of 28.134 men between 50 and 69 years of age and living in the region of Stockholm (Sweden) invited to respond to a web-based questionnaire and to provide blood for prostate cancer testing. The men were randomly allocated according to birth of date to receive either: (a) a pre-notification postcard or not; (b) a shorter or a longer invitation letter; (c) a shorter or a longer web-based questionnaire, and (d) a reminder or not. The effects of the survey design factors were tested using chi-square.

Results

The use of a pre-notification (p < 0.0001), a longer questionnaire (p = 0.004) and the use of a reminder (p = 0.02) were associated with an increase in overall participation, i.e. responding to the questionnaire or providing blood for PCT or performing both components.

Conclusions

The results of this pilot study justified the use of a pre-notification and a reminder in the following large population based study since the benefits of increased participation traded off against the greater costs incurred. Furthermore, we were able to use the longer version of the questionnaire, which allowed us to collect more information without risking a lower response rate.

Keywords

  • Participation rate
  • Response rate
  • Pre-notification
  • Invitation letter
  • Questionnaire length
  • Reminder

Background

Web-based surveys offer many advantages for improving data quality compared to written questionnaires [1]. Skipping irrelevant sections which are conditional on the responses to previous questions, minimising invalid responses, and requiring answers to missed questions are examples of how web-based surveys may increase the rate of complete surveys [2]. Not only are they improving data quality, but also, more cost-effective than postal or telephone surveys [3]. This is especially attractive for large population studies since material and staff costs tend to be proportional to the number of respondents. In addition, with Internet access being now widespread, barriers to electronic data gathering are diminishing.

At the same time, participation in epidemiological studies has decreased over the last few decades [2, 4]. Refusal rates are increasing, whereas making contact with potential respondents is becoming more strenuous [5]. For surveys directed to specialised populations such as professional groups or students, sending an e-mail with a direct hyperlink is usually possible. However, the lack of such email lists for the general population entails for an initial contact to be made through the use of conventional mail and not e-mail.

A large body of research has sought to identify various factors possibly affecting participation rates [57]. Factors affecting internet response rates show general similarities to those found for other survey modes [6]. Response rates for web-based questionnaires have been reported to be lower than for paper-based questionnaires [8]. Nonetheless, given the rapid changes within electronics and internet-literacy, and as suggested by Hohwü et al. [9], the platform used to collect data should reflect the development of electronic devices. Hence, web-based questionnaires could replace paper-based questionnaires with minor effects on response rates [9]. Thus, there is a need to examine participation rates in large population studies using a log-on web-based questionnaire coupled with conventional mail as mode of initial contact.

Between 2012 and 2014, a large population-based study of prostate cancer testing was carried out in Stockholm, Sweden [10]. Initial contact was made by conventional mail, inviting participants to log-on to a web-based survey and to provide blood sample for diagnostic measurements of prostate cancer risk. Because the study aimed to invite over 100.000 men, concerns were raised over the optimal invitation procedures.. Those procedures included use of a pre-notification card, length of the invitation letter, length of the questionnaire and use of a reminder. The aim of the study was to compare participation rates between the different modes of invitation and recruitment approaches.

Methods

Study context

The study was embedded in the pilot study for the Stockholm 3 trial (STHLM3), a large population-based diagnostic study of men aged 50–69 years investigating prostate cancer testing (PCT) [10]. The pilot study was conducted in order to test the feasibility of the invitation and survey design, as well as the procedures for data collection. Participation consisted of two components: completion of a web-based questionnaire and provision of a blood sample for prostate cancer testing (PCT). PCT required a visit to a hospital or a health centre for blood sampling. The participants could either decline to partake, or participate in one or both components. The PCT risk estimates were based on the blood samples only. The answers to the web-based questionnaire were collected to amount information about the participants in terms of physical activity, diet, knowledge about prostate cancer in the population, attitudes toward PCT, and health related quality of life.

Study sample and procedures

Invitations were sent out between September 2012 and November 2012 during six consecutive weeks to a total of 28.134 men without a previous prostate cancer diagnosis, between 50 and 69 years of age, living in the region of Stockholm. They were randomly selected by date of birth from the Swedish Population Register, kept by the Swedish Tax Agency. The invitations were sent on a weekly basis to all men who met the criteria and who had their birthday that same week, thus rendering this sampling quasi-randomised.

The men in the study sample were allocated to receive: (a) a pre-notification postcard or not; (b) a shorter or a longer invitation letter; (c) a shorter or a longer web-based questionnaire, and (d) a reminder or not. The allocations were combined into six study arms. Each arm represents one dispatch week and the distribution is presented in Table 1.
Table 1

Distribution of men randomized to each of the six modes of allocation

Study Arm

Pre-notification

Invitation

Questionnaire

Reminder

N

1.

Yes

Short

Short

Yes

3575

2.

Yes

Long

Short

Yes

3601

3.

Yes

Short

Short

No

6130

4.

Yes

Short

Long

No

4835

5.

No

Short

Long

No

4956

6.

No

Short

Short

No

5037

Total

28,134

Dependent variables

Data on provision of blood samples and on questionnaire completion were collected from the STHLM3 database. The outcomes measured were the proportion of completed web-based questionnaires, the proportion of men who provided blood samples, and the proportion of participants who completed both. The dependent variables were assessed by the end of April 2013.

Independent variables

Pre-notification

The pre-notifications were sent out as postcards one week before the mailed invitation letters. The postcard described the forthcoming invitation to participate in the STHLM3 trial, and, thereby, getting a free PCT, as well as more information about STHLM3.

Postal invitation letter

Both the short and the long invitation letters consisted of one sheet of paper. The front page contained information about the study, as well as the individual credentials to login to the web-survey, and stated that the web-based questionnaire would take approximately 20 min to fill out. The long version (406 words in Swedish) was approximately twice as long as the short version (218 words in Swedish). The back of the sheet contained a checklist explaining what to bring when giving the blood sample and how to respond to the web-survey. The checklist was made out of 274 words and was included in both the short and the long version of the invitation letter. Along with the letters, an extensive brochure with more in-depth information about the STHLM3 study and its procedures were included in both versions of the letters. Consequently, both invitation packages contained all in all the same information. The differences consisted in the format of the letter and the amount being mentioned in the front page of the letter. Both letters had the login information of the front page. An English translation of the letters can be found in the supplementary files (see Additional file 1).

Web-based questionnaire

The questionnaire was designed for the study and largely similar to the one used later on in STHLM3 [10]. The short web-based questionnaire consisted of 500 items; whereas, the long web-based questionnaire consisted of 1000 items. The short web-based questionnaire took an average of 18 min to fill out, whereas, the long web-based questionnaire took, on average, 47 min to complete. The layout and the issues tapped by the long and the short questionnaires were similar. The differences laid in how many follow-up or sub-questions there were.

Reminder

Those who were assigned to receive a reminder were sent a reminder two weeks after receiving the invitation letter, regardless of whether they had already participated or not. Apart from stating that it was a reminder, the letter used the same wording as the short invitation letter.

Data analysis

Descriptive statistics were performed to present the study sample. The participation rates between the different arms were compared by Chi-square tests. All tests were two-sided and significance level was set to 0.05.

Results

Out of the 28.134 men invited to participate, 9.543 men (34%) participated to one or two components. Of those participants, 7.302 men (77%) participated in both components, i.e., responded to the web-based questionnaire and provided blood for PCT; 1.744 men (18%) only provided blood for PCT; and 497 men (5%) only responded to the web-based questionnaire.

Use of a pre-notification (p < 0.0001), a longer questionnaire (p = 0.004) and a reminder (p = 0.02) was associated with an increase in overall participation, i.e. participation to one or two components. The length of the invitation letter was not associated with participation (Fig. 1).
Fig. 1
Fig. 1

Participation rates for each dispatch arms. * P-values of participation to one or two components are shown. a + = presence of pre-notification;  = absence of pre-notification. b + = long invitation letter;  = short invitation letter, c + = long questionnaire;  = short questionnaire. d + = presence of reminder;  = absence of reminder

Additionally, out of the 7.302 men that participated to both components, pre-notification (p = 0.0007) and a longer questionnaire (p = 0.0003) were associated with an increase in participation.

Finally, among the 1.744 men who only responded to the questionnaire, none of the four factors were associated with participation. Pre-notification was the only factor that increased participation (p = 0.002) among the 497 men who only provided blood for PCT.

Discussion

In the present study, inviting almost 30,000 men, associations on participation rates between four invitation and survey design factors (pre-notification, invitation letter length, questionnaire length, and reminder) were investigated within the STHLM3 setting. These four factors have been shown to play a substantial role in influencing response rates for surveys in previous studies [6, 7]. Use of a pre-notification, a reminder and a longer questionnaire were associated with increase in overall participation, whereas the length of the invitation did not appear to have any impact.

Sending a pre-notification turned out to be associated with increased participation rates. This finding is in concordance with previous research, stating that pre-notifications increase the odds of response [11]. However, we do not know whether that increase is due to the content of the postcard, the format of the postcard, or the pre-notification itself. The sentence “Join the study STHLM3 and get a free prostate cancer test”, which was included in the pre-notification postcard, could be seen as an incentive, and could have contributed to the positive association.

The length of the invitation letter was not associated with participation rates. This could be due to the fact that the overall information provided in the invitation packages was similar for both the short and the long invitation letter. The difference laid in the format of the front page, and the amount of information delivered on that front page. Hence, it seems that the participants receiving the shorter invitation letter were given enough information.

Previous population-based studies suggest an inverse association between questionnaire length and response rate [1113]. Conversely, in our study, the longer questionnaire was associated with higher participation rates. A possible explanation is that the men who spent longer time responding to the questionnaires were more involved and the items might have led to a greater interest in participation in PCT. Another possible explanation as to why questionnaire length did not show a negative association with response rate in our study could be in the statement of the length of the web-based questionnaire in our invitation letter. Galesic et al. [14] manipulated the stated length and showed that a longer stated length reduces the response rate. In our study, however, 20 min was the stated length for both questionnaires. Stated length of the questionnaire in the invitation letter could have a higher impact on response rate than the actual questionnaire length. Thus, it is likely that once a man had started to respond to the questionnaire, he completed it despite the amount of time required.

A web-based questionnaire was used, and the Swedish population can be considered as a highly Internet-literate population with almost full access to the Internet. Additionally, according to de Bernardo et al. 2013 [15], web survey as a tool of data collection is as effective as paper surveys in populations above 50 years of age. Therefore, it is unlikely that the participation rates were affected by a lack of computer knowledge or by a lack of access to Internet.

The participation rates were higher for PCT than for the web survey, even though the PCT required attendance to a medical centre, whereas the web survey could be completed at home or at work. A possible explanation for the higher participation rates for PCT is that the men perceived a personal benefit of participating in PCT. By participating they received information about their individual risk for prostate cancer whereas responding to the questionnaire did not result in any benefit for the individual man.

The strengths of the present study include the large sample, the randomized design and the use of the STHLM3 database, where reliable data were available. In addition, the questionnaires used were based on international studies on PCT.

The results from this study on factors of importance for participating in a PCT study were used for the STHLM3 main trial and seemed to substantially having increased the participation rates. The rates increased from 34% observed in the present pilot study to 42% achieved in the large population based STHLM3 study [10]. The results also highlight the importance to test, when possible, the study design of an upcoming trial, especially if it will be delivered to a large population sample.

Conclusion

The results of this pilot study suggest that web-based questionnaires coupled with pre-notification may be an alternative to costly traditional paper questionnaires, and that there is room for improvement in response rates.

The results of this pilot study justified the use of a pre-notification and a reminder in the following large population based study since the benefits of increased participation traded off against the greater costs incurred. Furthermore, we were able to use the longer version of the questionnaire, which allowed us to collect more information without risking a lower response rate. No negative effects on participation rates were observed by the use of the longer invitation letter or the longer questionnaire. The results of this study were used in the STHLM3 study, resulting in an improvement of the participation rate.

Abbreviations

PCT: 

Prostate cancer testing

PSA: 

Prostate specific antigen

Declarations

Acknowledgements

The authors thank all the participating men, the STHLM3 project manager, Ola Steinberg, for study coordination and administration, as well as the data manager, Astrid Björklund.

Funding

This study was mainly supported by the Cancer Risk Prediction Center (CRisP), a Linneus Centre (Contract ID 70867901) financed by the Swedish Research Council; but also the Swedish Cancer Society; and the Swedish Research Council for Health, Working Life, and Welfare (FORTE).

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

MK led the analysis and the writing of the manuscript. ME assisted with the analysis and with the writing of the manuscript. HG assisted with the design of the study, data collection, trial administration, and analysis. JA and YB assisted with design, data analysis and writing of the manuscript. All authors contributed to the interpretation of the findings, critical revision of the manuscript and approved the final version.

Ethics approval and consent to participate

Ethical approval for this study was obtained from the Regional Ethical Review Board in Stockholm (DNR 2012/572–31/1). All participants provided written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Oncology-Pathology, Karolinska Institutet. Karolinska University Hospital, Z1:00, 171 76 Stockholm, Sweden
(2)
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Boxes 281, 171 77 Stockholm, Sweden
(3)
Department of Clinical Science Intervention and Technology, Karolinska Institutet, 171 77 Stockholm, Sweden

References.

  1. Ekman A, Litton JE. New times, new needs; e-epidemiology. Eur J Epidemiol. 2007;22(5):285–92.View ArticlePubMedGoogle Scholar
  2. van Gelder MM, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol. 2010;172(11):1292–8.View ArticlePubMedGoogle Scholar
  3. Scott A, Jeon SH, Joyce CM, Humphreys JS, Kalb G, Witt J, Leahy A. A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Med Res Methodol. 2011;11:126.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Morton LM, Cahill J, Hartge P. Reporting participation in epidemiologic studies: a survey of practice. Am J Epidemiol. 2006;163(3):197–203.View ArticlePubMedGoogle Scholar
  5. Galea S, Tracy M. Participation Rates in Epidemiologic Studies. Annals of Epidemiology. 2007;17(9):643–53.View ArticlePubMedGoogle Scholar
  6. Fan W, Yan Z. Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior. 2010;26(2):132–9.View ArticleGoogle Scholar
  7. Hartge P. Participation in population studies. Epidemiology. 2006;17(3):252–4.View ArticlePubMedGoogle Scholar
  8. Zuidgeest M, Hendriks M, Koopman L, Spreeuwenberg P, Rademakers J. A comparison of a postal survey and mixed-mode survey using a questionnaire on patients’ experiences with breast care. J Med Internet Res. 2011;13(3):e68.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Hohwu L, Lyshol H, Gissler M, Jonsson SH, Petzold M, Obel C. Web-based versus traditional paper questionnaires: a mixed-mode survey with a Nordic perspective. J Med Internet Res. 2013;15(8):e173.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Gronberg H, Adolfsson J, Aly M, Nordstrom T, Wiklund P, Brandberg Y, Thompson J, Wiklund F, Lindberg J, Clements M, et al. Prostate cancer screening in men aged 50–69 years (STHLM3): a prospective population-based diagnostic study. Lancet Oncol. 2015;16(16):1667–76.View ArticlePubMedGoogle Scholar
  11. Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;3:MR000008.Google Scholar
  12. Edwards P, Roberts I, Sandercock P, Frost C. Follow-up by mail in clinical trials: does questionnaire length matter? Control Clin Trials. 2004;25(1):31–52.View ArticlePubMedGoogle Scholar
  13. Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D. iConnect c: Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial. BMC medical research methodology. 2011;11:62.View ArticlePubMedPubMed CentralGoogle Scholar
  14. Galesic M, Bosnjak M. Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey. Public Opinion Quarterly. 2009;73(2):349–60.View ArticleGoogle Scholar
  15. de Bernardo DH, Curtis A. Using Online and Paper Surveys: The Effectiveness of Mixed-Mode Methodology for Populations Over 50. Research on Aging. 2013;35(2):220–40.View ArticleGoogle Scholar

Copyright

Advertisement