Shortening a survey and using alternative forms of prenotification: Impact on response rate and quality

1.1. Questionnaire length effects

One of the hypotheses applied to survey participation is the notion of opportunity cost [7]. In the context of increasingly hectic lives, surveys that are perceived to take too long to complete may not be viewed favorably and may bring about diminished response. Indeed, evidence from 56 trials showed that the odds of response to a mailed survey were 60% higher in shorter versus longer questionnaires (OR = 1.64; 95% CI 1.43 to 1.87) [8]. However, what is considered long versus short appears to have changed over time. Whereas a 12 page cut-off appeared to have differentiated long versus short in the 1970s [9], subsequent speculation has suggested any questionnaire longer than four pages ought to be considered long [10]. Among physicians, response rates tend to decrease if a questionnaire exceeds a threshold of 1000 words [11]. Some have posited a curvilinear relationship between response propensity and questionnaire length whereby the likelihood of response is lowest when the questionnaire is overly long and when it is perceived to be too short [12]. The anticipated negative effect of a short questionnaire is thought to be motivated by a lack of importance attached to this type of survey vis-à-vis a longer and more comprehensive counterpart [13].

There exists some suggestive evidence in support of the notion of questionnaires being too short. For example, Asch and colleagues [14] found that mailed surveys with more pages had higher response rates than shorter surveys, although this response effect disappeared when length was measured by the number of questions rather than pages. Champion and Sear [15] found that response rates were significantly higher for a 9-page questionnaire than 3- or 6-page questionnaires. Similarly, Mond et al.[13] found the overall response rate to be higher for their long form questionnaire (14 pages) than their short form questionnaire (8 pages). On the shorter end of the questionnaire spectrum, Goldstein [16] found that the odds of response to a one page questionnaire decreased by half (OR = 0.47; 95% CI 0.34 to 0.66) when a double postcard was used. Although the preponderance of evidence falls squarely on the side of using shorter versus longer questionnaires to increase response, these findings suggest that may not always be the case, especially when considering questionnaire lengths beneath the threshold of four pages.

1.2. Prenotification effects

Prenotification, or the act of contacting prospective respondents before they are mailed an actual questionnaire, has been shown to be an effective way to increase response in both telephone surveys [17] and mailed surveys [8]. Prenotification works because it underscores the legitimacy of the survey, takes away suspicion, communicates the value of the survey, and evokes the principles of social exchange [17]. For telephone surveys, a recent meta-analysis found that prenotifcation increased participation from 58 percent (no prenotification) to 66 percent (prenotification) [17]. Prenotification may have an even larger effect in mailed surveys as Edwards et al.[8] found that the odds of response for a mailed survey were substantially higher with prenotification (OR = 1.45; 95% CI 1.29 to 1.63) than without. However, the best method of prenotification remains unclear. Virtually all of the studies reviewed in the Edwards et al. [8] meta-analysis utilized letters as the form of prenotification, some utilized telephone contact, and very few investigated the effect of postcard prenotification; none directly compared letter versus postcard. If postcard prenotification is found to be equally efficacious in terms of eliciting response, then cost savings can accrue to investigators as postcards are much less expensive to mail. Lessons from the few studies comparing the relative merits of prenotification via letter versus postcard in the context of the telephone survey suggest that postcards may be as effective in increasing response as letters [18, 19], although they are slightly less likely to be read [19, 20]. However, there is not an over-abundance of research on postcard prenotification in the telephone survey area either and there have been calls for more research into postcards as a form of prenotification [17].

1.3. The current study

Although there has been a fair amount of research undertaken studying the effects of questionnaire length on response rates, most of it has focused on manipulations in length at the higher end of the spectrum (viz. longer than four pages). Edwards et al.[8] indicates, "...that questionnaire length has a substantial impact on non-response, particularly when questionnaires are very short." (p. 11), but do not provide any direct comparisons of the impact of different lengths of surveys within the range of what is considered short. In addition, the extant literature on the effects of prenotification has mainly considered the effect of letters or telephone contact (versus none) as the primary prenotification vehicle. Very few have studied the viability of postcard prenotification even though use of postcards brings about rather substantial cost savings relative to other forms; what information exists comes from studies undertaken in the context of a telephone survey. How well these latter findings translate to mailed surveys is unclear. Therefore, we tested the effect of questionnaire length (2 pages versus 4 pages) crossed with prenotification type (letter versus postcard) on response rates, response times, and missing data totals in the context of a large population-based mail survey. To our knowledge, no published study has tested the effect of questionnaire length and prenotification type simultaneously in a factorial design.

2.1. Sample

This study was undertaken as part of a larger pilot study designed to determine the impact of different recall durations (3 months vs. 1 year) on individual gastrointestinal symptoms and functional gastrointestinal disorder (FGID) diagnoses. The sampling strategy and its associated power calculations were indexed off the principal aims of that parent study. Further details of this larger study and its findings can be found elsewhere [21]. Briefly, we randomly selected 900 residents of Olmsted County, Minnesota, aged 25-65 years old using the Rochester Epidemiology Project (REP). The REP is a comprehensive medical records linkage system that captures medical data from electronic and paper medical and autopsy records for patients using the Mayo Clinic, Olmsted Medical Center, their affiliated hospitals, or one private practice provider. Because most Olmsted County residents receive their medical care from one of those providers, it is possible to conduct population-based research on disease incidence, mortality, and use of health services in the region [22]. Importantly, from this sample frame we know the gender and age of both responders and non-responders, allowing us to assess how their distribution potentially differs across experimental conditions.

The sample was stratified by age and gender. Those that had previously participated in any gastrointestinal-related survey conducted by two of the authors (Talley, Locke) were excluded. Also excluded were subjects with significant illnesses, a major psychotic episode, mental retardation, dementia, inmates in the Federal Medical Center (a prison managed by the U.S. Federal Bureau of Prisons), and those that had previously refused general authorization to review their medical record for research (less than 4 percent of Olmsted County residents) [23]. These exclusions were done prior to the random assignment described in the next section. The survey was mailed in July 2008.

2.2. Procedure

Figure 1 provides a flowchart of the study sample, data collection process, and random assignment. Subjects were randomly assigned to four conditions using the RANUNI function in SAS v. 9.1. software according to a 2 × 2 factorial design to enable us to simultaneously assess the effects of 4 and 2 page versions of the questionnaire and the effect of pre-notification type (letter or postcard). The subjects were first randomized to length and then notification type within group. Approximately 225 cases were assigned to each of the four conditions. After randomization, it was discovered that 120 cases were ineligible due to residence outside of Olmsted County or deceased status. As such, 780 cases were available for data collection.

The varied questionnaire length versions were based on the Talley Bowel Disease Questionnaire (Talley-BDQ; [24]). The Talley-BDQ was designed as a self-report instrument to measure symptoms experienced over the past year and to collect past medical history. For this experiment, the full 16 page Talley-BDQ was shortened to a 4 page version and then to a 2 page version aimed to comprise only those questions needed to achieve the pre-defined specific goals. For that purpose, a sequential procedure was followed: (1) variables derived from each question were listed, (2) variables needed of the specific targeted research projects were selected, (3) all unnecessary questions were deleted, (3) remaining list was reviewed by investigators, (4) remaining questions were formatted in a 4 page questionnaire, (5) questions strictly needed to achieve the objective of one project were further refined to fit 2 page questionnaire. After the shortening process, a 2 page version of the questionnaire was drawn with 18 questions (7 questions about abdominal pain and related changes in bowel habits; 9 questions about usual bowel pattern; 2 questions relative to consultation). The 4 page version of the questionnaire included 17 additional questions (one for fecal incontinence, 11 for upper GI symptoms, and 5 for medications used) plus a short version of somatic symptom checklist (SSC -6 items). With the exception of the last question, the items were identical across the first two pages of each survey. The last question on the two page survey was, however, identical to the last question on the four page survey.

The letter and postcard prenotifications contained the same text. Both identified the survey sponsor and described the purpose of the study, how subjects were chosen, the importance of responding, the anticipated completion time (10 minutes or less), and how confidentiality will be protected. The letter and postcard also asked prospective respondents to mark a box if they wished to receive a report of the study results and alerted them to the fact that a book titled "Mayo Clinic on Digestive Health" would be included in the survey packet to come as a token of appreciation. The main differences between notification type was that the letter, but not the postcard, contained a salutation to a specific individual and included the primary investigator's signature (Locke).

All subjects were sent either a letter or a postcard one week prior to mailing the survey package. A week after pre-notification, a survey package was sent to all potential respondents. The package included a cover letter, the book, a pen incentive, and one of two versions of the modified Talley-BDQ. Reminder letters, along with another survey, were sent to nonresponders 4 weeks after the first mailing. Subjects who indicated at any point that they did not want to be contacted further were excluded from the study. All consent and study procedures were approved by the Mayo Clinic Institutional Review Board; the survey data collection was conducted by the Mayo Clinic Survey Research Center.

2.3. Statistical Methods

Sample characteristics were summarized with frequencies and percentages for categorical data (gender, age group, race), means and standard deviations for age (continuous), and medians and inter-quartile range (IQR) for time to response. Response rates (RR) were calculated overall as well as within each survey condition as the number of surveys returned divided by the number of surveys sent. Time to response (among responders) was calculated as the number of days between the initial survey mailing and response date. The primary outcome for the analysis was whether or not a survey recipient responded. Overall differences in response rates between factors (survey length and pre-notification type) and characteristics (gender, age group, and race) were assessed with chi-square tests (or Fisher's exact tests where appropriate). Race was categorized as "white", "non-white" (American Indian/Alaskan Native, Black or African American, Native Hawaiian/Pacific Islander, and Asian), and "other/unknown" (those that specifically indicated "other" or chose not to disclose). Differences in time to response among responders between survey conditions were compared with pair-wise Wilcoxon rank-sum tests. As an assessment of item non-response, the percentage of respondents with missing data for each question in common to the 2-page and 4-page surveys was compared with Fisher's exact tests.

Logistic regression was used to assess the effects of survey length and pre-notification type on likelihood to respond adjusted for each other as well as for age (continuous), gender, and race. The first model included main effects only (notification type, survey length, age, gender, and race). Two-way interactions between the survey characteristics and demographics were then assessed, and the second model included a significant age-by-survey length interaction. Odds ratios with 95% confidence intervals were reported. For the second model, to illustrate how the effect of survey length differed by age, the odds ratio for survey length was calculated at different points across the age range (ages 25, 35, 45, and 55). All analyses were performed using SAS v. 9.1 software (Cary, NC). A p-value of < 0.05 was regarded as statistically significant.

3.1. Sample characteristics

3.2. Response rates, response times, and missing data totals

3.3. Logistic regression analysis