Comparison of response rates on invitation mode of a web-based survey on influenza vaccine adverse events among healthcare workers: a pilot study

Background Web-based surveys have become increasingly popular but response rates are low and may be prone to selection bias. How people are invited to participate may impact response rates and needs further study as previous evidence is contradictory. The purpose of this study was to determine whether response to a web-based survey of healthcare workers would be higher with a posted or an emailed invitation. We also report results of the pilot study, which aims to estimate the percentage of adults vaccinated against influenza who report recurrent systemic adverse events (the same systemic adverse event occurring successively following receipt of influenza vaccines). Methods The pilot study was conducted in November 2016 in Toronto, Canada. Members of a registry of adults (18 years and older and predominantly healthcare workers) who volunteered to receive information regarding future studies about influenza were randomly assigned to receive either an email or postal invitation to complete a web-based survey regarding influenza vaccinations. Non-respondents received one reminder using the same mode of contact as their original invitation. Results The overall response rate was higher for those sent the invitation by email (34.8%) than by post (25.8%; p < 0.001) and for older versus younger participants (ptrend < 0.001). Of those who responded, 387/401 had been vaccinated against influenza at least once since adulthood. Of those responding to the question, 70/386 (18.1%) reported a systemic adverse event after their most recent influenza vaccine including 22 (5.7%) who reported a recurring systemic event. Systemic adverse events were reported more often by males 18–49 years old than by other groups (p = 0.01). Recurrent systemic adverse events were similar by age and sex with muscle ache being the most commonly reported recurrent reaction. More respondents who reported only a local adverse event (93.1%) planned to be vaccinated again next year than those with a systemic adverse event (69.7%; p = 0.04). Conclusions In this convenience sample of registry volunteers, response rates were generally low, but were higher for the emailed than posted invitations and for older than younger adults.


Background
Web-based surveys are becoming increasingly popular with the expansion of access to the internet [1]. Advantages over paper-based surveys include reduced cost [2,3], user convenience for automatic skip patterns [4], and reduced transcription error and data entry costs as responses are entered directly into a database [5]. However, technical issues can frustrate potential participants if sites are not easily accessible, properly designed, and well maintained. Emailed invitations to web-based surveys have many attractive advantages including direct links to survey sites, ease of sending and responding, and lower costs [6]. However, delivery and open rates suffer due to incorrectly recorded or outdated addresses, blockage by spam filters, and lack of perceived legitimacy [7]. Also, since email addresses may not be available for all members of the population of interest, there are concerns about selection bias [8]. In contrast, postal invitations may appear more legitimate and attract greater attention, but are somewhat more burdensome, requiring more participant effort when asked to respond to an online questionnaire [9], Postal invitations can also be more costly to the researchers, can suffer from non-delivery due to outdated or incorrect addresses and low response due to perceived importance.
Previous evidence comparing the impact of different invitation modes on response rates of web-based and paper-based surveys is contradictory. Several research reports have demonstrated the higher response rates to posted invitations for paper-based surveys compared to emailed invitations for web-based surveys [1,10], with the lower rates seen for postal invitations to web-based surveys [11]. Conversely, two studies of student populations reported higher rates of response to emailed invitations for web-based surveys than posted invitations for paper-based ones [12][13][14] suggesting possible age and cohort effects.
In a similar contradictory manner, posted invitations to web-based surveys had higher [6,8] or similar [6,9,15,16] response rates to emailed invitations for those same surveys. See Dykema et al. [6] for a more extensive review. Meanwhile, a large campus-wide project reported different results across subpopulations: emailed invitations yielded a significantly higher response rate to the web-based survey than postcard invitations for faculty (32.6% vs. 20.9%; p < 0.01), while there was no difference in response rates by invitation mode for non-faculty staff or for students after two invitations [17]. Dykema et al. [6] reported that response to mailed invitations to an online survey was more representative of the invitation group than those invited by email.
Influenza vaccination is strongly recommended in people working in healthcare facilities and an increasing number of facilities require employees to be vaccinated as a condition of employment or to be vaccinated or wear a surgical mask while providing care to patients during the influenza season. Despite evidence supporting vaccination against influenza for healthcare workers to protect their vulnerable patients [18,19], only about 50% are vaccinated annually in Canada [20,21]. Meanwhile, vaccine hesitancy and refusal appears to be increasing [22], with adverse events being a top concern of vaccine hesitant people. Numerous studies confirm higher rates of systemic adverse events in people receiving influenza vaccine than in those receiving a placebo [23], but there are no studies on the recurrence of systemic adverse events nor the effect that recurring systemic adverse events have on healthcare workers' decisions to be vaccinated again. Since annual revaccination is necessary due to the drifting and shifting of the circulating strains of influenza, these vaccines are distinct from other vaccination programs making the adverse events of particular importance.
The primary objective of this study was to determine the impact of invitation mode (emailed vs. posted) on response rates for a web-based survey regarding influenza vaccination. A secondary objective was to estimate the percentage of people receiving the influenza vaccine who reported recurrent systemic adverse events.

Study participants
The sample for this study was drawn from members of a registry maintained by the investigator. The registry consists of adults aged 18 years and older, mainly (84.4%) healthcare workers, who participated in previous studies conducted by the Infectious Disease Epidemiology Research Unit at Mount Sinai Hospital, Toronto, Canada. The registry contains the volunteer's name, year of birth, sex, and email and postal addresses. Eligibility for this study was limited to registrants with both an email and a postal address. The study was approved by the Sinai Health System research ethics board (REB #16-0267-E).

Study design and methodology
Eligible individuals were randomly assigned to one of eight groups. Four groups received an invitation by email with a direct link to the survey site while the other four groups received an invitation by post with the URL to the survey site included in the letter. Each invitation arm was randomized to receive one of four versions of the questionnaire: short or long with either open-or close-ended questions. A unique 6-digit personal identification (PID) code was assigned to each individual using a random number generator and used to ensure that only invited people were able to participate and to track responses so responding individuals did not receive unnecessary reminders. Individuals accessing the survey site first landed on the consent page. If they consented, they were brought to the PID code verification page. Once the code was verified, they were taken directly to the appropriate questionnaire. All questionnaires were implemented using SimpleSurvey® (OutSideSoft Solutions Inc., Quebec).
Invitations asked potential participants to help us learn "how to best ask questions" and their experiences with adverse events following immunization with influenza vaccines. The visual design and subject matter was consistent across the four versions of the questionnaire. Questionnaire version distinctions were made on the basis of i) overall length: short ( [24][25][26]. Questions included age, sex, occupation, work site, as well as influenza vaccination history, factors affecting the decision to be vaccinated, experience with side effects, and impact of side effects. To determine if there were signals that might predict adverse events in certain populations, there was a series of questions asking about non-influenza vaccines received and resulting side effects, as well as allergies or sensitivities to medications, foods, or environmental allergens. Skip logic was used to present respondents with follow-up questions based on previous responses. Open-ended responses were categorized by one person (XT) using the closed-ended items as a guide.

Data collection
The initial invitations were sent by post or email on November 22, 2016 to allow for the maximum number of respondents to be vaccinated against influenza for the season but to allow for reminders and responses before the Christmas holiday vacation period. Each invitation was personalized (e.g., "Dear Jane") and contained a direct URL to the survey, a brief introduction to the study to increase salience, and the PID code. The invitation emails were sent using the SimpleSurvey® interface which generated a list of email addresses to which the invitations had failed to send (e.g., address errors, inactive accounts). If errors were identified and a second email address was listed in the registry, a second attempt was made via email; if not, no further contact was attempted. If an invitation letter was returned by the postal service (e.g., invalid address), no further attempt was made to contact the subject. The research assistant answered telephone and email queries from potential respondents who had questions about the study or problems accessing the survey site.
Reminder emails were sent to all non-respondents seven days after the initial email while reminder letters were sent 14 days after the initial letter. The reminder letter was delayed by 7 days compared with the reminder email to allow for delivery by the postal system, which is commonly 2-4 business days. The online surveys were deactivated on December 28, 2016 and response data were downloaded the same day.

Statistical analysis
Response rates by invitation mode (i.e., post vs. email) were calculated based on the final disposition codes defined by the American Association for Public Opinion Research (AAPOR) using response rate 2 (RR2), which counts both complete and partial responses in the numerator [27].
(Abbreviation: RR, response rate; I, complete interview; P, partial interview; R, refusal and break-off; NC, non-contact; O, other; UH, unknown if household/occupied HU; UO, unknown, other.) Adverse events following influenza vaccination were categorized into four groups: no adverse event, local adverse event only, systemic adverse event (most recent only), and recurrent systemic adverse event. An adverse event was defined as any untoward, unfavourable, or unintended symptom or disease following influenza immunization. A local adverse event was defined as any local reaction at the injection site including pain, soreness, numbness, bruising, redness, or swelling. A systemic adverse event was any of: upper respiratory illness (with or without sore throat, coughing, stuffed or runny nose, etc.), fever/feverishness, headache, new joint pain, malaise/feeling generally unwell, new & unexplained muscle aches, tiredness/fatigue or sweating within 7 days, a rash with onset within 48 h, or an allergic reaction with onset within 24 h of vaccine receipt [28]. A recurrent systemic adverse event was defined as at least one of the same systemic adverse event symptoms occurring following the receipt of the two most recent influenza vaccines. Respondent's ages were categorized into four groups (18-34; 35-49; 50-64; 65+ years).
Chi-square or Fisher's exact tests were used as appropriate while non-parametric trend tests were used to assess by ordered age groups. All tests were two-tailed with a p-value of < 0.05 considered statistically significant using Stata SE version 11 (StataCorp, 2009). Multiple logistic regression analyses were used to determine the association between invitation mode and response rate. Regression diagnostics were performed to identify influential observations and outliers using Pearson residuals, deviance residuals and Pregibon leverage graphs. Since effect modification by sex was detected, sex-specific regression analyses were carried out adjusted for age.

Study sample
As shown in Fig. 1, of 1364 people in the registry, 1334 participants aged 22-94 years had both an email and postal address and were randomized to 8 groups by invitation mode, survey length (long vs. short), and response type (open-vs. close-ended). There was no significant difference by age or sex in those who were and were not eligible to be randomized. After excluding 6 ineligible individuals, there were 649 and 679 participants eligible for contact by email and post, respectively. The eligible sample had a median age of 48 years (95% CI: 47, 49) and 80% were female, which is reflective of the healthcare personnel in Canada [29].
There were no differences by median age (48 vs. 48 years; p = 0.89) or sex (78.7% vs. 81.6% female; p = 0.18) between the two invitation mode groups (email and post, respectively).
There were 559 (41.9%) potential participants with two email addresses available in the registry. After invitations were sent, 67 (10.3%) initial emails and 60 (8.8%) letters were returned as improperly addressed (p = 0.35). Twenty-three (76.7%) emailed invitations were successfully re-sent to an alternate email address. There was ultimately no statistically significant difference in the percentage of returned invitations by mode (44 (6.8%) email vs. 60 (8.8%) post) (p = 0.17).

Response rates
As shown in Table 1, the overall response rate was higher for those sent the invitation by email (34.8%) than by post (25.8%; p < 0.001). Older participants were more likely to respond to the survey than younger ones, overall (median age 53 vs. 46 years, p < 0.001) and by invitation mode (p trend < 0.001). Participants responding to emailed invitations were younger (median of age: 51 years) than those who responded to postal invitations (55 years; p = 0.005). Although males and females were equally likely to respond overall, females were more likely to respond to the email (36.0%) than the posted letter (24.4%; p < 0.001). This was driven by females younger than 50 years of age who were significantly more likely to respond to the email versus the postal invitation (32.1% vs. 15.9%, p < 0.001) while there was no difference for older women. Males 50 years and older were more likely to respond to the posted invitation (51.7%) than females in that age group (34.8%; p = 0.02) while there was no difference in response rates to emailed invitations (41.1 and 35.6%, respectively).
As shown in Table 2, there were some statistically significant differences in response rate by the length and/or format (open-ended vs. close-ended) of the questionnaire, with those randomized to the longer and open-ended questionnaire more likely to respond if they were invited by email. However, since the potential participants were not aware of the questionnaire they would receive, the differences are likely due to statistical chance. There was no statistical association between the length and type of questionnaire and the likelihood of providing an incomplete survey. Of the 6 (1.5% of 401) respondents who submitted partially completed surveys, 2 received long, open-ended, 3 received long close-ended, and 1 received short open-ended versions (p = 0.53).
For women, the odds of responding to the survey was higher (OR 1.77) by email than post (p < 0.001). On the other hand, there was no difference in the odds of responding by email or post for males while holding the effects of age constant (p = 0.90). While holding the method of response (email verus post) constant, both older men and older women were more likely to respond than younger ones (see Table 3).
As shown in Fig. 2, the number of responses from the emailed invitation reached a peak on the day the first invitation was sent with a second peak on the day the reminder emails were sent. In comparison, the peak for the posted invitations was 2 days after the first letters were sent with a second peak 2 days after the reminder letters were sent. The response rate after the initial invitation (up to date of second invitation) was 21.3 and 14.4% for those who received email and postal invitations, respectively (p < 0.001). Older participants were more likely to respond after initial invitations than younger groups (p < 0.001) but there were no difference by sex (p = 0.27).

Adverse events following immunization
Of the 401 people who completed a questionnaire, 96.5% had been vaccinated against influenza at least once during adulthood and 76.0% were vaccinated every year since their initial vaccination. Of those who had not been vaccinated during adulthood, the main reasons stated were: the effect- As shown in Table 4, there were no differences (at P < 0.025, to adjust for multiple testing) in the percentage of respondents reporting recurrent systemic adverse events by invitation mode, question format (open vs. closed), length of the survey, age, sex, or adverse event recall of non-influenza vaccines. Of the participants who recalled their experience with non-influenza vaccines, 67.8% (208/ 307) reported no adverse events after any vaccines while 8.1% (25/307) reported an adverse events (local or systemic) after both influenza and non-influenza vaccinations. There was also no correlation between known allergies or sensitivities and the occurrence of recurrent systemic adverse events.
A higher percentage of respondents who reported only a local adverse event (27/29 or 93.1%) than those who reported a systemic adverse event (46/66 or 69.7%; p = 0.04) planned to be vaccinated again next season. Of the 66 people who reported systemic adverse events, an equal percentage of those with an event after only their most recent vaccination (29/45 or 64.4%) and those with recurring events (17/21 or 81.0%; p = 0.38) planned to be vaccinated again next season. Respondents whose adverse event affected their daily activities or required a medical consultation were significantly less likely to plan to be vaccinated next year (12/25 or 48.0%) than those with adverse events that did not (61/70 or 87.1%; p < 0.001). Participants who reported no side effects were not asked about their intentions to receive the influenza vaccination next season; a

Discussion
The overall response rate to our web-based survey was low (30%), but higher for subjects 50 years and older compared with younger adults. Response rates were also higher for those sent an email invitation and email reminder with a clickable link to the web-based survey than for those sent a postal invitation and reminder which contained the survey site address which then had to be entered into the internet browser's address bar. Our response rate was consistent with responses to other surveys conducted with health professionals [1,6,11]. McPeake, Bateson and O'Neil hypothesized that health professionals may be overburdened by surveys and may complete only surveys they feel are absolutely necessary or are of particular interest to them [30]. The level of interest or salience may explain why subjects 50 years and older were more likely to respond to the survey than younger ones. In Canada, people 50 years of age and older are more likely to be vaccinated against influenza than younger ones [21]. Although one web-based survey about knowledge of arts education reported that participants 50 years and older were significantly more likely than their younger counterparts to respond [31], many others reported no difference in response rates by age [10,12,14]. However, the registry was small (only 1328 eligible) and results should be interpreted with caution.
Contrary to the current evidence that emailed invitations produce a lower or non-superior response rate to posted invitations [6,8,15,16], findings from our study suggest that emails may yield higher response rates in some populations. This finding may reflect our registry population who, for the most part, were healthcare workers who had participated in studies requiring the use of the internet. Unfortunately, with the increased use of email for advertising, phishing, and malware, it is possible that response rates will decrease as has been the experience with telephone surveys [32]. It is likely that our response rates would have been better if we had used more reminders, particularly if those reminders were done by post [33]. Newberry and Israel reported that response rates for adults in a state-run client registry jumped from < 30% after the third contact to over 55% after the fifth [34]. Although the first three modes of contact varied (post only, post followed by 2 emails, or three emails), the final two contacts in each arm were mailed letters with questionnaires.
Non-delivery rates for both contact modes were an issue in this study; one that we were unable to accurately measure. Although there was no difference in the percentage of known non-delivery by mode, we have no knowledge of how many invitations were filtered prior to delivery due to spam-filters [35] or were not returned by the current resident at the mailing address. We were also unable to ascertain how many individuals did not respond due to disinterest [15] or lack of engagement and/or salience.
Not unexpectedly, the rate of recurrent systemic adverse events following influenza immunization (5.7%) was higher  [36]. We found no association between recurrent systemic adverse events and respondent age, sex or reactions to non-influenza vaccines or allergens. However, the number of respondents was low and this result should be interpreted with caution. The most common recurrent adverse events were muscle aches and fatigue. This finding corresponds with results from an influenza vaccine safety surveillance study in which Canadian healthcare workers reported fatigue/myalgia as the most common systemic symptom interfering with daily activities or requiring healthcare consultation [37].
The fear of adverse events is an often-stated reason for rejecting the annual influenza vaccine [38,39]. In our study, respondents who experienced a systemic adverse event were significantly less likely to plan to be vaccinated against influenza in the future than those who had only a local reaction. This is similar to Turkish medical residents of whom 20/29 (69%) experiencing side effects were not planning to be vaccinated the following season compared with 13% of those who did not [40]. Contrary to this, a 1980s study found no difference in the percentage of outpatients who were actually revaccinated the following year based on whether they had a local or systemic reactions or neither [41]. Of note, participants who experienced recurring systemic adverse events were equally likely to plan future influenza vaccinations as people who experienced a systemic adverse event in the most recent season only. This suggests that people may overcome their initial decision regarding revaccination once the chance of an adverse event is weighed against not being protected against influenza the following season. Strengths of this study include random allocation of registrants and equivalent numbers and types of contacts by invitation mode. One limitation is that the registry was seldom updated which increased the probability for non-delivery. Although this affected the overall response rate, it would not have had an impact on the response by invitation mode. The registry was limited to volunteers in previous influenza studies, which may have increased the overall response rate due to interest in the topic. The high rate of previous influenza vaccination is a product of the volunteers within the registry. We were unable to assess response rates by factors beyond age and sex as the registry had no information on occupation or other factors.

Conclusions
In our population, emailed invitations yielded a higher response rate and shorter response times than posted invitations. Older respondents were more likely to respond to the survey in general than their younger counterparts while participants responding to emailed invitations were younger. The most common recurrent systemic adverse events were myalgia and fatigue.