- Research article
- Open Access
An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses
BMC Medical Research Methodology volume 17, Article number: 73 (2017)
Previous research has found that a “web-push” approach to data collection, which involves contacting people by mail to request an Internet survey response while withholding a paper response option until later in the contact process, consistently achieves lower response rates than a “paper-only” approach, whereby all respondents are contacted and requested to respond by mail.
An experiment was designed, as part of the Millennium Cohort Family Study, to compare response rates, sample representativeness, and cost between a web-push and a paper-only approach; each approach comprised 3 stages of mail contacts. The invited sample (n = 4,935) consisted of spouses married to U.S. Service members, who had been serving in the military between 2 and 5 years as of October, 2011.
The web-push methodology produced a significantly higher response rate, 32.8% compared to 27.8%. Each of the 3 stages of postal contact significantly contributed to response for both treatments with 87.1% of the web-push responses received over the Internet. The per-respondent cost of the paper-only treatment was almost 40% higher than the web-push treatment group. Analyses revealed no meaningfully significant differences between treatment groups in representation.
These results provide evidence that a web-push methodology is more effective and less expensive than a paper-only approach among young military spouses, perhaps due to their heavy reliance on the internet, and we suggest that this approach may be more effective with the general population as they become more uniformly internet savvy.
When conducting sample surveys it is often seen as advantageous to collect survey responses over the web. There are many advantages of online surveys including less time and cost associated with data collection and processing, ability to implement complex skip patterns, and other ‘smart’ features that reduce erroneous responses. However, previous research in general public surveys shows that response rates are lower and samples tend to be less representative for web-only surveys compared with paper surveys .
One major issue for researchers conducting online surveys of sampled populations is that email addresses are often unavailable, so survey requests must be limited to postal mail. Considerable research has investigated the effectiveness of using various methods to encourage survey participation for studies that have multiple response modes, such as web and paper, when email addresses are not available. One method, referred to as web-push, requests that individuals respond to an online survey and only offers the option of a paper questionnaire late in the survey cycle. Another method, referred to as paper-only, requests that individuals complete the survey via paper only and includes a paper questionnaire in two to three postal contacts.
Previous survey research conducted on ten different household samples indicated that paper-only strategies produce higher response rates than web-push strategies . Specifically, when individuals were required to go from a postal request to providing an Internet response, response rates were about 10 percentage points lower on average than when a paper response to an enclosed questionnaire was requested. Further, these studies demonstrated that the initial stages of the web-push approach, when only web surveys were completed, resulted in some response bias. Therefore, it was necessary to provide a paper questionnaire in a later contact in order to increase sample representativeness. The paper option stimulated responses from individuals who were older, less educated, and who had lower incomes. An exception was found for a national survey of college graduates, where the web-push approach obtained about the same response rate as a paper-push procedure (mail followed by later web request) without discernible differences in the demographics of respondent .
Research also has shown that when invited individuals are contacted by postal mail only, offering a simultaneous choice of mail and web response does not increase response rates and may lower them [3–5]. In laboratory studies, Schwartz  concluded that offering respondents a choice may simply make deciding what to do more difficult. Furthermore, when a choice of survey response modes is offered through a postal mail communication, along with the ability to respond via paper or web immediately, it leads to most of the responses (70-80%) coming in by paper rather than over the Internet .
While evidence from previous research indicates that a paper-only strategy yields higher response rates and a more representative sample with the general public than a web-push strategy, this may not hold true for military families. Heavy reliance on the Internet by military families for communicating and carrying out military tasks, suggests that this population could be more like a college educated population with regard to Internet experience than the previously studied general public. Therefore, the aim of this study was to compare a web-push methodology with a paper-only methodology for surveying spouses married to U.S. Service members, for whom only mail contact information was available. We were interested in determining the most efficacious way to enroll spouses in the Millennium Cohort Family Study (Family Study). To assess and compare the two methods, we examined 1) final response rates, 2) incremental increases in response after every stage of postal contact, 3) cost, and 4) sample representativeness.
This experiment was conducted using a sample of spouses eligible to join the Family Study via their marriages to service members who enrolled during the 2011-2013 survey cycle of the Millennium Cohort Study. The Millennium Cohort Study began in 2001 to examine how deployment and other military-related experiences affect the long-term physical and mental health of US Service members and veterans. Over the last decade, the Millennium Cohort Study has become the largest population-based prospective health study in US military history, providing significant contributions to the understanding of the health impacts of military service [7–11]. The Millennium Cohort Study was expanded in 2011, to include the Family Study, in order to understand the interrelated health and well-being effects of military service on families – including the service member, spouse, and children.
When the Millennium Cohort and Family Study survey cycle was launched in 2011, married Millennium Cohort members who completed a baseline survey (2011-2013) were requested to provide contact information about their spouse, so the spouse could be invited to join the Family Study via email and postal mail. Approximately a third of those who completed the Millennium Cohort questionnaire provided their spouses email and postal contact information. In cases where contact information was not provided, but the service member did not specifically decline the request to contact his/her spouse, the study team invited spouses to participate using available postal mail addresses. This experiment consisted of a sample of these spouses for whom no email address was available (n = 4,935). They were randomly assigned to either a web-push (n = 2,472) or paper-only (n = 2,463) treatment group.
Family Study spouses self-reported sociodemographic characteristics, physical health, mental health, military experiences, and family characteristics on the 100-item baseline Family Study survey. Unified mode construction principles  were followed in constructing both the web and paper questionnaires in order to produce similar stimuli and responses to these two visual modes of surveying. Service members’ demographic and military characteristics (e.g., military pay grade, service component, deployment) were obtained from the DoD electronic personnel files managed by Defense Manpower Data Center (DMDC).
Three stages of contacts, each consisting of two closely spaced postal mailings, were sent over a 10-week period to both the web-push and paper-only treatment groups. The web-push contacts, including sample questions in mailing 3 and paper questionnaire sent in mailing 5 are shown in Fig. 1. The paper-only contacts (not shown) differed by including a paper questionnaire in mailings 1, 3 and 5 without directions for responding over the Internet. Each stage included a primary mailing, and within a week, a follow-up mailing that complemented and reinforced the message of the primary mailing. The follow-up mailings for the web-push and mail-only treatment groups were identical, except for information on the mode of responding (see Table 1).
Considerable attention was given to designing each contact for this study, so that it would be opened and mentally garner attention from the respondent. Each contact was designed in a manner that encouraged it to be processed by the recipient and to connect with mailings that preceded or followed. The theoretical basis for designing each contact drew heavily from social exchange theory as presented by Dillman, Smyth, and Christian . In addition to pre-incentives (magnet picture frame, $5 Starbucks gift card), invitees were promised a $10 gift card (Starbucks, Subway, or Shutterfly) upon completion of the survey in order to be consistent with the Millennium Cohort Study of service members.
Across the 3 stages of mailings, a deliberate attempt was made to include the same response-inducing elements into procedures for the two treatment groups, except where inherent differences in the web-push and paper-only suggested the need for differences. All of the paper-only mailings encouraged participation using the paper survey without mention of the web survey option. However, printed on back cover of the questionnaire was the study website address where one could obtain additional information regarding the study and a link to the web survey. Therefore, it was possible for individuals in the paper-only treatment group to complete the online survey.
Two additional differences between the treatment groups occurred. There was a necessary six week delay in commencing the paper-only treatment due to the time associated with printing the paper surveys. Thus, the web-push mail contacts began as scheduled on August 2 with the last being sent on October 12th, whereas the paper-only implementation began September 13th with the last contact being sent on November 15th. In addition, an error occurred with the final stage of paper-only contacts, with the postcard reminder being sent slightly ahead of the final survey. Based on reasoning described in the Discussion, it seems unlikely that these differences impacted the results of the experiment.
Response rates for the paper-only and web-push treatment groups were compared for each stage of postal contacts. Descriptive analyses, including frequencies and chi-square tests, were used to compare characteristics between the two treatment groups. To investigate whether adding a paper response option to the web-push treatment group affected sample representation, a comparison of web and paper responders in the web-push treatment group was conducted examining demographic and military variables. To determine if treatment group impacted representation, interactions between treatment group and 11 survey non-response variables were tested. Lastly, a major reason for pursuing a web-push strategy is the effort to reduce costs, so we compared the costs associated with each treatment. All data cleaning and analyses were performed using SAS software (Version 9.3, SAS Institute, Inc., Cary, NC, USA).
The web-push strategy produced a significantly higher response rate (32.8%) compared with the paper-only strategy (27.8%, p < .001, Table 2). In addition, the increase in response rate after each stage of postal contacts, as well as the cumulative response rate at each stage, were higher for the web-push strategy. Stage 1 contacts established a one percentage point lead over the paper-only response. The other two stages of paired contacts each had a substantial effect on response, together producing around 60% of the final responses. Fig. 2 shows cumulative response rates by the elapsed number of days from the first mailing for each treatment group. This figure clearly reveals the impact of each stage of contacts for the web-push group, and a similar pattern for the paper-only group, but with a delay due to mailing and survey processing time.
Comparison of demographic, military, and health characteristics
A comparison of the responders by Family Study participant demographic and military characteristics did not reveal any significant differences between treatment groups (Table 3). In addition, when examining the service member characteristics of the responding Family Study spouses (i.e., military spouse’s service record and Millennium Cohort Study survey data) by treatment group, chi-square tests revealed that none of the characteristics were statistically different (data not shown). An additional comparison of treatment groups across 14 different physical, mental, and behavioral health indicators revealed only one significant difference (data not shown); more paper-only participants screened positive (7.4%) for anxiety than web-push participants (3.7%; p-value 0.002).
Overall, 95.2% of the paper-only responders completed a paper survey, with the remaining 4.8% responding via the online survey. Of the web-push responders, 87.1% completed the survey online, with the remaining 12.9% responding by paper.
In order to determine whether the addition of a paper response option to the web-push treatment group impacted sample representation, a comparison of web and paper responders in the web-push group were compared across demographic and military variables. Of the 12 demographic and military characteristics examined, only one significant difference was detected. Spouses of Reserve/National Guard members were significantly more likely to respond by paper (18.5%), compared with spouses of active duty members (11.4%, p = .014).
To determine if treatment group impacted sample representation, we tested the interactions between treatment group with key service member demographic and military characteristics. These characteristics were ones used to assess for non-response in the entire Family Study population (See Assessing and Adjusting for Non-response in the Millennium Cohort Family Study for additional information) . Using a non-response model developed for the entire Family Study sample, we found that none of these interaction terms were significant (p < .05) in our study sample after Bonferroni correction. In other words, treatment group did not significantly modify the relationship between service member characteristics and spouse survey response.
The cost of printing, packaging, and postage for each mailing, along with the cost of incentives, and salary for data entry of paper surveys were summed for each treatment group and then divided by the number of completed questionnaires per group. The per respondent cost of the paper-only treatment was almost 40% higher than the web-push treatment group as a result of the increased cost associated with printing and mailing a lengthy questionnaire, as well as the additional cost of data entry and cleaning. Specifically, the costs were $89.50 per respondent for the paper-only strategy vs. $61.50 per respondent for the web-push treatment. Examination of the cost details suggested that the major contributor to cost for the paper-only strategy was paper questionnaire processing and data entry, which was only a small part of the cost associated with web-push respondents.
This experiment showed that the web-push strategy for obtaining responses to an in-depth health survey was more effective for military spouses that the paper-only approach. It produced a 5 percentage point higher final response rate, showed no meaningfully significant differences in sample representation, and was more cost effective than the paper-only strategy. Moreover, the web-push strategy led most respondents to complete an online survey (87%). For complex surveys with extensive branching (i.e., skip patterns) such as the Family Study, web surveys are more desirable from the stand-point of achieving better compliance with branching instructions than could be obtained by paper questionnaires.
Our findings contrast with previous studies of US general populations that have shown paper-only outperforming web-push. The spouses of military members may be more likely than the general public to respond over the web because they are on average younger and have greater familiarity with web-based technologies than the general public [14–16]. In addition, communication by email is part of military culture and is facilitated and encouraged by the military to promote organizational efficiency and family connections while on deployment. Military families also use social media with remarkably high frequency, even more so than civilians, with reported rates of Facebook use as high as 96% . Thus, our findings may be helpful in anticipating future uses of these survey methods as the general society becomes more familiar with web-based technologies.
It is also possible that the web-push approach was more effective than the paper-only strategy for our study based on respondent perceptions of the length of the questionnaire. The paper version of the survey appeared quite long (a 36-page booklet). This was partly the result of having to include all possible questions, many of which individual respondents would be directed to skip. The higher response for the web-push group may have occurred because the questionnaire length was not easily seen by the web-push recipients (until the Stage 3 contact), whereas the paper-only recipients saw the length of survey at the first mailing. In a social exchange sense, the perceived “cost” of responding may have seemed higher for the paper-only recipients, while for the web-push recipients, the length of the web survey was not easily apparent. However, this possibility may be somewhat diminished by the inclusion of the 8-page sample questionnaire booklet mailed during Stage 2 that would have suggested to web-push recipients that the questionnaire was longer than 8 pages.
Due to the design of the study and sampling strategy, we had the unique capability to examine numerous demographic, military, and health characteristics between the treatment groups. We found no significant differences between treatment groups with regard to the demographic and military characteristics of the Family participant or the Service member spouse. Of the 14 health factors we examined, only one was significantly different by treatment group. Paper-only respondents were significantly more likely to screen positive for anxiety. However, given the lack of a theoretical explanation for this finding and the fact that so many variables were examined, it is quite plausible that this single difference occurred by chance. Using data from the entire Family Study cohort (n = 9,872), an additional analysis found no significant difference of screening positive for anxiety by mode (6.2% for paper vs 5.6% for web, p = 0.39); providing further support that overall representation was not meaningfully different by treatment group. Overall, based on the number and variety of variables investigated, these findings suggest that there were no meaningful differences between the treatment groups with regard to representation.
Our study also confirms the potential importance of utilizing paper surveys to increase web responses and sample representativeness. During the final stage of postal contact to the web-push group (introduction of the paper survey) there was an 8 percentage point increase in the response rate, produced by nearly equal numbers of web and paper respondents. Offering a paper questionnaire in a later stage of a web-push approach has been found to increase the number of web responses in addition to producing paper questionnaire returns [17, 18]. Based on our analyses, adding the paper questionnaire during the web-push approach seemed to increase representativeness of only one factor/group (e.g. service component). Reserve/National Guard members were more likely to respond by paper survey than web, indicating that adding the paper response option may have helped with the overall sample representation of the web-push approach among this group. Although this may have been a result of chance, it seems plausible that the Reserve/National Guard respondents are settled in civilian communities and less attached to the military culture that fosters email use as a means of rapid communication. Consequently, they and their families may behave more like the general public than the active duty military members comprising the majority of participants in this study. Therefore, providing a paper option as part of a web-push strategy may be important for increasing response as well as sample representation.
While it was not possible to separate the effects of each mailing, it was feasible to separate with reasonable precision the effects of the different stages of postal contact. One of the challenges of comparing response rates for web-push and paper-only methodologies comes from differences in processing time. Whereas web responses can be seen almost immediately after a postal request is received, completed paper surveys need time to be mailed in before responses can be counted. However, incremental response associated with each stage of contact was apparent for both approaches. That is, each mailing stage produced a considerable improvement in response rates, confirming the importance of each of the three contact stages for both approaches.
The findings from this experiment are subject to certain limitations. First, the lack of availability of the paper questionnaire made it necessary to delay implementation of the paper-only treatment for about six weeks after the web-push treatment had started (August 2 vs. September 20). However, previous research does not suggest that this particular time-of-year shift would confound results, and as shown in Table 2, most of the difference in response rates came in the first two stages of implementation. Also, a mix-up in mailing dates resulted in all of the Stage 3 mailings to the paper-only treatment group going out in reverse order. Although, we cannot ascertain whether this affected overall response rates for this group, there was already a significant difference in response rates after the Stage 2 contacts (p = .03) which remained significantly higher for the web-push group until the end of the experiment (p < .001). We do not believe that this shift in the final two contacts would have been powerful enough to produce a response advantage for the paper-only strategy for three reasons: 1) The mailings were scheduled only one week apart as part of a paired set of contacts sent by different delivery mechanisms (FedEx and Postal mail); 2) The mailings were sent in weeks 9 and 10, so the overall impact on response rates was expected to be diminished compared to the first 2 stages of contact; and 3) The paper-only group was provided a 3rd copy of the survey, in contrast to the web-push group that was provided with a new way to respond to the survey (paper), which in previous research has produced a significant improvement in response. An additional limitation of this experiment is that it was conducted among a sample of military spouses in which their service member partner had 2-5 years of military service. Therefore, these findings generalize only to junior military spouses.
Findings from our study indicate that when only postal addresses are available, a web-push methodology is more effective than a paper-only strategy (for both cost and response rate reasons) among military spouses. This suggests that as Internet use continues to increase in importance for the general population that web-push data collection may be increasingly desirable for that population as well. However, it remains to be seen whether our web-push response results will generalize more broadly to future general populations.
The pressures on surveyors for shifting to Internet data collection methods is growing rapidly and increasing numbers of government surveys, including the Census Bureau’s American Community Survey and the 2020 Decennial Census, are developing web-push data collection procedures . This study of military spouses shows contrary to published research on general public populations, web-push methodologies may outperform paper-only methodologies without negatively impacting representation and data quality, while also lowering survey costs significantly.
Defense Manpower Data Center
Department of Defense
- Family Study:
Millennium Cohort Family Study
Dillman DA, Smyth JD, Christian LM. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 4th ed. New Jersey: John Wiley and Sons; 2014.
Finamore J, Dillman DA. How mode sequence affects response by internet, mail, and telephone in the National Survey of College Graduates. In: Paper presented at the European Survey Research Association Biennial Conference. Ljubljana, Slovenia. 2013.
Smyth JD, Dillman DA, Christian LM, O’Neill A. Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century. Am Behav Sci. 2010;53:1423–48.
Gentry R. Offering respondents a choice of survey mode. In: Paper presented at CMOR Respondent Cooperation Workshop: March 2008; Las Vegas,NV. 2008.
Medway R, Fulton J. When more gets you less: A meta-analysis of the effect of concurrent web options on mail survey response rates. Public Opin Q. 2012;76(4):733–46.
Schwarz N, Donsbach W, Traugott MW. The psychology of survey response. In: The Sage handbook of public opinion research. Newbury Park, CA: Sage; 2008. p. 374–87.
Jacobson IG, Ryan MA, Hooper TI, et al. Alcohol use and alcohol-related problems before and after military combat deployment. J Am Med Assoc. 2008;300(6):663–75.
Ryan MA, Smith TC, Smith B, Amoroso P, Boyko EJ, Gray GC, Gackstetter GD, Riddle JR, Wells TS, Gumbs G, et al. Millennium Cohort: enrollment begins a 21-year contribution to understanding the impact of military service. J Clin Epidemiol. 2007;60(2):181–91.
Smith TC, Jacobson IG, Hooper TI, LeardMann CA, Boyko EJ, Smith B, Gackstetter GD, Wells TS, Amoroso PJ, Gray GC, et al. Health impact of US military service in a large population-based military cohort: findings of the Millennium Cohort Study 2001–2008. BMC Public Health. 2011;11:69.
Smith TC, Ryan MA, Wingard DL, Slymen DJ, Sallis JF, Kritz-Silverstein D. New onset and persistent symptoms of posttraumatic stress disorder self-reported after deployment and combat exposures: prospective population based US military cohort study. Br Med J. 2008;336:366–71.
Wang L, Seelig A, MacDermid S, McMaster H, Alcaraz JE, Crum-Cianflone NF: Associations of military divorce with mental, behavioral, and physical health outcomes. BMC Psychiatry. 2015;15(128). doi:10.1186/s12888-015-0517-7.
Dillman DA, Smyth JD, Christian LM: Internet, mail and mixed-mode surveys; The tailored design method, 3rd edition. New Jersey: Wiley; 2008.
Corry NH, Williams CS, Battaglia M, McMaster HS, Stander VA. Assessing and adjusting for non-response in the Millennium Cohort Family Study. BMC Medical Research Methodology. 2017;17:16. doi:10.1186/s12874-017-0294-8.
DoD demographic profile of the military community, Office of the Deputy Under Secretary of Defense (Military Community and Family Policy), produced by ICF International. 2012. Available at http://download.militaryonesource.mil/12038/MOS/Reports/2012_Demographics_Report.pdf. Accessed 10 Dec 2016.
Pew Research Center. Summer tracking survey August 7 - September 6. In: Pew Internet and American Life Project. 2012.
Greentree V, Bradbard D, Dagher L, Levingston K, Elquist Lore C, Taylor JA. 2013 Military Families Lifestyle Report. 2013. Blue Star Families.
Millar MM, Dillman DA. Improving Response to Web and Mixed Mode Surveys. Public Opin Q. 2011;75(2):249–69.
Millar MM: Determining whether research is intediscipinarly: An analysis of new indicators (Technical Report No. 13-049). Pullman Washington State University; 2013.
National research council: Reducing response burden in the American community survey: Proceedings of a workshop. The National Academies Press. 2016.
The authors express gratitude to the other contributing members of the Millennium Cohort Family Study Team from the Deployment Health Research Department, Naval Health Research Center, including Valerie Stander, PhD; Jackie Pflieger, PhD; Carlos Carballo, MPH; Teresa Powell, MS; Kelly Woodall, MPH, Evelyn Sun, MPH; Lauren Bauer, MPH; and William Lee, as well as the contributions of team members from Abt Associates, including Nida Corry, PhD and Christianna Williams, PhD. The authors gratefully acknowledge the members of the Millennium Cohort Family Study Team from the Center for Child and Family Health including John Fairbank, PhD; Robert Murphy, PhD; Ernestine Briggs-King, PhD; Ellen Gerrity, PhD; and Robert Lee, MS. In addition, the authors want to express their gratitude to the Family Study participants, without whom this study would not be possible.
This work was funded by the US Defense Health Program and the Military Operational Medicine Research Program. This work was supported by the Bureau of Medicine and Surgery under work unit no. 60002. The funders were not involved in the design of the study, data collection, analysis, interpretation of data, or in writing the manuscript. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of the Army, Department of the Air Force, Department of Veterans Affairs, Department of Defense, or the U.S. Government. U.S. Government Work (17 USC §105). This work was prepared as part of our official duties under the contract OMNIBUS W911QY-11-d-0053-0010. Title 17, U.S.C. §105 provides the “Copyright protection under this title is not available for any work of the United States Government.” Not copyrighted in the United States.
Availability of data and materials
The datasets analyzed during the current study are not publicly available due to institutional regulations protecting service member survey responses but are available from the corresponding author on reasonable request (may require data use agreements to be developed).
HM conceived of the study, collaborated on the design and execution of the study, provided data interpretation, and drafted the manuscript. CL performed the statistical analyses, provided data interpretation, and was integral in revising the manuscript. SS was critical to the execution of the study and data interpretation. DD was instrumental in the design and execution of the study, provided subject matter expertise, statistical guidance, and data interpretation, and helped in drafting the manuscript. All authors read and approved the final manuscript. HM, CL, and SS are contractors of the U.S. Government.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Human subjects participated in this study after giving their free and informed written consent. This research has been conducted in compliance with all applicable Federal Regulations governing the Protection of Human Subjects in Research. This study was approved by the Institutional Review Board of the Naval Health Research Center, San Diego, CA (Protocol NHRC.2000.0007).
About this article
Cite this article
McMaster, H.S., LeardMann, C.A., Speigle, S. et al. An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Med Res Methodol 17, 73 (2017). https://doi.org/10.1186/s12874-017-0337-1
- Study methodology