Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches

Modes of contact for community-based epidemiological research traditionally include mail-outs to residential addresses, and telephone contact based on public directories or random-digit-dialing (RDD) to landline telephone numbers. In addition to deciding on contact mode(s), a choice needs to be made between personalised (directed to a named person) or generic (to the Householder/Occupant) approaches. However, the population coverage of personalised data sources based on public telephone directory listings has declined in recent years due to the rapid transition to wireless (mobile/cell) phones [1, 2], and higher proportions of unlisted landline numbers [3]. The rising prevalence of households with only wireless telephone connections has also impacted adversely on the coverage of generic RDD methodology for landline telephone numbers [2].

In addition, researchers are encountering greater problems in making contact with potential respondents and rising refusal rates among those contacted [4]. Declining response rates have been reported for long established population health surveys including the U.S. Behavioral Risk Factor Surveillance Study (BRFSS), as well as public opinion and consumer surveys [4]. A review of the RDD monthly Survey of Consumer Attitudes by the University of Michigan [5] showed non-contacts accounted for less than 5% of non-response between 1979 and 1985, but increased to more than 15% by 2003. The rate of refusals among contacted households also increased from 19% in 1979 to 27% in 2003, and the overall response rate fell from 72% to 48%. The dual challenges of shrinking population coverage of traditional sampling frames and falling response rates underline the need to explore new sampling frames and alternative modes of data collection for community-based surveys and to document their utility.

The total target number for completed surveys via all recruitment strategies combined was 600, with about 200 responses to be derived from each of the three modes of contact (postal, internet and telephone). The target area included approximately 128,000 residential postal delivery addresses in 27 contiguous suburbs in the southeast region of Melbourne. The selected area contained a high percentage of houses with gardens relative to the inner city, and therefore a higher likelihood of greywater use given the prevailing restrictions on outdoor tap water use imposed by prolonged drought conditions. The target area included suburbs spanning a range of socioeconomic levels defined by the Australian Bureau of Statistics (ABS) using Socio-Economic Indexes for Areas (SEIFA). The SEIFA Postal Area Index of Relative Socio-Economic Advantage and Disadvantage 2006 was used to categorise suburbs. This is a continuum of advantage (high values) to disadvantage (low values) and is derived from several Census variables, including factors such as tertiary education and household income [29].

The recruitment strategy ensured that each household received only one approach and this was accomplished by reserving two suburbs for the generic internet postcard mail out and one suburb for the generic postal survey mail out. For the remaining suburbs, separate non-overlapping data files were obtained for the personalised approaches (one data extract comprising 15,500 randomly selected names and addresses for the postal survey and another data extract containing 2,060 randomly selected names, addresses and telephone numbers for the telephone survey).

Printing, packaging and postal lodgement of survey material was carried out by a mailing company. Personalised postal surveys were sent to 5,500 householders, generic postal surveys to 4,000 householders, personalised internet postcards to 10,000 householders, generic internet postcards to 7,000 householders, and personalised letters informing householders they would be called for a telephone interview were sent to 2,060 households. Households selected for the telephone survey had a two week period to contact the Study Centre and decline participation before interviews began (a condition of MUHREC approval). Numbers listed in the Australian Do Not Call registry (a national opt-out list for telemarketing calls) were deleted prior to commencing telephone interviews. The generic postal surveys and internet postcards were delivered by the postal service to all households in the targeted suburb(s) along with conventional mail, rather than being bundled with “junk mail” deliveries. Householders had four weeks to respond to the postal and internet surveys, and no reminder mail outs or telephone calls were made. Telephone interviewing was contracted to a telemarketing company, and calls were made between 4 pm and 8 pm on Monday to Thursday of one week. Data entry for postal surveys was carried out in-house, and respondents entered their own data for the internet survey.

The survey enquired about collection and reuse of greywater (defined as water from the laundry, bathroom or kitchen) at any time in the last five years. For those who had used greywater, details were requested about specific greywater use practices. To enable demographic comparisons to be made, information was collected from all participants about postcode of residence, household type (family, couple, single, group), home ownership (own/purchasing, renting) and dwelling type (detached house, semi-detached house, apartment). The survey required a maximum of 10 min to complete, and the majority of questions required checkbox answers with some optional short text fields. Respondents to all survey modes (whether greywater users or non-users) could choose to enter a lottery draw for 20 retail gift vouchers to the value of $70 each.

A web database was established for data entry from all survey modes. The database programming included branching, so that questions deemed irrelevant based on a previous response were not presented to participants in the internet or telephone modes, minimising completion time. Programming also ensured that respondents could not skip relevant questions, thereby enhancing survey completeness and validity. Postal surveys included instructions for respondents to move to the next relevant section depending on their prior answers.

Response rates for the different survey administration modes varied, as did greywater use rates among respondents to different modes, perhaps reflecting differing levels of motivation and interest in the topic. Similar to other community studies using postal invitations and comparing response rates for different survey modes [8, 10], we found that response rates to the internet survey were significantly lower than other modes even after correction for levels of internet access. The crude response rates in our survey were significantly higher for personalised versus generic approaches with both postal and internet modes, but for the postal surveys there was no significant effect of personalisation when comparing areas of the same socioeconomic level, suggesting little if any benefit to overall response in this instance. However personalisation allows preliminary contact by letter prior to telephone interviews, and this has been shown to significantly increase response rates [31].

Our overall response rates were lower than has been documented in some other studies, but still enabled sufficient recruitment numbers for our purposes. This may in part reflect the specialised research topic, as well use of a single contact attempt with no follow up. Additionally, changing weather conditions (heavy rains before the survey period) and a relaxation in tap water restrictions in Melbourne probably reduced levels of interest in the survey topic. The response time permitted for postal and internet surveys was short (four weeks), and although we recognise that follow up of non-responders significantly enhances response rates to both postal [32] and internet surveys [6], we were unable to follow up non-respondents because of study time constraints. Similarly, the short time frame available for telephone interviews (four days) limited the ability to call back households who asked to defer the survey.

Our survey was not intended to assess the prevalence of greywater use in Melbourne but rather to collect detailed information on greywater use practices from households which had used this water source during a recent drought. Representative data on household greywater use in Australia are limited to the prevalence in state capitals and other areas, and garden watering as a major use [33]. For this reason it is difficult to ascertain whether our respondents are representative of all urban greywater users. The household characteristics of survey respondents were consistent with expectations, with greywater use being strongly associated with the presence of a garden and with home ownership. Our respondents spanned a range of socioeconomic levels, with no clear trend in response rates relative to socioeconomic status (data not shown). However, response required a good command of spoken or written English, and the effect of this restriction can be estimated from the telephone survey where 10.7% of contacted households were excluded because of language difficulties.

Analysis of costs per valid survey returned showed that the postal survey mode was the most cost-effective, with only minor differences between personalised and generic approaches, but internet and telephone modes were more costly. Our internet-based survey database was constructed by a professional programmer, and cost savings could be achieved by using a “do-it-yourself” web survey tool, provided that the survey structure is not too complex. However, costs need to be balanced against generalisability, and if the lower motivation required for telephone surveys means they are associated with increased response rates, they may also provide a more representative range of respondents than other survey modes.

The impact of changes in household telecommunications and other factors on the coverage and availability of different data sources mean that researchers must now consider use of more than one sampling frame and/or response mode in attempts to survey or recruit representative population samples [3, 34]. Current evidence suggests that telephone and postal modes of survey administration remain the most effective for community surveys, but this may change as levels of household internet access and familiarity increase. There is also a need to develop more cost-effective ways of inviting community participation in internet surveys. Paid advertising on social media websites or pop-up adverts triggered by particular internet search terms have been successfully used for recruitment of specific target groups including young women [35] or people seeking quit smoking information [36], but it is not yet clear how such methods might be adapted to broader community recruitment efforts.

Regardless of the sampling frame or mode of contact, low response rates in comparison to historical levels seem likely to continue in the future. While high response rates have often been viewed as a safeguard against non-response bias, low response rates of themselves do not necessarily indicate a high degree of bias in study results. A detailed analysis of the relationship between non-response rates and non-response bias in 235 separate estimates from 30 published studies found that the response rate alone was not a very good indicator of the magnitude of non-response bias [37]. While very high response rates reduced the probability that non-response bias was present, this did not mean that the degree of bias was low, in cases when it did occur. In addition, there was often considerable variation in the degree of bias among different variables within any individual study.

Our findings suggest that postal surveys remain the most economic option for community surveys, with similar costs for personalised and generic approaches. Telephone surveys may produce higher response rates but also entail higher costs, and the internet is not currently an effective mode for community-based surveys.

In Australia, as in many other jurisdictions, it is becoming increasingly difficult to obtain data sources which provide a representative population sampling frame for inviting community members to participate in epidemiological surveys. The optimal recruitment strategy for a specific study requires consideration of the relative importance of response rates, response biases, cost-effectiveness and generalisability. Even when a representative sampling frame is used, some degree of response bias is inevitable depending on demographic and socioeconomic factors, literacy and language constraints, the personal level of interest/relevance of the topic under study, the demands of the survey (length/complexity/sensitive topics), and the mode(s) of contact employed. When publishing study outcomes it is important that researchers specify the limitations of the data sources and sampling strategies used for recruitment, and document the potential biases in the target and respondent populations and how this may affect the data collected.

MM is currently located at the Epidemiology Unit, Ministry of Health, Colombo, Sri Lanka.