- Research article
- Open Access
- Open Peer Review
Does sampling using random digit dialling really cost more than sampling from telephone directories: Debunking the myths
BMC Medical Research Methodology volume 6, Article number: 6 (2006)
Computer assisted telephone interviewing (CATI) is widely used for health surveys. The advantages of CATI over face-to-face interviewing are timeliness and cost reduction to achieve the same sample size and geographical coverage. Two major CATI sampling procedures are used: sampling directly from the electronic white pages (EWP) telephone directory and list assisted random digit dialling (LA-RDD) sampling. EWP sampling covers telephone numbers of households listed in the printed white pages. LA-RDD sampling has a better coverage of households than EWP sampling but is considered to be more expensive due to interviewers dialling more out-of-scope numbers.
This study compared an EWP sample and a LA-RDD sample from the New South Wales Population Health Survey in 2003 on demographic profiles, health estimates, coefficients of variation in weights, design effects on estimates, and cost effectiveness, on the basis of achieving the same level of precision of estimates.
The LA-RDD sample better represented the population than the EWP sample, with a coefficient of variation of weights of 1.03 for LA-RDD compared with 1.21 for EWP, and average design effects of 2.00 for LA-RDD compared with 2.38 for EWP. Also, a LA-RDD sample can save up to 14.2% in cost compared to an EWP sample to achieve the same precision for health estimates.
A LA-RDD sample better represents the population, which potentially leads to reduced bias in health estimates, and rather than costing more than EWP actually costs less.
Computer assisted telephone interviewing (CATI) is widely used for health surveillance surveys. The main advantages of CATI over the traditional face-to-face interviewing are timeliness and substantial cost reduction to achieve the same sample size and geographical coverage via residential telephones. In Australia, there is little bias in population-based health estimates due to a high residential telephone coverage of 97.5%,  combined with post stratification weighting to the population.
Even though full coverage of household telephones can be achieved by simple random digit dialling (RDD), it is not efficient due to inclusion of too many numbers that are not currently household telephone numbers (out-of-scope). The Waksberg Method improves the efficiency of RDD by using a two-stage sampling procedure, [2, 3] and this method is widely used in studies outside Australia . When relatively good quality lists of household telephones are available, and the list itself does not have satisfactory coverage of the target population, information in the list (such as prefixes) can be used to develop a sampling frame that has a better coverage of households: this is called list assisted random digital dialling (LA-RDD). LA-RDD further improves the efficiency of simple RDD by removing banks of numbers with no listed household telephone numbers.
Two major CATI sampling procedures are used in health surveys: sampling directly from the electronic white pages (EWP) telephone directory, [5–7] and LA-RDD sampling [7, 8]. The EWP covers telephone numbers of households who choose to have their telephone listed in the printed white pages telephone directory. The proportion of households who choose to not have their telephone listed in the white pages telephone directory (that is, unlisted or silent numbers) is estimated to be at least 15% in Australia, on the basis of a comparison of the estimated number of households with telephones and the number of listings in the EWP .
The LA-RDD sampling surveys have a better coverage of households than sampling directly from the EWP but are considered to be more expensive due to interviewers dialling more out-of-scope numbers. Bennett and Steel estimate that, compared with sampling directly from the EWP, using lists of active telephone numbers for each telephone exchange increases total interviewer time by about 20%, and if LA-RDD were used the increase in interviewing time could be about 12% . Wilson and Starr argue that the EWP is preferred due to low cost based on number of calls to get an interview (1.6 calls using the EWP sampling methodology compared with 6.5 calls using the Waksberg RDD sampling methodology) .
Results of studies examining the effects of bias due to sample methodology (EWP versus RDD) on health estimates for the general population have varied. For example, Bennett and Steel found that EWP sampling led to significant bias in unweighted estimates for households who had moved, single parent households, and households composed of unrelated people . Wilson and Starr found some difference, however, in demographic profiles between RDD and EWP sampling methodologies, but minimal bias was found in the studied health estimates from the EWP sample compared with a RDD sample after post-stratification weighting .
In addition to the limitations of no post stratification weighting in Bennett and Steel, and using the Waksberg RDD sampling methodology in Wilson and Starr, neither study used survey-quality and design-efficiency measures. These survey-quality and design-efficiency measures include:
• coefficients of variation in weights, where a larger coefficient of variation in weights means more extreme adjustment is needed for the sample to match the population benchmark;
• design effects, which are ratios of variance of estimates of a sampling method compared with a simple random sampling design.
In both Bennett and Steel and Wilson and Starr, the comparative costs of RDD and EWP sample surveys were based on achieving the same sample size, which is misleading when the design effects were different between the two survey designs.
The aims of this study are to use the 2003 data from the New South Wales Population Health Survey, a CATI health survey in New South Wales, Australia, to:
• examine the demographic profiles of weighted and unweighted LA-RDD and EWP samples;
• compare a series of health estimates from the LA-RDD and EWP samples;
• compare the costs of LA-RDD and EWP on the basis of achieving the same precision of estimates, rather than on the basis of achieving the same sample sizes.
A stratified sample was generated using the LA-RDD method for the New South Wales Population Health Survey in 2003. The LA-RDD sampling frame was created by using four digit prefixes that contained connected residential numbers from the EWP. All possible suffixes were then added to create a list of possible telephone numbers. This list was checked against the EWP so that banks of 10 continuous numbers with no numbers in the EWP were removed from the sampling frame. The list was further matched with an electronic business telephone directory to remove business numbers and fax numbers, to improve the efficiency of the final LA-RDD sampling frame. Each selected household telephone number was flagged as either EWP listed or unlisted (silent). So the resulting LA-RDD sampling frame includes all the EWP listed household telephone numbers and other potentially high productive telephone numbers with the four digit prefixes appearing in the EWP.
Letters of approach introducing the survey were sent to selected households with EWP listed telephones. Using a CATI facility, up to seven calls were made to establish initial contact with a household, and up to five more calls were made in order to contact a selected respondent and conduct an interview. Within a selected household, one person was selected at random from the usual residents in the selected household. Children were included in the survey, with information provided by a person aged over 16 years.
Sample and weighting
The total number of completed interviews for the 2003 survey is 15,837, among which 12,403 had EWP listed household telephones. Of these, 13,008 interviews were adults (aged 16 years or over), among which 10,386 had EWP listed household telephones. The same method of weighting was applied to the whole LA-RDD sample and the EWP sample. Weighting for samples was by probability of selection, then post-stratification weighting was applied to adjust the sample distribution to match the Australian Bureau of Statistics 2003 mid-year residential population estimates jointly by gender, five-year age groups (0–4 years to 80 years and over), and geographical regions .
After weighting, other health related demographics, such as country of birth, speaking a language other than English at home, having indigenous origin, marital status, and quartiles of socioeconomic index for areas, were compared to the New South Wales population using the Australian Bureau or Statistics Census 2001 figures.
Telephone call outcomes
Operational data for the survey were downloaded using SAWTOOTH WinCati version 4.1. The data included the following information for each attempted 'telephone' number, including connected and non-connected numbers: the number dialled; the number of attempts of dialling to that number; the starting and ending time for each dialling attempt to the number; whether or not the number is listed in the EWP; and whether the number dialled has led to a completed interview, or no answer, or a refusal, or a non-connected number, or any kind of out-of scope number (including non-connected numbers, fax machines, unusual tones, business-institution numbers, and households not eligible). Diallings to a non-connected number, or a number with no answer, or a number with unusual tone, were treated as non-costed diallings. For each attempted number, the sum of calling time over all attempts, the total number of attempted diallings, and the total number of costed diallings were calculated. The total costed diallings was zero for a non-connected number, or a number with no answer, or a number with unusual tone. Then the total calling time, total number of diallings, total number of costed diallings, and total number of completed interviews, were calculated over all the attempted numbers to obtain the whole LA-RDD sample and over all the EWP listed numbers to obtain the EWP sample respectively. Finally, the average calling time, average number of diallings, and average number of costed diallings, were calculated per LA-RDD completed interview and per EWP completed interview.
Response rates were calculated as the number of completed interviews divided by the sum of the number of completed interviews and number of refusals.
Prevalence estimates were generated using PROC SURVEYMEANS in SAS version 8.2. Pairs of estimates on 23 health-related indicators for males, females, and overall persons aged 16 years and over were calculated from a weighted EWP sample and a weighted LA-RDD sample. Differences between the point estimates were examined. Detailed information on the definition of these indicators can be found in the electronic report for the New South Wales Population Health Survey in 2003 .
Coefficient of variation and design effects
Coefficients of variation in weight were produced by PROC SURVEYMEANS in SAS version 8.2 from the EWP sample and LA-RDD sample respectively. The design effect for each indicator was calculated by dividing the variance of the prevalence estimate (the square of the standard error obtained from PROC SURVEYMEANS in SAS) by the estimated variance of the corresponding estimate from an assumed simple random sample, with the same sample size, Var srs = (1 - )/n. Coefficient of variation in weights and overall design effects for individual indicators and average design effect across these indicators were compared between the two samples.
Comparative cost data for the EWP sample and the LA-RDD sample were extracted, including number of dialling, number of costed dialling, and length of calling time. The EWP sample and LA-RDD sample costs were compared on the basis of achieving the same precision of estimates as that for an assumed simple random sample of 1,000 respondents, taking into consideration the different design effects of the EWP sample and RDD sample. Estimated costs are in Australian dollars using the average charge per telephone call of A$0.20, and the calling cost of A$20.00 an hour for a CATI interviewer, with the overall cost calculated as:
Cost = A$0.20 × number of costed dialling + A$20 × calling time in hours.
Quality of sampling frames and response rates
Among all the numbers dialled to get the 15,837 completed interviews, 48% were out of scope, which included fax machine numbers, not connected numbers, unusual tone numbers, business or institution numbers, and household numbers that were not in New South Wales. Among the EWP listed numbers dialled, 26% were out of scope numbers.
Among the 15,837 completed interviews, 3,434 (21.7%) were from households with non-EWP-listed telephone numbers. Of the 13,008 interviews on adults (aged 16 years or over), 2,622 (20.2%) were from households with non-EWP-listed telephone numbers.
The response rate for the EWP sample was 71.4% and 69.0% for the overall LA-RDD sample.
Distribution of age group and gender of unweighted samples
The proportions for males in both the unweighted LA-RDD sample (43.1%) and the EWP sample (42.7%) were under-represented compared to the overall New South Wales population (49.7%). In both males and females the LA-RDD sample was a closer fit to the population five-year age group distribution than the EWP sample (Figures 1 and Figure 2).
The coefficients of variation in weights are estimated as 0.96 and 1.13 for the LA-RDD sample and EWP sample respectively. The coefficient of variation in weights are 1.03 and 1.21 for the LA-RDD adult sample and EWP adult sample respectively. As shown, a larger coefficient of variation of weights for the EWP sample indicates that more adjustment is needed for the EWP sample to match the population benchmark than for the LA-RDD sample.
Demographics of weighted samples
Both the LA-RDD and the EWP weighted samples over-estimate people who were born in Australia, indigenous Australians, and married people, and under-estimated people who speak a language other than English at home (Table 1). For each of the demographics, however, the weighted LA-RDD sample was closer to the New South Wales population . The weighted LA-RDD sample is also slightly closer to the New South Wales population in the proportion of socioeconomic index quartiles .
Comparison on estimates for health related indicators for adults
The 23 health indicators examined included health behaviours, health status, health services, and health related social capital for adults aged 16 years and over. The weighted prevalence estimates for these health indicators from the LA-RDD sample and the EWP sample are listed in Table 2 for males, females, and persons respectively.
Overall, for the adult population, the weighted prevalence estimates from the EWP sample compared to the LA-RDD sample was higher by:
• 1.0% or more for three indicators: usual use of low or reduced fat or skim milk, trust most people, and visit neighbours;
• 0.5–0.9% for three indicators: any alcohol risk drinking, use of public water as usual source of water, and smoke-free household;
• less than 0.5% for six indicators: self-rated health, positive emergency department rating, positive hospital care rating, attended at least one community event in the previous six months, recommended daily fruit intake, and adequate physical activity.
Overall, for the adult population, the weighted prevalence estimates from the EWP sample compared to the LA-RDD sample was lower by:
• 1.0% or more for two indicators: hand washing when handling raw meat, and current daily or occasional smoking;
• 0.5–0.9% for three indicators: ever diagnosed with asthma, overweight or obesity, and food insecurity in the last 12 months;
• less than 0.5% for six indicators: high alcohol risk drinking, diabetes or high blood sugar, high or very high psychological distress, difficulties in getting health care when needing it, recommended daily vegetable intake, and current asthma.
On average, the design effect is 2.00 for the LA-RDD sample and 2.38 for the EWP sample across the 23 indicators studied (Table 3). Both the mean and median design effect ratios (design effect of EWP sample divided by design effect of LA-RDD sample) is 1.19. Therefore, on average, 19% more of the LA-RDD sample size is needed for the EWP sample to achieve the same precision for the health estimates. The largest five design effect ratios are for overnight hospital care rated as excellent, very good or good (1.35); high or very high psychological distress (1.32); current daily or occasional smoking (1.25); any high alcohol risk drinking (1.23); and trust most people (1.23). This means that, to achieve the same precision of the estimates on these indicators, 23% to 35% more of the LA-RDD sample size is needed for EWP sample.
According to data from the New South Wales Population Health Survey in 2003, 14.9 diallings, 8.3 costed diallings, and 35.4 minutes of calling time are required to get one completed interview using LA-RDD sample, while 9.9 dialling, 6.6 costed diallings and 31.5 minutes of calling time are required to get one completed interview using EWP sample. To achieve the same precision as an assumed simple random sample of 1,000 respondents (an effective sample size is 1,000), on average twice the respondents are required for a LA-RDD sample (design effect = 2.0) and over two-and-one-third times more respondents are required for an EWP sample (design effect = 2.38).
In Table 4, costs were compared on the basis of the number of respondents required for LA-RDD and EWP samples per 1,000 effective sample size. On average, 892 more costed diallings occur in the LA-RDD sample; however, 69.5 hours less calling time are needed for the LA-RDD sample, leading to the LA-RDD sample costing A$1,212 (4.3%) less than the EWP sample per 1,000 effective sample. In the example using the high psychological distress indicator, which has a design effect ratio of 1.32, the LA-RDD sample is estimated to cost $4,350 (14.2%) less than the EWP per 1,000 effective sample.
The distribution of age groups by gender for the unweighted LA-RDD sample is closer to the distributions for the population than the unweighted EWP sample in this study resulting in a smaller coefficient of variation in weights for the LA-RDD sample. After weighting, the demographic profile of the weighted LA-RDD sample is closer to the research population than the weighted EWP sample. The weighted EWP sample tends to cover the more socioeconomically advantaged respondents than the weighted LA-RDD sample. Therefore, LA-RDD sample could better retain the relationship among multiple variables, and potentially benefit both descriptive health estimates and analytical analysis in terms of reducing bias and improving precision of estimates.
Differences in estimates
Even though post stratification weighting removes most of the potential bias for estimates from the EWP sample compared with the LA-RDD sample, difference in estimates still exist between the weighted EWP sample and weighted LA-RDD sample in this study. The differences between the weighted prevalence estimates from the EWP sample compared to the LA-RDD sample were up to 1.0% or more higher for three indicators (usual use of low or reduced fat or skim milk, trust most people, and visit neighbours in the past week) and 1.0% or more lower for two indicators (hand washing when handing raw meat, and current daily or occasional smoking) in the overall adult population.
Overall, there were larger design effects for the EWP sample than the LA-RDD sample across all but one of the 23 health indicators examined with the average design effect being 2.00 for LA-RDD sample and 2.38 for EWP sample (design effect ratio of 1.19).
Based on this study, it is estimated that with a design effect ratio of 1.19 up to 1.32, the LA-RDD sample would save up to 14.2% in costs, A$4,350 less than the EWP sample per 1,000 effective sample. Previous studies in Australia have either focused on the number of diallings or calling time [9, 10]. Although the number of diallings is substantially more in LA-RDD sampling than EWP sampling, because of the increased number of out-of-scope numbers, this does not equate to increased dialling cost because many of these dials have no dialling cost (non-connected number and dialling of number without an answer). These previous studies also did not calculate design effect differences between RDD and EWP sampling or how they would affect the overall cost. The comparison of costs involved in costed calls and paid calling time for interviewers on the basis of achieving same precision level gives a clearer evaluation of cost-effectiveness in a LA-RDD survey and an EWP survey.
This study shows that LA-RDD sampling has the advantage of obtaining a better representative sample of the population, leading to population health estimates with potentially reduced bias compared with EWP sampling, and would benefit other analyses involving multiple variables. On the basis of achieving the same precision for health estimates, LA-RDD sampling surveys costs less than EWP sampling surveys per 1,000 effective sample.
random digital dialling
list assisted random digital dialling
electronic white pages
computer assisted telephone interviewing.
Australian Bureau of Statistics: Population Survey Monitor. Catalogue no 4103.0. 1996, Canberra: ABS
Lepkowski JM: Telephone sampling methods in the United States. Telephone Survey Methodology. Edited by: Groves R, Biemer P, Lyberg L, Massey J, Nicholls W, Waksberg J. 1988, New York: John Wiley and Sons, 73-98.
Waksberg J: Sampling Methods for Random Digit Dialling. J Am Stat Assoc. 1978, 73: 40-46.
Iachan R, Schulman J, Powell-Griner E, Nelson DE, Mariolis P, Stanwyck C: Pooling State Telephone Survey Health Data for National Estimates: The CDC Behavioural Risk Factor Surveillance System, 1995. Proceedings of the Seventh Conference on Health Survey Research Methods. 2001, Publication No. (PHS) 01-1013. Hyattsville, MD: Department of Health and Human Services
Taylor A, Dal Grande E, Daly A, Wilson D, D'Espaignet E, Measey M-A: Collaborative Health and Wellbeing Survey 2000. Report prepared for the Commonwealth Department of Health and Ageing. 2003, Adelaide: South Australian Department of Human Services, Centre for Population Studies in Epidemiology
Queensland Health: Chronic Diseases Surveys: Diabetes Prevalence and Management Report. 2002, Brisbane: Queensland Health
Centre for Epidemiology and Research New South Wales Department of Health: New South Wales Adult Health Survey 2003. N S W Public Health Bull. 2004, 15 (S-4):
Victorian Government Department of Human Services: Victoria Population Health Survey 2001. 2002, Melbourne: Rural and Regional Health and Aged Care Services Division, Victorian Government Department of Human Services
Bennett DJ, Steel D: An Evaluation of a Large-Scale CATI Household Survey Using Random Digit Dialling. Aust N Z J Statist. 2000, 42 (3): 255-270. 10.1111/1467-842X.00126.
Wilson D, Starr G, Taylor A, Dal Grande E: Random Digit Dialling and Electronic White Pages Samples Compared: Demographic Profiles and Health Estimates. Aust N Z J Public Health. 1999, 23: 627-33.
Steel D: New South Wales Population Health Survey: Review of the Weighting Procedure. Commissioned Report to the Centre of Epidemiology and Research. 2004, Department of Health of Australia
Centre for Epidemiology and Research New South Wales Department of Health: Electronic Report of the New South Wales Adult Health Survey. 2003
Australian Bureau of Statistics: Basic Community Profile. 2002, ABS Catalogue no. 2001.0. Sydney: Commonwealth of Australia
Australian Bureau of Statistics: Social Economic Indexes for Areas. 2002, ABS Catalogue no. 2039.0. Sydney: Commonwealth of Australia
Australian Bureau of Statistics: Basic 2001 Census Time Series Profile: 1 New South Wales, Australia. 2002, Sydney: Commonwealth of Australia
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/6/6/prepub
The authors wish to thank: Professor David Steel at the School of Mathematics and Applied Statistics, University of Wollongong, New South Wales, Australia, for his insightful editorial comments on an earlier draft of this article; and Doctor Michael Giffin, Centre for Epidemiology and Research, New South Wales Department of Health, for editing the article.
The author(s) declare that they have no competing interests.
Baohui Yang and Margo Eyeson-Annan contributed equally to this work
Baohui Yang and Margo Eyeson-Annan contributed equally to this work.
About this article
- Design Effect
- Random Digit Dialling
- High Psychological Distress
- Computer Assisted Telephone Interview
- Post Stratification Weighting