Summary of findings
It is feasible to identify system users using a population survey. Although the telephone survey used market research quota sampling and obtained a low response rate of 8.5%, it appeared to perform better than the postal survey in terms of representativeness by age, gender and minority ethnic communities, and estimating use of different services in the system. It also cost less and suffered less from missing values.
A recent randomised controlled trial of a postal versus telephone survey did not identify one mode of administration as superior[28]. This supported an earlier review of surveys in health care which identified four randomised controlled trials of postal and telephone surveys, revealing little consensus about the benefits of one over the other [29]. However a higher rate of missing values was found in the postal survey for both the recent trial [28] and the one trial in the earlier review which measured this[29]. This supported our findings. There is some evidence that telephone surveys can elicit more extreme responses and more positive responses than postal surveys,[28] although this is by no means a consistent finding [29]. This is an important issue to bear in mind if telephone surveys are to be recommended for obtaining the patient perspective of the emergency and urgent care system. However, it is less important when using telephone surveys to monitor patient views over time within systems because the focus would be changes in values rather than absolute values.
Use of the system was estimated as 10% or 8.5% in a four week period for the postal and telephone surveys respectively. Other researchers have estimated use of urgent care in a month to be 23% of adults seeking care for themselves or someone else, with the figure rising to 56% when the time period was the previous year[30]. Use of unscheduled care in the previous four weeks, which is similar to urgent care, has been estimated at 16%[4]. Our estimates are smaller than those found elsewhere but we did validate reported use of some services in the system and our estimates appeared to be accurate. Four weeks appears to be an accurate recall period for estimating event occurrence but limits the number of events identified. Three months underestimates utilisation of the system but offers more events for users to describe their experience and views. We have shown that there is little difference in the experiences and views of events described in the early and late recall periods and therefore recommend that a three month period of recall is used to assess experiences and views of the system.
The telephone survey cost £10,000 to identify 150 users of the system, that is, £67 per user. Although this is a large cost per user, the telephone survey methodology described here offers an unbiased approach to identifying a comparable group of people over time, thus allowing health care commissioners to monitor changes in their system over time.
Biases in postal and telephone surveys
Bias arises when non-response in a survey is related to the outcome being measured. Here, the most important outcome is satisfaction with the system. The potential for non-response bias differed for the postal and telephone population surveys. For the postal survey the potential for bias was introduced by the exclusion of general practices unwilling to participate, the exclusion of those not registered with a general practice or registered but recently moved, the screening of individuals' names by health professionals for those who might be distressed by the contact (this might include frequent users of the urgent care system such as people with mental health problems), the likely exclusion of people with literacy problems, and low saliency of the questionnaire for the majority of people (non-users of the system). For the telephone survey the potential for bias was introduced by the exclusion of those without telephones or landlines, those who screened their telephone calls, and people who do not spend much time at home to receive telephone calls.
A comparison of a postal survey with a probabilistic random digit dialling telephone survey in the United States identified some advantages of the telephone approach in health research[31]. The telephone survey had higher proportions of respondents from ethnic minority communities and less educated groups, and a lower proportion of missing values. This supports our findings about the representativeness of the telephone survey. The United States study also found problems with the telephone survey which have been identified previously in some literature, namely lower response to sensitive questions and use of extremes of response sets to offer socially desirable answers. Our questionnaire did not contain any obviously sensitive questions, for example we did not ask for details of the health problem for which help was sought apart from whether it was an illness, injury or other type of problem. However, questions about health and health care may be perceived as sensitive by respondents and thus the telephone survey may have been affected by this. It is possible that there was social desirability bias due to respondents worrying whether their use of health services appeared to be appropriate, and concerns about not being seen to complain about health services. These problems are less problematic if telephone surveys are used to measure change over time rather than prevalence of behaviour or attitudes. However, these problems must be borne in mind when interpreting survey findings.
Interestingly, the United States study described above identified that the 6% of postal survey responders who were living in households without a landline, that is, used mobile phones only, had health behaviours different from those with landlines. They were more likely to participate in risk taking behaviour for HIV. This bias may be context specific to a survey undertaken in 2005 in the United States but raises the important concern that the prevalence of mobile phone only households may increase over time in the United Kingdom and thus exclude people from our telephone survey. In the United Kingdom, ownership of landlines fell from 94% to 89% of households between 1997 and 2007, while 78% of households had mobile phones[32]. We could not find any UK data on mobile only households, especially the age and socio-economic groups most likely to be without landlines. However, young people, and students in particular, might be more likely to live in households without landlines than other subgroups of the population. Telephone survey methodology will need to address this potential exclusion in the future.
Strengths and limitations
The strength of this study is that different approaches to methodology have been tested empirically. Limitations include the low power for some of the statistical comparisons made, the low response rate of the telephone survey although this is typical of these types of surveys,[25–27] the use of English only versions of both surveys so that those who could speak English well enough for the telephone survey and read and write it well enough for the postal survey were included, and the potential for 'primacy' effects of selection of the response options nearer the beginning of any list to operate in the postal questionnaire while 'recency' effects of selection of the response options closer to the end of the list operated in the telephone survey[33]. Finally, population estimates were based on the 2001 Census and the demographics of the area could have changed by 2007.
The postal and telephone surveys were undertaken at different times - the telephone survey was undertaken in July, covering system use in a three month period between April and June, and the postal survey between September and December with the three month period lying between June and November. There is seasonal variation in the use of some services in the system. In particular, general practice consultations for respiratory and influenza-like symptoms, and pressures on emergency hospital beds, increase enormously in the winter months of December to February while remaining steady in other months. This peak use of services was not included in either the telephone or postal survey recall periods but part of it was likely to have affected the postal survey and this could account for some of the higher reporting of system use in this survey. It is also the case that the routine data which the survey were compared with covered the whole period of 2006/7 which would include the winter peak. Because of this we would expect both the telephone and postal survey to underestimate average system use because they did not include the winter peak.
Implications of using the telephone survey
The telephone survey does not make use of NHS sampling frames and therefore does not need approval from an NHS Ethics Committee in the UK. However if it is used for research purposes then ethics approval must be sought. We have used it since this study and sought ethics approval from our university. If the survey is used by those managing an emergency and urgent care system then this would be classified as service evaluation and would not need NHS ethics committee approval in the UK.
It is important to bear in mind the limitations of the telephone survey in terms of excluding mobile phone only households and homeless people. It also assumes that adults completing the survey on behalf of a child know about that child's use of health care. We tested an English only version of the survey but some market research companies offer a translation service for telephone administered surveys. This is a useful approach so that people who do not speak English - and who may have difficulties using the emergency and urgent care system because of this - can be included in the survey.