A randomized trial of mailed questionnaires versus telephone interviews: Response patterns in a survey

Assignment

The allocation sequence for the randomization was generated by a computer assistant using the SAS routine RANUNI and a subsequent sorting procedure. The allocation lists were delivered to the head of the project.

Masking

Neither participants, nor surveyors or assessors were blind to group assignment.

Those randomised to receive a mailed questionnaire were sent one at their current address. Non-respondents received two postal reminders at intervals of three weeks. The second reminder included a questionnaire. For the group randomised to telephone contact, phone numbers were searched electronically in the database of the national phone company and manually in additional databases. Persons with a confidential phone number were sent a letter with an invitation to call back. The telephone interviews were computer assisted; they were preceded by a letter and were primarily conducted Monday-Thursday between 4 pm and 9 pm and Sunday between noon and 4 pm. The interviewers were student assistants and paid by the hour. They received a short course, a written manual and conducted three test interviews. The interviewers were instructed to accept item refusals and act neutral, but whether they complied completely with these instructions is uncertain, just as it is uncertain whether all response categories consistently were read out. Non-respondents were followed up with at least three calls at different times of the day/evening. Those unwilling to participate in a telephone interview were offered a mailed questionnaire. The latter are treated as non-respondents here. In both of the randomised groups, the study was presented as concerning health, well-being and, for those having a job, the psychosocial work environment. No incentives were used in either of the two randomised groups.

The questionnaire was 27 pages long and contained 257 items. The duration of telephone interviews was typically 30–45 minutes (cf. English translation[13]). This paper deals with the 47 items concerning health and well-being that all respondents were asked, irrespectively of their labour market status. These 47 questions are on the first 8 pages of the questionnaire. The questions have between three and six response categories and are primarily from the Stress Profile[14] and the SF-36 questionnaire[15]. The order of the questions was the same as in additional file 1: Table A.

The study has been notified to and registered by the Danish Data Protection Agency (Datatilsynet). According to Danish law, questionnaire based studies do not need approval by ethical and scientific committees, nor informed consent.

Protocol

The original protocol for the survey did not specify any objectives or hypotheses (or primary and secondary outcome measures) concerning the effect of mode of data collection. We thus tested the hypotheses that we identified in the literature and listed in the introduction, and we estimated any differences. Questions concerning loneliness, well-being, self-esteem, unhappiness, nervousness, worry, depression, or trouble sleeping are mentioned in the literature as sensitive[1, 7, 16]. We categorised our 47 items as belonging under one of the four mentioned self assessments: well-being, self-esteem, depression or the additional heading stress, or under one of two behaviours: smoking habits and use of medicine. Thus all 47 questions are either sensitive or concerning socially undesirable behaviour. The categorization appears from the additional file 1: Table A (cf. the "Theme" column) and 2 and the analyses are presented accordingly. The distinction between self assessments and behaviours were added post-hoc.

Statistical analysis

We used the chi-square test to test the hypothesis of telephone interviews resulting in higher response rates than mailed questionnaires (Hypothesis 1) and logistic regression to assess differential bias between mode of data collection, age and gender; marginal associations between mode of data administration and demographic data were examined using the chi-square test of independence.

We tested the hypotheses of higher rates of missing responses among questionnaire respondents than among telephone respondents (Hypothesis 2) and of greater willingness to answer sensitive questions in questionnaires than in telephone interviews (Hypothesis 3) by Fishers exact test (two-sided for all items since both hypotheses are relevant for all items).

We used the chi-square test to test whether telephone respondents had a preference for extreme (i.e. the highest possible and the lowest possible) response categories compared to questionnaire respondents (Hypothesis 4) and whether the level of reported health and well-being differed according to mode of data collection (Hypothesis 5) (The null hypotheses of these hypotheses are the same, while the alternative hypotheses differ). The tendency to use the extreme response categories among phone respondents relative to mail respondents was quantified by the odds ratio for extreme replies versus the pooled middle categories, and the tendency to reply more positively was quantified by the odds ratio estimated in the proportional odds model. Responses in the "don't know" category were treated as missing values.

The same statistical tests are performed on 47 items so the traditional 5% level of statistical significance is replaced by a 0.1% level, i.e. a Bonferroni correction of the 5% level[17].

Participant flow and follow up

Response rates

Self reported demographic data were compared for the two types of respondents. No differences were found in the marginal distribution of schooling (p = 0.13), labour market status (p = 0.43), number of children living at home (p = 0.20), or number of children under the age of six living at home (p = 0.77). Significant differences were found for vocational training (p = 0.004) and cohabitation (p = 0.001). Among telephone respondents, there was a larger percentage of persons with a further and higher education than among mail respondents (11.8% versus 7.3%) and a larger percentage of persons who had always lived alone (9.1% versus 5.3%).

Missing

The "don't know" option

Response pattern

Adjustment for the observed differences in gender and age response rates did not alter the magnitude of the estimates appreciably (Data not shown).

If the "don't know" category was analysed as the neutral middle response, the estimates were increased and the conclusion was strengthened (Data not shown).

Findings

No differences were found in the overall response rate but a breakdown on age and gender showed that women and persons in their thirties had a higher response rate to the mailed questionnaire compared to the telephone survey. The rate of missing responses is clearly higher among questionnaire respondents than among telephone respondents. In relation to questions on behaviour, no difference between the two modes of data collection was found in the use of extreme response categories and no shift in the level was found between the modes. In relation to questions involving self assessments, telephone respondents had a strong preference for extreme response categories compared to questionnaire respondents and most items were shifted towards a more positive reporting among the telephone respondents.

When it comes to utilizing extreme response categories and to shift in level of response, the odds ratio estimates were above 2 for several items, i.e. as large as many effects studied in epidemiology.

Strengths and weaknesses of our study

The unique opportunity in the presented data set to study mode sensitivity on a range of questions has so far been unused. The difficulties we met trying, retrospectively, to construct a flowchart and to verify the coding of variables emphasise the importance of completing these tasks concurrently with conducting the study[18]. These deviations from the ideal, however, do not seem to cause bias in the conclusions of our study. Our categorisation of individual items under the themes mentioned in the literature as sensitive might have been done differently by other researchers. However, we have preferred to maintain our pre-analysis categorisation. We do not suspect this categorisation to have any major influence on our results.

To counteract the multiplicity of analyses and outcomes we applied a conservative Bonferroni correction of the p-values. Nevertheless, the vast majority of statistical tests were significant and the emerging pattern of differences was fairly consistent.

The sample was a stratified sample of an adult population. The results are thus fairly generalizable compared to samples of more specific subgroups. However, the analysed questions concerned only health behaviour and self assessment items so generalization to other types of questions should be done cautiously.

Comparison with other studies

Compared to most previous studies, our randomized design is strong.

It has previously been reported both that mailed questionnaires resulted in a lower[2, 6, 8], a higher[4], or the same[7] response rate as telephone interviews. The response rate will probably depend highly on the specific strategy and persistence in each trial arm[19]; the telephone response rate may also dependent on administrative accessibility of phone numbers and changes in phone technology.

The highly significant difference[4, 7–9] and the missing rate of almost nil among the telephone respondents[4, 8] are in accordance with other findings.

The cognitive demands on telephone respondents are higher than the cognitive demands on questionnaire respondents. Questionnaires enable the respondent to control both the pace and the order of interview[7]. A tendency for telephone respondents to prefer extreme response categories has also been reported in other studies[7, 8] that primarily covered self assessments.

Due to the few missing responses among telephone respondents, the hypothesis of greater willingness to answer sensitive questions and questions concerning socially undesirable behaviour in questionnaires than in telephone interviews could not be confirmed by our data.

Possible mechanisms

A majority of the questions considered here are sensitive according to the literature, and if interviewers are instructed not to accept item refusals it may be at the cost of sensitive questions being answered randomly or perhaps overly positive.

As already mentioned, most of the self assessment items were shifted towards a more positive reporting among the telephone respondents. However, the pattern was slightly more complex, since six of the mode-sensitive items shifted in the opposite direction. These six items all appear in blocks of questions formulated as statements with response categories indicating how well the statement fits the respondent's situation ("Correct", "Almost correct", "Somewhat correct", "Only slightly correct", "Incorrect"). Some statements are positive (E.g. "I feel I understand most of what is going on in my everyday life") and some are phrased using a negation (E.g. "So far, I have not had any clear direction or purpose in life"); five of the deviating items included negations and four of these concerned self-esteem. In Danish, the questions concerning self-esteem are strikingly longer and more long-windedly phrased than the rest of the questions. The long-windedness, the mix of positive statements and statements using negations, or both might explain the deviation from the overall pattern. However, these are post-hoc speculations.

According to the theory of response order effects, a preference for the first response categories is to be expected in visually presented questionnaires whereas a preference for the last mentioned response categories is to be expected in telephone interviews[20]. A shift in response pattern supporting this theory was observed for half of the items. (This theory coincides with the hypothesis of a shift toward a more positive reporting among the telephone respondents for questions using a negation and coincides with the hypothesis of a shift toward a more negative reporting among telephone respondents for questions positively phrased). Thus the mechanism of more positive reporting among the telephone respondents is stronger in our study than the proposed response order effect.

Future research

In our study response patterns were similar for questions on health behaviour regardless of mode of data collection; whereas, response patterns were significantly different for questions on health self assessments. This finding was not one of our pre-specified hypotheses so an independent test of our finding would be welcome. Also similar tests and quantifications of mode sensitivity in a wider set of item types, e.g. items unrelated to health issues and items including opinions and values would be useful.

Costs were not booked in our study but ought to be booked in future studies. It is important additional information in decision making when comparing the two data collection methods in relation to differences in response rates, patterns of missing responses and response patterns and when optimizing reminder procedures. Calibration of the measurements from the two modes of data collection is also an important issue. This has to some extent been dealt with elsewhere[21].