Designing questionnaires: healthcare survey to compare two different response scales

Participants and setting

From November 2011 until January 2012, every in-hospital patient above 16 years of age who was discharged from the University Hospital in Basel, Switzerland, had the opportunity to participate in this healthcare survey. Patients received a letter with both questionnaires and were requested to accept the redundancy of five questions on two different response scales and to answer all questions. If the discharged patients did not return the questionnaire, no reminders were sent out. The questionnaire was sent out in the following five languages: German, French, Italian, English and Turkish. If a patient had a different native language, they received the German version of the questionnaire.

The following socio-demographic variables were assessed for all patients: Age, gender, nationality, native language, emergency versus elective hospital admission, length of hospital stay, individual department where patients received their main treatment and hospital readmissions within the same study period. Since our anonymisation procedure was irreversible, re-admission information was limited to all in-hospital patients during the study period and was not available for the subpopulation of patients returning the complete questionnaire.

The anonymisation procedure of this quality control survey was approved by the local data protection committee (“Datenschutzbeauftragter des Kantons Basel-Stadt”, reference number 10_0147). Since the primary purpose of the survey was the quality control of in-hospital patients at the University Hospital Basel, the local ethics committee (“Ethikkommission beider Basel”) exempted the survey from formal ethical committee approval. Moreover, confidentiality was preserved by separating the data analysis team from the healthcare providers.

Questionnaires

1) Questionnaire using a numeric answering scale (NS)

In 2011, the Swiss National Association for Quality Development in Hospitals and Clinics (ANQ, http://www.anq.ch) has developed a questionnaire with the aim that every member of this association uses the questionnaire during one defined month of the year, leading to a yearly national quality evaluation of all hospitals in Switzerland. In order to obtain a larger sample, we have extended the survey to three months.

The five questions (Figure 1) covered four important domains, which indirectly rate patient satisfaction on an evaluation scale. The questions were ordered as follows: 1) behavioural intent to return to the hospital (one question), 2) quality of treatment (one question), 3) quality of medical information (two questions), and 4) question concerning judgement whether the patient was treated with respect and dignity (one question). The response scale is displayed on an 11-point numeric scale (NS) with anchors at both ends, presenting the negative answers first.

Statistical analyses

Statistical analyses were conducted using Intercooled Stata Version 11.2 for Macintosh (StataCorp, College Station, TX, USA). Graphs were performed in Intercooled Stata Version 11.2 for Macintosh and in R system, version 2.14.2 (R Foundation for Statistical Computing. Vienna, Austria). We report 95% confidence intervals (CI), rather than p-values, in order to emphasise clinical relevance over statistical significance, because in this large data set, irrelevant differences are also statistically significant. According to the current guidelines for reporting observational data (STROBE) [13], we avoided significance tests for the evaluation of differences in baseline characteristics.

Baseline characteristics were summarised for all patients, questionnaire responders, non-responders and responders with no missing items. The latter group corresponds to the patients included in this analysis.

The answers on both response scales were summarised in a frequency table for comparison of the ceiling effect. The Cronbach’s alpha for the five questions on each response scale was calculated as a measure for internal consistency. Since the five items are all measuring patient satisfaction (i.e. are congeneric), the alpha is an estimate of the lower end of reliability [14]. Therefore, only the lower one-sided 95% confidence interval was computed to see whether it is significantly greater than some minimal value [15, 16]. Furthermore, the data were investigated using graphical displays (histogram, scatterplot) and the Spearman’s rank correlation coefficient with its 95% confidence intervals (CI) [17] to determine the amount of correlation between both response scales (NS and LS) for each item.

We calculated an overall percentage score for each of the two questionnaires’ results as a summary measure with a common metric. In the numeric scale questionnaire, we calculated the sum of the five answers, ranging from 5 to 55. In the adjectival scale questionnaire, we attributed values from 1 to 4 or 1 to 3, respectively, to the answer categories, resulting in a range from 5 to 17. Here, we assumed equal distance between the categories. The percentage score was calculated by dividing the sum of the answers by the maximum possible sum. For the calculation of the Spearman’s correlation coefficient and the overall percentage score, those patients who did not have any questions in the third and fourth questions were disregarded.

These analyses were repeated in an explorative way in the following prespecified subgroups: Elective versus emergency patients and short versus long hospitalisation, as defined by median split. We have chosen these subgroups since it is known from the literature that hospital length of stay and type of admission have an impact on satisfaction with hospital care [18, 19].

Patients’ characteristics

Of the 7158 patients to whom the questionnaire was sent, 2957 (41%) responded. Out of these, 2400 (81%) completely filled out all questions on both response scales (Figure 2). The number of all patients who stayed more than once in the hospital during the study period was 603 (8.3%).

Response scale comparison

1) Questionnaire results, mapping and ceiling effect

Summary statistics for the answers on the NS and the LS show a very high and satisfactory rating in all domains (Table 2 and Figure 3). As shown in the mapping of the answers of each category of the LS to the NS (Figures 4 and 5), each response category of the adjectival response scale is depicted by at least four levels on the numeric response scale. Comparing both response scales, the percentages of best category ratings in the LS are slightly higher for questions with four answering categories as compared to the NS (e.g. 73% in LS vs. 67% in NS for the “Return-Question”). This ceiling effect was clearly more pronounced in the LS with only three answering categories (e.g. 80% on LS vs. 48% on NS for “Question to Nurses”). The floor effect was more pronounced in the LS with the lowest two to four categories of the NS being represented by the lowest category of the LS, independently of the number of answering categories in the LS.

	Numeric Scale (NS)												Labelled Scale (LS)
Questions	0	1	2	3	4	5	6	7	8	9	10	NA	A	B	C	D	NA
Would Return	1%	<1%	1%	1%	1%	2%	1%	4%	11%	13%	67%	-	1%	2%	23%	73%	-
Quality	<1%	<1%	<1%	1%	1%	1%	2%	5%	15%	22%	52%	-	1%	3%	39%	57%	-
Question to Physicians	<1%	<1%	<1%	1%	1%	2%	2%	4%	14%	15%	54%	6%	1%	15%	77%	-	8%
Question to Nurses	<1%	<1%	<1%	1%	1%	2%	3%	6%	16%	17%	48%	7%	1%	14%	80%	-	5%
Respect and Dignity	<1%	<1%	<1%	1%	1%	1%	1%	3%	8%	15%	69%	-	1%	9%	90%	-	-

NA = I didn’t have any questions for staff.

The answering scale in the NS had anchoring labels at 0 and 10 of ‘Certainly not – Yes, of course’ for the first question, ‘Poor – Excellent’ for the second question, and ‘No, never – Yes, always’ for questions 3 to 5, respectively.

The categories of the answering scale in the LS were labelled with ‘Of course not (A), No, I don’t think so (B), Yes, I think so (C), Yes, of course (D)’ for the first question; ‘Poor (A), Fair (B), Good (C), Excellent (D)’ for the second question; and ‘No (A), Yes, sometimes (B), Yes, always (C) for questions 3 to 5, respectively.

3) Response scale comparison on the individual item level

The results of the correlation of the individual item answers between both response scales are displayed in Table 3 (all questions) and Figures 6 and 7 for two representative questions (quality of treatment and quality of information provided by physicians; Webappendix: Additional files 4, 5 and 6 for the remaining three questions). There was a strong correlation between the response scales regarding questions about intent to return, quality of treatment and judgement whether the patient was treated with respect and dignity, but a lower correlation concerning satisfactory information transfer by physicians or nurses.

	All patients (n = 2400)
Question	Correlation coefficients	95% confidence intervals
Return	0.60	0.58, 0.63
Quality	0.59	0.56, 0.61
Question to Doctors	0.32	0.29, 0.36
Question to Nurses	0.33	0.29, 0.37
Respect and Dignity	0.52	0.49, 0.55

Spearman’s rank correlation coefficient for each item between both response scales (NS and LS): all patients (n = 2400 for question 1, 2, and 5, n = 2140 for question 3 and n = 2173 for question 4, respectively).

Corr. Coeff. Correlation Coefficient. CI Confidence Interval.

4) Response scale comparison on an overall level using a percentage score

The overall percentage score showed a comparable distribution (median 96 (interquartile range (IQR) 87.5 – 100) for the NS vs. 94 (IQR 88 – 100) for the LS, respectively) with a wider spread of lower satisfaction in the NS (Figures 8 and 9).

Response scale comparison in pre-specified subgroups

Patient satisfaction was rated higher in patients after elective, than after emergency admission and after shorter, compared to longer, hospital stay. Consequently, the ceiling effect was more pronounced on both response scales in patients after elective admission and shorter stay (Table 2). The results of the comparison of Cronbach’s alpha, the individual items and of the overall percentage score did not greatly differ from overall results (Webappendix: Additional file 7).

Strengths of the study

Our study is one of the largest studies including all departments of a university hospital. It belongs to a small number of studies, using a pragmatic (as opposed to an empirical) approach [3]. Moreover, this study is unique since it enables a within-patient comparison of the perception of hospital care. This allows refining the properties of the response scale in the general questions in order to reduce floor and ceiling effect which in turn leads to a better discrimination at the high end rating and to improved internal consistency.

Limitations of the study

Our study presents some limitations. The main limitation was that only overall questions on patient satisfaction were available for the response scale comparison, since the nationwide questionnaire was limited to these five evaluating questions. A further limitation was a response rate of only 41%, which however is in line with response rates between 42 and 48% in previous studies [7, 20]. Resulting bias is unlikely since the baseline characteristics of responders and non-responders only differed slightly and the percentage of missing items was very small compared to other studies in similar settings [21, 22] still allowing for a certain generalisability. Moreover, several studies could show that non-respondents are generally not significantly different from respondents in terms of satisfaction scores [23], or in terms of sociodemographic characteristics [24]. A further limitation consists of differences between the questions on both response scales with regard to exact wording, the polarity of the response scale, the order of the questions and that the five questions on the LS correspond to a subset of a 17-items questionnaire, whereas the nationwide survey only included these five general questions. This may partly explain the divergent answers on both answering scales, which results in lower correlation coefficients and could be taken as a consequence of the pragmatic approach.

Findings in relation to other studies

The high patient satisfaction in the present study is in line with the findings of an international evaluation [25], in which Switzerland was found to stand out by having very high quality of care ratings compared to other European countries and the United States.

Questionnaires are convenient for monitoring in-patient satisfaction, but reliability and validity of the findings are reduced by skewed response distributions [8]. Different methods have been proposed to reduce the ceiling effect. In our study, we investigated the influence of the length of scale together with labelled categories versus end-anchors. We found a less pronounced ceiling effect with a persistent albeit less left-skewed response distribution on the longer end-anchored numeric scale than on the shorter adjectival scale with labelled categories. It was rather unexpected that the skewedness on the numeric answering scale (compared to LS) was nearly as marked. The different polarity of both answering scales, with the LS having positive answers on the left-hand side and the NS on the right-hand side, might partially explain the higher ceiling effect found in the answers on the LS, since there is evidence of bias towards the left side [26]. People tend to use the first satisfactory response option in a presented questionnaire, as shown in studies which directly compared a reversed to a not-reversed answering scale [27, 28]. However, had the reversed answering scale of both questionnaires a strong influence in our setting, we would expect the response distribution of the numeric scale (with negative answers on the left-hand side) to be more symmetrical. Moret et al. [20] achieved a reduction in ceiling effect by extending the response scale from a four-point to a five-point format, but with an unbalanced scale including three positive and two negative choices. In contrast to our results, Garratt et al. [7] showed that a 10-point scale produced a highly skewed distribution compared to a balanced 5-point scale, which showed a fairly symmetrical distribution. However in both cited studies, the ceiling effect was less pronounced than in our data [7, 20]. Similarly, in a randomised comparison of four patient satisfaction questionnaires, Perneger et al. [29] found the lowest ceiling effect in a questionnaire using an unbalanced five-point Likert scale. These results suggest that an unbalanced five or six points scale might outperform both our three- to four-point scale as well as our 11-point end-anchored scale. Alternatively, an even stronger imbalance with four positive categories is necessary to render our high rating approximately normally distributed. A normal distribution of questionnaire results can also be achieved by using an overall score [30, 31]. However, even by combining all five answers to an overall percentage score, we did not obtain a fairly symmetric distribution. It might be argued that it is not possible to achieve a symmetrical distribution with general questions about patient satisfaction [32, 33], rendering the selection of the questions used for the national benchmark disputable.

Apart from changing the length of the answering scale, the two scales also differed in the number of labelled categories. The use of an adjectival scale for comparative purposes can be limited by its lack of sensitivity for detecting small changes and may strongly depend upon the choice of the wording and the literacy of the patient. Accordingly, Downie et al. showed an improvement of discrimination by using a numerical rating scale as compared to a four-point descriptive scale and a continuous scale with two end-anchors [34]. On the other hand, Streiner and Norman provided evidence that there is a tendency of end-labelled scales to pull responses to the end [35]. This is in line with our data where a fairly big difference could be found between the percentages in the highest category (10) and the next one (9) in the responses on the numerical scale using end-anchors. Garratt et al. argued that the higher ceiling effect of their longer 10-point scale could be explained by this phenomenon of end-anchors pulling responses to the end [7]. On the other hand, one could expect an end-aversion bias, especially if the endpoints are labelled with absolute words such as ‘Always’ as opposed to ‘Almost always’.

Our results further suggest that the internal consistency (Cronbach’s alpha) was higher for the NS than for the LS, which has been confirmed in the study of Hendriks et al. [21]. However, with a lower limit of the Cronbach’s alpha above 0.75, both scales can formally be accepted as a valid measure of patient satisfaction.

Comparing both answering scales, we found a high individual item level correlation in three out of five questions. These three questions cover the domains of intention to return, quality of treatment and respectful behaviour. Laerhoven et al. [36] found Spearman rank correlation coefficients comparing three different types of response scales, which all laid above the plausible range of values from our study. This finding might be explained by the above-mentioned additional differences in both questionnaires apart from the length and type of scale (e.g. polarisation and order of questions). There is consistent evidence that the most important factor affecting patient satisfaction is the patient-practitioner relationship, including information provision [2, 3]. In the present study, this was the domain in which the correlation between both response scales diverged most. From the collected information, we are unable to determine which of the questionnaires contains the valid individual response. On the other hand, it is not unexpected that the correlation is lower in these two information questions. Whereas questions about treatment with respect and dignity and intention to return imply almost binary answers, in the sense that you are either willing to return or not, a question about provision with adequate information is much more complex, depending on the content as well as the wording of the given information. Moreover, the adjectival response scale only consists of three categories not allowing for much differentiation.

Implications for daily practice and further research

In order to further improve the nationwide benchmark at this point, the questions and answering scale need to be refined to enhance the discriminative power for the high end of patient satisfaction, since both response scales showed a strong ceiling effect. This is especially relevant, because patient satisfaction may influence many important outcomes such as compliance, overall well-being and consumer choice. The choice of these five overall questions for a national benchmark should be challenged and a different set of reporting questions defining quality aims tested.