The association between survey timing and patient-reported experiences with hospitals: results of a national postal survey

Abstract

Background

Research on the effect of survey timing on patient-reported experiences and patient satisfaction with health services has produced contradictory results. The objective of this study was thus to assess the association between survey timing and patient-reported experiences with hospitals.

Methods

Secondary analyses of a national inpatient experience survey including 63 hospitals in the 5 health regions in Norway during the autumn of 2006. 10,912 (45%) patients answered a postal questionnaire after their discharge from hospital. Non-respondents were sent a reminder after 4 weeks. Multilevel linear regression analysis was used to assess the association between survey timing and patient-reported experiences, both bivariate analysis and multivariate analysis controlling for other predictors of patient experiences.

Results

Multivariate multilevel regression analysis revealed that survey time was significantly and negatively related to three of six patient-reported experience scales: doctor services (Beta = -0.424, p< 0.05), information about examinations (Beta = -0.566, p < 0.05) and organization (Beta = -0.528, p < 0.05). Patient age, self-perceived health and type of admission were significantly related to all patient-reported experience scales (better experiences with higher age, better health and routine admission), and all other predictors had at least one significant association with patient-reported experiences.

Conclusions

Survey time was significantly and negatively related to three of the six scales for patient-reported experiences with hospitals. Large differences in survey time across hospitals could be problematic for between-hospital comparisons, implying that survey time should be considered as a potential adjustment factor. More research is needed on this topic, including studies with other population groups, other data collection modes and a longer time span.

Background

Patient experiences are an important part of health-care quality [1, 2]. Surveys are frequently used to measure patient experiences and satisfaction with health care [3, 4], but their value is subject to several methodological challenges. One particular methodological challenge relates to the time it takes from the health-care encounter to the patient receives the questionnaire. Questionnaires might be distributed immediately after a healthcare encounter, a short time after the encounter or a long time after the encounter. Since factors like clinical and health-related quality-of-life outcomes might produce different patient evaluations depending on when the survey is applied to patients [3], decisions about survey time might have substantial effects. Following a longitudinal study, two different theoretical models for patient satisfaction were suggested: (i) an immediate post-visit satisfaction model that includes demographics, patient expectations, patient functioning and patient-doctor interaction; and (ii) a model for 2-week/3-month satisfaction that includes demographics, expectations, patient functioning and symptom improvement [5]. This suggests that the results of evaluations of patients would vary with the survey time point.

Several studies have investigated the association between survey timing and patient evaluation [6–16], and most of them found that patient evaluation is poorer when measured at a longer time after the encounter [6–9, 11, 14–16]. However, a closer investigation of these studies shows that in all except one [11], the data collection mode changed between the different measurements; the aforementioned timing effects might therefore have been due to changes in data collection mode. In fact, the best-designed study concerning the association between survey timing and patient satisfaction found little association between survey timing and patient satisfaction measures [12]. That study grouped patients into three different mailing intervals: 1, 5 and 9 weeks after discharge. All groups received and completed a postal survey at home, and so the data-collection mode was standardized between the groups.

The objective of this study was to assess the association between survey timing and patient-reported experiences with hospitals. The Norwegian Knowledge Centre for the Health Services conducted a national postal patient experience survey among adult inpatients discharged from Norwegian hospitals in 2006. The data set included survey data, administrative data from the hospitals including discharge dates, and practical survey variables including the dates for first postal mailing and for response registration in the Knowledge Centre. The availability of these data made it possible to assess the association between survey timing and patient-reported experiences while simultaneously controlling for other known predictors of patient experiences.

Methods

Data

The national survey included adult inpatients discharged from Norwegian hospitals between September 1 and November 23, 2006. The response rate to the survey was 45%, with responses being received from 10,912 patients. In total, 24,141 patients were included in the study; 345 patients were not eligible. The study is described in more detail in another publication [17].

The Norwegian Regional Committee for Medical Research Ethics, the Data Inspectorate and the Norwegian Directorate of Health and Social Affairs approved the survey.

Questionnaire

The questionnaire comprised 29 items about patient experiences and satisfaction, 14 questions about quality of life and 10 background questions. The patient experience questions were based on the Patient Experiences Questionnaire [18], but the response scale was changed to improve data quality [19]. For 27 of the 29 experience items, a five-point response format was used, ranging from "not at all" to "to a very large extent" The national report used the following six scales with good evidence for data quality, reliability and validity [20]: doctor services (three items), nursing services (four items), information about examinations (two items), organization (three items), hospital and equipment (two items) and contact with next of kin (two items). These scales were used in the present study. Scale scores were transformed to a scale of 0-100, where 100 is the best possible rating.

Statistical analysis

Patients discharged from one hospital department to another were included in the national survey. For these patients we only saved the latest discharge date, and consequently it is inappropriate to use the discharge date in the computation of survey time. Therefore, patients with more than one department stay were excluded from this study (n = 362).

The survey-time variable was computed as the difference between the date of the first postal mailing and the discharge date. The survey time was 11.8 ± 5.7 days (mean ± SD; minimum: 1 day; maximum: 41 days). According to the protocol, the survey time should have been 1-15 days, but delays in transfers from hospitals resulted in a greater variation. Survey time is a continuous variable and was included as such in the regression analysis described below. In bivariate analysis, the survey-time variable was grouped by week, which gave large groups for statistical comparisons. These bivariate analyses are secondary and no other tests were conducted to assess the appropriateness of this grouping. The survey-time variable was divided into the following groups: ≤ 1 week, 1-2 weeks, 2-3 weeks and > 3 weeks. Survey-time groups were compared across six variables: gender, age, education, self-perceived health, admission type and number of admissions in the previous 2 years. Pearson's chi-square was used for statistical testing, except for age, for which one-way analysis of variance (ANOVA) was used.

Response time has been shown to be associated with patient-reported experiences [21]. Response time was computed as the difference between the dates of response registration and first postal mailing. This variable is influenced by how rapidly each individual responded to the questionnaire, and was included as a covariate in the regression models described below. The main focus of the regression analysis was the association between survey timing and the level of patient-reported experiences, adjusted for all the other predictors including response time.

Multilevel linear regression analysis was used to assess the association between survey timing and the six patient experience scales, both bivariate analysis and multivariate analysis controlling for gender, age, self-perceived health, education, admission type, number of admissions in the previous 2 years, response time and hospital. Patient clustering within hospitals might inflate t values in ordinary linear regression models so as to produce a type I error, which was the reason for using multilevel regression. The multilevel model divides the total variance in patient-reported experiences into variance at the hospital (macro) level versus the patient (micro) level. The hospitals were included as random intercepts, and all variables from the ordinary regression as fixed effects at the patient level. Standardized variables at level 1 were used in the regression; consequently, standardized regression coefficients were computed. SPSS version 15.0 was used for statistical analyses.

Results

Three of the six background variables differed significantly across the survey-time groups (Table 1): gender (p < 0.05), self-perceived health (p < 0.05) and number of admissions in the previous 2 years (p < 0.05). Bivariate multilevel regression analysis showed that survey time was significantly and negatively related to three of six patient-reported experience scales: organization (Beta = 0.528 p < 0.05), information about examinations (Beta = -0.631, p < 0.01) and contact with next of kin (Beta = -0.669, p < 0.05) (Table 2).

Table 1 Background variables for the survey-time groups (n = 10,717)

Table 2 Bivariate multilevel regression models: associations between survey-time and patient-reported experience scales

Multilevel regression analysis revealed that several background variables were significantly associated with patient-reported experiences (Table 3). Survey time was significantly related to three of the six patient-reported experience scales: doctor services (Beta = -0.424, p < 0.05), information about examinations (Beta = -0.566, p < 0.05) and organization (Beta = -0.528, p < 0.05). All associations were negative, indicating that the patient-reported experience scores declined with increasing survey time. Patient age, self-perceived health and type of admission were significantly related to all patient-reported experience scales (better experiences with higher age, better health and routine admission), and all other predictors had at least one significant association with patient-reported experiences.

Table 3 Multilevel linear regression models: associations between independent variables and patient-reported experience scales

Discussion

Research on the effect of survey timing on patients' evaluations of health services has produced contradictory results [3]. This study found that patients report worse experiences for 3 of 6 patient-reported experience scales when survey time is longer. Individual response time was also negatively related to patient-reported experiences, so regardless of reason increasing time since discharge seems to result in poorer patient-experience scores.

Studies assessing the effect of survey time need to standardize the data-collection mode in order to avoid confounding time and mode effects. Almost all studies showing a worsening in patient evaluation over time changed the data-collection mode between the different measurements [6–9, 11, 14–16]. Therefore, the timing effects might be related to the mode change rather than being actual timing effects. The current study standardized the data-collection mode and found a significant association between survey time and patient-reported experiences for three of the six scales. This is in line with the aforementioned studies, but contradicts another study from Switzerland in which the data-collection mode was also standardized [12]. However, the Swiss study only included one hospital, a specific patient group and a relatively small sample. Consideration of all of the available data suggests that there is a negative association between survey timing and patient-reported experiences and satisfaction. However, more research is needed including studies with other population groups, other data-collection modes and a longer time span.

The first limitation of the current study relates to the distribution of the survey-time variable. The data-collection protocol means that most patients were sent a questionnaire 0-3 weeks after discharge, with a maximum of 41 days for individual patients. It would be useful to have knowledge about the potential effects of a longer time span, such as the effects of a model with surveys sent 1 month versus 2 months after discharge. The second potential limitation of the current study is the response rate. In general, postal surveys have lower response rates than other data collection modes [3]. The response rate in the current study is in line with other Norwegian national patient experience surveys. Non-response bias occurs when the main variables differ systematically between respondents and respondents [22]. In our study, the main question relates to differences between respondent groups, and hence non-response bias is of less concern. However, response rates might be lower in surveys with a longer time between discharge and postal mailing [12, 13]. Consequently, the association between survey timing and the response rate is an important consideration when designing data-collection procedures. A Swiss study found that the response rate was significantly lower for the 9-week group [12] but not for the 5-week group, indicating that satisfactory results regarding response rate can be achieved with surveys mailed 0-5 weeks after discharge.

A third possible limitation of the study concerns its observational research design, causing uncertainty related to potential confounding variables. The gold standard for effect research is randomized trials, in which the aim is for only random variations to exist between study groups and for there to be a direct link between intervention and effect. However, a multicentre randomized trial on this topic would present large practical and methodological challenges, both regarding which time frames to use (intervention) and how to apply sample frames and randomization across hospitals. Another suitable design could have been a longitudinal approach, but that was not possible in this study. The present study adjusted for the most important sociodemographic predictors of patient experiences and patient satisfaction, reducing the probability of confounding effects related to variables not included. The study was based on data from all hospitals in Norway, and the survey-time variable was registered and analyzed as a continuous variable at the individual level. The former feature increased the external relevance of the study, and the latter gave the opportunity to use survey time in days in the analysis, providing more detailed information than only groups based on, say, weeks or months.

Conclusions

Survey time was significantly and negatively related to three of the six scales for patient-reported experiences with hospitals. Large-scale hospital comparisons of patient-reported experiences should consider survey time as an adjustment factor if it is not standardized across hospitals. The generalizability of the survey-time effect to other topics and other modes is uncertain, but a negative association has been found in most of the other studies referenced, including patient populations in primary care and hospital in- and outpatient care. However, more high-quality research on this topic is needed, including studies with other population groups, other data collection modes and a longer time span.