The Hawthorne Effect: a randomised, controlled trial

The so-called 'Hawthorne Effect' may be an important factor affecting the generalisability of clinical research to routine practice, but has been little studied. Consider the following scenario: a clinician assesses a patient with Alzheimer's disease, decides to prescribe a treatment to improve cognition and reassesses the patient six months later. At follow-up, the patient appears marginally worse, so, the clinician, mindful of the fact that in published dementia trials, people receiving active medication often appear better or stable stops the drug, believing this to be a treatment failure. But the clinician has not taken into account the possible impact of the Hawthorne Effect on the trial results that guide his practice- patients in clinical trials appear to fare better than those in routine practice by virtue of their participation. He may have stopped his patient's medication unnecessarily.

The Hawthorne Effect was first reported following an extensive research programme investigating methods of increasing productivity in the Western Electrical Company's Hawthorne Works in Chicago during the 1920s and 30s [1, 2]. The finding of enduring interest was that no matter what change was introduced to working conditions, the result was increased productivity. For example, improving or reducing the lighting in the production areas under test produced similar effects. It has been defined as 'an increase in worker productivity produced by the psychological stimulus of being singled out and made to feel important' [3]. Subsequently the definition has been broadened; here it refers to treatment response rather than productivity.

Although first reported in industrial research, the Hawthorne Effect clearly may have its implications for clinical research and its generalisability to routine practice. If there is a demonstrable benefit from participating in clinical research, for whatever reason, then this has implications for good clinical practice and for improving care [4, 5]. The Hawthorne Effect is a component of the non-specific effects of trial participation, but is not controlled for by usual controlled trial designs.

Most clinical trials are unable to quantify the magnitude of the Hawthorne Effect because its defining features, such as extra attention by researchers and higher levels of clinical surveillance, apply equally to treatment and control arms. Although the Hawthorne Effect should not affect assessment of the difference between intervention and control, it may result in an inflated estimate of effect size in routine clinical settings by over-estimating response in both groups.

Hawthorne Effects have been suggested in previous clinical trials in dementia but as far as we are able to determine, no attempt has been made to quantify them. To quantify the magnitude of the Hawthorne Effect, in the context of a placebo-controlled study of an intervention for treating mild-moderate dementia, we compared intensive to minimal follow-up in a randomised fashion. Our aim was to determine whether intensive follow-up had an effect on cognition, quality of life, behavioural functioning and/or psychopathology, as compared to minimal follow-up.

Our primary null hypothesis for this component of the trial was that compared to minimal follow-up, intensive follow-up would have no additional effect on participants' in a placebo controlled trial of Ginkgo biloba for treating mild-moderate dementia in a community setting, as measured by the ADAS-Cog (the Alzheimer's Disease Assessment Scale – Cognitive subscale) [6] and the participant's quality of life, as measured by the QOL-AD (Quality of Life in Alzheimer's Disease assessment scale) [7].

We conducted a 2 × 2 factorial, community-based, pragmatic, randomised, double-blind, parallel-group trial in which participants were treated, double-blind with an high-purity extract of Ginkgo biloba (120 mg daily) or placebo for six months (the DIGGER trial: Dementia In General Practice Ginkgo Extract Research). In addition to randomisation to treatment, participants were randomised to intensive or minimal follow-up (see below for details). The factorial design was economical in terms of the numbers of patients required to address the two questions. The trial was approved by South West Multi-Centre Research Ethics Committee (ref:MREC/02/6/35) and was registered with Current Controlled Trials(ISRCTN45577048). The trial was sponsored by London West Mental Health Research and Development Consortium, and funded by the Alzheimer's Society. There was extensive consumer involvement, through the Alzheimer's Society, at all stages.

Figure 1 details participant flow through the trial for the Hawthorne analysis. Recruitment took place between February 2003 and June 2005. Table 1 presents baseline demographic and clinical characteristics of the participants and baseline demographic characteristics of their carers. 119 GP practices, representing 388 individual GPs, recruited for the trial. Of the 176 participants in the trial, 132 (75%) were recruited through their GP.

Data on effectiveness of Ginkgo versus placebo, and adverse events will be reported elsewhere. To summarise these findings however, in the ANCOVA model with baseline score as a co-variate (n = 176), we found no evidence that a standard dose of high purity Ginkgo biloba confers benefit over placebo in mild-moderate dementia over six months.

We found that more intensive follow-up of individuals in a placebo-controlled clinical trial of Ginkgo biloba for treating mild-moderate dementia resulted in a better outcome than minimal follow-up, as measured by their cognitive functioning. Participants in our study had a maximum of four assessment points at 2 monthly intervals: less intensive than some other trials. For example participants in two trials of donepezil had 7 assessments in 14 weeks and 26 weeks respectively [10, 11], Such assessment protocols may result in a greater Hawthorne Effect. This is the first example of a randomised controlled investigation of trial effects where both groups have comparable (baseline) characteristics. Our findings accord with previous literature [4, 5] – which suggest that there may be a small, positive effect of trial participation on response – and our results suggests that the intensity of follow-up could be a factor here.

The mean difference between the follow-up groups was two points on the ADAS-Cog. This may not be regarded as particularly clinically significant, but is of similar magnitude to that reported in randomised controlled trials of cholinesterase inhibitors for dementia [12].

We also found a significant difference in participant-rated quality of life score in the main analysis, suggesting quality of life was worse in participants who were intensively followed-up. This may seem to be contrary to the impact on cognition, as improved quality of life may be expected to mirror better cognition scores. However, improved ADAS-cog scores in this study may not reflect better cognition (participation is a study per se is unlikely to improve cognitive performance), but be due to Hawthorne or learning effects. Evidence of a possible Hawthorne effect may be supported by the finding that more intensive contact leads to lower reported quality of life. Previous literature [13] suggests that more intensive contact may lead to a better recognition of needs. This may in part be due to more 'honest' reporting of needs if intensively followed-up as there is a better relationship with the care provider. A similar honesty premium may have operated in the intensively followed up group here. Another possibility is that more intensive contact with study personnel led to greater awareness of the diagnosis and resultant disability, impairing perception of quality of life. This explanation may account for the lack of differences in carer QOL.

Some researchers have criticised the interpretation of the original research about the Hawthorne Effect [14]. Other possible variables which may have contributed to the observed effect include managerial input; the fear of losing one's job during The Great Depression; the duration of rest breaks; and the changing of participants and human relations, among many factors contributing to the overall lack of scientific rigour. Indeed the first statistical analysis of the findings found most of the variation was explained by other factors [3]; other researchers argue that the effect was minimal at best [15, 16]; and yet others have not managed to replicate the findings [17]. As a result the usefulness of the term has been questioned; as it is effectively a catch-all term for labelling psychological and social factors that weren't controlled for or that cannot be explained. Parsons [18] defined it as 'the confounding that occurs if experimenters fail to realize how the consequences of subjects' performance affect what subjects do". Nevertheless and perhaps erroneously, the term 'Hawthorne Effect', coined by French [19] many years after the original experiments, has come to be understood as, the effect on outcome through the participation in research.

There are limitations to our study. We have looked at a specific group: individuals with mild-moderate dementia in a community setting, and the effect may be limited to this group especially as learning, and/or habituation to the assessment process over time, may be significant components of improved outcome in dementia trials. A Hawthorne Effect has been previously suggested in dementia trials [20, 21], which drove our investigation. It has been suggested [20, 21] that an observed effect amongst such a group may be due to a learning effect whereby familiarity with the instruments results in better performance. To minimise learning effects, we used different word lists for the memory items (sections one and seven of the ADAS-Cog) at each follow-up point. These items account for the majority of variation in ADAS-Cog scores amongst individuals with mild-moderate dementia [22]. Furthermore, the overall test-retest coefficient was found to be 0.91 over a six week period [23], which does not suggest there is a significant learning effect associated with the ADASCog. The opposite argument can also be made: people with dementia have a reduced ability to learn and learning effects may thus be smaller in this group than others. Our results do not allow us to rule out either possibility, and if this is the case, the Hawthorne effect may be larger in non-dementia trials.

Another limitation is the lack of blinding of participants and researchers to follow-up group. This may have a 'nocebo effect' in that those minimally followed-up may feel that they are not getting the 'best care' during their participation. However as Table 2 shows, two-thirds of the participants and over half of the carers expressed no preference in respect of follow-up frequency. Furthermore, the distribution of preference was similar between the groups (i.e. half of those participants who preferred minimal follow-up received minimal follow-up). This suggests that such an effect may not be of great import. A further possible limitation is that we assessed participants in their own home, rather than in hospital as most studies do, and this may have reduced the impact of trial participation.

The situation around adherence to protocol is complex. There is an imbalance in the follow-up rates between the groups: intensive follow up patients were less likely to be followed up according to the protocol. This may be due to the greater demands on participants and their carers in this group. This may introduce bias, with more compliant participants remaining in the study. However the main analysis is based on evaluable data, with similar numbers in the groups providing data at six months (71 in the minimal, 69 in the intensive). The imbalance in numbers followed-up according to protocol is due to missed two or four month assessments in the intensive follow-up group, making their assessment schedule more similar to that of the minimal follow up group. This reduces the difference between groups and is likely to underestimate the effect. Rates of attrition and compliance to the trial medication did not differ significantly between groups.

It is possible that there is a threshold effect: any participation in a clinical trial is associated with a different outcome from that of routine care, but the difference between levels of follow up within a clinical trial is small. This is perhaps the most relevant question in terms of generalisability of clinical trial findings to routine practice, but we did not attempt to answer it, and a different methodology would have been required to do so.

We draw two main implications from our findings. The first relates to how treatment effects may be over-estimated as a result of follow-up in trials. If there is an uneven rate of drop-out between groups, the benefit to the group with less attrition may be inflated because of the added effect of follow-up. In dementia research, more people tend to drop-out of the treatment arm; this would lead to an underestimate of the effect size of the intervention. Secondly it may have implications for best practice because if trial participation improves outcome (for whatever reason), it can be argued that routine treatment should be carried out under similar conditions to clinical trials [5]. This, of course, may have resource implications and the issue might be best considered in the context of maximising the non-specific effects of treatment. If our findings are replicated, the presence of a Hawthorne Effect should be considered when interpreting the results of treatment trials in dementia.

Hrobjartsson and Gotzsche [24] in a systematic review of placebo effects found placebo had minimal clinical effect compared to usual treatment. Braunholtz et al [4] systematically reviewed whether participation in clinical trials is beneficial or otherwise to patients, they defined separately treatment, protocol, care, Hawthorne and placebo effects, but remarked that it is difficult to tease such effects apart. The conclusion was that there is weak and inconclusive evidence of benefit of a Hawthorne Effect. A 'structured' review [5] compared outcomes in cancer patients treated within and outside clinical trials and found little evidence in support of a beneficial trial effect. Both these reviews focussed on cancer and both groups of authors agreed on the paucity of good quality evidence, considering the reason for this to be the methodological difficulties inherent in trying to make such evaluations in an ethical manner, with full and informed consent while maintaining comparable trial and non-trial groups.

In a randomised, non-blind comparison of intensive and minimal follow up of community dwelling patients suffering from mild-to-moderate dementia, we found evidence of a small 'Hawthorne Effect'. This may be due to effects other than being observed, such as learning effects of repeated exposure to the ADAS cog or greater familiarity with the research process. In any event, a non-treatment driven effect remains intriguing and clinically relevant, whatever the cause.