- Open Access
- Open Peer Review
Novel study design to assess the utility of the copd assessment test in a primary care setting
BMC Medical Research Methodology volume 13, Article number: 63 (2013)
The quality of a consultation provided by a physician can have a profound impact on the quality of care and patient engagement in treatment decisions. When the COPD Assessment Test (CAT) was developed, one of its aims was to aid the communication between physician and patient about the impact of COPD. We developed a novel study design to assess this in a primary care consultation.
Primary care physicians across five countries in Europe conducted videoed consultations with six standardised COPD patients (played by trained actors) which had patient-specific issues that the physician needed to identify through questioning. Half the physicians saw the patients with the completed CAT, and half without. Independent assessors scored the physicians on their ability to identify and address the patient-specific issues, review standard COPD aspects, their understanding of the case and their overall performance. This novel study design presented many challenges which needed to be addressed to achieve an acceptable level of robustness to assess the utility of the CAT. This paper discusses these challenges and the measures adopted to eliminate or minimise their impact on the study results.
The quality of a consultation provided by a physician can have a profound impact on the quality of care and patient engagement in treatment decisions . The most effective consultations are those in which doctors most directly acknowledge and respond to patients’ problems and concerns . Limited time for consultations forces primary care physicians to focus on the fundamental problems, and patients often do not present all of their problems and concerns in a consultation, which can lead to poor consultation outcome . Thus, tools to improve the communication between patient and physician have the potential to improve consultation outcomes.
The COPD Assessment Test (CAT) is a new, patient completed questionnaire designed to provide a simple and reliable measure of health status in a patient with COPD . The CAT questionnaire is formed of 8 questions covering the most burdensome symptoms of COPD. The CAT has undergone robust validation testing, and has been shown to have very similar properties to the more complex health status questionnaires, the St George’s Respiratory Questionnaire (SGRQ)  and the Chronic Respiratory Questionnaire . However, it is shorter, making it suitable for routine clinical use.
In general, once a patient reported outcome measure (PROM) has been developed and validated, there needs to be an assessment of whether it improves patient care . When the CAT was developed, one of its aims was to aid the communication between physician and patient on the impact of COPD, and thus aid physicians to optimise the patients’ care . However, to date, this aspect of the CAT has not been tested. We therefore set out to conduct a study to assess the impact of a PROM, the CAT, on physician-patient communication.
Such a research question presents several challenges to researchers: How to ensure a fair comparison between arms; how to assess the impact of the PROM; and how to conduct a study large enough (both sample size and geographical spread) to give robust and generalisable results? This paper discusses the feasibility of a novel study which was designed to address many of these challenges.
This was a single visit, randomised (1:1) open, parallel group study (Figure 1). The UK’s National Research Ethics Services confirmed ethical approval was not required, and physicians consented to their participation in the study.
The CAT had been launched by the time the study commenced and was already being used by some secondary care respiratory specialists. The study therefore focused on the primary care setting to aid recruitment and to assess the CAT where the impact of its use is unknown. Physicians across five European countries (United Kingdom (UK), Ireland, France, Germany and Austria) were contacted by a local market research agency and screened by telephone interview. Those reporting experience of managing COPD patients (at least three), but not of using the CAT, were invited to participate in a physician-patient communication study. A total of 168 physicians agreed to participate in the study, of which three failed to attend their allotted filming session. The selection of the physicians was not random due to the practicalities of the study, which required twelve physicians who were willing to participate in the study and within travelling distance of suitable filming facilities. Potential biases of location (rural versus urban-based physician) and specific interest or expertise in respiratory diseases were considered. Location (rural versus urban within an individual country) is unlikely to affect physician communication skills, but knowledge of COPD and the CAT may have done. The screening questions therefore also covered experience of managing asthma, use of Hospital Anxiety and Depression Score, and Framingham risk such that physicians were unaware that the study was about COPD and the CAT until they attended their allocated sessions. Physicians were asked again about whether they had a working knowledge of the CAT when the attended the filming session to identify any physician who had used the CAT following the screening interview. A total of nine physicians reported experience of using the CAT on the day of filming. These physicians were excluded from the primary analysis, along with any physicians who reported insufficient experience of managing COPD patients on the day of filming.
Two or four geographically-spread locations were used in each country, depending on the number of physicians recruited (Table 1). Physicians attended the sessions in groups of five or six, with two groups recruited in each location. Each group was randomised to see the patient with (“CAT+” arm) or without (“no CAT” arm) the completed CAT during the consultation. A 2-level hierarchical design was used, with a randomisation block size of two, without stratification, such that one group was randomised to each arm at each location. The recruiters were blinded to the randomisation.
Physicians received brief training on COPD and those in the CAT+ arm also received brief training on the CAT. The training was provided in the form of reading material, which for the CAT+ groups included background information on the tool, how to interpret overall scores and how to identify specific areas of concern for the patient. Participants were given around 20 minutes to complete the training, and were encouraged to discuss the information between them. This level of training may be inadequate to provide sufficient understanding for physicians to change their behaviour based on the CAT results, but is often reflective of time and training available to primary care physicians in real life. Practical educational approaches generally encourage more behavioural changes , but this would not have been practical to implement in this study. No specific guidelines were provided on actions to take based on the CAT score as such advice was not available at the time.
Physicians then undertook videoed consultations with six standardised COPD patients. Each physician conducted all their consultations either with or without the CAT so that non-CAT consultations were not influenced by questions / practices adopted in consultations with the CAT. After the consultation, physicians were asked to record to camera their impression of the case and recommended course of action – as if they were making notes in the patient’s records.
Because of the practicalities of the study, the physicians were meeting the patient actor for the first and only time, which may have driven a different kind of consultation compared with consecutive consultations. A physician first needs to build a relationship with the patient, and then become acquainted with all his/her diagnoses, of which COPD may be just one (and not always the most important). Additionally, a maximum time was allowed for each consultation, with discussions being interrupted and asked to finish after ten minutes, limiting the time the physicians had to build that relationship. However, this is representative of real-life time pressures in the primary care setting.
Interestingly, during validation work on the Clinical COPD Questionnaire it was noted that physicians changed their practices with experience of using the PROM . If this phenomenon is representative, the utility of CAT may be different when the relationship between the patient and physician is more established, and in which the physician has a better understanding of the patient’s history and situation.
Standardised patient cases
Pendleton’s tasks  are often used to assess the quality of a consultation and have previously been used to assess the communication of a healthcare provider with their patient . The tasks include “understanding the patient issues”, and “involving the patient in the choice of action”. Similar descriptions of what a good consultation should achieve have been described by Howie  and Mauksch , which include skills such as “topic tracking”, “emphatic response to cues” , “patient priorities” and “sharing decision making” . The expected impact of CAT is on the specific COPD content of the discussion, rather than physicians’ values and personality. We therefore decided to assess the ability of the physicians to identify ‘patient issues’, COPD specific issues and management of these issues with and without the CAT. These issues represented aspects of the patients’ medical history that needed to be addressed by the physicians, such as depression, review of lifestyle or therapy, and compliance issues.
The authors (including active clinicians from a variety of different environments) constructed six cases to test the utility of CAT across a range of scenarios, disease burden and COPD patient issues (Table 2, Figure 2). Some of the social aspects of each case were amended slightly for each country, to ensure they resonated with the physicians. The CAT scores of each case were independently verified. Each case included four or five ‘patient issues’. While this allowed us to test the utility of the CAT in the areas we believed it may impact, they may not have represented valid patient scenarios, for example the presence of several issues in each case. Alternative methods of creating the standardised cases, such as selecting actual COPD patient cases, and have the actors portray those individual cases would have been equally valid.
Actors (n=20) fluent in the relevant language performed the role of the COPD patients. Fluency was confirmed by native speakers. Where language skills allowed, actors were used in multiple countries (seven actors only performed in one country, eleven actors performed in two countries, and two actresses performed in 3 countries), increasing the consistency with which the cases were presented to the physicians. Many of the actors were either native to, or had spent significant time living in the countries where the study was conducted. In addition, they spent time with local COPD patients to ensure they portrayed a COPD patient as socially and linguistically accurately as possible. The actors were trained in clinical aspects of COPD, details of their case and not to proactively raise their ‘patient issues’ with the physician, who needed to uncover them by direct enquiry.
Using actors to portray standardised cases provides confidence that differences seen in a study are due to the intervention, rather than differences in case presentations, especially as the study needed to run over several countries and several months. The use of trained actors to play standardised patients has been successfully used before in other studies assessing the behaviours of physicians [14, 15], and by the Royal College of General Physicians in the assessment of primary care physicians in the UK. There is often a concern that simulated surgeries compromise the realism of the case, and the behaviour of the physician may be altered by knowledge of being observed (Hawthorne effect). However, physicians frequently report face-validity of patient actors , and the use of simulated consultations for the testing of an instrument’s properties is widely accepted and considered useful . Additionally any impact of loss of realism could reasonably be expected to be similar for physicians in both arms.
Independent assessors (n=24) with experience of assessing physician performance (e.g. through primary care qualification assessments, physician education programs), had experience of COPD management, but who were not part of the development of the CAT, assessed the videoed consultations. German and UK assessors also assessed Austrian and Irish physicians respectively (Table 1).
All independent assessors were trained by HM through individual or small group telephone discussions. Assessors scored each physician on whether they identified and addressed the relevant ‘patient issues’ and reviewed ten standard COPD aspects such as cough and smoking history (0 = none, 1 = some, 2 = high). Scores were captured on an online score sheet, and the patient issues scores (sub-score A, out of 20) and COPD review scores (sub-score B, out of 20) were calculated. A global score of sub-score A plus sub-score B was calculated as a composite endpoint (Figure 3). A ‘some’ score for sub-score A and sub-score B was given when the physician had gained a superficial appreciation of the patient issue; while a ‘high’ score was awarded when the physician asked a number of insightful questions to fully understand the issue, the burden on the patient and how the situation might be improved. The independent assessors also rated the physician’s understanding of the case from their description to camera (“understanding score”: poor, acceptable, accurate) and their overall performance (very poor, poor, good, very good). Similar to the sub-score grading scheme, an ‘acceptable’ understanding score was given when the physician had understood the key elements of the case, while an ‘accurate’ score was awarded when the physician demonstrated a deep and full understanding of the case. Overall performance summarised the whole consultation including, for example how the physician behaved in the consultation, empathy etc.
Although the assessors from UK and Germany were trained to judge Irish and Austrian physicians in the same way as UK and German physicians, it is possible that their assessment differed due to cultural or health care system differences though it is unlikely that any impact from these differences would be different in the two arms.
These assessment criteria were prospectively created specifically for this study to test the utility of the CAT in the areas that we believed it might impact. They were not tested for assessor interpretation, inter- and intra- assessor consistency of marking or sensitivity (relative importance of each item on the quality of the consultation). The global score of sub-score A plus B was chosen as the primary endpoint as this is the most conservative endpoint, so we were not setting up the study for guaranteed success.
Alternative methods of assessing physician behaviour have been described previously [16–18]. The Consultation Quality Index-2 (CQI-2) measures the holistic quality of the consultation, including patient enablement, empathy and patient satisfaction , and so may not have been sufficiently sensitive to measure the impact upon a consultation that we expected the CAT to achieve. Similarly, the Roter’s Interaction Analysis System (RIAS)  focuses on the physicians psychosocial behaviour and quality of management of the patient:physician relationship, and as such may miss the expected impact of the CAT. The Medication-Related Consultation Framework (MRCF) tool assesses medication related consultations and focuses on the discussion content between a patient and pharmacist . This tool includes sections on “data collection and problem identification” and “actions and solutions”, with the questions primarily assessing the patient:pharmacist discussion around medication use. The questionnaire is quite long (46 items), aimed at pharmacists, and - as medication compliance is only a small part of primary care consultations – would miss other discussion that the CAT may influence. We therefore felt that none of these established consultation assessment tools would have provided a sound basis of assessment of the impact of the CAT in the primary care setting.
Actor feedback is frequently used in studies of physician behaviour and is often found to be predictive of real patient feedback , with acceptably small variance between different actors . Therefore the actors provided feedback following each consultation on whether they felt the physician addressed their issues by scoring 5 questions (such as “I felt the physician understood my issues”, “The physician helped me address my issues”) as “no”, “yes, but unsatisfactory” or “yes”. These questions were based on a similar questionnaire used to gather feedback on the healthcare service provided to UK COPD patients. The actors also provided feedback on the length of the consultation, and their overall satisfaction.
We conducted a pilot study to confirm the feasibility and inform the sample size of the study. The methodology as described above was used with ten UK Physicians. Only minor technical issues were identified with the recruitment of physicians, consultation filming, and assessments, which were resolved in time for the main study.
The assessment of the physicians in the pilot study was conducted by some of the authors (SH, JR, GN and DL). The mean global score across the cases ranged between 12.8 and 19.0 in the no CAT arm, and 16.0 and 25.3 in the CAT+ arm (Table 3); and the difference between the arms, ranged between 3.2 and 12.3. The percent of “good” or “very good” consultations rose by up to 40 percentage points in the CAT+ arm.
The results from the pilot study indicate the assessment items and scoring system employed were sufficient to identify differences between good and poor consultations.
Assessor scoring variability
A large number of independent assessors, from 3 different countries, were required to assess the volume of consultations which introduced significant complexity to the analysis of the study. Additionally, it was not possible to blind the assessors to the presence of the CAT from the consultation, which may have biased their scoring. Alternative methodologies, such as audio taped consultations, would not necessarily have resolved this. We conducted a benchmarking exercise to gauge whether these aspects would impact upon the final result of the study. The intention was to assess the variability in the assessors’ scores and to identify any outlying assessor(s). Each assessor therefore reviewed two high-scoring and two low-scoring consultations from the pilot study, with and without CAT.
The actual scores given by each assessor varied significantly, and a difference in mean total scores across the countries were identified; however the ranking and differences between the high and low scoring assessments were generally consistent (Figure 4). The inter-assessor reliability was analysed. The intraclass correlation coefficient (ICC) was 0.68 [95% CI 0.38; 0.97], p<0.001, and no atypical assessor was identified.
The benchmarking analysis also suggested that ‘case’ was as much a factor in the different scores as the assessor – i.e. variability in the actual score across the assessors was not obscuring the difference between the cases. To account for these effects, assessor and case were included as adjustment variables in the statistical analyses.
The primary endpoint for the study was mean global score (combined sub-scores A and B, which had a scale of 0 (worst) to 40 (best)). For the one case which only had four patients issues, compared to the other cases with five, sub-score A results were scaled up (multiplied by 1.25) to make it comparable. Since the primary endpoint had not been previously trialled, the sensitivity or potential magnitude of difference between the arms was unknown. The investigators agreed that a difference of 10% (i.e. 4 points) between the arms would be convincing as a true difference. The pilot study data indicated that this difference was realistic, and that a conservative estimation of standard deviation was 12 points. Therefore, in order to achieve 90% power to detect a difference of at least 3 points in global score in the main study 752 consultations were needed, allowing for 10% missing data. No adjustments for confounders were included. Given the number of patient consultations created at each session and location, the number of planned patient consultations was 960.
Sub-scores A and B, were also designated as secondary outcomes; all scores were analysed using repeated-measures analysis of variance with a linear mixed model. Differences in items of global score, and of sub-scores A and B were tested with a chi-square test or a Fisher’s exact test. ‘Understanding score’ and ‘overall assessment’ grading were analysed with a generalized estimating equations model. To account for the assessor effect, as identified in the benchmarking exercise, both assessor and case were included as adjustment variables in the models. The order in which physicians saw the cases was also included to account for any training effect. A secondary analysis where physician characteristics, including country, were explored as potential confounders was conducted to confirm the generalisability of the study results across multiple countries. Similarly a sensitivity analysis including all physicians who participated in the study was conducted. The statistical analysis was conducted using SAS v9.1.
This novel study was designed to assess the impact of the CAT on the behaviour of a physician in their consultation with a COPD patient. At the time the study was run there was no guidance available on managing COPD patients based on their CAT scores. As such we were not able to assess whether the CAT impacted the therapy and management choices made by the physician and patient. This situation has subsequently changed as CAT is now forms part of the Global Initiative for Chronic Obstructive Lung Disease (GOLD) 2011 COPD assessment framework, alongside spirometry and exacerbation history . Additionally, this methodology would not be able to evaluate the impact of CAT on long-term management and outcomes. Further studies on the CAT specifically are warranted.
To ensure a fair comparison between the arms in this study, we needed to take steps to ensure that potential confounders such as underlying knowledge of the physicians, suitability of the assessment criteria, and variability of the assessors were as controlled as possible. Although there are still some limitations, the study design allows standardisation of the cases and analysis, and could be used to assess the utility of other PROMs by direct observation of clinical practice.
COPD Assessment Test
Chronic Obstructive Pulmonary Disease
Consultation Quality Index-2
Intraclass correlation coefficient
Medication-Related Consultation Framework
Patient reported outcome measure
Roter’s Interaction Analysis System
St George’s Respiratory Questionnaire
Coulter A, Ellins J: Patient-focused interventions: A review of the evidence. 2006, Oxford, UK: Picker Institute Europe; Health Foundation
Freeman G, Horder JP, Howie JGR, Hungin AP, Hill AP, Shah NC, Wilson A: Evolving general practice consultation in Britain: issues of length and context. BMJ. 2002, 324: 880-882. 10.1136/bmj.324.7342.880.
Barry CA, Bradlet CP, Britten N, Stevenson FA, Barber N: Patients’ unvoiced agendas in general practice consultations: qualitative study. BMJ. 2000, 320: 1246-1250. 10.1136/bmj.320.7244.1246.
Jones P, Harding G, Wiklund I, Berry P, Leidy N: Improving the process and outcome of care in COPD: development of a standardised assessment tool. Prim Care Resp J. 2009, 18: 208-215. 10.4104/pcrj.2009.00053.
Jones PW, Harding G, Berry P, Wiklund I, Chen W-H, Leidy NK: Development and first validation of the COPD Assessment Test. Eur Respir J. 2009, 34: 648-654. 10.1183/09031936.00102509.
Jones P, Harding G, Wiklund I, Berry P, Tabberer M, Yu R, Leidy NK: Tests of the Responsiveness of the Chronic Obstructive Pulmonary Disease (COPD) Assessment Test™ (CAT) following acute exacerbation and pulmonary rehabilitation. Chest. in press
Levine MN, Ganz PA: Beyond the development of quality of life instruments: Where do we go from here?. J Clin Oncol. 2002, 20: 2215-2216.
Greenhalgh J, Long A, Flynn R: The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory. Soc Sci Med. 2005, 60: 833-843. 10.1016/j.socscimed.2004.06.022.
Kocks JWH, Kerstjens HAM, Snijders SL, de Vos B, Biermann JJ, van Hengel P, Strijbos JH, Bosveld HEP, van der Molen T: Health status in routine clinical practice: validity of the clinical COPD questionnaire at the individual patient level. Health Qual Life Outcomes. 2010, 8: 135-10.1186/1477-7525-8-135.
Pendleton D, Tate ST, Havelock PP: The Consultation - An approach to learning and teaching. 1993, Oxford: Oxford University Press
Efraimsson EO, Klang B, Larsson K, Ehrenberg A, Fossum B: Communication and self-management education at nurse-led COPD clinics in primary health care. Patient Educ Couns. 2009, 77: 209-217. 10.1016/j.pec.2009.03.033.
Howie JGR, Heaney D, Maxwell M: Quality, core values and the general practice consultation: issues of definition, measurement and delivery. Fam Pract. 2004, 21: 458-468. 10.1093/fampra/cmh419.
Mauksch LB, Dugdale DC, Dobson S, Epstein RM: Relationship, communication and efficiency in the medical encounter. Arch Intern Med. 2008, 168: 1387-1395. 10.1001/archinte.168.13.1387.
Brown RF, Butow PN, Henman M, Dunn SM, Boyle F, Tattersall MHN: Responding to the active and passive patient: flexibility is the key. Health Expect. 2006, 5: 236-245.
Gerner B, Sanci L, Cahill H, Ukoumunne OC, Gold L, Rogers L, McCallum Z, Wake M: Using simulated patients to develop doctors’ skills in facilitating behaviour change: addressing childhood obesity. Med Educ. 2010, 44: 706-715. 10.1111/j.1365-2923.2010.03686.x.
Abdel-Tawab R, James DH, Fichtinger A, Clatworthy J, Horne R, Davies G: Development and validation of the Medication-Related Consultation Framework (MRCF). Patient Educ Couns. 2011, 83: 451-457. 10.1016/j.pec.2011.05.005.
Benzing JM, Dronkers J: Instrumental and affective aspects of physician behaviour. Med Care. 1992, 30: 283-297. 10.1097/00005650-199204000-00001.
Mercer SW, Howie JGR: CQI-2 – a new measure of holistic interpersonal care in primary care consultations. Br J Gen Pract. 2006, 56: 262-268.
Lurie SJ, Mooney CJ, Nofziger AC, Meldrum SC, Epstein RM: Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments. Med Educ. 2008, 42: 662-668. 10.1111/j.1365-2923.2008.03080.x.
Global Initiative for Chronic Obstructive Lung Disease (GOLD): Global Strategy for the diagnosis, management, and prevention of COPD. 2011, Available from http://www.goldcopd.org/
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/13/63/prepub
The authors would like to thank the film crew, actors, assessors and physicians who participated in the study, James Dodd for supporting the actor training, and those COPD patients who helped shape the study design and train the actors. Funding for this study was provided by GlaxoSmithKline.
KGJ has acted as a consultant for and spoken on behalf of GSK, AZ, Chiesi, Boehringer Ingelheim, MSD, Muni Pharma/Napp, Allmirall, Novartis, Sandoz. SH has received speaker fees, travel grants and honoraria for advisory board from AZ, Boehringer Ingelheim, Chiesi, GSK, MSD, Napp, Novartis and Nycomed. PK has received honoraria for advisory board, travel grants and speaker fees from AZ, Boehringer Ingelheim, Chiesi, GSK, MSD, Novartis, Nycomed RE has received honoraria for advisory board, travel grants and speaker fees from AZ, Boehringer Ingelheim, Chiesi, GSK, MSD, Novartis, Nycomed RDN reports no conflict of interest JR has received speaker fees, travel grants and honoraria for advisory board from AZ, Boehringer Ingelheim, Chiesi, GSK, MSD, Novartis, Teva. PJ has received fees from pharmaceutical companies, including GlaxoSmithKline, for speaking at meetings and participating in advisory board meetings, and has received support for research from pharmaceutical companies, including GlaxoSmithKline. HM, GN and DL are employees of GlaxoSmithKline, who funded this study.
All authors heavily inputted into the design of the study, from designing the patient cases and assessment criteria to defining the endpoints and statistical analysis used in the study. HM wrote this paper and all other authors have critically reviewed and approved it. Additionally, KGJ, SH, PK, RE, JR, GN and DL assisted in the assessment of the consultations, and PJ assisted design the COPD patient cases. All authors read and approved the final manuscript.