Conceptualisation of fidelity and guiding framework
To guide our methodology we used the NIH BCC framework [27, 28] and were guided by the work of Toomey et al. [26]. We therefore aimed to consider how to both enhance and assess fidelity for each of the five NIH BCC domains. As described above we conceptualised fidelity as a process, such that fidelity at earlier phases was assumed to impact on fidelity at latter stages.
Borelli (2011) [22] has suggested comprehensive strategies to enhance fidelity within each of the five domains (see Fig. 1). We used these from the outset (in developing our grant application) as a guide for intervention development. An iterative process was undertaken by the lead author supported by a multidisciplinary team (consisting of health psychologists, academic general practitioners, public health professionals, qualitative researchers and health service researchers) whereby the evolving intervention was reviewed and refined in light of recommendations. The intervention was reviewed by the team post piloting to consider if further enhancement of fidelity was possible. Figure 2 illustrates the key enhancement strategies under each of the five fidelity domains and supplementary table 1 provides a full description of strategies and links these to recommendations by Borrelli et al. [22] of how fidelity should be enhanced.
In order to standardise delivery, and enhance fidelity in the control group, all control participants were provided with the same consistent educational information in addition to receiving a referral for PR assessment. This comprised the British Lung Foundation (BLF) publicly available DVD: ‘Living with COPD’/ ‘Stay Well Stay Active’ and BLF COPD information on exercise and a PR booklet which provides advice in accordance with national guidelines [29] to facilitate the control group receiving best standard care.
To target treatment differentiation, as mentioned above the second component of fidelity [22], and to avoid treatment burden, the trial design specifically excluded individuals who were currently receiving a psychological intervention, or who had received one within the preceding six months.
Assessment of fidelity domains
Design
To monitor intervention and assessment delivery and detect protocol deviations a case report form was developed in line with SPIRIT recommendations [30] and has been reported previously [16]. In brief this form collected when each activity was conducted, by whom, and the duration of the task. In this way any deviations from protocol were detected.
Training
Hiring facilitators
A job specification and description for the Facilitator role was developed. The role was advertised through social media (i.e., Twitter) and respiratory networks such as the Association of Respiratory Nurse Specialists, and at events such as the Primary Care Respiratory Society annual conference. Interested professionals were invited to submit a curriculum vitae and a personal statement to the study team, and those fulfilling criteria (e.g. registered respiratory professional; able to commit a day a week to TANDEM) were invited to a structured telephone interview. The interview was conducted by one of the chief investigators with a second interview with a health psychologist if deemed necessary. All potential Facilitators had to demonstrate a commitment to a biopsychosocial approach to treatment, willingness to travel and readiness to see participants in their homes before a place on the training programme was offered. A study log was kept of the full recruitment process.
Training facilitators
A standardised initial training programme was developed with slide sets, demonstration videos and specified exercises for use by all trainers. All training sessions were video recorded to enable comparison with the protocol if resources allowed. In addition, as skills often develop and must be maintained over time, we required all Facilitators to have regular individual supervision of approximately thirty minutes every fortnight. It was possible for Facilitators to use their audio-recordings of sessions if desired, although this was not a specific requirement of supervision. The number of planned supervision sessions received was measured to indicate dosage of supervision received.
In order to assess Facilitator therapeutic competence these post-initial training, all Facilitators underwent an individual face-to-face, video-recorded assessment with a trained actor playing the role of a patient. The task was to conduct an initial formulation, using a cognitive behavioural approach as taught in the training, and present this back to the patient. The same actor was used for all assessments and they received training on the scenario to be acted before delivery. Use of standardized role plays in training for cognitive behavioural approaches has been recommended previously [31, 32].
The video-recording of each Facilitator was independently coded by two psychologists (LS & VW) using the Cognitive First Aid Rating Scale (CFARS) [33]. The CFARS is a 10 item scale based upon the Revised Cognitive Therapy Rating Scale (CTS-R) [34] but adapted to be more appropriate for health practitioners who are delivering a cognitive behavioural approach, rather than full cognitive behavioural therapy. The CFARS was developed, and has demonstrated reliability and validity, in the context of palliative care practitioners who had received a brief training course on cognitive behavioural skills [33]. This was judged appropriate to TANDEM which provided brief training for a cognitive behavioural approach with a population with physical health difficulties, in line with the previously defined competencies [35, 36]; in addition the CFARS had been used with a COPD population [14, 15].
For each of the 10 items on the CFARS there is a 7-point scale (0 = incompetent, to 6 = expert) giving a 60-point total. We omitted item 9 ‘application of appropriate change techniques’ from our assessment scoring as we did not require Facilitators to show application of change techniques within the evaluated role-play. In line with previous guidance we required a minimum total score of 50% (raw score 27) and no item falling below 2 as an indication of sufficient competence [33]. Where there was disagreement between coders which could not be resolved through discussion typically the mean score was taken, or if necessary, a third coder (SCT) arbitrated.
All Facilitators received one to one, face to face, feedback on their assessment videos to enhance their therapeutic competence and fidelity to treatment delivery [37].
Treatment delivery
All sessions delivered by TANDEM Facilitators were audio-recorded (with participant permission) and a random sample of full cases (i.e. all sessions delivered to that participant) coded for fidelity of treatment delivery. If Facilitators delivered the intervention to nine or fewer participants one full case was randomly selected from their first five cases for review. If Facilitators delivered to ten or more participants then two full cases were reviewed (one randomly selected from their first five cases and a second randomly selected from their 10-15th participant to allow for possible change over time to be seen). Overall, we intended to assess 10% of cases.
All of the sessions within the randomly selected cases were coded by a psychologist (VR) trained in behavioural interventions who was independent of the study team. Twenty percent of these were second coded by a member of the study team (LS) to ensure quality of the main coder; however, to ensure consistency, the scores of the primary coder were used for analysis.
Both therapeutic competence and treatment adherence should be assessed in the fidelity evaluation of treatment delivery [23]. In order to assess therapeutic competence the full 10-item CFARS was used and treatment adherence was measured by a TANDEM specific treatment adherence scale.
Development of the TANDEM Specific treatment adherence scale
A review of the literature did not identify a suitable validated tool for assessment of adherence to treatment delivery. This was expected as TANDEM is a novel and tailored intervention, hence it was necessary to create a bespoke checklist which was developed in line with recommendations by Walton et al. [25] which describes a 5-step process involving reviewing previous measures (as noted above this did not identify a suitable measure), analysing the intervention, developing a coding system, checking wording with the team and piloting.
Three individuals (LS, VW, ST) completed a comprehensive analysis of the TANDEM Facilitators Handbook (study manual) and all other patient materials to identify the core and topic specific elements of the delivered TANDEM intervention. Thus, an initial coding framework was developed. An attempt was made at this stage to code each task according to the Michie Behaviour Change Taxonomy v1 [38], however this was not successful given insufficient cognitive techniques (e.g. challenge thoughts) in the taxonomy.
In understanding adherence there is a further distinction that needs to be recognised, that between behaviours/tasks that should be delivered, and content/information that should be delivered. As TANDEM is a tailored intervention, certain behaviours (which we describe as core tasks) were required every session, however the content i.e. topic this related to, was dependent on the needs of the individual (see Steed et al. [17] for further discussion on tailoring within the TANDEM intervention). The TANDEM specific treatment adherence scale was therefore split into two sections i) for core tasks that were required repeatedly e.g., set agenda, discuss homework, refer to hot cross bun (this is a CBT term referring to a formulation based on how thoughts, feelings, behaviours and symptoms inter-relate) and deliver intervention based on the hot cross bun and ii) topic-specific content e.g. discuss why exercise is important in COPD, discuss prevalence of anxiety and depression. To assess quality of delivery of each core task (section i) a 5-point Likert scale ranging from ‘not delivered at all’ to ‘delivered completely’ was agreed. As content (section ii) could only be considered as delivered or not delivered this was rated on a 0–1 scale. The treatment adherence scale was then presented to the process evaluation team for consideration of wording and understanding of items. It was iteratively piloted by the fidelity team on approximately 10 h of recordings until a workable and consistent treatment adherence scale was produced. One key change was the simplification of the 5-point quality scale for core tasks to a 3-point scale due to complexity in rating the 5 point scale.
Final TANDEM treatment delivery fidelity tool
The final method to assess treatment delivery comprised an initial section on therapeutic competence (as measured by the CFARS scale) and the specific treatment adherence scale with measurement of core tasks on a 1–3 scale (1 = not at all, 2 = partial, 3 = complete) and measurement of topic-specific content on a 1 = delivered, 0 = not delivered scale.
Analysis of the final TANDEM treatment delivery fidelity tool
It is intended that therapeutic competence will be scored by using the total competence scores from CFARS. Therapeutic competence will be analysed for total item mean and standard deviations, as well as median and interquartile range on each item of the CFARS. The proportion of Facilitators achieving fidelity (a score of > 30) across all of the ten items can also be reported. Treatment specific adherence will be scored separately from therapeutic competence. For each core task we will report the percentage of cases achieving high fidelity (> 80%). For topic specific adherence it is necessary to account for tailoring in scoring. Therefore, where items are not applicable (e.g., acceptance not discussed if a participant had good acceptance of their COPD) these will be omitted and the final score adjusted accordingly. The final score reflects the percentage of tailored content that should have been delivered that was delivered. The proportion of content that reaches greater than 80% fidelity is then calculated. A list of any ‘non TANDEM’ content delivered will also recorded.
Following piloting a fidelity handbook (available from the authors upon request) was produced with guidance on how each aspect of fidelity would be evaluated and specific detail on coding of the intervention specific treatment adherence scale was provided.
Treatment receipt
To measure treatment receipt, the number of sessions (and topics) each participant received, and all intervention materials provided, such as TANDEM handouts and BLF leaflets, was recorded. The minimum specified dose of the intervention was receipt of at least two CBA sessions and the handout on mood and COPD.
Interviews
Further information on treatment receipt was examined in qualitative interviews with Facilitators and patients. These were conducted as part of the process evaluation [21]. Patients were purposively sampled to reflect individuals in the TANDEM intervention arm who had completed versus dropped out of the CBA sessions, and attended versus not attended PR sessions. We aimed for a target of five participants per cell. Interviews were conducted after the 6 month assessment and were in person or telephone based on participant preference. The topic guide focussed on current experience of COPD/breathlessness, experience of being in the TANDEM study, relationship and working with the TANDEM Facilitator, experience of attending PR, suggested improvements to the TANDEM experience, perspectives on receiving TANDEM as part of routine care.
All facilitators were invited to participate in an individual interview with a target sample of fourteen interviews sought from different professional groups where possible. The number of patients seen by Facilitators (1–4; 5–8; 9 +) was also a factor in sampling. The main topics were training sessions, CBA sessions with patients, supervision, professional identity, perspectives on post-trial implementation.
Treatment enactment
No direct fidelity measure of treatment enactment and whether patients used the skills delivered was possible, however changes in key measures such as attendance at pulmonary rehabilitation and social outcomes were considered proxy measures of treatment enactment. In addition, qualitative interviews (as outlined above) with patients explored whether patients were enacting the skills learnt in the intervention.