This paper describes the assessment of fidelity to treatment delivery of the ICONS systematic voiding programme (SVP). Assessment of fidelity to treatment delivery was based upon data collected from daily clinical logs completed by healthcare staff. Significant issues arose when the planned method of data analysis was trialled on the actual data collected: the voiding interval was often either missing or incorrectly documented and the schedule of proposed voiding times was either incomplete or incorrect. Both the voiding interval and schedule of proposed voiding times are prerequisites for undertaking the SVP. Therefore, the clinical logs had not been completed as intended. This meant that the a priori planned method for data analysis was unfeasible because it was often not possible to calculate the differences between actual and proposed voiding times. A new method was developed, in which key quality indicators were identified and an assessment made for each clinical log regarding its performance against each key quality indicator. This method appeared to work well and revealed overall a relatively low level of fidelity to treatment delivery.
The first objective of this study was to explore the feasibility of a method to assess fidelity to treatment delivery in the ICONS trial. The issues highlighted in the results from Phase I suggest that researchers should not assume that components of their intervention will necessarily be delivered in exactly the way they expect. In this trial, the voiding interval and schedule of proposed voiding times were not documented as expected. It is important for researchers to consider for every component of a complex intervention ways in which actual practice may deviate from the protocol, and take steps to minimise this. Researchers should also ensure that measurement of fidelity is sufficiently sensitive to capture meaningful variations to ‘core components’ of the intervention [23].
BCC recommendations state that interventions should be assessed against “a priori criteria” [1]. For the ICONS trial, a priori criteria were set, namely that actual voiding times should be within 30 min of their proposed voiding times. However, it was not foreseen that errors would arise in the documentation of both the voiding interval and the proposed voiding times, two prerequisites for both successfully undertaking the SVP and enabling the calculation of differences between actual and proposed voiding times. These issues rendered the planned method of fidelity assessment unfeasible. This demonstrates that, whilst it is important to develop plans for the assessment of fidelity to treatment delivery at the study design stage, it is necessary to remain flexible and adapt the planned method in light of unforeseen issues. Fidelity assessment should be an iterative process within trials [7].
The revised method of assessing fidelity to treatment delivery through the evaluation of clinical logs according to key quality indicators provided a feasible alternative. Despite the complexities of the information, data could be extracted from all clinical logs and used to assess fidelity to treatment delivery. However, this method of analysis has some drawbacks.
Clinical logs with a missing or incorrect voiding interval or an incomplete or incorrect schedule of proposed voiding times were not examined further. There were some clinical logs that contained only a small number of errors, for example just one or two incorrect proposed voiding times, but that otherwise demonstrated reasonably high levels of fidelity to treatment delivery, for example, by including a number of actual voiding times documented as within 30 min of their proposed voiding times. However, these data were not captured and consequently not included in the analysis. In a future trial, it would be useful to consider whether this method of data input could be amended to account for varying degrees of fidelity, rather than simply accepting or rejecting clinical logs at earlier stages of the process.
In this trial, documentation of delivery of the SVP was used as a proxy measure for fidelity to actual treatment delivery. Findings from other components of the process evaluation, for example interview data, suggest that healthcare staff did not always document everything that they did [17], and therefore it is likely that a method of fidelity assessment based upon healthcare staff recording of activity underestimates the true level of fidelity to treatment delivery. Similarly, it also appeared that at times healthcare staff found the amount of paperwork associated with the intervention onerous [17]; it is therefore worth considering whether the clinical logs could be simplified while still providing a structure for SVP delivery.
Some aspects of the clinical logs, such as documenting whether encouragement had been given, involved more subjective self-reporting by healthcare staff. There is therefore a risk of self-report bias in the use of clinical logs to measure fidelity, as self-report measures can often be inaccurate and may over-estimate fidelity [24–26]. Additionally, there is evidence from the clinical logs that selective documentation was sometimes undertaken, in which data were more likely to be missing if the required intervention component had not been achieved. For example, it appeared that sometimes healthcare staff would leave the answer to a question such as “Was encouragement given?” as missing rather than document “no”. In the future trial a more objective measure of fidelity may be appropriate, for example direct observation. However, this could affect intervention delivery as a consequence of providers being observed [26] and may not be feasible due to ethical issues inherent in observing patients receiving personal care.
A key limitation of the assessment of fidelity to treatment delivery in this trial was that data were not available regarding the total number of clinical logs per trial arm that should have been completed. It was not possible to calculate these numbers due to the complexities inherent in the delivery of the SVP. For example, sometimes patients would start and stop the SVP at various time points during their stay due to needing a catheter for a short period, or due to becoming temporarily medically unstable, and the dates for these variations were not consistently recorded. This meant that it was not possible to reliably calculate the theoretical numbers of clinical logs that should have been received. The first step in fidelity to delivery of the SVP was the completion of a clinical log and therefore a comparison of the numbers of clinical logs expected and received could add another stage to the assessment of fidelity to treatment delivery. In a future trial it will be important to develop a process that will ensure it is possible to calculate the number of clinical logs that should have been completed per sampling period, per trial arm.
A second objective of this study was to obtain preliminary evidence of fidelity to treatment delivery within the ICONS trial. Documentation of an appropriate voiding interval and a correct schedule of proposed voiding times (Stages 1 and 2) was often not done correctly. The original intention was to determine whether a patient had voided within 30 min of the proposed voiding time, but when this method proved unfeasible it was necessary to reconsider what constituted a reasonably valid method to assess ‘fidelity’. It was decided that the documentation of an appropriate voiding interval and a correct schedule of proposed voiding times were the first steps in the successful undertaking of the SVP and should therefore be the first steps in the assessment of fidelity to treatment delivery. The clinical logs that did not contain both these components (61.1 % in the Intervention sites; 68.1 % in the Supported Implementation sites) therefore lacked the necessary information for healthcare staff to be able to follow the SVP throughout the day and this constitutes a lack of fidelity. It is unclear why this happened, but one possible explanation is that healthcare staff misunderstood the principles of the SVP. In the future trial this needs to be addressed through detailed consideration of methods of implementation of the SVP, for example giving healthcare staff time to become familiar with and practice using the SVP before the trial intervention period begins [23, 27]. The future definitive ICONS trial will include a detailed plan for improved training provision, uptake and monitoring.
Whilst a key principle of the ICONS SVP is that voiding should occur within 30 min of a proposed voiding time, other aspects of treatment delivery were also important: healthcare staff should have given sufficient consideration to the most appropriate voiding interval for a particular patient on a particular day, and should also have attempted to undertake the SVP throughout the day, from 7.30 am until 9.30 pm. These components were only examined for clinical logs that achieved Stages 1 and 2. It was found that the length of the voiding interval did vary across days and across patients (data not presented), and this suggests that there was some individualisation of voiding interval. The mean number of proposed voiding times per clinical log was also relatively high for each arm, at around five for both arms, given that the voiding interval was almost always between 2-hourly and 3-hourly (as shown in Fig. 1) and that the SVP only ran from 7.30 am until 9.30 pm. This suggests that, on the whole, the SVP was undertaken through most of the day. Fidelity to these two components of the SVP therefore seems relatively high for clinical logs that passed Stages 1 and 2.
For clinical logs that achieved Stages 1 and 2, the percentage of occasions on which the actual voiding time occurred within 30 min of the proposed voiding time was around 55 %. A key principle of the SVP is that voiding occurs at a regular, consistent interval as part of a progressive voiding schedule [28, 29]; on a substantial proportion of occasions ICONS patients did not void at, or close to, the scheduled time. Fidelity to this principle of the SVP therefore appeared to be relatively low. One possible explanation for the relatively low fidelity is that returning to all patients on the SVP at their individualised times posed the most challenges to practice and was not always achievable. Additionally, there is evidence that staffing levels associated with weekend working can affect clinical outcomes [30], and it is possible that documentation of the SVP and fidelity to it could have been affected by different patterns of working/staffing at weekends versus in the week. However, exploring such patterns was beyond the scope of this paper. When fidelity to the intervention is low, the degree to which trial outcomes can be attributed to the intervention is compromised [31].
Fidelity to two ‘best practice’ components of the SVP, giving the patient encouragement and asking the patient if they were wet, was relatively high, although data for these components were extracted only from clinical logs that achieved Stages 1 and 2. Whilst these results are encouraging, they should be interpreted with caution as healthcare staff simply documented ‘yes’ or ‘no’ in relation to the two components, and, without asking for further detail to be provided about the manner in which these elements were provided, it is difficult to assess how accurate these measures are. There is also the risk of positive bias in self-reporting these more subjective elements of the programme [25].
A limitation of this study is that the analysis does not reflect the sampling strategy, which had two levels of clustering – by time and by patient. In our analysis we grouped together the clinical logs per site and then per trial arm and presented pooled results. The data were merged in this fashion in order to simplify the analysis and subsequent presentation. This method is unlikely to have affected our overall conclusions in relation to treatment fidelity, but would have implications if we were to perform any inferential statistics.
Key lessons have been learnt from the assessment of fidelity to treatment delivery in the ICONS trial. Real-life practice may often deviate from the protocol: the ICONS protocol for the SVP was often not followed and this highlights how important it is for researchers to consider how real-life practice may deviate from what was intended and explore ways in which to minimise this. An iterative approach to fidelity assessment is important in trials of complex interventions: ideally, fidelity data should be examined and analysed whilst the trial is ongoing, allowing issues to be addressed as the trial progresses. Different ways of defining and assessing fidelity should be considered: trials of complex interventions should consider the relative merits of subjective, self-reported measures of fidelity versus more objective measures, such as direct observation, and select the method most feasible and appropriate for the study. More comprehensive implementation strategies will be important in a future evaluation trial in order to improve fidelity, given that the overall fidelity to treatment delivery in the ICONS feasibility trial appeared to be relatively low.
