Development of an integrative coding framework for evaluating context within implementation science

Abstract

Background

This research aims to explore an identified gap in implementation science methodology, that is, how to assess context in implementation research. Context is among the strongest influences on implementation success but is a construct that is poorly understood and reported within the literature. Consequently, there is little guidance on how to research context. This study addresses this issue by developing a method to account for the active role of context during implementation research. Through use of a case study, this paper demonstrates the value of using our context coding framework.

Methods

The developed context coding framework was guided by the sub-elements of the Consolidated Framework for Implementation Research (CFIR). Employing a constructivist approach, this framework builds on the CFIR and enables a deeper exploration of context at multiple levels of the health system. The coding framework enables the collation of various data sources such as organisational reports, culture audits, interview, survey, and observational data. It may be continuously updated as new data emerge and can be adapted by researchers as required. A pre-existing rating criterion has been integrated to the context coding framework to highlight the influence and relative strength of each contextual factor prior to and during implementation.

Results

It is anticipated that the context coding framework will facilitate a standardised approach to assessing context. This will provide a deeper understanding of how to account for the influence of context, ultimately providing guidance that should increase the likelihood of implementation success. The coding framework enables implementation progress to be monitored, facilitating the identification of contextual changes and variations across settings at different levels of the healthcare system. It is expected this framework will inform the selection of appropriate implementation strategies and enable the monitoring of such strategies regarding their impact on local context.

Conclusions

This research contributes to the extant literature by advancing methodologies for the consideration and assessment of context in implementation research. This context coding framework may be used in any setting to provide insight into the characteristics of particular contexts throughout implementation processes.

Background

It is widely acknowledged that the uptake of evidence-based healthcare interventions is challenging [1,2,3,4] with it being reported that only 50–60% of care in the past decade has been delivered in line with the best available evidence [5]. It is suggested that this is due to a naïve assumption that the implementation of a new intervention will be self-evident, disseminating automatically through a linear ‘pipeline model’ whereby an intervention moves from the laboratory, into an effectiveness trial before sustained application in routine practice [4, 6, 7]. However, this simplistic model does not account for the complexity associated with health systems. Context has been cited as a key concept impacting the translation of evidence into practice [8,9,10,11] and accounting for it is critical in interpreting and generalising findings [12,13,14]. Yet despite this noted importance, context is often overlooked by researchers within the field of implementation science which has led to an insufficient understanding of this construct.

This poor understanding has been attributed to the variability in how context is defined and assessed across studies [6, 15,16,17,18,19,20]. However, Rogers et al.’s [20] systematic review confirms that inconsistencies exist in the extant literature when defining context. This review also identifies commonalities across the papers included to develop a broad operational definition for the construct. Using a complexity science lens, this conceptualisation recognises the interconnections of system components, defining context as

“…a multi-dimensional construct encompassing micro, meso and macro level determinants that are pre-existing, dynamic and emergent throughout the implementation process. These factors are inextricably intertwined, incorporating multi-level concepts such as culture, leadership, and the availability of resources”.

Buttney [21] suggests that “we need to take context as that which can be empirically investigated rather than some factor that is a priori assumed to be at work”. To address this gap, a method to adequately account for context within implementation research is necessary This is echoed by Murdoch [22] and Fernandez et al. [23] who argue that understanding the dynamics of context requires its conceptualisation to be translated into a practical method of assessment. However, Rogers et al.’s [20] systematic review observed considerable heterogeneity in the approaches used to assess context. Over 40 methods were identified within the 64 papers included which led to the recommendation that a standardised approach for assessing context is required integrating a qualitative method informed by a comprehensive framework. This approach is considered the most suitable to ensure an in-depth assessment of context is achieved.

To address this identified gap, a context coding framework has been developed. This paper outlines how the context coding framework was created and highlights the value of employing this approach within implementation science research, using an illustrative case study to demonstrate its application.

Methods

Study background

This study is part of a wider body of research that uses a case study design to investigate the active role of context during the implementation of a healthcare initiative. The introduction of a collective leadership intervention was chosen as the implementation case study and is the primary focus of this research. This intervention aims to introduce collective leadership to healthcare teams using a suite of interventions to improve team performance and safety culture [24]. The collective leadership interventions have been piloted over a one-year period with four heterogeneous healthcare teams. The results presented in this paper incorporate data from one of the four participating teams as an illustrative case study to highlight the value of applying the context coding framework during the implementation of a healthcare initiative. Characteristics of the chosen case are summarised in Table 1.

Table 1 Description of Case A

Measures

As outlined in the introduction, contextual conditions are the pre-existing, dynamic and emergent factors of implementation that encompass multi-level determinants. To gain a greater understanding of implementation, it is essential to monitor these contextual conditions in research as the data generated is context dependent. Literature suggests that theories, models and frameworks provide greater insight into the mechanisms of implementation [15, 25]. The Consolidated Framework for Implementation Research (CFIR) is one of the most widely operationalised frameworks that aids in the characterisation of contextual determinants [23, 26,27,28]. Given the comprehensive nature of the CFIR and the vast quantity of frameworks available within the field of implementation science [27], CFIR was chosen to inform the development of the context coding framework. By synthesising overlapping constructs, the CFIR provides a repository of implementation related concepts which are organised into five major domains; intervention characteristics, outer setting, inner setting, individual characteristics and the implementation process [29].

Using a constructivist approach, the context coding framework builds on the CFIR to enable a deeper exploration of contextual factors independent of the intervention. By grounding the analysis across system levels, the framework accounts for team-level contextual factors previously overlooked in implementation science [20] and rearranges CFIR determinants to the appropriate system level (individual, team, organisational, and system levels) (Table 2). Figure 1 outlines the process which identified contextual factors relating to these system levels. Firstly, all five CFIR domains were reviewed to determine the applicability of each construct in describing the surrounding context. For example, the relative advantage construct within the Intervention characteristics domain can outline improvements in the surrounding context which participants attribute to the intervention. Within the context coding framework this CFIR construct is depicted as the individual-level determinant individual attitudes (Table 2). Next, the extant literature was appraised to collate contextual determinants not explicitly referenced by the CFIR [20, 29,30,31,32,33,34,35,36]. Rogers et al.’s [20] recent review provided a diverse range of novel constructs including team-level contextual determinants (Table 2). Finally, the contextual determinants of CFIR were integrated with the findings from the literature to refine the established CFIR constructs. For example, leadership engagement within the Inner setting domain was revised to highlight the influence of leadership at multiple system levels; team (local leadership engagement), organisational (organisational leadership engagement), and system (political environment) (Table 2). Subsequently the context coding framework provides a revised collection of contextual features that are pertinent to implementation and maps these determinants to the micro, meso and macro levels of the health system (Table 2).

Table 2 Context Coding Framework

Fig. 1

Context coding framework development

The coding framework enables the collation of various data sources. The data utilised in this wider body of research are presented in Table 3. As data is collected the context coding framework can be continuously updated and adapted by the researcher. This approach aligns with Stake’s [37] guidance on completing a case study which advises that a chronological description of a case is reported. Given the complexity of the construct, using multiple methods facilitates a rich and nuanced understanding of context. However, for some researchers, staffing, funding, and time restrictions may hinder the use of multiple methods and repeated analyses. Therefore, researchers should consider the context and complexity of their project and adapt the framework accordingly.

Table 3 Data sources utilised within the context coding framework

A pre-existing rating criteria developed by Damschroder and Lowery [38] has also been integrated to our context coding framework (Table 4). The ratings reflect the influence (positive, negative, or neutral) and the strength of each contextual feature prior to and during the implementation of the collective leadership intervention. However, additional contextual evaluations may be included as applicable to the research (e.g. multiple periods of data collection as part of a longitudinal evaluation of an intervention).

Table 4 Criteria used to assign ratings to constructs-Adapted from Damschroder & Lowery (2013) (http://creativecommons.org/licenses/by/4.0/)

Data analysis

Comparable to the extant literature which derives analytical codes from CFIR constructs [38,39,40], the context coding framework operates as a codebook during the evaluation process. Consistent with the CFIR Guide (available at www.cfirguide.org), the context coding framework incorporates detailed definitions and examples to ensure the consistent interpretation of codes. However, unlike previous analysis techniques, the framework employs a blended evaluation approach that combines aspects of rapid evaluation and in-depth analysis. Aligned with previous rapid evaluations, the context coding framework provides a visual display that succinctly collates multiple data sources [40, 41]. This method enhances the simplicity of applying Damschroder and Lowry’s rating process [38]. Using an iterative approach, data collection and analysis are concurrent rather than successive [37, 42]. Comparable to more in-depth qualitative evaluations, rather than summarising the data from the outset [40], the framework formally codes each data source as a first analytic step. Although the flexibility of the framework offers remit for an inductive, exploratory analysis, unlike more traditional qualitative approaches, a deductive template is predominantly used to structure the ‘real-time’ analysis of data as it is collected [43]. Consequently, the framework prompts the assessment of potentially influential contextual determinants from the outset. For example, by appraising the construct team workload prior to implementation, researchers can ensure the appropriate resources are available to strengthen the likelihood of implementation success (e.g. need for increased staffing/ researcher support). Thus, the context coding framework offers a structured approach to support intermediary rapid and in-depth evaluations which generates actionable and detailed findings. Figure 2 outlines the process of applying the context coding framework.

Fig. 2

Process of Application

Step 1

Each data source was analysed by one researcher (LR) using line-by-line thematic coding as outlined by Braun and Clarke [44]. As data emerged, statistics and verbatim quotes were extracted and entered in the relevant section of the context coding framework. Data within each section of the framework was categorised by method of data collection (i.e., observations, organisational reports, interviews, survey data) and recorded chronologically. By employing a triangulation of qualitative research methods a rich, detailed, comprehensive understanding of the context was achieved, enhancing the credibility of the analysis [45, 46]. To manage the large volume of interview data, positive and negative participant quotes were categorised separately to assist with the analysis of each framework domain. Reflecting the frequency and quality of the coded data, LR gave a provisional rating of influence to each contextual determinant within the framework. As new data emerged this rating was discussed and deliberated with researchers familiar with the data and the settings (ADB and EMC were involved with data collection and analysis for the wider body of research evaluating the effectiveness of the collective leadership intervention) and ratings were updated as required. Although double coding of all data sources would be preferable, this approach was unfeasible due to the large volume of data collected and the limited resources available to this study. However, ADB double coded 10% of observation and interview transcripts and all authors engaged in regular coding checks during team meetings throughout the evaluation process (Table 5). Additionally, the dependability of these findings was further heightened by maintaining an audit trail of all decisions and changes in thought processes throughout the analysis process [46].

Table 5 Techniques used to enhance the scientific rigour of the context coding framework

Step 2

Following the completion of data collection at each phase (prior to implementation and post-implementation), the extracted findings were summarised by LR under each section of the context coding framework. This summary included illustrative quotes and exemplar statistics. Reflecting this summarised narrative, a final rating was applied by the authors to indicate the relative influence of each contextual determinant on the implementation process. We acknowledge that this is a subjective rating based on mostly qualitative data, thus, to achieve actionable results while maintaining scientific rigor, Lincoln and Guba’s [46] trustworthiness criteria were applied to enhance the credibility and dependability of approach. Table 5 presents the techniques applied throughout the evaluation process and identifies additional strategies that researchers may employ to further enhance the scientific rigour associated with the framework’s application.

Ethical approval

Favourable ethical opinion for the research has been obtained from the University College Dublin Research Ethics Committee (ref: HREC-LS-16-116,397) and participating hospital sites. All participants provided written informed consent and the data collected was inputted into the context coding framework with all potentially identifiable characteristics removed to maintain anonymity.

Results

When data collection and analysis were complete, data were summarised to give an overarching narrative of the contextual conditions revealed. By employing the context coding framework, the structure enabled innovative insights to be generated throughout the implementation process which facilitated a transparent account of context to be documented and monitored over time.

Considering the influence of context prior to implementation

Prior to implementation, the context coding framework supported the identification of potentially influential contextual determinants requiring consideration. Table 6 exemplifies one feature of the framework at an organisational and team level to illustrate the value inherent in this approach. Although a facilitative organisational culture was portrayed within hospital documents, survey and interview data obtained at a team level highlighted that a traditional hierarchical context existed that would likely impede implementation success. Some team members felt intimidated by colleagues while others suggested that silo working was apparent with some consultants “dismissive of any other disciplines”. This context suggests that the introduction of the collective leadership intervention was timely However, the need to gain and sustain consultant support throughout the implementation process was apparent as consultants appeared to be highly influential in determining team operations and processes; “what they say goes”.

Table 6 Context coding framework pre-implementation: Culture (Case A)

The context coding tool can also identify variations in context across settings throughout an intervention’s implementation. As mentioned in the introduction, the collective leadership intervention was implemented in four healthcare teams. While Case A is documented as demonstrating a hierarchical team structure (Table 6), characterised by consultant control, silo working and “fear”, this was not representative of the culture within each of the four teams. By applying the context coding framework to another case with similar structural characteristics, a very different culture may be effectively identified. For instance, aligned with the organisation’s public documents and reports, prior to implementation, Case B characterised its culture as “open” with a “multidisciplinary focus” in which “every member is valued, and their input is welcomed”. When introducing an initiative to multiple sites, the rich information gained from applying the context coding framework helps to inform researchers of the differing contextual forces that may exist in each setting throughout an implementation effort. This should ultimately support the likelihood of implementation success across sites as researchers can tailor the implementation strategies they employ.

Understanding the dynamics of context

Context is made up of multidirectional forces [6, 34]. Its constructs are “suspended in a complex web of relevance to and relationship with other constructs” [47]. The developed context coding framework facilitates the exploration of these relationships and the dynamism of the concept as each construct is collectively assessed over time. For example, Tables 7 and 8 confirm the association between the culture of Case A, and the relationships and communication of team members within the organisation. Table 8 clearly identifies the effects of this hierarchical environment (Table 6) which is depicted by inadequate communication and hostility across and within professional groups. The framework also highlights once again the possible disconnect between senior management and frontline staff. Numerous opportunities are listed for team members to voice their frustrations at an organisational level (Table 7), however, these are not operationalised to support staff within their local department (Table 8). Alternatively, it may be interpreted that while the wider organisation is characterised as a supportive environment, the participating team may be an isolated, divergent case. Therefore, the variation in how the organisation at a meso level and the team at a micro level understand and illustrate their context is clearly evident within this framework.

Table 7 Context coding framework pre- and post-implementation: Networks and Communication (Case A)

Table 8 Context coding framework pre-and post-implementation: Teamwork (Case A)

Monitoring contextual dynamics over time

The framework also enables contextual factors to be monitored longitudinally, highlighting not only the impact of context on the intervention’s implementation but also the intervention’s possible influence on the surrounding context. Table 8 documents that while communication remains “disjointed” between some professions, there is consensus that communication has “opened a bit more” due to improved relationships among the team. Following the introduction of the collective leadership intervention, informal relationships have been established, meaning staff perceive each other “in a different light”. This has enabled team members to approach senior colleagues more comfortably as they feel “allowed to say it {their concern/opinion} and voice it”. It also appears to have broadened the narrow perspectives of senior members who observe that following the implementation of the intervention “anybody now can talk to you…there is no limit. There’s nothing between us”. Therefore, the complex interplay between the intervention and the context in which the change is implemented is clearly illustrated over time within the context coding framework.

Informing the selection of implementation strategies

By enabling the influence of context to be mapped over time, the context coding framework also assists researchers in selecting and tailoring the appropriate implementation strategies for their context of study. Implementation strategies are defined as the “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” [48]. As outlined previously, consultants were identified as the dominant leaders of Case A prior to implementation. By appraising this information researchers can ensure their implementation effort includes educational sessions or meetings tailored to these individuals, so they are fully informed of the intervention’s benefits. By gaining their support, this will likely impact the adoption of the collective leadership intervention due to the reported influence they have on the team.

Additionally, the context coding framework can aid in clarifying the association between contextual determinants and change strategies as a transparent account of this interaction can be reported. For example, consultant engagement can be monitored throughout the implementation of the collective leadership intervention. If this remains poor and is observed to be impacting adoption, the researchers could tailor their strategy and perhaps involve the executive board to provide incentives for participation or disincentives for poor engagement. By continuously monitoring the implementation strategies employed, the context coding framework, assists researchers in documenting the effects of these strategies on the local context, aiding in their refinement, modification or adoption of alternative approaches, as required.

Discussion

This paper has described the value of using our developed context coding framework as a methodology to explore and understand an often-neglected area of research within the field of implementation research, the study of context. Despite the acknowledged influence of context on implementation processes [10, 11, 13, 49, 50], it is reported that historically studies tend to be “acontextual” in character or represent context as a stable entity or a unidirectional set of influences [30, 34]. However, this case study illustrates the benefit of using the context coding framework as it offers a means for exploring the active role of context and the dynamism of the concept “in a world on the move” [51]. It also responds to previous literature [17,18,19,20], by providing a standardised approach to improve the reporting of context during implementation.

By accounting for the complexity of context, Tsoukas [52] suggests that the researcher is providing a meaningful contribution as organisational realities are exposed rather than constrained by a simplified, limited representation. However, this complexity, although powerful [12,13,14], is purported to impede the study of context [53]. Through developing a standardised approach, the context coding framework assists researchers by making the implicit aspects of context more observable. By considering these multifaceted contextual conditions together across system levels, an interpretable, transparent understanding of context can be elucidated, subsequently advancing the field of implementation research.

Although some studies recognise the multifaceted nature of context [20, 34, 54], this is not a universal perspective. Nielsen and Miraglia [55] describe the distinction between omnibus (stable characteristics of an organisation) and discrete contextual factors (changes taking place throughout implementation). Etymologically, the term context means to knit together or to make connections [14]. Therefore, the context coding framework challenges the simplicity and dichotomy of presenting contextual factors as omnibus or discrete. As illustrated within this case study, contextual factors cannot be depicted as stable, isolated entities. It is the dynamic relationships between these complex constructs rather than their individual forces that have a significant influence on implementation processes. Fitzgerald and McDermott [54] suggest that to capture these interactionist aspects of the concept, context needs to be analysed across multiple system levels. In response to the dearth of guidance provided within the extant literature, the context coding framework provides a structure to enable these dynamic and multidirectional interactions to be documented across the health system.

Fitzgerald [56] suggests that developments are needed to progress research from post hoc analyses of context, toward the ability to predict between alternative scenarios. As illustrated by this case study, the context coding framework provides researchers with a method of weighing the possible influence, affect and effect of differing contextual conditions prior to implementation. Using the integrated rating criteria [38], researchers can identify potential ‘receptive’ and ‘non-receptive’ contexts for change and select the most appropriate implementation strategies relevant to the context(s) of study.

Determining the most appropriate implementation strategy to address an identified contextual determinant has been cited as a fundamental challenge of implementation science [57]. Powell et al. [58] provide a list of 73 discrete implementation strategies that can be utilised by researchers to enhance their implementation effort. These strategies are context dependent signifying that the success of a strategy in one context may fail in another. However literature suggests that few studies tailor their implementation strategies to the context of study [57, 59, 60] or apply them to the appropriate system level (e.g. organisational-focused strategy used to address a contextual determinant at the team level) [59, 61]. Aligned with Grol and Bosch’s [60] suggestion, the context coding framework provides a systematic and rigorous method to help clarify the association between contextual determinants and change strategies as a transparent account of this interaction can be reported across system levels. This builds on the work of Waltz et al. [57], who have developed the CFIR-Expert Recommendations for Implementation Change (ERIC) Implementation Strategy Matching Tool. This tool supports researchers in choosing the best implementation strategy to address CFIR-based barriers [57]. By enabling researchers to document an accurate narrative of the context longitudinally, the context coding framework enables the improved use of the CFIR-ERIC Implementation Strategy Matching Tool. By continuously monitoring these strategies over time further refinements and modifications can be conducted depending on their observed effects on the local context which will likely support and enhance implementation success.

While this methodology is useful in understanding context prior to implementation, its value is not limited to a solitary phase of the implementation process. The Greek philosopher Herachitus argues that “the only thing that is constant is change”. Therefore, it is vital to monitor the changing context of healthcare when introducing an initiative. Pettigrew et al. [30] concluded that studies predominantly fail to “allow the change process to reveal itself in any kind of substantially temporal or contextual manner”. The context coding framework responds to this criticism by offering a means to monitor the local context longitudinally. This makes it a powerful tool for documenting both the impact of context on the implementation effort and the effect of the intervention on the local context over time. Additionally, although the CFIR conceptualises context as a multilevel construct, our developed framework accounts for the ongoing change that occurs within the health system. Therefore, the context coding framework builds on the CFIR by enabling this ongoing measurement of context and the assessment of its influence longitudinally, across system levels.

Limitations

There are some important limitations to consider when using the context coding framework. “To understand anything well we must grasp it in its context” [12]. However, it is difficult to grasp all features of every context in one framework. While this method offers a means for studying the active role of contextual determinants, researchers are cautioned against drawing a boundary around the phenomenon by only including the contextual factors listed herein. Although the context framework has been informed by the extant literature and the comprehensive nature of the CFIR, it is also shaped by the researchers’ experience of the case studies employed. Therefore, it is recommended that when using the framework, researchers are open and attentive to the inclusion of additional contextual factors that are relevant to their context of study, adapting the framework accordingly. Research is also acknowledged to be “a product of its time” [14]. Although the context coding framework provides a transparent account of the dynamism of context longitudinally, it is important that the timing of data collection is considered and documented. With the passage of time the meaning of these contextual constructs and their influence can change.

Additionally, while the framework enables the collation of multiple data sources and offers a high-level overview of implementation influence (positive, negative, neutral), further analysis is required to reveal the mechanisms through which context influences implementation success. However, by using the context coding framework researchers will obtain a comprehensive understanding of the data which will likely accelerate any additional in-depth analyses. We recommend applying Proctor et al.’s [62] implementation outcomes during the next phase of analysis to advance researcher knowledge of the interrelationships between implementation outcomes and contextual determinants.

Furthermore, we acknowledge that this approach is likely to be resource intensive, given the data required to ensure a rich understanding of the complex construct of context. Although the cost of the framework was not formally assessed, it is worth detailing the labour associated with application of this approach in the wider research project. LR invested 100% of her time to this project over a two-year period as part of her doctoral studies. The researcher’s involvement in the framework’s development and their familiarity with the data sources (involved in data collection and transcription), likely expedited the analysis process. However, while future researchers may require training in line-by-line coding or other forms of qualitative analysis, learning to populate the tool requires limited guidance. Larger teams will also have greater capacity to distribute workload to account for any competing priorities. Therefore, before applying this method, researchers should consider the compatibility of the approach with the project aims, team logistics, and project deadlines. Adapted versions of the framework may also be used if researchers are interested only in specifics dimensions of context.

Recommendations for future use

Aligned with Chambers [63], we recommend the need to use shared measures of context consistently over time in diverse settings to comprehensively understand the complexity of the construct and its influence. This research has addressed an identified gap in implementation science methodology by creating a resource suitable for both researchers and non-researchers (e.g. healthcare professionals, policy makers) to investigate an underdeveloped area of study, our understanding of context in implementation research. It is suggested that by using the context coding framework as a standardised tool across multiple settings, we will likely gain an enhanced knowledge of the construct from multiple perspectives. This will offer valuable guidance to researchers during their future implementation efforts while also providing insight into how context is perceived from differing vantage points (e.g., researcher vs non-researcher perspective, or between healthcare professions). If we do not begin to use a standardised approach consistently, our ability to understand and capture the dynamism of context and its multiple levels will remain severely limited.

Conclusion

This study described how the context coding framework was developed, outlining the benefits and limitations of employing this approach. The illustrative case study highlighted the value of using this approach as a method to (a) highlight the influence of context prior to implementation, (b) understand the dynamics of context throughout the implementation process, (c) monitor these contextual dynamics longitudinally and (d) inform the selection of the most relevant implementation strategies. This research advances the field by providing a practical methodology that focuses on an overlooked area of implementation science, the study of context. By helping to appropriately assess and report context, the robustness and learning acquired from implementation research will be enhanced, aiding in the translation of evidence-based healthcare interventions into routine practice. We welcome comments and critiques that will refine and further enhance the context coding frameworks utility to progress implementation research and practice.