An innovative sequential mixed-methods approach to evaluating clinician acceptability during implementation of a standardized labor induction protocol

Background Implementation outcomes, including acceptability, are of critical importance in both implementation research and practice. The gold standard measure of acceptability, Acceptability of Intervention Measure (AIM), skews positively with a limited range. In an ongoing hybrid effectiveness-implementation trial, we aimed to evaluate clinician acceptability of induction standardization. Here, we describe an innovative mixed-methods approach to maximize the interpretability of the AIM using a case study in maternal health. Methods In this explanatory sequential mixed methods study, we distributed the validated, 4-question AIM (total 4–20) to labor and delivery clinicians 6 months post-implementation at 2 sites (Site 1: 3/2021; Site 2: 6/2021). Respondents were grouped by total score into tertiles. The top (“High” Acceptability) and bottom (“Low” Acceptability) tertiles were invited to participate in a 30-minute semi-structured qualitative interview from 6/2021 to 10/2021 until thematic saturation was reached in each acceptability group. Participants were purposively sampled by role and site. Interviews were coded using an integrated approach, incorporating a priori attributes (Consolidated Framework for Implementation Research constructs) into a modified content analysis approach. Results 104 clinicians completed the initial survey; 24 were interviewed (12 “High” and 12 “Low” Acceptability). Median total AIM scores were 20/20 IQR[20–20] in the High and 12.5/20 IQR[11–14] in the Low Acceptability groups. In both groups, clinicians were enthusiastic about efforts to standardize labor induction, believing it reduces inter-clinician variability and improves equitable, evidence-based care. In the Low Acceptability group, clinicians stated the need for flexibility and consideration for patient uniqueness. Rarely, clinicians felt labor induction could not or should not be standardized, citing discomfort with medicalization of labor, and concerns with “bulldozing” the patient with interventions. Suggested strategies for overcoming negative sentiment included comprehensive clinician education, as well as involving patients as active participants in the protocol prenatally. Conclusions This study utilized AIM in an innovative sequential mixed-methods approach to characterize clinician acceptability, which may be generalizable across implementation endeavors. By performing this work during a hybrid trial, implementation strategies to improve acceptability emerged (clinician education focusing on respect for flexibility; involving patients as active participants prenatally) for year 2, which will inform future multi-site work.

Text box 1. Contributions to the literature • Acceptability of Intervention Measure (AIM) scores skew positively, thus limiting how well they inform optimization of implementation approaches.
• This sequential mixed-methods approach is an innovative way to utilize AIM in evaluating clinician acceptability in healthcare innovation.
• In using this methodology to evaluate clinician acceptability of an implementation endeavor to standardize labor induction practices, several implementation strategies to improve acceptability were identified.
In prior work using the AIM to evaluate acceptability of a variety of healthcare interventions and/or implementation strategies, scores skew positively with a limited range, thus potentially hindering the utility of the assessment to optimize interventions and/or implementation strategies when used alone [6].For example, work evaluating a communication program for adult child caregivers of parents with a blood cancer utilized the AIM measure.As no available cutoffs for "acceptable" versus "unacceptable" exist, the authors deemed the intervention would be acceptable if participants had mean scores of 4 or higher on the AIM items.The authors met their acceptability goal [7].In a project evaluating a huddle to review ongoing quality improvement work in a pediatric intensive care unit, AIM was used prior to implementation.Despite lack of a clear cutoff, the authors determined their intervention to be adequately acceptable with a mean total AIM score of 15.2, and moved forward with implementation [8].Yet, a measure like AIM likely has value beyond developing a cutoff for acceptability, which most interventions would likely meet.
In addition, prior work evaluating acceptability using qualitative methods has generally focused on the perspectives of clinicians thought to be key stakeholders due to their role, limiting the capture of a full range of sentiments [9][10][11].One possible method to select interview respondents to obtain richer data involves purposeful sampling using deviant cases [9].A purposeful sampling approach utilizing deviant cases involves selecting participants at extreme ends of a spectrum in manifestations of belief or behavior around a healthcare intervention for inclusion in a sample for in-depth qualitative study.The logic of purposeful sampling utilizing deviant cases is that the approach facilitates the inclusion of individuals in a sample that manifest the phenomenon of interest with sufficient intensity to produce variation in expression, thus increasing the analytic value of the sample.This is a pragmatic and innovative way to overcome barriers to assembling a sample in a qualitative study of acceptability that contains people whose daily experience will be influenced by healthcare implementation who hold both positive and negative views about the proposed evidence-based practice.In particular, interviewing positive deviants around healthcare implementation, such as those who have been most successful at enacting change, has yielded valuable results.However, this method has not yet been applied to acceptability [10][11][12].
Here, we describe an innovative explanatory sequential mixed methods approach to evaluate the implementation outcome of acceptability, which harnesses the validity of AIM to provide a deeper look at acceptability.We utilized this method during a 2-year pre-and postimplementation type I hybrid effectiveness-implementation trial incorporating a standardized labor induction protocol into routine care at 2 urban labor and delivery units.The goal of this protocol intervention is to reduce the cesarean rate overall, as well as reduce racial and ethnic disparities in cesarean [13,14].We performed this acceptability assessment after one year of implementation to optimize the second year.

Parent hybrid type I effectiveness-implementation trial
This explanatory sequential mixed-methods (QUAN->QUAL) study was embedded into a broader type I hybrid effectiveness-implementation trial evaluating implementation of a standardized protocol for the management of labor induction into routine care at 2 urban labor and delivery units.Site #1 is an academic, high acuity labor and delivery unit delivering approximately 4200 patients per year staffed primarily by obstetriciangynecologists and maternal-fetal medicine specialists, with associated residency and fellowship programs.Site #1 also has family practice attending physicians, residents, and a small group of midwives.Site #2 is a community based program delivering approximately 5000 patients per year staffed by obstetrician-gynecologists and an associated residency program, as well as a large midwifery program.
The hybrid trial is a 2-year stepped pre-and postimplementation trial, beginning October 1, 2018 and concluding December 31, 2022.The protocol was initiated at Site #1 on October 1, 2020, and at Site #2 on January 1, 2021.The standardized labor induction protocol was approved by a multidisciplinary institutional obstetrical committee prior to its initiation and includes recommendations for frequent cervical exams and interventions such as oxytocin and amniotomy at particular time points, with a focus on active management of labor induction.Implementation strategies for the initial implementation plan included conducting local consensus discussions, conducting educational meetings with clinicians in the 3 months leading up to implementation at each site, and audit and feedback reports, which detailed site-level compliance with the protocol and were distributed to sites every 3 months post-implementation.
The hybrid trial has 2 aims: (1) to determine the effectiveness of implementing a standardized labor induction protocol at improving obstetric outcomes, specifically cesarean delivery rate, and (2) to evaluate implementation outcomes relevant to standardized labor induction protocol (acceptability, penetration, and fidelity) using a mixed-methods approach.Here, we will specifically discuss our methodology and results related to the evaluation of acceptability within the context of the broader trial.The University of Pennsylvania Institutional Review Board approved all study procedures; informed consent was obtained prior to initiation of survey and interview components of the study.SRQR guidelines were utilized in preparing this manuscript.

Quantitative methods
The validated, 4-item AIM was modified to our intervention ([Triple P/Implementation Strategy] replaced with "the standardized labor induction protocol") and distributed using clinician listservs 6 months after implementation at each of the 2 participating sites (Site #1: 3/2021; Site#2: 6/2021).In addition to the AIM questions, clinicians were asked demographic items including role, age, gender identity, and years in practice; survey responses were otherwise anonymous.At the completion of the survey, clinicians were asked if they would be willing to participate in the qualitative portion of the study, and, if willing, asked for their name and e-mail address to connect their survey response to identifying information.
After survey closure, AIM scores were totaled and grouped into "tertiles": highest 1/3 of scores ("High" Acceptability), middle 1/3 of scores ("Middle" Acceptability), and lowest 1/3 of scores ("Low" Acceptability) among all respondents.Tertiles were selected in order to ascertain positive and negative deviants and thus determine who would be the most data rich respondents for the qualitative section of our work.Skew of the AIM total score was determined using skew and kurtosis tests for normality.

Qualitative methods
Clinicians who stated interest in participating were contacted from each of the High and Low Acceptability tertiles, with purposive sampling by clinician role (attending physician, trainee physician, nurse-midwife, nurse) and site until thematic saturation was achieved for both the High and Low Acceptability groups.We monitored for thematic saturation in both acceptability groups by noting the emergence of themes by interview question at the conclusion of each interview.We determined the groups to be saturated when we ceased to observe novel themes in the answers to each interview question.A summary of this QUAN->QUAL approach is shown in Fig. 1.
The Consolidated Framework for Implementation Research (CFIR) was used to create the interview guide [15].Interview questions elicited (1) acceptability of efforts to standardize labor induction, (2) barriers and facilitators to implementation of standardized labor induction practices generally, (3) acceptability of, as well as barriers and facilitators to implementation of specific components of the labor induction protocol (such as recommendations for frequent cervical exams and interventions like oxytocin and amniotomy at particular time points), (4) acceptability of current implementation strategies around the standardized labor induction protocol (including local consensus discussions, clinician education meetings, and audit and feedback) and suggestions for improved implementation.A summary of interview transcripts were then uploaded to NVivo 12 software for management and coding.A research coordinator trained in qualitative methods used an inductive process of iterative coding to ascertain recurrent relationships, themes, and categories and to develop the codebook.Then, we used an integrated analysis approach [16], identifying a priori attributes (CFIR constructs) and also using modified content analysis approach.Two trained research personnel (MN and MW) applied the codebook to the transcripts and periodically refined the themes and definitions based on inter-rater reliability tests in order to facilitate analysis.20% of transcripts were double-coded (k = 0.83).The researcher coordinators then synthesized the outputs of the coding and identified the key themes described in this manuscript.Analysis was completed in February 2022.

Quantitative results
104 clinicians across both sites completed the AIM survey, 61.5% from Site #1 and 38.5% from Site #2.Clinicians participating in the survey were evenly split among physicians and nurses, were primarily under 45 years of age, and 87.5% identified as female (Table 2).

Qualitative results
A total of 24 interviews were performed, 12 in the High and 12 in the Low Acceptability groups.Interviewees included 15 physicians (8 trainees and 7 attending level; 13 obstetrician-gynecologists and 2 family medicine physicians), 2 certified nurse-midwives, and 7 registered nurses.Median total AIM scores were 20/20 [IQR 20-20] in the High and 12.5/20 [  in the Low Acceptability groups, consistent with the positive skew of this value.Characteristics by acceptability group enrolled in the qualitative portion of the study are also shown in Table 2.

Acceptability of efforts to standardize labor induction
Broadly, participants reported positive sentiment regarding efforts to standardize obstetric care and to reduce rates of cesarean delivery.Regardless of acceptability group, providers were generally enthusiastic about the induction protocol and efforts to standardize labor induction.Participants cited a range of reasons for valuing standardized labor management, such as feeling that the intervention will make patient care more equitable and reduce variability between clinicians, that they appreciate being guided by evidence-based practices, or that they like having a tool for learning or to refer to if they receive pushback from patients or other providers.

I love it. I find that there is a lot going on the labor floor, and so whenever there's something [like] a protocol to rely on and actually physically point to, or just have at our disposal is very helpful.
While most participants reported they believed labor could be standardized, those in the Low Acceptability group were more likely to add nuance to this belief, noting that providers also need to be flexible and consider the specific needs of each unique patient.

I think it's good [to standardize labor practices], I
think that aim to make sure that everyone is being cared for on the same level and given the same attention is good.I do think that there has to be room for specifications to the situation and to the

Barriers and facilitators to implementation of standardized labor induction practices in general
Participants discussed several key barriers to protocol use, which were primarily focused on the CFIR constructs of Inner Setting (such as lack of adequate staffing or time) and Characteristics of Individuals (including clinician and patient factors).Almost all participants, regardless of acceptability group, described their floors as being very busy, with high patient volume, or not having sufficient staff to meet all patient needs in a timely manner.Participants tended to say these demands on their or their peers' time impeded their ability to implement the protocol because they simply needed to be elsewhere, other cases were more urgent and needed to take priority, or that juggling so many tasks led them to forget a protocol step.
[The barrier] is the volume [of patients] and the lack of providers being available at the right time.I feel like as a nurse, there are definitely times-like just yesterday-where…they said [the patient's next cervical exam should be] around 1:00…And then gener-ally, it's like, 'Oh, well, there are a couple of deliveries, but we'll be in here soon.' I'm like, 'You're on the list and we know we're supposed to check you at 1:00' , but there isn't anybody to check.
In regards to clinician factors, those in the High Acceptability group, made up of more physicians than the Low Acceptability group, more commonly spoke about lack of buy-in to the protocol from other clinicians.
Especially when we first started, there was a lot of grumbling … about treating patients the same, when they are different people.
Those in the Low Acceptability group were more likely to speak about patient resistance, rather than clinician resistance, to the protocol.These participants spoke about patients who, despite undergoing labor induction, wanted to let their body labor naturally or were distrustful of the interventions proposed.
There are some patients...that come with a birth plan, they come with a doula, and they are pushing back on treatment or exams.
Participants often described the culture of the included labor and delivery units as facilitative to protocol implementation.
I think our culture has some problems, but instituting protocols is not one of them.We are very used to that.I think our culture is also that, we want to improve outcomes for our patients, but sometimes we just feel a little helpless as to the best way to do that, because some of our patients do come in with a lot of co-morbidities.So, to have toolkits that we know actually do help our patients, I think we're very willing to embrace.

Acceptability of current implementation strategies around the standardized labor induction protocol and suggestions for improved implementation
Many participants, particularly in the High Acceptability group, felt that they were well educated on the protocol going into implementation, and had the opportunity to ask questions with open discussion.
I think it made perfect sense… when I was presented with the information and the data, it was all sensible.
From both groups, participants agreed that nursing and resident trainee education around the protocol could be improved.In particular, there was focus on the number of residents rotating through labor and delivery, largely on a monthly basis, and that reminders were needed with rotation changes.
[In reference to nursing], I am sure there has been some education, but there should be even more.If there is education, about why we do [these induction steps], then I think some of the resistance to each step might be reduced.I think constant rotation of residents makes it hard to implement a protocol, like they start every month, and then they forget that there's even a protocol.
Participants from both groups agreed that better education of patients on the labor induction protocol is needed, with a focus on counseling about the protocol prior to admission for labor and delivery.
Well, I think education is really important … I think the nurses are doing a lot of education on the spot, but it would be great if the patients had more education about induction on the front end.If there was something, some sort of handout, that would make it really easy.
When asked about the audit and feedback reports clinicians received regarding their fidelity to the protocol, participants all reported that they were both helpful and motivating to improve compliance.
I think that we're driven by trying to be better and I think by seeing the things that we are not good at, it's motivating to be better at them.

Discussion
Measuring the acceptability of interventions accurately has proven to be difficult, despite its importance in implementation research.While the Acceptability of Intervention Measure is a validated and increasingly used measure in implementation science, the tendency for both interventions and their implementation strategies to receive high scores on the AIM assessment limits its utility.This study utilizes AIM in an innovative, explanatory sequential mixed-methods approach that harnesses participants with positive and negative responses to the AIM measure.This work was performed during the first year of a two year type I hybrid implementation-effectiveness study, and allowed our team to obtain a deep understanding of acceptability, both of the intervention itself (a standardized labor induction protocol), as well as the implementation strategies.As a result, we were able to develop concrete strategies targeted to improving acceptability to optimize the intervention and strategies for its implementation for year 2 of the parent trial.
In a study most similar to this work, Hoskins et al. evaluated parent acceptability of a S.A.F.E.Firearm Program in pediatric care.Here, a sequential mixed-methods approach was also used, beginning with the quantitative AIM survey, and finishing with semi-structured interviews with a convenience sample of respondents [17].Their work also found uniformly high AIM scores.Our work expands upon the methodology set forth by Hoskins et al., interviewing deviant AIM samples rather than a convenience sample of survey respondents.
The notion of studying closest those with the most positive and/or negative attitudes towards a healthcare concept or intervention is not new.Positive deviance studies have focused on surveying and interviewing individuals successfully implementing interventions, with the goal of ascertaining valuable data to support others in similar implementation endeavors [10][11][12].Similarly, negative deviance studies focus on those struggling to implement or unengaged with implementation, in order to catalog barriers and develop targeted responses [18].Here, we focus on both positive and negative deviance in acceptability, to meet the objectives of both types of studies simultaneously.
In interviewing only those grouped into High (positive deviants) or Low Acceptability (negative deviants), we aimed to interview only those most data-rich respondents.As a result, we were able to develop 2 concrete strategies to improve acceptability in our work.The first focused on education.We initiated routine, multidisciplinary clinician re-education, to foster an understanding as well as community around protocol implementation.Next, we focused on involving patients as active participants prenatally.We developed a plan to educate patients, involving printed and electronic patient-centered information on labor induction, distributed to patients when scheduled for induction, as well as initiate ongoing conversations about labor induction and our methodologies throughout the third trimester, allowing patients to ask questions and prepare.We will evaluate the success of these implementation strategy adaptations in year 2 of our parent hybrid trial.
The strength of this study is in innovative measurement of acceptability using a mixed-methods approach, resulting in tangible changes to an ongoing implementation study.This study has several limitations.First, those who elected to complete the AIM survey may represent a selfselecting sample, which may already be biased towards High or Low Acceptability.Our methodology also assumes that the most data-rich individuals are those at both the extreme ends of acceptability, without evaluating the opinions of those in the Middle.We included clinicians from 2 sites, although both diverse and disparate from each other, which do not necessarily reflect the large variation in geography and practice model of labor and deliveries across the US.

Conclusions
In conclusion, this sequential quantitative to qualitative methodology provided insightful data for the implementation outcome of acceptability around a standardized protocol for labor induction.The mixed-methods process described is likely applicable to assessing acceptability in other healthcare implementation endeavors, with benefits beyond either quantitative or qualitative evaluation alone.After application to other disciplines and interventions, this methodology may be refined and validated.Thus, we may be able to advance the scientific rigor and sophistication of the study of acceptability in implementation science, with the goal of eventually reaching a superior "gold standard" for acceptability evaluation.

Table 2
I don't think women are standard.I think it's very challenging to work with sort of a checklist checkbox model of labor, induction, birth, any of those things.I feel like a lot of our patients feel very vulnerable and almost assaulted by the process and in some ways I feel I'm almost in a position of needing to protect my patients from sometimes excessive intervention and rushing of interventions that they may not be fully educated about or really informed of, or consentto.Like people just walking in the room and saying, "We're going to break your water now."