- Technical advance
- Open access
- Published:
How to specify healthcare process improvements collaboratively using rapid, remote consensus-building: a framework and a case study of its application
BMC Medical Research Methodology volume 21, Article number: 103 (2021)
A Correction to this article was published on 29 July 2021
This article has been updated
Abstract
Background
Practical methods for facilitating process improvement are needed to support high quality, safe care. How best to specify (identify and define) process improvements – the changes that need to be made in a healthcare process – remains a key question. Methods for doing so collaboratively, rapidly and remotely offer much potential, but are under-developed. We propose an approach for engaging diverse stakeholders remotely in a consensus-building exercise to help specify improvements in a healthcare process, and we illustrate the approach in a case study.
Methods
Organised in a five-step framework, our proposed approach is informed by a participatory ethos, crowdsourcing and consensus-building methods: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvement in light of learning from the exercise. We tested the approach in a case study that sought to specify process improvements for the management of obstetric emergencies during the COVID-19 pandemic. We used a brief video showing a process for managing a post-partum haemorrhage in women with COVID-19 to elicit recommendations on how the process could be improved. Two Delphi rounds were then conducted to reach consensus.
Results
We gathered views from 105 participants, with a background in maternity care (n = 36), infection prevention and control (n = 17), or human factors (n = 52). The participants initially generated 818 recommendations for how to improve the process illustrated in the video, which we synthesised into a set of 22 recommendations. The consensus-building exercise yielded a final set of 16 recommendations. These were used to inform the specification of process improvements for managing the obstetric emergency and develop supporting resources, including an updated video.
Conclusions
The proposed methodological approach enabled the expertise and ingenuity of diverse stakeholders to be captured and mobilised to specify process improvements in an area of pressing service need. This approach has the potential to address current challenges in process improvement, but will require further evaluation.
Background
The last three decades have seen clinical guidelines, defined as “systematically developed statements informed by a systematic review of evidence and an assessment of the benefits and harms of care options designed to optimize patient care” [1], become a cornerstone of evidence-based practice. Production of guidelines is based on well-established methodologies, including synthesis of scientific evidence, expert opinion, and stakeholder consultation, and is supported by an infrastructure of national and international bodies (e.g. government agencies and professional associations) [2]. Clinical guidelines are not, of course, self-implementing. Getting clinical guidance into practice is typically complex, requiring multi-modal approaches, and is the subject of a burgeoning science and associated literature [3, 4]. It is now clear, however, that the implementation of evidence-based practices (what should be done) depends crucially on process improvement – changes to how things are done [5, 6]. For this reason, the specification of process improvements (i.e. identifying and defining the changes in processes that need to be made to deliver good care) is a key task. In this article, we offer an approach for developing specifications for process improvements using rapid, remote, consensus-building methods, and we illustrate it using a case study conducted during the COVID-19 pandemic.
We start by noting that there is no consensual definition of process improvement, but it is distinguished by its focus on how to improve the underlying processes (such as workflows, task design, role allocations, communication techniques, resources required, and so on) for delivering care – rather than, as in the case of clinical guidelines, defining ideal clinical standards. Process improvement has a particular role in ensuring that work systems are optimised, for example by helping to define the activities from beginning to end of a clinical process or pathway; to explain how these activities can most effectively be undertaken; to clarify tasks, roles, and skills needed; to characterise the decisions to be made and the support needed to make and implement those decisions; and to identify the equipment, resources and other tools required [7, 8].
The various methods for process improvement are often gathered together under the rubric of quality improvement [9, 10]. Some, such as the Model for Improvement, Lean and Six Sigma, amongst others [11], have been adapted from industry techniques [12]. Other approaches have been developed from the design and engineering disciplines and draw on socio-technical systems principles. For example, human factors and systems engineering use structured methods to change existing work systems based on the analysis of the interactions between people, tasks, tools, technology and the environment [13,14,15].
Common to most of these approaches is the need to specify process improvements. Specification requires identifying and defining the changes that need to be made (for example, an amendment to task or role design, or use of a new piece of equipment) to bring about improvement. A major challenge, however, is that the large-scale infrastructure for developing specifications for process improvement in healthcare has, in contrast with clinical guideline development, remained under-developed. The work of process improvement specification has instead remained largely locally-led, conducted within individual organisations.
Local leadership of process improvement has, of course, many advantages, including the potential to customise a solution to local circumstances and to imbue a sense of local ownership, but it is also associated with some disadvantages. One is that each individual organisation facing circumstances requiring change may develop their own process improvement specification in isolation [16, 17], but a multiplicity of approaches to the same area of practice may cause problems of its own. First, it may be wasteful, with each organisation reinventing the wheel [17, 18]. Second, it may be sub-optimal: reaching the best possible solution requires inputs from multiple disciplines, but individual organisations may lack access to the fullest range of expertise, particularly when it is rare (e.g. specialist human factors knowledge) [19,20,21]. Third, destandardisation creates learning overheads and new risks, for example when personnel moving between institutions have to learn new ways of doing things while unlearning previous ones. But while exclusively bottom-up development of ways of working may not be ideal, top-down imposition of particular practices can generate other pathologies [22], including failures of implementation, truculent and dysfunctional compliance, inadequate customisation to local circumstances, and creation of perverse incentives.
The deficiencies in some of the current infrastructure for process improvement have been vividly surfaced by the COVID-19 pandemic, which has caused massive disruption in the organisation and delivery of healthcare [23, 24]. Rapid innovation has been a feature of the response, prompted by the need to make adaptations to established clinical processes to address the infection risks and other challenges associated with the virus, as well as many other changes to clinical pathways and practices [25,26,27,28,29,30,31,32,33,34,35]. While the scale and speed of response to the pandemic has been impressive, an important risk is that some efforts may unhelpfully reproduce some of the challenges previously identified in the field of quality improvement [17, 36], including those discussed above of duplication of effort, waste, de-standardisation, and inability to engage sufficiently diverse expertise.
Given the challenges of both top-down and bottom-up quality improvement, a more collaborative alternative is likely to be of value. In this article, we address this need. We propose a methodological approach designed to enable large-scale remote engagement and mobilisation of multiple forms of expertise to build rapid consensus on specifications of process improvements, and we describe a case study of its application.
Methods
In developing our approach, we built on a participatory ethos, principles of crowd-sourcing, and consensus-building methods.
Participatory ethos
A participatory ethos – an approach that values the perspectives of the full range of groups of people affected by an issue – is an important guiding value in healthcare improvement [37]. But it is also of practical significance: securing participation may be more likely to result in solutions that are satisfying, workable, informed by professional values and clinical expertise, capable of being customised for specific situations, and capable of being implemented through collective effort rather than harsh, externally-imposed sanctions [38, 39]. Participatory approaches may also be more likely to lead to sustainable impacts by generating a sense of local ownership and commitment [40, 41]. Participatory approaches may be especially useful in encouraging practitioners to engage with evidence and its creation [42], as well as generating findings that have impact on practice [43].
Crowdsourcing
By drawing on the collective intelligence of many individuals, crowdsourcing can enable data to be collated on a much greater scale than would otherwise be possible [44], creating potential to solve problems by drawing on a wider range of perspectives and diverse experiences and knowledge [45]. Recent advances such as online engagement platforms [46,47,48] are now facilitating engagement of large, diverse and geographically dispersed stakeholders remotely as collaborators in co-constructing solutions [44, 49,50,51].
Consensus-building methods
Consensus-building methods are well established as ways of promoting deliberation, inclusion, and participation in situations where there may be multiple perspectives, interests and communities [52,53,54,55,56]. Methods widely used in developing guidance in healthcare include the nominal group technique [57, 58], the consensus development conference [59, 60], the RAND/UCLA appropriateness method [61, 62], and the Delphi method [63, 64]. These approaches are commonly recommended and used for developing clinical guidelines [55, 65] and reporting guidelines [66, 67], but application of consensus-building in process improvement has remained much more limited, not least because of the tendency (discussed above) to see process improvement as the domain of local teams. Yet consensus-building is potentially of value for process improvement in helping to build shared understanding, to include diverse forms of expertise, and to produce agreements about process improvement that might otherwise remain elusive.
The Delphi method offers considerable promise in this respect, as one of the best known and most widely used approaches for consensus-building in healthcare contexts [52, 54]. Using group communication that brings together and synthesises knowledge, participants are typically involved in a number of rounds of rating or voting on a set of propositions, and may then adjust their initial ratings based on feedback from the group in a number of subsequent iterations [68, 69]. Though Delphi can include a large number of individuals across diverse locations and areas of expertise [54], many Delphi exercises for healthcare have only involved relatively small and homogeneous panels of approximately 10 to 30 participants [70]. There is evident scope for including larger and diverse groups of participants [54, 70]. New methodological approaches are needed to use the method effectively for large-scale remote consensus-building to specify process improvements, while adhering to a participatory ethos and minimising time and effort required of participants.
A proposed methodological approach for developing specifications for process improvements using rapid, remote, consensus-building methods
The approach we propose for rapid consensus-building of process improvement specifications involves five steps, which we have organised into a framework (Table 1). The steps are: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvements in light of learning from the exercise.
(1) Define scope and objective of the process improvement
A first step is to identify and characterise the problem to be solved and the goal to be achieved so that an assessment can be made of whether consensus-building is an appropriate method [71]. If consensus-building is selected, the next step is to define the scope and objective of the process improvement, similar to current practices for developing clinical practice guidelines [72] and Core Outcome Sets [54]. These activities require close collaboration with key stakeholders, in accordance with the participatory principle [37, 73, 74].
(2) Produce a draft or prototype of the proposed process improvement specification
Rapid consensus-building on the specification of process improvements can be facilitated by producing a draft or prototype version, which might be informed by rapid literature reviews, existing guidelines or practices, ideas sourced from specialist groups, or stakeholder surveys, interviews and focus groups [70, 75]. Feedback can then be sought on this draft/prototype, informing subsequent rounds of Delphi consensus-building. The draft/prototype may take a range of forms [75], for example a conceptual framework, an existing standard operating procedure, a video-based simulated scenario, or an artefact (e.g. equipment or software). The most appropriate form can be selected based on the technical aspects of the healthcare process, alignment with the goals of the final product, expectations of participants’ available time, questions of how to maximise participant engagement, and possibilities and limitations of (mobile) devices and platforms on which the draft/prototype will be presented [76, 77].
(3) Identify participant recruitment strategy
Eligibility
Determining participant eligibility criteria requires consideration of the need for triangulation and bringing together the views of different types of stakeholders [54, 55, 78,79,80]. Seeking diversity can reduce risk of bias and provide a richer variety of views [78, 81,82,83]. Eligibility criteria might include specialists working in the selected field of clinical practice (e.g. maternity staff) or those with specialist expertise (e.g. infection prevention, human factors). Importantly, for many process improvement activities, patients and the public are key stakeholders whose expertise and perspectives should be included [42, 84].
Sample size
The current literature provides no set standards for required sample sizes for Delphi exercises, meaning that pragmatic choices have to be made [54, 85]. One approach is to estimate the sample size based on the number that would likely result in stable ratings across the Delphi rounds, accommodating for dropout [85, 86]. This may require establishing a minimal sample size for each stakeholder group included.
Sampling and recruitment
Sampling strategies for consensus-building exercises are generally informed by available time and resources, and include convenience, purposive or criterion sampling [71, 85]. Recruitment strategies can, for example, be grounded in the voluntary contribution of willing individuals who wish to contribute to the production of scientific knowledge [87,88,89]. The principles of participatory research may inform successful strategies for increasing participation of minority groups [89, 90].
Retention
Measures to support retention across the different phases of the consensus-building exercise may involve strategies such as e-mail reminders [91] that include reference to the importance of the participant’s contributions [87]. This could be enhanced by the use of online engagement platforms [46,47,48], which may help encourage feelings of being part of a research community.
(4) Design and conduct a remote consensus-building exercise
Ethics
The ethical principles for participatory consensus-building exercises have much in common with any other quality improvement activity [92, 93], or indeed guideline development, in that they may often be classified as service improvement activities not requiring specific ethical review. Where these exercises are conducted as research studies rather than improvement activities, different considerations may apply, such as the requirement for oversight and/or approval by an Institutional Review Board or Research Ethics Committee [92, 93].
Idea generation
Delphi rounds are typically informed by an exercise to generate ideas [63, 64]. This might be done by inviting participant feedback on the draft/prototype of the process improvement specification, which can then be synthesised in a set of propositions to be rated in the subsequent Delphi rounds. If this approach is taken, using open-ended, short-answer options may work best for a rapid response with minimal burden for participants and analysts [94]. The synthesis method depends on the type of feedback generated, and needs to be clearly documented [75]. Synthesis might involve, for example, removal of duplicates and merging of responses with similar wording [95]; using thematic analysis to create concepts, categories or themes [96]; or coding individual responses to themes and triangulating the coding among multiple researchers [95, 97]. The aim is to balance the consolidation of responses (i.e. creating a list that is reasonable for experts to navigate in subsequent rounds, e.g. about 20 items [94, 95]) while avoiding excessive abstraction [97].
Delphi rounds
The propositions derived from the idea generation phase can be used in two or more subsequent Delphi rounds of iterative surveys in which participants state their level of agreement with propositions on a numeric scale. Responses are aggregated and participants have the opportunity to revise their judgments in the light of feedback that includes their own and the group’s judgment, with the aim of exploring or reaching group consensus [98]. Two rounds is often sufficient to reach consensus and may reduce burden for participants [70, 75], though more rounds may sometimes be used. Use of visually appealing forms of feedback, such as interactive graphs that show the distributions of ratings across one or more stakeholder groups, can facilitate response across rounds.
(5) Produce a final specification of the process improvements in light of learning from the exercise
The aim of the Delphi exercise is to produce recommendations that can inform the specification of process improvements that can be implemented at scale. A plan should be in place for developing and disseminating supporting resources that reflect the specified process improvements. One advantage of the collaborative approach is that it is likely to facilitate engaged dissemination and implementation by those who have participated [99].
Case study
We tested the proposed methodological framework using an example from emergency maternity care. The need for process improvement specification in maternity care during the COVID-19 crisis was particularly urgent given that, in contrast with some other areas of care, it is not possible to defer or reschedule births [100, 101]. During the first few months of the pandemic [23, 24], approximately 100 pregnant women with suspected or confirmed COVID-19 were admitted each week to obstetric units across the UK [102]. This meant that many existing areas of care where good clinical practice was well-established (e.g. through clinical guidelines) required process improvement to adapt to the need for: infection control, the challenges of communication and teamwork likely to be posed by use of personal protective equipment (PPE), and other demands of making clinical processes COVID-safe. It was important, for example, that maternity staff were able to adapt quickly to the new infection prevention requirements of donning PPE to minimise any delays to providing prompt clinical treatment.
The result was that maternity units were all, individually, urgently seeking COVID-19-specific resources and training for obstetric emergencies. Responding to the emerging but still limited guidance for dealing with COVID-19 in maternity care (e.g. [103]), NHS maternity professionals expressed a need for clearer e-learning resources relating to PPE skills, and more training on COVID-19-specific emergency drills [104]. One priority area concerned how to manage an obstetric emergency such as post-partum haemorrhage (PPH) in a woman with suspected or confirmed COVID-19. PPH is an emergency that complicates 1.2% of births in high-income settings [105, 106]. It is one of the most frequent and severe maternal complications after birth [107,108,109], and a cause of intensive care admission in the UK [110]. It is a classic example of where the requirements for managing situation clinically are well understood and communicated through clinical guidelines (e.g. [111, 112]), but where process improvement was needed to ensure that the underlying processes of delivering care were adapted to deal with the challenges imposed by the pandemic.
Addressing this problem required rapid, remote consensus-building among multi-professional stakeholder groups to specify process improvements for managing an obstetric emergency in a woman with suspected or confirmed COVID-19. To employ a consensus-building exercise, we used Thiscovery (https://www.thiscovery.org/about), an online research and development platform created and developed by THIS Institute at the University of Cambridge. One of its founding goals is that of facilitating inclusive, multi-stakeholder involvement while offering maximum flexibility and minimum burden.
Results
We applied the five steps of the framework to the case study. We: (1) defined the scope and objective of the project with key stakeholders; (2) produced a draft video showing a simulation of processes for handling an obstetric emergency in a COVID scenario; (3) recruited three expert groups (maternity care, infection prevention and control, and human factors specialists); (4) designed and conducted an exercise to reach consensus on recommendations to improve the processes illustrated in the video; and (5) produced a final specification of the process improvement, informed by the consensus-built recommendations, and illustrated this specification in an updated video and other resources .
(1) Scope and objective
The project had its origins early in the pandemic when key stakeholder organisations, including the Royal College of Midwives and Royal College of Obstetricians and Gynaecologists, identified that while clinical guidance on how to handle obstetric emergencies such as PPH remained sound, the underlying processes required adaptation for women with confirmed or suspected COVID-19. Many maternal deaths related to PPH in healthcare settings can be avoided through effective clinical management [106, 109], including prompt initiation of several simultaneous actions such as uterine massage, intravenous fluid resuscitation, and administration of medication (tranexamic acid to treat major haemorrhage and uterotonics to contract the uterus). Treatment delay can result in poor outcomes [113, 114], so delivering these clinical interventions requires highly optimised underlying processes, including effective teamwork, coordination, communication, and access to appropriate supplies. All of these processes require adaptation for a COVID-19 scenario, which might demand, for example, donning and doffing of personal protective equipment, changes in the tasks undertaken and their sequencing, and forms of communication suitable for a situation where masks and visors may inhibit verbal and non-verbal exchange.
This problem appeared well-suited to consensus-building that could rapidly generate a consistent approach. The overall objective of our project was defined as: to develop specifications for the process improvements needed to manage an obstetric emergency (such as PPH) in a woman with suspected or confirmed COVID-19, using rapid, remote consensus-building among multi-professional stakeholder groups. Target audiences included healthcare professionals working in maternity care in UK NHS trusts, including midwives, obstetricians, and managers of maternity services.
(2) Draft of the proposed process improvement specification
We started by producing Version 1 of a video that showed a simulation of a maternity ward team managing PPH in a woman with suspected or confirmed COVID-19. The video illustrated processes that included: how the team communicated with each other, the woman and her partner; PPE donning and doffing procedures; and use of obstetric-specific procedures (e.g. PPH ‘grab bag’, treatment algorithms) in a COVID-19 context. The processes illustrated were based on: the emerging national guidance on COVID-19 infection prevention and control in a clinical setting (April 2020); clinical guidelines for managing obstetric emergencies such as PPH [e.g. 111, 112]; and ways of working established in one of the safest maternity units of the UK [115, 116]. Though the processes shown thus represented initial reasonable specifications for process improvements to facilitate handling of an obstetric emergency in a COVID-19 scenario, it was also likely that these specifications could be further optimised.
The use of a video format to elicit suggestions for improvement was intended to enhance participant engagement [76, 77], minimise cognitive burden (e.g. not having to study a written manual) [117], and align maternity professionals’ desire to have more COVID-specific e-resources available [104]. The video format was also expected to work well for the study participants using a range of different technologies, including mobile devices [76, 118].
(3) Participant recruitment strategy
Eligibility
We drew on the expertise of three expert groups: maternity teams to provide clinical and practical views; infection prevention and control staff for specialist knowledge on infection guidance; and healthcare human factors specialists for their perspective on the interaction between people and work systems in healthcare settings [21, 119]. The invitation asked potential participants to provide their main professional background, including an “other” option to avoid forcing people into categories that did not suit them. Participants were not further screened for expertise, in order to ensure a rapid response, minimal burden to participants, and maximal inclusivity.
The consensus-building exercise consisted of a recommendation generation exercise followed by two Delphi rounds (Fig. 1). For the first Delphi round, only those who had taken part in the recommendation exercise and had stated that they were happy to be contacted again were eligible, and for the second Delphi round, only those who had taken part in the first Delphi round were eligible.
Design of the consensus-building exercise
Sampling and recruitment
We made use of convenience and snowball sampling to rapidly recruit a sample for the three expert groups [120]. Geographical representation was maximised using recruitment conducted through nationwide email networks of specialists in maternity care, infection prevention and control, and healthcare human factors. We anticipated that most participants would engage in response to the email (convenience sampling), while other participants might become involved as a result of colleagues alerting them to the study (snowball sampling). To maximise response rates and minimise attrition bias, reminders were sent through the email networks and Thiscovery, referring to the importance of their (continued) engagement [87, 91].
Sample size
We expected that we would require three panels of stakeholders with distinct areas of professional expertise [85], with at least seven experts each, in accordance with conventional recommendations on sample size of an expert panel [61]. In the design of our consensus-building exercise (Fig. 1), we assumed that approximately half of the participants of the initial recommendation exercise would take part in the first Delphi round (50% response rate), with an attrition rate of 20% for the second Delphi round [54]. Accordingly, we aimed to recruit at least 20 stakeholders for each of the three expert groups for the recommendation exercise.
(4) Remote consensus-building exercise
The recommendation generation exercise and Delphi rounds were designed and then integrated into Thiscovery. All exercises were user tested to optimise participant experience on computer, tablet and smartphone platforms. Data collection and analysis for the consensus-building exercise (including the recommendation generation exercise and Delphi rounds) was completed in 6 weeks.
Ethics
Participants registered for an account on Thiscovery after consenting to the platform’s privacy policy and terms of use (https://www.thiscovery.org/register/). Participants confirmed their consent before each new round of the consensus-building exercise. All data was captured and processed without any personal identifiable information. Review by an institutional review board was not applicable, as the project was a consultation and engagement exercise classified as a quality improvement activity [92, 93], in which all of the participants were invited to join the authorship group and to be acknowledged in the project’s outputs.
Recommendation exercise
We started with a recommendation generation exercise to inform the subsequent Delphi consensus-building rounds (Fig. 1). This initial exercise consisted of asking participants for their feedback on a draft of good practice in undertaking the process, as illustrated in a video. After seeing the video, participants were asked, using open-ended questions, to draw on their professional expertise to provide recommendations to improve the practice illustrated in relation to: 1) donning PPE; 2) management of the emergency in the context of COVID-19, e.g. use of PPH “grab bag”; 3) doffing PPE; and 4) any other areas.
This first stage of the project – the recommendation exercise – was completed by 105 participants (Table 2). There were 912 responses from 103 participants (two participants did not provide recommendations). Of these 912 responses, 94 were coded as general comments (e.g. “The order of doffing is very important”) or supportive statements (e.g. “The process shown appeared very proficient”) and were excluded from further analysis. The remaining 818 responses were recommendations relating to improvement of practice illustrated in the video, and were synthesised: three analysts worked in parallel on an iterative analysis process of assigning recommendations to pre-defined categories, coding them, re-assigning them to categories more closely fitting with the codes, and merging similar recommendations (Supplement 1). This led to a total of nine categories including 74 synthesised recommendations. Of these, 26 recommendations were identified as the most frequently raised across all participants (Supplement 1).
A final clinical stakeholder review by authors TD and CW identified two recommendations as only focusing on the video format, rather than on improving the processes illustrated in the video, and were excluded from the consensus-building exercise. This stakeholder review also identified recommendations that could be further combined due to sufficient overlap. We were therefore able to classify the remaining 22 recommendations into five categories (Table 3).
Delphi rounds
All but one of the 105 participants agreed to be contacted again for the Delphi rounds and all 104 were invited. About two-thirds (n = 71) of those invited took part in the first Delphi round, of whom 57 took part in the second Delphi round (Table 2). Retention from the first to the second Delphi round was 80% for the total group: 70% for maternity care, 71% for infection prevention and control, and 88% for human factors (Table 2). The risk of attrition bias was low, as ratings in the first Delphi from participants completing both Delphi rounds and from participants who did not respond to the second round were similar.
In the Delphi rounds, the most frequently raised recommendations were presented to participants, with the option to review the video at any point. Participants were asked to rate each recommendation in response to the statement: “This recommendation should be implemented.” A nine-point scale was used with the anchors of “strongly disagree” (rating = 1), “uncertain” (rating = 5), and “strongly agree” (rating = 9). Consensus to implement was defined as > 70% of participants rating a recommendation with a 7, 8 or 9, and < 15% of participants rating it with a 1, 2 or 3, in accordance with core outcome set methodology [54, 121]. An additional criterion we used was an interquartile range ≤ 2, to further validate consensus among the group [98].
Only recommendations that did not reach consensus in the first Delphi round were taken forward to the second [75]. In the second round, the participant was presented with their original rating for each respective recommendation, along with the distribution of ratings from each stakeholder group [122, 123], as shown in an interactive bar chart (see Supplement 2). The participant was asked to consider the ratings of the others, and whether they would like to change their rating or stay with their original rating from the first round.
Of the 22 recommendations rated in the Delphi rounds, 16 reached consensus that they should be implemented (Table 3), with consensus on 12 achieved in the first round and a further four in the second round. The other six recommendations received higher ratings in the second round, but insufficient to reach consensus (only 39 to 68% of participants rated them with a 7, 8 or 9, as shown in Table 3).
Risk of bias due to unequal stakeholder group size was low: analysis showed that consensus across the whole group was similar to that found within the three stakeholder groups, i.e. when each stakeholder group was analysed independently, consensus was reached for an almost identical set of recommendations. For example, “Perform hand hygiene prior to donning PPE” was rated with a 7, 8 or 9 by 74% of maternity care, 71% of infection prevention and control, and 72% of human factor participants (see Supplement 2).
(5) Final specification of process improvements
The recommendations that reached consensus were reviewed by the project team and used to inform the specification of process improvements for optimised management of obstetric emergencies during the COVID-19 pandemic. These specifications included the processes shown in Version 1 of the video amended based on the consensus-built recommendations (Table 3). A new video (Version 2) to illustrate the specified process improvements was produced [124], along with an infographic [125], and a brief overview of key points [126]. The video and other resources were endorsed by leading organisations who had supported the project, including royal colleges, specialist societies, and quality improvement bodies, and were widely shared. We also shared the resources directly with the participants. We acknowledged the participants’ contribution as collaborators [48].
Discussion
This article presents a proposed methodological approach aimed at realising a commitment to broad participation, collaboration and consensus-building in process improvement in healthcare. Our case study shows that it is possible to deploy the approach successfully to specify process improvements in an area of pressing need during the COVID-19 pandemic. Subject to further evaluation, this approach has potentially wide application beyond the specific context in which it was tested, and may enable forms of inclusion and collaboration and listening that are otherwise very difficult to do remotely – and do so on a much greater scale. A particular strength of the approach is its ability to support mobilisation of the expertise and ingenuity of people in healthcare systems. This capability can help to enhance the currently limited infrastructure for collaborative building of specifications for process improvement in quality and safety of healthcare. It thus may have potential to address many of the problems of duplication of effort, waste, de-standardisation, and inability to engage sufficiently diverse expertise that currently characterise many quality improvement efforts in healthcare [17, 36].
Our case study showed that the proposed methodological approach can be used successfully to develop specifications for the process improvements needed to ensure high quality care, and may support the production of the kinds of high quality resources that professionals particularly value [104]. The systematic approach to participatory consensus-building that we propose is rich in potential for use in other areas that would benefit from specifying process improvement for clinical scenarios. It may help prevent the characteristic dysfunctions associated with exclusively bottom-up or top-down innovation for quality improvement [16, 17, 22]. Including a large “crowd” of stakeholders can further help mobilise the ingenuity of people in the system (e.g. patients, staff), strengthen the credibility of the consensus-built solution, and enhance feelings of ownership. It may help address lack of access to specific expertise common in locally led, bottom-up approaches [19, 36], and reduce the risks of the excessive focus on compliance associated with top-down approaches [22].
The case study included a wide range of stakeholders as collaborators, thus triangulating the perspectives and expertise of infection prevention and control experts, maternity professionals and healthcare human factors specialists in a way that would not have been possible for a single maternity unit. This blend of expertise was reflected in the participants’ recommendations: many were not tied to the specifics of PPH, but touched on important principles established by infection prevention and human factors specialists [15, 119, 127, 128]. These process improvement specifications are likely to be useful and relevant to multiple clinical communities, particularly those handling emergencies during the pandemic. Using this approach is likely to reduce waste in process improvement, since it not only produces a solution that can be used at scale within the maternity care community, it also generates many core elements of a solution that can be customised for different clinical scenarios outside of maternity. For instance, the relevance of principles of non-technical skills for teams, role assignment, leadership, and ergonomic workspace design when responding to medical emergencies can be generally applicable [127, 129].
A further strength of the approach is that it was possible to undertake this work relatively rapidly – in less than 6 weeks, despite pandemic conditions – and we anticipate that with the formalisation of our framework and with gains in experience, it may be possible for others to replicate the approach in other contexts at an even more rapid pace. Further examples of use cases would help to refine the approach and build a repository that could be used to evaluate it. Over time, the approach may facilitate further work to strengthen the infrastructure for participatory approaches in process improvement, similar to efforts over the last decades for building the infrastructure for clinical guidelines and Core Outcome Set development [1, 2, 54, 72].
Building this infrastructure is critical, because there is an ethical requirement to optimise approaches to process improvement (including specification of improvements) to reduce the risk that people may be avoidably exposed to poorer care associated with sub-optimal processes [36, 130]. Yet results of quality improvement in healthcare are typically mixed [18], suggesting the need to improve how improvement is done. Creating infrastructures for large-scale, collaborative improvement will, of course, require further development, refinement and evaluation of the approach we propose. The participatory ethos on which our approach is built may increase acceptability, uptake and impact of process improvement, but that remains to be tested. It will, for example, be important to study participant experience of taking part in these exercises (e.g. the degree of participation intensity, motivation, engagement, sense of ownership and empowerment [131, 132]), and to assess how far it is possible to reach agreements that stakeholders understand and accept. Evaluation might also examine the potential of using real-time Delphi to avoid having sequential rounds [133,134,135]. This might improve efficiency, reduce dropout, and minimise participant time, potentially without comprising user experience, inclusivity and robustness of consensus results [136] – though the implications of such an approach for the ability of different groups, such as patients and carers, to participate would require careful evaluation. Other innovations might include application of the principles of gamification to further enhance user experience [137], and provide intuitive, low-burden ways of gaining qualitative feedback on ratings in the Delphi rounds if required [138].
Given our positive experiences in using the approach in the case study, it has potential value in optimising many processes, including, for example, clinical tools and pathways. The approach may also have a role in facilitating implementation of guidance that is written at a high level of generality and requires customisation at local level for particular clinical scenarios, or in assembling relevant elements (e.g. relating to infection control and teamwork) that may be distributed across several guidelines but require integration to achieve patient care goals for specific scenarios [139,140,141].
In future work, it will important to identify the kinds of applications the approach works best for and where its limits lie. One issue, for example, is that the approach is likely well suited for specifying process improvements; locally, organisations will still need to do the work of implementing the process improvements, for example through cycles of change and monitoring implementation over time [9, 10]. Another issue concerns for what else the approach could work well for: our case study has used it to specify process improvements, but the broader consensus-building, participatory principles and methods may also have wider potential in, for example, optimising the design of equipment, forms, clinical pathways, and other applications. Evaluation should also establish when application of the approach has created sufficient learning for process improvements in one clinical area such that its learning can be applied to other related clinical scenarios without having to conduct another consensus-building exercise [129, 142, 143]. Finally, there will be an ongoing need to evaluate the process improvements specified through our proposed approach, for examining impacts on implementability, efficiency, staff and service user experience, acceptability, sustainability of change, impact on clinical outcomes, and any unintended consequences. Mixed-methods approaches are likely to be particularly useful in this regard [144].
Strengths and limitations
The case study illustrated a number of the strengths of our proposed approach. For example, it demonstrated the approach’s effectiveness in engaging a relatively large number of stakeholders as collaborators in a consensus-building exercise to specify process improvements. It also showed the feasibility of rapidly gaining feedback and reaching consensus on the process improvement (< 6 weeks), even during the first peak of the COVID-19 pandemic in the UK, with relatively little attrition (≤; 20%) between the Delphi rounds.
A limitation of the case study is that it did not involve users of maternity services themselves. However, because the methodology facilitates engagement of multiple stakeholder groups [cf. 42, 84], it is potentially well-suited to including service users, patients and carers in future projects. A further limitation is that the three stakeholder groups in our case study were not equal in size. Notwithstanding, the risk of bias appeared to be low: when each stakeholder group was analysed independently, consensus was reached for an almost identical set of recommendations as that determined for the total group. Due to the need to minimise participant burden, we did not collect demographic information that could be used to verify the representativeness of the stakeholders for the population working for or in the NHS. We also had to rely on non-probability sampling techniques that could have created some representation bias. We did, however, ensure that invitations were sent to stakeholder networks across the UK, bolstering our confidence that a representative sample was included. Finally, the case study was conducted during a fast-moving situation where the science (e.g. on infection control) was evolving very rapidly, posing the risk that recommendations made by participants could have been out of date by the time the exercise was completed. This risk was managed by close involvement of clinical expertise from the project team, and by ensuring that process improvements were specified such that they could have enduring relevance (e.g. referring to principles and national guidance on donning and doffing procedures rather than rigidly specifying them).
We were not able to evaluate the implementation or impact of the specified process improvements in the time available or compare our approach with alternative approaches to the specification of process improvements. Both are areas for future development.
Conclusion
We developed and tested a methodological approach to specifying process improvements that employed a participatory ethos and remote consensus-building methods. The approach was used successfully during pandemic conditions to build consensus among different stakeholder groups on specifying process improvements for managing an obstetric emergency in women with suspected or confirmed COVID-19. The methodological approach has significant potential to support rapid and transparent consensus-building for facilitating process improvement in various healthcare settings using online methods that can be standardised, replicated and scaled when needed, but will require further evaluation.
Availability of data and materials
Queries about the dataset should be directed to the corresponding author.
Change history
29 July 2021
A Correction to this paper has been published: https://doi.org/10.1186/s12874-021-01345-3
Abbreviations
- PPH:
-
Post-partum haemorrhage
- NHS:
-
National Health Service
- PPE:
-
Personal protective equipment
References
Brouwers MC, Spithoff K, Kerkvliet K, Alonso-Coello P, Burgers J, Cluzeau F, et al. Development and validation of a tool to assess the quality of clinical practice guideline recommendations. JAMA Netw Open. 2020;3(5):e205535.
Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8. https://doi.org/10.1093/intqhc/mzv115.
Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4(1):212. https://doi.org/10.1186/1756-0500-4-212.
Grol R, Wensing M, Grimshaw J. Improving patient care: the implementation of change in health care (third edition). Oxford: Wiley; 2020.
Chassin MR, Loeb JM. High-reliability health care: getting there from Here. Milbank Q. 2013;91(3):459–90. https://doi.org/10.1111/1468-0009.12023.
Holden RJ, Carayon P, Gurses AP, Hoonakker P, Hundt AS, Ozok AA, et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics. 2013;56(11):1669–86. https://doi.org/10.1080/00140139.2013.838643.
Alfred M, Catchpole K, Huffer E, Fredendall L, Taaffe KM. Work systems analysis of sterile processing: decontamination. BMJ Qual Saf. 2020;29(4):320–8. https://doi.org/10.1136/bmjqs-2019-009422.
Antonacci G, Reed JE, Lennox L, Barlow J. The use of process mapping in healthcare quality improvement projects. Health Serv Manag Res. 2018;31(2):74–84. https://doi.org/10.1177/0951484818770411.
The Health Foundation. Quality improvement made simple: What everyone should know about health care quality improvement. 2013. Available at: https://www.health.org.uk/publications/quality-improvement-made-simple.
Boaden R, Harvey G, Moxham C, Proudlove N: Quality improvement: theory and practice in healthcare. 2008. Available at: https://www.england.nhs.uk/improvement-hub/publication/quality-improvement-theory-practice-in-healthcare/.
Hughes RG, Ronda G. Tools and strategies for quality improvement and patient safety. In: Patient safety and quality: An evidence-based handbook for nurses. US: Agency for Healthcare Research and Quality; 2008.
Johnson JK, Barach P. Tools and strategies for continuous quality improvement and patient safety. In: Surgical Patient Care: Springer; 2017. p. 121–32.
Clegg CW. Sociotechnical principles for system design. Appl Ergon. 2000;31(5):463–77. https://doi.org/10.1016/S0003-6870(00)00009-0.
Karsh BT, Waterson P, Holden RJ. Crossing levels in systems ergonomics: a framework to support 'mesoergonomic' inquiry. Appl Ergon. 2014;45(1):45–54. https://doi.org/10.1016/j.apergo.2013.04.021.
Royal Academy of Engineering. Engineering better care: A systems approach to health and care design and continuous improvement. 2017. Available at: https://www.raeng.org.uk/publications/reports/engineering-better-care.
Dixon-Woods M, Pronovost PJ. Patient safety and the problem of many hands. BMJ Qual Saf. 2016;25(7):485–8. https://doi.org/10.1136/bmjqs-2016-005232.
Dixon-Woods M. Harveian oration 2018: improving quality and safety in healthcare. Clin Med. 2019;19(1):47–56. https://doi.org/10.7861/clinmedicine.19-1-47.
Dixon-Woods M, Martin GP. Does quality improvement improve quality? Future Hosp J. 2016;3(3):191–4. https://doi.org/10.7861/futurehosp.3-3-191.
Shorrock S. The loneliest profession in healthcare. 2018. Available at: https://humanisticsystems.com/2018/05/11/the-loneliest-profession-in-healthcare/.
Gurses AP, Tschudy MM, McGrath-Morrow S, Husain A, Solomon BS, Gerohristodoulos KA, et al. Overcoming COVID-19: what can human factors and ergonomics offer? J Patient Saf Risk Manage. 2020;25(2):49–54. https://doi.org/10.1177/2516043520917764.
Hignett S, Jones EL, Miller D, Wolf L, Modi C, Shahzad MW, et al. Human factors and ergonomics and quality improvement science: integrating approaches for safety in healthcare. BMJ Qual Saf. 2015;24(4):250–4. https://doi.org/10.1136/bmjqs-2014-003623.
Mukamel DB, Haeder SF, Weimer DL. Top-down and bottom-up approaches to health care quality: the impacts of regulation and report cards. Annu Rev Public Health. 2014;35(1):477–97. https://doi.org/10.1146/annurev-publhealth-082313-115826.
Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–33. https://doi.org/10.1056/NEJMoa2001017.
World Health Organization. Timeline: WHO’s COVID-19 response. 2020. Available at: https://www.who.int/news-room/detail/29-06-2020-covidtimeline.
Indini A, Aschele C, Cavanna L, Clerico M, Daniele B, Fiorentini G, et al. Reorganisation of medical oncology departments during the novel coronavirus disease-19 pandemic: a nationwide Italian survey. Eur J Cancer. 2020;132:17–23. https://doi.org/10.1016/j.ejca.2020.03.024.
Hanna TP, Evans GA, Booth CM. Cancer, COVID-19 and the precautionary principle: prioritizing treatment during a global pandemic. Nat Rev Clin Oncol. 2020;17(5):268–70. https://doi.org/10.1038/s41571-020-0362-6.
Royal College of Surgeons of England. Recovery of surgical services during and after COVID-19. 2020. Available at: https://www.rcseng.ac.uk/coronavirus/recovery-of-surgical-services/.
Coimbra R, Edwards S, Coimbra BC, Tabuenca A. Resuming elective surgical services in times of COVID-19 infection. Trauma Surg Acute Care Open. 2020;5(1):e000511. https://doi.org/10.1136/tsaco-2020-000511.
American College of Surgeons, American Society of Anesthesiologists, Association of periOperative Registered Nurses, American Hospital Association. Joint statement: roadmap for resuming elective surgery after COVID-19 pandemic. 2020. Available at: https://www.facs.org/covid-19/clinical-guidance/roadmap-elective-surgery.
Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiatry. 2020;7(9):813–24. https://doi.org/10.1016/S2215-0366(20)30307-2.
Royal College of Midwives, Royal College of Obstetricians & Gynaecologists. Guidance for provision of midwife-led settings and home birth in the evolving coronavirus (COVID-19) pandemic. 2020. Available at: https://www.evidence.nhs.uk/document?id=2273544&returnUrl=search%3Fq%3Dhomebirth.
NHS England. Clinical guide for the temporary reorganisation of intrapartum maternity care during the coronavirus pandemic. 2020. Available at: https://madeinheene.hee.nhs.uk/Portals/0/Clinical%20guide%20for%20the%20temporary%20reorganisation%20of%20intrapartum%20maternity%20care.pdf.
Semaan A, Audet C, Huysmans E, Afolabi B, Assarag B, Banke-Thomas A, et al. Voices from the frontline: findings from a thematic analysis of a rapid online global survey of maternal and newborn health professionals facing the COVID-19 pandemic. BMJ Glob Health. 2020;5(6):e002967. https://doi.org/10.1136/bmjgh-2020-002967.
Centers for Disease Control and Prevention (CDC). Considerations for inpatient obstetric healthcare settings. 2020. Available at: https://www.cdc.gov/coronavirus/2019-ncov/hcp/inpatient-obstetric-healthcare-guidance.html.
Chen D, Yang H, Cao Y, Cheng W, Duan T, Fan C, et al. Expert consensus for managing pregnant women and neonates born to mothers with suspected or confirmed novel coronavirus ( COVID −19) infection. Int J Gynecol Obstet. 2020;149(2):130–6. https://doi.org/10.1002/ijgo.13146.
Dixon-Woods M. How to improve healthcare improvement—an essay by Mary Dixon-Woods. BMJ. 2019;367:l5514.
Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801. https://doi.org/10.1111/hex.12888.
Aveling EL, Martin G, Armstrong N, Banerjee J, Dixon-Woods M. Quality improvement through clinical communities: eight lessons for practice. J Health Organ Manage. 2012;26(2):158–74. https://doi.org/10.1108/14777261211230754.
Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89(2):167–205. https://doi.org/10.1111/j.1468-0009.2011.00625.x.
Fowler A. Authentic NGDO partnerships in the new policy agenda for international aid: dead end or light ahead? Dev Chang. 1998;29(1):137–59. https://doi.org/10.1111/1467-7660.00073.
Kahssay HM, Oakley P, World Health Organization. Community involvement in health development: a review of the concept and practice. 1999. Available at: https://apps.who.int/iris/handle/10665/42151.
Marjanovic S, Ball S, Harshfield A, Dimova S, Prideaux R, Carpenter A, Punch D, Simmons RK. Involving NHS staff in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/involving-nhs-staff-in-research/.
Eyre L, Farrelly M, Marshall M. What can a participatory approach to evaluation contribute to the field of integrated care? BMJ Qual Saf. 2017;26(7):588–94. https://doi.org/10.1136/bmjqs-2016-005777.
Parks S, d'Angelo C, Gunashekar S. Citizen science: generating ideas and exploring consensus. The Healthcare Improvement Studies (THIS) Institute 2018. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/citizen-science-generating-ideas-exploring-consensus/
Guazzini A, Vilone D, Donati C, Nardi A, Levnajić Z. Modeling crowdsourcing as collective problem solving. Sci Rep. 2015;5(1):–16557. https://doi.org/10.1038/srep16557.
Franzoni C, Sauermann H. Crowd science: the organization of scientific research in open collaborative projects. Res Policy. 2014;43(1):1–20. https://doi.org/10.1016/j.respol.2013.07.005.
Breidbach CF, Brodie RJ. Engagement platforms in the sharing economy. J Serv Theory Pract. 2017;27(4):761–77. https://doi.org/10.1108/JSTP-04-2016-0071.
Cappa F, Laut J, Porfiri M, Giustiniano L. Bring them aboard: rewarding participation in technology-mediated citizen science projects. Comput Hum Behav. 2018;89:246–57. https://doi.org/10.1016/j.chb.2018.08.017.
European Citizen Science Association. Ten principles of citizen science. 2015. Available at: https://osf.io/xpr2n/.
Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018. https://doi.org/10.2307/j.ctv550cf2.
Lichten C, Ioppolo R, D'Angelo C, Simmons RK, Jones MM. Citizen science: crowdsourcing for research. The Healthcare Improvement Studies (THIS) Institute 2018. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/citizen-science-crowdsourcing-research/.
Jones J, Hunter D. Qualitative research: consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80. https://doi.org/10.1136/bmj.311.7001.376.
Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2008;26(1):5–14. https://doi.org/10.1111/j.1365-2710.2001.00331.x.
Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(S3):280.
Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):1–88.
Innes JE. Planning through consensus building: a new view of the comprehensive planning ideal. J Am Plan Assoc. 1996;62(4):460–72. https://doi.org/10.1080/01944369608975712.
Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94. https://doi.org/10.1111/j.1440-172X.2012.02017.x.
Van De Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–42. https://doi.org/10.2105/ajph.62.3.337.
Lomas J, Anderson G, Enkin M, Vayda E, Roberts R, MacKinnon B. The role of evidence in the consensus process: results from a Canadian consensus exercise. JAMA. 1988;259(20):3001–5. https://doi.org/10.1001/jama.1988.03720200023027.
Institute of Medicine (US) Council on Health Care Technology, Goodman C, Baratz SR. Improving consensus development for health technology assessment: An international perspective. Washington: National Academies Press (US); 1990.
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. The RAND/UCLA appropriateness method user's manual. Santa Monica: Rand Corp; 2001.
Brook RH. Assessing the appropriateness of care—its time has come. JAMA. 2009;302(9):997–8. https://doi.org/10.1001/jama.2009.1279.
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67. https://doi.org/10.1287/mnsc.9.3.458.
Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang. 2006;73(5):467–82. https://doi.org/10.1016/j.techfore.2005.09.002.
Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48. https://doi.org/10.1177/135581969900400410.
Banno M, Tsujimoto Y, Kataoka Y. The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines. J Clin Epidemiol. 2020;124:50–7. https://doi.org/10.1016/j.jclinepi.2020.04.010.
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of Health Research reporting guidelines. PLoS Med. 2010;7(2):e1000217. https://doi.org/10.1371/journal.pmed.1000217.
Hsu C-C, Sandford BA. The Delphi Technique: Making Sense of Consensus. Pract Assess Res Eval. 2007;12:10.
Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12. https://doi.org/10.1111/j.1365-2648.2006.03716.x.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. https://doi.org/10.1371/journal.pone.0020476.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.
Brouwers MC, Kerkvliet K, Spithoff K. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016:i1152. https://doi.org/10.1136/bmj.i1152.
Graham ID, Kothari A, McCutcheon C. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13(1):1.
Brereton L, Ingleton C, Gardiner C, Goyder E, Mozygemba K, Lysdahl KB, et al. Lay and professional stakeholder involvement in scoping palliative care issues: methods used in seven European countries. Palliat Med. 2017;31(2):181–92. https://doi.org/10.1177/0269216316649154.
Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. https://doi.org/10.1177/0269216317690685.
Herodotou C, Aristeidou M, Sharples M, Scanlon E. Designing citizen science tools for learning: lessons learnt from the iterative development of nQuire. Res Pract Technol Enhanc Learn. 2018;13(1):1.
Wald DM, Longo J, Dobell AR. Design principles for engaging and retaining virtual citizen scientists. Conserv Biol. 2016;30(3):562–70. https://doi.org/10.1111/cobi.12627.
Boers M, Kirwan JR, Wells G, Beaton D, Gossec L, D'Agostino M-A, et al. Developing Core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67(7):745–53. https://doi.org/10.1016/j.jclinepi.2013.11.013.
Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x.
Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82. https://doi.org/10.1046/j.1365-2648.2003.02537.x.
Hussler C, Muller P, Rondé P. Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technol Forecast Soc Chang. 2011;78(9):1642–53. https://doi.org/10.1016/j.techfore.2011.07.008.
Ioannidis JPA. Professional societies should abstain from authorship of guidelines and disease definition statements. Circulation. 2018;11(10):e004889.
Hutchings A, Raine R. A systematic review of factors affecting the judgments produced by formal consensus development methods in health care. J Health Serv Res Policy. 2006;11(3):172–179H. https://doi.org/10.1258/135581906777641659.
Marjanovic S, Harshfield A, Carpenter A, Bertscher A, Punch D, Ball S. Involving patients and the public in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/involving-patients-public-research/.
Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5(1):37.
Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7(1):52. https://doi.org/10.1186/1471-2288-7-52.
West S, Pateman R. Recruiting and retaining participants in citizen science: what can be learned from the volunteering literature? Citiz Sci. 2016;1(2):15.
Brouwer S, Hessels LK. Increasing research impact with citizen science: the influence of recruitment strategies on sample diversity. Public Underst Sci. 2019;28(5):606–21. https://doi.org/10.1177/0963662519840934.
Fiske A, Prainsack B, Buyx A. Meeting the needs of underserved populations: setting the agenda for more inclusive citizen science of medicine. J Med Ethics. 2019;45(9):617–22. https://doi.org/10.1136/medethics-2018-105253.
De Las ND, Hacker K, DiGirolamo A, Hicks LS. A systematic review of community-based participatory research to enhance clinical trials in racial and ethnic minority groups. Health Serv Res. 2012;47(3pt2):1363–86.
Hsu C-C, Sandford BA. Minimizing non-response in the Delphi process: How to respond to non-response. Pract Assess Res Eval. 2007;12(1):17.
Ogrinc G, Nelson WA, Adams SM, O'Hara AE. An instrument to differentiate between clinical research and quality improvement. Irb. 2013;35(5):1–8.
Healthcare Quality Improvement Partnership (HQIP). Guide to managing ethical issues in quality improvement or clinical audit projects. 2021. Available at: https://www.hqip.org.uk/resource/guide-to-managing-ethical-issues-in-quality-improvement-or-clinical-audit-projects/#.YGTi769KhnI.
Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decis Sci. 1997;28(3):763–74. https://doi.org/10.1111/j.1540-5915.1997.tb01330.x.
Schmidt R, Lyytinen K, Keil M, Cule P. Identifying software project risks: an international Delphi study. J Manag Inf Syst. 2001;17(4):5–36. https://doi.org/10.1080/07421222.2001.11045662.
Brady SR. Utilizing and adapting the Delphi method for use in qualitative research. Int J Qual Methods. 2015;14(5):160940691562138. https://doi.org/10.1177/1609406915621381.
Hirschhorn F. Reflections on the application of the Delphi method: lessons from a case in public transport research. Int J Soc Res Methodol. 2019;22(3):309–22. https://doi.org/10.1080/13645579.2018.1543841.
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9. https://doi.org/10.1016/j.jclinepi.2013.12.002.
Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19(1):173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173.
Renfrew MJ, Cheyne H, Craig J, Duff E, Dykes F, Hunter B, et al. Sustaining quality midwifery care in a pandemic and beyond. Midwifery. 2020;88:102759. https://doi.org/10.1016/j.midw.2020.102759.
Coxon K, Turienzo CF, Kweekel L, Goodarzi B, Brigante L, Simon A, et al. The impact of the coronavirus (COVID-19) pandemic on maternity care in Europe. Midwifery. 2020;88:102779.
Knight M, Bunch K, Vousden N, Morris E, Simpson N, Gale C, et al. Characteristics and outcomes of pregnant women admitted to hospital with confirmed SARS-CoV-2 infection in UK: national population based cohort study. BMJ. 2020:m2107. https://doi.org/10.1136/bmj.m2107.
Royal College of Obstetricians & Gynaecologists, Royal College of Midwives, Royal College of Paediatrics and Child Health, Public Health England, Public Health Scotland. Coronavirus (COVID-19) infection in pregnancy. 2020. Available at: https://www.rcog.org.uk/coronavirus-pregnancy.
Rimmer M, Al Wattar B, Barlow C, Black N, Carpenter C, Conti-Ramsden F, et al. Provision of obstetrics and gynaecology services during the COVID-19 pandemic: a survey of junior doctors in the UK National Health Service. BJOG Int J Obstet Gynaecol. 2020;127(9):1123–8. https://doi.org/10.1111/1471-0528.16313.
Mehrabadi A, Hutcheon J, Lee L, Kramer M, Liston R, Joseph K. Epidemiological investigation of a temporal increase in atonic postpartum haemorrhage: a population-based retrospective cohort study. BJOG Int J Obstet Gynaecol. 2013;120(7):853–62. https://doi.org/10.1111/1471-0528.12149.
van Stralen G, von Schmidt auf Altenstadt JF, Bloemenkamp KW, van Roosmalen J, Hukkelhoven CW. Increasing incidence of postpartum hemorrhage: the Dutch piece of the puzzle. Acta Obstet Gynecol Scand. 2016;95(10):1104–10. https://doi.org/10.1111/aogs.12950.
Knight M, Bunch K, Tuffnell D, Jayakody H, Shakespeare J, Kotnis R, et al. Saving Lives, Improving Mothers’ Care - Lessons learned to inform maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2014–16. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2018.
Dunning T, Harris JM, Sandall J. Women and their birth partners’ experiences following a primary postpartum haemorrhage: a qualitative study. BMC Pregnancy Childbirth. 2016;16(1):80.
Say L, Chou D, Gemmill A, Tunçalp Ö, Moller A-B, Daniels J, et al. Global causes of maternal death: a WHO systematic analysis. Lancet Glob Health. 2014;2(6):e323–33. https://doi.org/10.1016/S2214-109X(14)70227-X.
Zwart JJ, Dupuis JRO, Richters A, Öry F, Van Roosmalen J. Obstetric intensive care unit admission: a 2-year nationwide population-based cohort study. Intensive Care Med. 2010;36(2):256–63. https://doi.org/10.1007/s00134-009-1707-x.
Mavrides E, Allard S, Chandraharan E, Collins P, Green L, Hunt B, et al. Prevention and management of postpartum Haemorrhage. BJOG Int J Obstet Gynaecol. 2016;124(5):e106–49.
National Institute for Health and Care Excellence (NICE). Intrapartum care for healthy women and babies. 2017. Available at: https://www.nice.org.uk/guidance/CG190.
Gayet-Ageron A, Prieto-Merino D, Ker K, Shakur H, Ageron F-X, Roberts I, et al. Effect of trekeratment delay on the effectiveness and safety of antifibrinolytics in acute severe haemorrhage: a meta-analysis of individual patient-level data from 40 138 bleeding patients. Lancet. 2018;391(10116):125–32. https://doi.org/10.1016/S0140-6736(17)32455-8.
Shakur H, Roberts I, Fawole B, Chaudhri R, El-Sheikh M, Akintan A, et al. Effect of early tranexamic acid administration on mortality, hysterectomy, and other morbidities in women with post-partum haemorrhage (WOMAN): an international, randomised, double-blind, placebo-controlled trial. Lancet. 2017;389(10084):2105–16. https://doi.org/10.1016/S0140-6736(17)30638-4.
Liberati EG, Tarrant C, Willars J, Draycott T, Winter C, Chew S, et al. How to be a very safe maternity unit: an ethnographic study. Soc Sci Med. 2019;223:64–72. https://doi.org/10.1016/j.socscimed.2019.01.035.
Draycott T. Practical obstetric multi-professional training – PROMPT. 2013. Available at: https://improve.bmj.com/sites/default/files/resources/prompt.pdf.
Richmond H, Copsey B, Hall AM, Davies D, Lamb SE. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ. 2017;17(1):227.
Mobasheri MH, King D, Johnston M, Gautama S, Purkayastha S, Darzi A. The ownership and clinical use of smartphones by doctors and nurses in the UK: a multicentre survey study. BMJ Innov. 2015;1(4):174–81. https://doi.org/10.1136/bmjinnov-2015-000062.
Gosbee J. Human factors engineering and patient safety. Qual Saf Health Care. 2002;11(4):352–4. https://doi.org/10.1136/qhc.11.4.352.
Fricker RD. Sampling methods for web and e-mail surveys. In: Fielding N, Lee RM, Blank G, editors. The SAGE handbook of online research methods. London: SAGE Publications Ltd; 2008. https://doi.org/10.4135/9780857020055.n11.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. https://doi.org/10.1186/1745-6215-13-132.
Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409.
Maclennan S, Kirkham J, Lam TBL, Williamson PR. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. J Clin Epidemiol. 2018;93:1–8. https://doi.org/10.1016/j.jclinepi.2017.09.024.
The Healthcare Improvement Studies (THIS) Institute. COVID-19: managing an obstetric emergency (video). 2020. Available at: https://www.thisinstitute.cam.ac.uk/research-articles/covid-19-managing-an-obstetric-emergency/.
The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (Infographic). 2020. Available at: https://www.thisinstitute.cam.ac.uk/wp-content/uploads/2020/11/TI_Covid_Obstertric_Emergency_A4_Graphic.pdf.
The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (information sheet). 2020. Available at: https://mk0thisinstitut0r1om.kinstacdn.com/wp-content/uploads/2020/11/TI_Covid_Obstertric_Emergency_A4_Colour.pdf.
Catchpole K. Improving performance through human-centred reconfiguration of existing designs. BMJ Qual Saf. 2013;22(1):5–7. https://doi.org/10.1136/bmjqs-2012-001677.
National Institute for Health and Care Excellence (NICE). Infection prevention and control. 2014. Available at: https://www.nice.org.uk/guidance/qs61.
Brogaard L, Kierkegaard O, Hvidman L, Jensen K, Musaeus P, Uldbjerg N, et al. The importance of non-technical performance for teams managing postpartum haemorrhage: video review of 99 obstetric teams. BJOG. 2019;126:1015.
Watson SI, Dixon-Woods M, Taylor CA, Wroe EB, Dunbar EL, Chilton PJ, et al. Revising ethical guidance for the evaluation of programmes and interventions not initiated by researchers. J Med Ethics. 2020;46(1):26–30. https://doi.org/10.1136/medethics-2018-105263.
Tredick CA, Lewison RL, Deutschman DH, Hunt TA, Gordon KL, Von Hendy P. A rubric to evaluate citizen-science programs for long-term ecological monitoring. BioScience. 2017;67(9):834–44. https://doi.org/10.1093/biosci/bix090.
Kieslinger B, Schäfer T, Heigl F, Dörler D, Richter A, Bonn A. Evaluating citizen science-towards an open framework. In: Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018.
Gordon T, Pease A. RT Delphi: an efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Chang. 2006;73(4):321–33. https://doi.org/10.1016/j.techfore.2005.09.005.
Gnatzy T, Warth J, von der Gracht H, Darkow I-L. Validating an innovative real-time Delphi approach - a methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Chang. 2011;78(9):1681–94. https://doi.org/10.1016/j.techfore.2011.04.006.
Gordon TJ. The real-time Delphi method. Futures Res Methodol Version. 2009;3:19.
Rowe G, Wright G. The Delphi technique: past, present, and future prospects — introduction to the special issue. Technol Forecast Soc Chang. 2011;78(9):1487–90. https://doi.org/10.1016/j.techfore.2011.09.002.
Ponti M, Hillman T, Kullenberg C, Kasperowski D. Getting it Right or Being Top Rank: Games in Citizen Science. Citizen Science. Theory and Pract. 2018;3(1):1–12. https://doi.org/10.5334/cstp.101.
Veugelers R, Gaakeer MI, Patka P, Huijsman R. Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response. BMC Med Res Methodol. 2020;20(1):156.
Hall L, Eccles M. Case study of an inter-professional and inter-organisational programme to adapt, implement and evaluate clinical guidelines in secondary care. Br J Clin Gov. 2000;5(2):72–82. https://doi.org/10.1108/14664100010343863.
Cook DJ, Pulido JN, Thompson JE, Dearani JA, Ritter MJ, Hanson AC, et al. Standardized practice design with electronic support mechanisms for surgical process improvement: reducing mechanical ventilation time. Ann Surg. 2014;260(6):1011–5. https://doi.org/10.1097/SLA.0000000000000726.
Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implement Sci. 2018;13(1):72.
Patey R, Flin R, Fletcher G, Maran N, Glavin R. Developing a Taxonomy of Anesthetists’ Nontechnical Skills (ANTS). In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005. https://pubmed.ncbi.nlm.nih.gov/21250033/.
Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13(Suppl 1):i85–90. https://doi.org/10.1136/qshc.2004.010033.
Portela MC, Pronovost JP, Woodcock T, Carter P, Mary D-W. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24(5):325–36. https://doi.org/10.1136/bmjqs-2014-003620.
Acknowledgements
Thiscovery Group: André Sartori, Andy Paterson, Doro Unger-Lee, Joann Leeding, Luke Steer.
Obstetric Emergency Consensus Group: Amanda Andrews, Rita Arya, Sarah F. Bell, Denise Chaffer, Andrew Cooney, Rachel Corry, Mair G.P. Davies, Lisa Duffy, Caroline Everden, Theresa Fitzpatrick, Courtney Grant, Mark Hellaby, Tracey A. Herlihey, Sue Hignett, Sarah Hookes, Fran R. Ives, Gyuchan T. Jun, Owen J. Marsh, Tanya R. Matthews, Celine McKeown, Alexandra Merriman, Giulia Miles, Susan Millward, Neil Muchatata, David Newton, Valerie G. Noble, Pamela Page, Vincent Pargade, Sharon P. Pickering, Laura Pickup, Dale Richards, Cerys Scarr, Jyoti Sidhu, James Stevenson, Ben Tipney, Stephen Tipper, Jo Wailling, Susan P. Whalley-Lloyd, Christian Wilhelm, Juliet J. Wood.
We thank the participants in this project. We thank the Infection Prevention Society, Each Baby Counts, and the Clinical Human Factors Group for help with recruitment. We thank the volunteers who participated in both shoots of the video, and North Bristol NHS Trust and the PROMPT Maternity Foundation for supporting the videos.
We thank the Royal College of Midwives, the Royal College of Obstetricians and Gynaecologists, Each Baby Counts, the Infection Prevention Society, the Association of Obstetric Anaesthetists, the PROMPT Maternity Foundation and the Health Foundation for their endorsement and support of the resources.
We thank Becky Kenny for help with project management. We thank the communications team at THIS Institute for their work on developing the communications resources.
We thank Jenny George, Emily Ryen Gloinson and Carolina Feijao of RAND Europe for their contributions to data analysis.
Funding
This project was supported by the Health Foundation’s grant to the University of Cambridge for The Healthcare Improvement Studies (THIS) Institute. The Health Foundation is an independent charity committed to bringing about better health and health care for people in the UK. Mary Dixon-Woods is a National Institute for Health Research (NIHR) Senior Investigator (NF-SI-0617-10026). The funders had no role in study design, data collection and analysis, interpretation of the data, and preparation of the manuscript.
Author information
Authors and Affiliations
Consortia
Contributions
MD-W and TD came up with the idea for the project. JWvdS led the design of the project, co-led data collection and analysis, and co-led drafting of the manuscript. MW contributed to the design of the consensus-building exercise, co-led data collection and analysis, and drafting of the manuscript. AA contributed to the design of the consensus-building exercise, data collection, and drafting of the manuscript. TD contributed to the design of the project and drafting of the manuscript. CW contributed to the design of the project and drafting of the manuscript. GM contributed to the design and oversight of the project, and drafting of the manuscript. KK contributed to the drafting of the manuscript. NR contributed to the data analysis and drafting of the manuscript. RK contributed to the oversight of the project and contributed to drafting of the manuscript, and led the Thiscovery Group. MD-W contributed to the design of the project, supervised the project and co-led drafting of the manuscript. The final version of the manuscript was read and approved by all authors. The final version of the manuscript was also read and approved by the members of the Thiscovery Authorship Group and the Obstetric Emergency Authorship Group (see acknowledgements).
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The UK’s Health Research Authority decision tool (http://www.hra-decisiontools.org.uk/research/) showed that ethics approval was not required for the consensus-building exercise, as the project was a consultation and engagement exercise classified as a quality improvement activity [92, 93], in which all of the participants were invited to join the authorship group and to be acknowledged in the project’s outputs. Participants provided written online consent to the terms and conditions regarding the Thiscovery platform’s privacy policy and terms of use (https://www.thiscovery.org/register/). Participants were provided with information about the project, and re-confirmed their consent before each new round of the consensus-building exercise.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original online version of this article was revised: It has been found that Fig. 3B has an error of misplacement. In Fig. 3B, the images of Oil red O staining of liver sections from control (I) and palmitic acid (II) had been misplaced.
Supplementary Information
Additional file 1: Supplement 1.
Synthesis process of the 912 responses to the recommendation task.
Additional file 2: Supplement 2.
Interactive bar charts of ratings across the three stakeholder groups as presented to participants in the second Delphi round.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
van der Scheer, J.W., Woodward, M., Ansari, A. et al. How to specify healthcare process improvements collaboratively using rapid, remote consensus-building: a framework and a case study of its application. BMC Med Res Methodol 21, 103 (2021). https://doi.org/10.1186/s12874-021-01288-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12874-021-01288-9