Skip to main content

How to specify healthcare process improvements collaboratively using rapid, remote consensus-building: a framework and a case study of its application

A Correction to this article was published on 29 July 2021

This article has been updated



Practical methods for facilitating process improvement are needed to support high quality, safe care. How best to specify (identify and define) process improvements – the changes that need to be made in a healthcare process – remains a key question. Methods for doing so collaboratively, rapidly and remotely offer much potential, but are under-developed. We propose an approach for engaging diverse stakeholders remotely in a consensus-building exercise to help specify improvements in a healthcare process, and we illustrate the approach in a case study.


Organised in a five-step framework, our proposed approach is informed by a participatory ethos, crowdsourcing and consensus-building methods: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvement in light of learning from the exercise. We tested the approach in a case study that sought to specify process improvements for the management of obstetric emergencies during the COVID-19 pandemic. We used a brief video showing a process for managing a post-partum haemorrhage in women with COVID-19 to elicit recommendations on how the process could be improved. Two Delphi rounds were then conducted to reach consensus.


We gathered views from 105 participants, with a background in maternity care (n = 36), infection prevention and control (n = 17), or human factors (n = 52). The participants initially generated 818 recommendations for how to improve the process illustrated in the video, which we synthesised into a set of 22 recommendations. The consensus-building exercise yielded a final set of 16 recommendations. These were used to inform the specification of process improvements for managing the obstetric emergency and develop supporting resources, including an updated video.


The proposed methodological approach enabled the expertise and ingenuity of diverse stakeholders to be captured and mobilised to specify process improvements in an area of pressing service need. This approach has the potential to address current challenges in process improvement, but will require further evaluation.

Peer Review reports


The last three decades have seen clinical guidelines, defined as “systematically developed statements informed by a systematic review of evidence and an assessment of the benefits and harms of care options designed to optimize patient care” [1], become a cornerstone of evidence-based practice. Production of guidelines is based on well-established methodologies, including synthesis of scientific evidence, expert opinion, and stakeholder consultation, and is supported by an infrastructure of national and international bodies (e.g. government agencies and professional associations) [2]. Clinical guidelines are not, of course, self-implementing. Getting clinical guidance into practice is typically complex, requiring multi-modal approaches, and is the subject of a burgeoning science and associated literature [3, 4]. It is now clear, however, that the implementation of evidence-based practices (what should be done) depends crucially on process improvement – changes to how things are done [5, 6]. For this reason, the specification of process improvements (i.e. identifying and defining the changes in processes that need to be made to deliver good care) is a key task. In this article, we offer an approach for developing specifications for process improvements using rapid, remote, consensus-building methods, and we illustrate it using a case study conducted during the COVID-19 pandemic.

We start by noting that there is no consensual definition of process improvement, but it is distinguished by its focus on how to improve the underlying processes (such as workflows, task design, role allocations, communication techniques, resources required, and so on) for delivering care – rather than, as in the case of clinical guidelines, defining ideal clinical standards. Process improvement has a particular role in ensuring that work systems are optimised, for example by helping to define the activities from beginning to end of a clinical process or pathway; to explain how these activities can most effectively be undertaken; to clarify tasks, roles, and skills needed; to characterise the decisions to be made and the support needed to make and implement those decisions; and to identify the equipment, resources and other tools required [7, 8].

The various methods for process improvement are often gathered together under the rubric of quality improvement [9, 10]. Some, such as the Model for Improvement, Lean and Six Sigma, amongst others [11], have been adapted from industry techniques [12]. Other approaches have been developed from the design and engineering disciplines and draw on socio-technical systems principles. For example, human factors and systems engineering use structured methods to change existing work systems based on the analysis of the interactions between people, tasks, tools, technology and the environment [13,14,15].

Common to most of these approaches is the need to specify process improvements. Specification requires identifying and defining the changes that need to be made (for example, an amendment to task or role design, or use of a new piece of equipment) to bring about improvement. A major challenge, however, is that the large-scale infrastructure for developing specifications for process improvement in healthcare has, in contrast with clinical guideline development, remained under-developed. The work of process improvement specification has instead remained largely locally-led, conducted within individual organisations.

Local leadership of process improvement has, of course, many advantages, including the potential to customise a solution to local circumstances and to imbue a sense of local ownership, but it is also associated with some disadvantages. One is that each individual organisation facing circumstances requiring change may develop their own process improvement specification in isolation [16, 17], but a multiplicity of approaches to the same area of practice may cause problems of its own. First, it may be wasteful, with each organisation reinventing the wheel [17, 18]. Second, it may be sub-optimal: reaching the best possible solution requires inputs from multiple disciplines, but individual organisations may lack access to the fullest range of expertise, particularly when it is rare (e.g. specialist human factors knowledge) [19,20,21]. Third, destandardisation creates learning overheads and new risks, for example when personnel moving between institutions have to learn new ways of doing things while unlearning previous ones. But while exclusively bottom-up development of ways of working may not be ideal, top-down imposition of particular practices can generate other pathologies [22], including failures of implementation, truculent and dysfunctional compliance, inadequate customisation to local circumstances, and creation of perverse incentives.

The deficiencies in some of the current infrastructure for process improvement have been vividly surfaced by the COVID-19 pandemic, which has caused massive disruption in the organisation and delivery of healthcare [23, 24]. Rapid innovation has been a feature of the response, prompted by the need to make adaptations to established clinical processes to address the infection risks and other challenges associated with the virus, as well as many other changes to clinical pathways and practices [25,26,27,28,29,30,31,32,33,34,35]. While the scale and speed of response to the pandemic has been impressive, an important risk is that some efforts may unhelpfully reproduce some of the challenges previously identified in the field of quality improvement [17, 36], including those discussed above of duplication of effort, waste, de-standardisation, and inability to engage sufficiently diverse expertise.

Given the challenges of both top-down and bottom-up quality improvement, a more collaborative alternative is likely to be of value. In this article, we address this need. We propose a methodological approach designed to enable large-scale remote engagement and mobilisation of multiple forms of expertise to build rapid consensus on specifications of process improvements, and we describe a case study of its application.


In developing our approach, we built on a participatory ethos, principles of crowd-sourcing, and consensus-building methods.

Participatory ethos

A participatory ethos – an approach that values the perspectives of the full range of groups of people affected by an issue – is an important guiding value in healthcare improvement [37]. But it is also of practical significance: securing participation may be more likely to result in solutions that are satisfying, workable, informed by professional values and clinical expertise, capable of being customised for specific situations, and capable of being implemented through collective effort rather than harsh, externally-imposed sanctions [38, 39]. Participatory approaches may also be more likely to lead to sustainable impacts by generating a sense of local ownership and commitment [40, 41]. Participatory approaches may be especially useful in encouraging practitioners to engage with evidence and its creation [42], as well as generating findings that have impact on practice [43].


By drawing on the collective intelligence of many individuals, crowdsourcing can enable data to be collated on a much greater scale than would otherwise be possible [44], creating potential to solve problems by drawing on a wider range of perspectives and diverse experiences and knowledge [45]. Recent advances such as online engagement platforms [46,47,48] are now facilitating engagement of large, diverse and geographically dispersed stakeholders remotely as collaborators in co-constructing solutions [44, 49,50,51].

Consensus-building methods

Consensus-building methods are well established as ways of promoting deliberation, inclusion, and participation in situations where there may be multiple perspectives, interests and communities [52,53,54,55,56]. Methods widely used in developing guidance in healthcare include the nominal group technique [57, 58], the consensus development conference [59, 60], the RAND/UCLA appropriateness method [61, 62], and the Delphi method [63, 64]. These approaches are commonly recommended and used for developing clinical guidelines [55, 65] and reporting guidelines [66, 67], but application of consensus-building in process improvement has remained much more limited, not least because of the tendency (discussed above) to see process improvement as the domain of local teams. Yet consensus-building is potentially of value for process improvement in helping to build shared understanding, to include diverse forms of expertise, and to produce agreements about process improvement that might otherwise remain elusive.

The Delphi method offers considerable promise in this respect, as one of the best known and most widely used approaches for consensus-building in healthcare contexts [52, 54]. Using group communication that brings together and synthesises knowledge, participants are typically involved in a number of rounds of rating or voting on a set of propositions, and may then adjust their initial ratings based on feedback from the group in a number of subsequent iterations [68, 69]. Though Delphi can include a large number of individuals across diverse locations and areas of expertise [54], many Delphi exercises for healthcare have only involved relatively small and homogeneous panels of approximately 10 to 30 participants [70]. There is evident scope for including larger and diverse groups of participants [54, 70]. New methodological approaches are needed to use the method effectively for large-scale remote consensus-building to specify process improvements, while adhering to a participatory ethos and minimising time and effort required of participants.

A proposed methodological approach for developing specifications for process improvements using rapid, remote, consensus-building methods

The approach we propose for rapid consensus-building of process improvement specifications involves five steps, which we have organised into a framework (Table 1). The steps are: (1) define scope and objective of the process improvement; (2) produce a draft or prototype of the proposed process improvement specification; (3) identify participant recruitment strategy; (4) design and conduct a remote consensus-building exercise; (5) produce a final specification of the process improvements in light of learning from the exercise.

Table 1 Framework for rapid, participatory, remote consensus-building for process improvement specification

(1) Define scope and objective of the process improvement

A first step is to identify and characterise the problem to be solved and the goal to be achieved so that an assessment can be made of whether consensus-building is an appropriate method [71]. If consensus-building is selected, the next step is to define the scope and objective of the process improvement, similar to current practices for developing clinical practice guidelines [72] and Core Outcome Sets [54]. These activities require close collaboration with key stakeholders, in accordance with the participatory principle [37, 73, 74].

(2) Produce a draft or prototype of the proposed process improvement specification

Rapid consensus-building on the specification of process improvements can be facilitated by producing a draft or prototype version, which might be informed by rapid literature reviews, existing guidelines or practices, ideas sourced from specialist groups, or stakeholder surveys, interviews and focus groups [70, 75]. Feedback can then be sought on this draft/prototype, informing subsequent rounds of Delphi consensus-building. The draft/prototype may take a range of forms [75], for example a conceptual framework, an existing standard operating procedure, a video-based simulated scenario, or an artefact (e.g. equipment or software). The most appropriate form can be selected based on the technical aspects of the healthcare process, alignment with the goals of the final product, expectations of participants’ available time, questions of how to maximise participant engagement, and possibilities and limitations of (mobile) devices and platforms on which the draft/prototype will be presented [76, 77].

(3) Identify participant recruitment strategy


Determining participant eligibility criteria requires consideration of the need for triangulation and bringing together the views of different types of stakeholders [54, 55, 78,79,80]. Seeking diversity can reduce risk of bias and provide a richer variety of views [78, 81,82,83]. Eligibility criteria might include specialists working in the selected field of clinical practice (e.g. maternity staff) or those with specialist expertise (e.g. infection prevention, human factors). Importantly, for many process improvement activities, patients and the public are key stakeholders whose expertise and perspectives should be included [42, 84].

Sample size

The current literature provides no set standards for required sample sizes for Delphi exercises, meaning that pragmatic choices have to be made [54, 85]. One approach is to estimate the sample size based on the number that would likely result in stable ratings across the Delphi rounds, accommodating for dropout [85, 86]. This may require establishing a minimal sample size for each stakeholder group included.

Sampling and recruitment

Sampling strategies for consensus-building exercises are generally informed by available time and resources, and include convenience, purposive or criterion sampling [71, 85]. Recruitment strategies can, for example, be grounded in the voluntary contribution of willing individuals who wish to contribute to the production of scientific knowledge [87,88,89]. The principles of participatory research may inform successful strategies for increasing participation of minority groups [89, 90].


Measures to support retention across the different phases of the consensus-building exercise may involve strategies such as e-mail reminders [91] that include reference to the importance of the participant’s contributions [87]. This could be enhanced by the use of online engagement platforms [46,47,48], which may help encourage feelings of being part of a research community.

(4) Design and conduct a remote consensus-building exercise


The ethical principles for participatory consensus-building exercises have much in common with any other quality improvement activity [92, 93], or indeed guideline development, in that they may often be classified as service improvement activities not requiring specific ethical review. Where these exercises are conducted as research studies rather than improvement activities, different considerations may apply, such as the requirement for oversight and/or approval by an Institutional Review Board or Research Ethics Committee [92, 93].

Idea generation

Delphi rounds are typically informed by an exercise to generate ideas [63, 64]. This might be done by inviting participant feedback on the draft/prototype of the process improvement specification, which can then be synthesised in a set of propositions to be rated in the subsequent Delphi rounds. If this approach is taken, using open-ended, short-answer options may work best for a rapid response with minimal burden for participants and analysts [94]. The synthesis method depends on the type of feedback generated, and needs to be clearly documented [75]. Synthesis might involve, for example, removal of duplicates and merging of responses with similar wording [95]; using thematic analysis to create concepts, categories or themes [96]; or coding individual responses to themes and triangulating the coding among multiple researchers [95, 97]. The aim is to balance the consolidation of responses (i.e. creating a list that is reasonable for experts to navigate in subsequent rounds, e.g. about 20 items [94, 95]) while avoiding excessive abstraction [97].

Delphi rounds

The propositions derived from the idea generation phase can be used in two or more subsequent Delphi rounds of iterative surveys in which participants state their level of agreement with propositions on a numeric scale. Responses are aggregated and participants have the opportunity to revise their judgments in the light of feedback that includes their own and the group’s judgment, with the aim of exploring or reaching group consensus [98]. Two rounds is often sufficient to reach consensus and may reduce burden for participants [70, 75], though more rounds may sometimes be used. Use of visually appealing forms of feedback, such as interactive graphs that show the distributions of ratings across one or more stakeholder groups, can facilitate response across rounds.

(5) Produce a final specification of the process improvements in light of learning from the exercise

The aim of the Delphi exercise is to produce recommendations that can inform the specification of process improvements that can be implemented at scale. A plan should be in place for developing and disseminating supporting resources that reflect the specified process improvements. One advantage of the collaborative approach is that it is likely to facilitate engaged dissemination and implementation by those who have participated [99].

Case study

We tested the proposed methodological framework using an example from emergency maternity care. The need for process improvement specification in maternity care during the COVID-19 crisis was particularly urgent given that, in contrast with some other areas of care, it is not possible to defer or reschedule births [100, 101]. During the first few months of the pandemic [23, 24], approximately 100 pregnant women with suspected or confirmed COVID-19 were admitted each week to obstetric units across the UK [102]. This meant that many existing areas of care where good clinical practice was well-established (e.g. through clinical guidelines) required process improvement to adapt to the need for: infection control, the challenges of communication and teamwork likely to be posed by use of personal protective equipment (PPE), and other demands of making clinical processes COVID-safe. It was important, for example, that maternity staff were able to adapt quickly to the new infection prevention requirements of donning PPE to minimise any delays to providing prompt clinical treatment.

The result was that maternity units were all, individually, urgently seeking COVID-19-specific resources and training for obstetric emergencies. Responding to the emerging but still limited guidance for dealing with COVID-19 in maternity care (e.g. [103]), NHS maternity professionals expressed a need for clearer e-learning resources relating to PPE skills, and more training on COVID-19-specific emergency drills [104]. One priority area concerned how to manage an obstetric emergency such as post-partum haemorrhage (PPH) in a woman with suspected or confirmed COVID-19. PPH is an emergency that complicates 1.2% of births in high-income settings [105, 106]. It is one of the most frequent and severe maternal complications after birth [107,108,109], and a cause of intensive care admission in the UK [110]. It is a classic example of where the requirements for managing situation clinically are well understood and communicated through clinical guidelines (e.g. [111, 112]), but where process improvement was needed to ensure that the underlying processes of delivering care were adapted to deal with the challenges imposed by the pandemic.

Addressing this problem required rapid, remote consensus-building among multi-professional stakeholder groups to specify process improvements for managing an obstetric emergency in a woman with suspected or confirmed COVID-19. To employ a consensus-building exercise, we used Thiscovery (, an online research and development platform created and developed by THIS Institute at the University of Cambridge. One of its founding goals is that of facilitating inclusive, multi-stakeholder involvement while offering maximum flexibility and minimum burden.


We applied the five steps of the framework to the case study. We: (1) defined the scope and objective of the project with key stakeholders; (2) produced a draft video showing a simulation of processes for handling an obstetric emergency in a COVID scenario; (3) recruited three expert groups (maternity care, infection prevention and control, and human factors specialists); (4) designed and conducted an exercise to reach consensus on recommendations to improve the processes illustrated in the video; and (5) produced a final specification of the process improvement, informed by the consensus-built recommendations, and illustrated this specification in an updated video and other resources .

(1) Scope and objective

The project had its origins early in the pandemic when key stakeholder organisations, including the Royal College of Midwives and Royal College of Obstetricians and Gynaecologists, identified that while clinical guidance on how to handle obstetric emergencies such as PPH remained sound, the underlying processes required adaptation for women with confirmed or suspected COVID-19. Many maternal deaths related to PPH in healthcare settings can be avoided through effective clinical management [106, 109], including prompt initiation of several simultaneous actions such as uterine massage, intravenous fluid resuscitation, and administration of medication (tranexamic acid to treat major haemorrhage and uterotonics to contract the uterus). Treatment delay can result in poor outcomes [113, 114], so delivering these clinical interventions requires highly optimised underlying processes, including effective teamwork, coordination, communication, and access to appropriate supplies. All of these processes require adaptation for a COVID-19 scenario, which might demand, for example, donning and doffing of personal protective equipment, changes in the tasks undertaken and their sequencing, and forms of communication suitable for a situation where masks and visors may inhibit verbal and non-verbal exchange.

This problem appeared well-suited to consensus-building that could rapidly generate a consistent approach. The overall objective of our project was defined as: to develop specifications for the process improvements needed to manage an obstetric emergency (such as PPH) in a woman with suspected or confirmed COVID-19, using rapid, remote consensus-building among multi-professional stakeholder groups. Target audiences included healthcare professionals working in maternity care in UK NHS trusts, including midwives, obstetricians, and managers of maternity services.

(2) Draft of the proposed process improvement specification

We started by producing Version 1 of a video that showed a simulation of a maternity ward team managing PPH in a woman with suspected or confirmed COVID-19. The video illustrated processes that included: how the team communicated with each other, the woman and her partner; PPE donning and doffing procedures; and use of obstetric-specific procedures (e.g. PPH ‘grab bag’, treatment algorithms) in a COVID-19 context. The processes illustrated were based on: the emerging national guidance on COVID-19 infection prevention and control in a clinical setting (April 2020); clinical guidelines for managing obstetric emergencies such as PPH [e.g. 111, 112]; and ways of working established in one of the safest maternity units of the UK [115, 116]. Though the processes shown thus represented initial reasonable specifications for process improvements to facilitate handling of an obstetric emergency in a COVID-19 scenario, it was also likely that these specifications could be further optimised.

The use of a video format to elicit suggestions for improvement was intended to enhance participant engagement [76, 77], minimise cognitive burden (e.g. not having to study a written manual) [117], and align maternity professionals’ desire to have more COVID-specific e-resources available [104]. The video format was also expected to work well for the study participants using a range of different technologies, including mobile devices [76, 118].

(3) Participant recruitment strategy


We drew on the expertise of three expert groups: maternity teams to provide clinical and practical views; infection prevention and control staff for specialist knowledge on infection guidance; and healthcare human factors specialists for their perspective on the interaction between people and work systems in healthcare settings [21, 119]. The invitation asked potential participants to provide their main professional background, including an “other” option to avoid forcing people into categories that did not suit them. Participants were not further screened for expertise, in order to ensure a rapid response, minimal burden to participants, and maximal inclusivity.

The consensus-building exercise consisted of a recommendation generation exercise followed by two Delphi rounds (Fig. 1). For the first Delphi round, only those who had taken part in the recommendation exercise and had stated that they were happy to be contacted again were eligible, and for the second Delphi round, only those who had taken part in the first Delphi round were eligible.

Fig. 1
figure 1

Design of the consensus-building exercise

Sampling and recruitment

We made use of convenience and snowball sampling to rapidly recruit a sample for the three expert groups [120]. Geographical representation was maximised using recruitment conducted through nationwide email networks of specialists in maternity care, infection prevention and control, and healthcare human factors. We anticipated that most participants would engage in response to the email (convenience sampling), while other participants might become involved as a result of colleagues alerting them to the study (snowball sampling). To maximise response rates and minimise attrition bias, reminders were sent through the email networks and Thiscovery, referring to the importance of their (continued) engagement [87, 91].

Sample size

We expected that we would require three panels of stakeholders with distinct areas of professional expertise [85], with at least seven experts each, in accordance with conventional recommendations on sample size of an expert panel [61]. In the design of our consensus-building exercise (Fig. 1), we assumed that approximately half of the participants of the initial recommendation exercise would take part in the first Delphi round (50% response rate), with an attrition rate of 20% for the second Delphi round [54]. Accordingly, we aimed to recruit at least 20 stakeholders for each of the three expert groups for the recommendation exercise.

(4) Remote consensus-building exercise

The recommendation generation exercise and Delphi rounds were designed and then integrated into Thiscovery. All exercises were user tested to optimise participant experience on computer, tablet and smartphone platforms. Data collection and analysis for the consensus-building exercise (including the recommendation generation exercise and Delphi rounds) was completed in 6 weeks.


Participants registered for an account on Thiscovery after consenting to the platform’s privacy policy and terms of use ( Participants confirmed their consent before each new round of the consensus-building exercise. All data was captured and processed without any personal identifiable information. Review by an institutional review board was not applicable, as the project was a consultation and engagement exercise classified as a quality improvement activity [92, 93], in which all of the participants were invited to join the authorship group and to be acknowledged in the project’s outputs.

Recommendation exercise

We started with a recommendation generation exercise to inform the subsequent Delphi consensus-building rounds (Fig. 1). This initial exercise consisted of asking participants for their feedback on a draft of good practice in undertaking the process, as illustrated in a video. After seeing the video, participants were asked, using open-ended questions, to draw on their professional expertise to provide recommendations to improve the practice illustrated in relation to: 1) donning PPE; 2) management of the emergency in the context of COVID-19, e.g. use of PPH “grab bag”; 3) doffing PPE; and 4) any other areas.

This first stage of the project – the recommendation exercise – was completed by 105 participants (Table 2). There were 912 responses from 103 participants (two participants did not provide recommendations). Of these 912 responses, 94 were coded as general comments (e.g. “The order of doffing is very important”) or supportive statements (e.g. “The process shown appeared very proficient”) and were excluded from further analysis. The remaining 818 responses were recommendations relating to improvement of practice illustrated in the video, and were synthesised: three analysts worked in parallel on an iterative analysis process of assigning recommendations to pre-defined categories, coding them, re-assigning them to categories more closely fitting with the codes, and merging similar recommendations (Supplement 1). This led to a total of nine categories including 74 synthesised recommendations. Of these, 26 recommendations were identified as the most frequently raised across all participants (Supplement 1).

Table 2 Participants in the recommendation generation exercise and Delphi rounds

A final clinical stakeholder review by authors TD and CW identified two recommendations as only focusing on the video format, rather than on improving the processes illustrated in the video, and were excluded from the consensus-building exercise. This stakeholder review also identified recommendations that could be further combined due to sufficient overlap. We were therefore able to classify the remaining 22 recommendations into five categories (Table 3).

Table 3 Recommendations used in the Delphi rounds, median and interquartile (IQR) ratings of participants, and proportion of participants providing a rating of 7, 8 or 9. Legend: Bold numbers indicate recommendations on which consensus was reached. See Supplement 2 for more details

Delphi rounds

All but one of the 105 participants agreed to be contacted again for the Delphi rounds and all 104 were invited. About two-thirds (n = 71) of those invited took part in the first Delphi round, of whom 57 took part in the second Delphi round (Table 2). Retention from the first to the second Delphi round was 80% for the total group: 70% for maternity care, 71% for infection prevention and control, and 88% for human factors (Table 2). The risk of attrition bias was low, as ratings in the first Delphi from participants completing both Delphi rounds and from participants who did not respond to the second round were similar.

In the Delphi rounds, the most frequently raised recommendations were presented to participants, with the option to review the video at any point. Participants were asked to rate each recommendation in response to the statement: “This recommendation should be implemented.” A nine-point scale was used with the anchors of “strongly disagree” (rating = 1), “uncertain” (rating = 5), and “strongly agree” (rating = 9). Consensus to implement was defined as > 70% of participants rating a recommendation with a 7, 8 or 9, and < 15% of participants rating it with a 1, 2 or 3, in accordance with core outcome set methodology [54, 121]. An additional criterion we used was an interquartile range ≤ 2, to further validate consensus among the group [98].

Only recommendations that did not reach consensus in the first Delphi round were taken forward to the second [75]. In the second round, the participant was presented with their original rating for each respective recommendation, along with the distribution of ratings from each stakeholder group [122, 123], as shown in an interactive bar chart (see Supplement 2). The participant was asked to consider the ratings of the others, and whether they would like to change their rating or stay with their original rating from the first round.

Of the 22 recommendations rated in the Delphi rounds, 16 reached consensus that they should be implemented (Table 3), with consensus on 12 achieved in the first round and a further four in the second round. The other six recommendations received higher ratings in the second round, but insufficient to reach consensus (only 39 to 68% of participants rated them with a 7, 8 or 9, as shown in Table 3).

Risk of bias due to unequal stakeholder group size was low: analysis showed that consensus across the whole group was similar to that found within the three stakeholder groups, i.e. when each stakeholder group was analysed independently, consensus was reached for an almost identical set of recommendations. For example, “Perform hand hygiene prior to donning PPE” was rated with a 7, 8 or 9 by 74% of maternity care, 71% of infection prevention and control, and 72% of human factor participants (see Supplement 2).

(5) Final specification of process improvements

The recommendations that reached consensus were reviewed by the project team and used to inform the specification of process improvements for optimised management of obstetric emergencies during the COVID-19 pandemic. These specifications included the processes shown in Version 1 of the video amended based on the consensus-built recommendations (Table 3). A new video (Version 2) to illustrate the specified process improvements was produced [124], along with an infographic [125], and a brief overview of key points [126]. The video and other resources were endorsed by leading organisations who had supported the project, including royal colleges, specialist societies, and quality improvement bodies, and were widely shared. We also shared the resources directly with the participants. We acknowledged the participants’ contribution as collaborators [48].


This article presents a proposed methodological approach aimed at realising a commitment to broad participation, collaboration and consensus-building in process improvement in healthcare. Our case study shows that it is possible to deploy the approach successfully to specify process improvements in an area of pressing need during the COVID-19 pandemic. Subject to further evaluation, this approach has potentially wide application beyond the specific context in which it was tested, and may enable forms of inclusion and collaboration and listening that are otherwise very difficult to do remotely – and do so on a much greater scale. A particular strength of the approach is its ability to support mobilisation of the expertise and ingenuity of people in healthcare systems. This capability can help to enhance the currently limited infrastructure for collaborative building of specifications for process improvement in quality and safety of healthcare. It thus may have potential to address many of the problems of duplication of effort, waste, de-standardisation, and inability to engage sufficiently diverse expertise that currently characterise many quality improvement efforts in healthcare [17, 36].

Our case study showed that the proposed methodological approach can be used successfully to develop specifications for the process improvements needed to ensure high quality care, and may support the production of the kinds of high quality resources that professionals particularly value [104]. The systematic approach to participatory consensus-building that we propose is rich in potential for use in other areas that would benefit from specifying process improvement for clinical scenarios. It may help prevent the characteristic dysfunctions associated with exclusively bottom-up or top-down innovation for quality improvement [16, 17, 22]. Including a large “crowd” of stakeholders can further help mobilise the ingenuity of people in the system (e.g. patients, staff), strengthen the credibility of the consensus-built solution, and enhance feelings of ownership. It may help address lack of access to specific expertise common in locally led, bottom-up approaches [19, 36], and reduce the risks of the excessive focus on compliance associated with top-down approaches [22].

The case study included a wide range of stakeholders as collaborators, thus triangulating the perspectives and expertise of infection prevention and control experts, maternity professionals and healthcare human factors specialists in a way that would not have been possible for a single maternity unit. This blend of expertise was reflected in the participants’ recommendations: many were not tied to the specifics of PPH, but touched on important principles established by infection prevention and human factors specialists [15, 119, 127, 128]. These process improvement specifications are likely to be useful and relevant to multiple clinical communities, particularly those handling emergencies during the pandemic. Using this approach is likely to reduce waste in process improvement, since it not only produces a solution that can be used at scale within the maternity care community, it also generates many core elements of a solution that can be customised for different clinical scenarios outside of maternity. For instance, the relevance of principles of non-technical skills for teams, role assignment, leadership, and ergonomic workspace design when responding to medical emergencies can be generally applicable [127, 129].

A further strength of the approach is that it was possible to undertake this work relatively rapidly – in less than 6 weeks, despite pandemic conditions – and we anticipate that with the formalisation of our framework and with gains in experience, it may be possible for others to replicate the approach in other contexts at an even more rapid pace. Further examples of use cases would help to refine the approach and build a repository that could be used to evaluate it. Over time, the approach may facilitate further work to strengthen the infrastructure for participatory approaches in process improvement, similar to efforts over the last decades for building the infrastructure for clinical guidelines and Core Outcome Set development [1, 2, 54, 72].

Building this infrastructure is critical, because there is an ethical requirement to optimise approaches to process improvement (including specification of improvements) to reduce the risk that people may be avoidably exposed to poorer care associated with sub-optimal processes [36, 130]. Yet results of quality improvement in healthcare are typically mixed [18], suggesting the need to improve how improvement is done. Creating infrastructures for large-scale, collaborative improvement will, of course, require further development, refinement and evaluation of the approach we propose. The participatory ethos on which our approach is built may increase acceptability, uptake and impact of process improvement, but that remains to be tested. It will, for example, be important to study participant experience of taking part in these exercises (e.g. the degree of participation intensity, motivation, engagement, sense of ownership and empowerment [131, 132]), and to assess how far it is possible to reach agreements that stakeholders understand and accept. Evaluation might also examine the potential of using real-time Delphi to avoid having sequential rounds [133,134,135]. This might improve efficiency, reduce dropout, and minimise participant time, potentially without comprising user experience, inclusivity and robustness of consensus results [136] – though the implications of such an approach for the ability of different groups, such as patients and carers, to participate would require careful evaluation. Other innovations might include application of the principles of gamification to further enhance user experience [137], and provide intuitive, low-burden ways of gaining qualitative feedback on ratings in the Delphi rounds if required [138].

Given our positive experiences in using the approach in the case study, it has potential value in optimising many processes, including, for example, clinical tools and pathways. The approach may also have a role in facilitating implementation of guidance that is written at a high level of generality and requires customisation at local level for particular clinical scenarios, or in assembling relevant elements (e.g. relating to infection control and teamwork) that may be distributed across several guidelines but require integration to achieve patient care goals for specific scenarios [139,140,141].

In future work, it will important to identify the kinds of applications the approach works best for and where its limits lie. One issue, for example, is that the approach is likely well suited for specifying process improvements; locally, organisations will still need to do the work of implementing the process improvements, for example through cycles of change and monitoring implementation over time [9, 10]. Another issue concerns for what else the approach could work well for: our case study has used it to specify process improvements, but the broader consensus-building, participatory principles and methods may also have wider potential in, for example, optimising the design of equipment, forms, clinical pathways, and other applications. Evaluation should also establish when application of the approach has created sufficient learning for process improvements in one clinical area such that its learning can be applied to other related clinical scenarios without having to conduct another consensus-building exercise [129, 142, 143]. Finally, there will be an ongoing need to evaluate the process improvements specified through our proposed approach, for examining impacts on implementability, efficiency, staff and service user experience, acceptability, sustainability of change, impact on clinical outcomes, and any unintended consequences. Mixed-methods approaches are likely to be particularly useful in this regard [144].

Strengths and limitations

The case study illustrated a number of the strengths of our proposed approach. For example, it demonstrated the approach’s effectiveness in engaging a relatively large number of stakeholders as collaborators in a consensus-building exercise to specify process improvements. It also showed the feasibility of rapidly gaining feedback and reaching consensus on the process improvement (< 6 weeks), even during the first peak of the COVID-19 pandemic in the UK, with relatively little attrition (≤; 20%) between the Delphi rounds.

A limitation of the case study is that it did not involve users of maternity services themselves. However, because the methodology facilitates engagement of multiple stakeholder groups [cf. 42, 84], it is potentially well-suited to including service users, patients and carers in future projects. A further limitation is that the three stakeholder groups in our case study were not equal in size. Notwithstanding, the risk of bias appeared to be low: when each stakeholder group was analysed independently, consensus was reached for an almost identical set of recommendations as that determined for the total group. Due to the need to minimise participant burden, we did not collect demographic information that could be used to verify the representativeness of the stakeholders for the population working for or in the NHS. We also had to rely on non-probability sampling techniques that could have created some representation bias. We did, however, ensure that invitations were sent to stakeholder networks across the UK, bolstering our confidence that a representative sample was included. Finally, the case study was conducted during a fast-moving situation where the science (e.g. on infection control) was evolving very rapidly, posing the risk that recommendations made by participants could have been out of date by the time the exercise was completed. This risk was managed by close involvement of clinical expertise from the project team, and by ensuring that process improvements were specified such that they could have enduring relevance (e.g. referring to principles and national guidance on donning and doffing procedures rather than rigidly specifying them).

We were not able to evaluate the implementation or impact of the specified process improvements in the time available or compare our approach with alternative approaches to the specification of process improvements. Both are areas for future development.


We developed and tested a methodological approach to specifying process improvements that employed a participatory ethos and remote consensus-building methods. The approach was used successfully during pandemic conditions to build consensus among different stakeholder groups on specifying process improvements for managing an obstetric emergency in women with suspected or confirmed COVID-19. The methodological approach has significant potential to support rapid and transparent consensus-building for facilitating process improvement in various healthcare settings using online methods that can be standardised, replicated and scaled when needed, but will require further evaluation.

Availability of data and materials

Queries about the dataset should be directed to the corresponding author.

Change history



Post-partum haemorrhage


National Health Service


Personal protective equipment


  1. Brouwers MC, Spithoff K, Kerkvliet K, Alonso-Coello P, Burgers J, Cluzeau F, et al. Development and validation of a tool to assess the quality of clinical practice guideline recommendations. JAMA Netw Open. 2020;3(5):e205535.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4(1):212.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Grol R, Wensing M, Grimshaw J. Improving patient care: the implementation of change in health care (third edition). Oxford: Wiley; 2020.

    Google Scholar 

  5. Chassin MR, Loeb JM. High-reliability health care: getting there from Here. Milbank Q. 2013;91(3):459–90.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Holden RJ, Carayon P, Gurses AP, Hoonakker P, Hundt AS, Ozok AA, et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics. 2013;56(11):1669–86.

    Article  PubMed  Google Scholar 

  7. Alfred M, Catchpole K, Huffer E, Fredendall L, Taaffe KM. Work systems analysis of sterile processing: decontamination. BMJ Qual Saf. 2020;29(4):320–8.

    Article  PubMed  Google Scholar 

  8. Antonacci G, Reed JE, Lennox L, Barlow J. The use of process mapping in healthcare quality improvement projects. Health Serv Manag Res. 2018;31(2):74–84.

    Article  Google Scholar 

  9. The Health Foundation. Quality improvement made simple: What everyone should know about health care quality improvement. 2013. Available at:

  10. Boaden R, Harvey G, Moxham C, Proudlove N: Quality improvement: theory and practice in healthcare. 2008. Available at:

    Google Scholar 

  11. Hughes RG, Ronda G. Tools and strategies for quality improvement and patient safety. In: Patient safety and quality: An evidence-based handbook for nurses. US: Agency for Healthcare Research and Quality; 2008.

    Google Scholar 

  12. Johnson JK, Barach P. Tools and strategies for continuous quality improvement and patient safety. In: Surgical Patient Care: Springer; 2017. p. 121–32.

  13. Clegg CW. Sociotechnical principles for system design. Appl Ergon. 2000;31(5):463–77.

    Article  CAS  PubMed  Google Scholar 

  14. Karsh BT, Waterson P, Holden RJ. Crossing levels in systems ergonomics: a framework to support 'mesoergonomic' inquiry. Appl Ergon. 2014;45(1):45–54.

    Article  PubMed  Google Scholar 

  15. Royal Academy of Engineering. Engineering better care: A systems approach to health and care design and continuous improvement. 2017. Available at:

  16. Dixon-Woods M, Pronovost PJ. Patient safety and the problem of many hands. BMJ Qual Saf. 2016;25(7):485–8.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Dixon-Woods M. Harveian oration 2018: improving quality and safety in healthcare. Clin Med. 2019;19(1):47–56.

    Article  Google Scholar 

  18. Dixon-Woods M, Martin GP. Does quality improvement improve quality? Future Hosp J. 2016;3(3):191–4.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Shorrock S. The loneliest profession in healthcare. 2018. Available at:

  20. Gurses AP, Tschudy MM, McGrath-Morrow S, Husain A, Solomon BS, Gerohristodoulos KA, et al. Overcoming COVID-19: what can human factors and ergonomics offer? J Patient Saf Risk Manage. 2020;25(2):49–54.

    Article  Google Scholar 

  21. Hignett S, Jones EL, Miller D, Wolf L, Modi C, Shahzad MW, et al. Human factors and ergonomics and quality improvement science: integrating approaches for safety in healthcare. BMJ Qual Saf. 2015;24(4):250–4.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Mukamel DB, Haeder SF, Weimer DL. Top-down and bottom-up approaches to health care quality: the impacts of regulation and report cards. Annu Rev Public Health. 2014;35(1):477–97.

    Article  PubMed  Google Scholar 

  23. Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–33.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. World Health Organization. Timeline: WHO’s COVID-19 response. 2020. Available at:

  25. Indini A, Aschele C, Cavanna L, Clerico M, Daniele B, Fiorentini G, et al. Reorganisation of medical oncology departments during the novel coronavirus disease-19 pandemic: a nationwide Italian survey. Eur J Cancer. 2020;132:17–23.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Hanna TP, Evans GA, Booth CM. Cancer, COVID-19 and the precautionary principle: prioritizing treatment during a global pandemic. Nat Rev Clin Oncol. 2020;17(5):268–70.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Royal College of Surgeons of England. Recovery of surgical services during and after COVID-19. 2020. Available at:

  28. Coimbra R, Edwards S, Coimbra BC, Tabuenca A. Resuming elective surgical services in times of COVID-19 infection. Trauma Surg Acute Care Open. 2020;5(1):e000511.

    Article  PubMed  Google Scholar 

  29. American College of Surgeons, American Society of Anesthesiologists, Association of periOperative Registered Nurses, American Hospital Association. Joint statement: roadmap for resuming elective surgery after COVID-19 pandemic. 2020. Available at:

  30. Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiatry. 2020;7(9):813–24.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Royal College of Midwives, Royal College of Obstetricians & Gynaecologists. Guidance for provision of midwife-led settings and home birth in the evolving coronavirus (COVID-19) pandemic. 2020. Available at:

  32. NHS England. Clinical guide for the temporary reorganisation of intrapartum maternity care during the coronavirus pandemic. 2020. Available at:

  33. Semaan A, Audet C, Huysmans E, Afolabi B, Assarag B, Banke-Thomas A, et al. Voices from the frontline: findings from a thematic analysis of a rapid online global survey of maternal and newborn health professionals facing the COVID-19 pandemic. BMJ Glob Health. 2020;5(6):e002967.

    Article  PubMed  Google Scholar 

  34. Centers for Disease Control and Prevention (CDC). Considerations for inpatient obstetric healthcare settings. 2020. Available at:

  35. Chen D, Yang H, Cao Y, Cheng W, Duan T, Fan C, et al. Expert consensus for managing pregnant women and neonates born to mothers with suspected or confirmed novel coronavirus ( COVID −19) infection. Int J Gynecol Obstet. 2020;149(2):130–6.

    Article  CAS  Google Scholar 

  36. Dixon-Woods M. How to improve healthcare improvement—an essay by Mary Dixon-Woods. BMJ. 2019;367:l5514.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Aveling EL, Martin G, Armstrong N, Banerjee J, Dixon-Woods M. Quality improvement through clinical communities: eight lessons for practice. J Health Organ Manage. 2012;26(2):158–74.

    Article  Google Scholar 

  39. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89(2):167–205.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Fowler A. Authentic NGDO partnerships in the new policy agenda for international aid: dead end or light ahead? Dev Chang. 1998;29(1):137–59.

    Article  Google Scholar 

  41. Kahssay HM, Oakley P, World Health Organization. Community involvement in health development: a review of the concept and practice. 1999. Available at:

  42. Marjanovic S, Ball S, Harshfield A, Dimova S, Prideaux R, Carpenter A, Punch D, Simmons RK. Involving NHS staff in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at:

  43. Eyre L, Farrelly M, Marshall M. What can a participatory approach to evaluation contribute to the field of integrated care? BMJ Qual Saf. 2017;26(7):588–94.

    Article  PubMed  Google Scholar 

  44. Parks S, d'Angelo C, Gunashekar S. Citizen science: generating ideas and exploring consensus. The Healthcare Improvement Studies (THIS) Institute 2018. Available at:

  45. Guazzini A, Vilone D, Donati C, Nardi A, Levnajić Z. Modeling crowdsourcing as collective problem solving. Sci Rep. 2015;5(1):–16557.

  46. Franzoni C, Sauermann H. Crowd science: the organization of scientific research in open collaborative projects. Res Policy. 2014;43(1):1–20.

    Article  Google Scholar 

  47. Breidbach CF, Brodie RJ. Engagement platforms in the sharing economy. J Serv Theory Pract. 2017;27(4):761–77.

    Article  Google Scholar 

  48. Cappa F, Laut J, Porfiri M, Giustiniano L. Bring them aboard: rewarding participation in technology-mediated citizen science projects. Comput Hum Behav. 2018;89:246–57.

    Article  Google Scholar 

  49. European Citizen Science Association. Ten principles of citizen science. 2015. Available at:

  50. Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018.

    Book  Google Scholar 

  51. Lichten C, Ioppolo R, D'Angelo C, Simmons RK, Jones MM. Citizen science: crowdsourcing for research. The Healthcare Improvement Studies (THIS) Institute 2018. Available at:

  52. Jones J, Hunter D. Qualitative research: consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2008;26(1):5–14.

    Article  Google Scholar 

  54. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(S3):280.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):1–88.

    Article  Google Scholar 

  56. Innes JE. Planning through consensus building: a new view of the comprehensive planning ideal. J Am Plan Assoc. 1996;62(4):460–72.

    Article  Google Scholar 

  57. Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94.

    Article  PubMed  Google Scholar 

  58. Van De Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–42.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Lomas J, Anderson G, Enkin M, Vayda E, Roberts R, MacKinnon B. The role of evidence in the consensus process: results from a Canadian consensus exercise. JAMA. 1988;259(20):3001–5.

    Article  CAS  PubMed  Google Scholar 

  60. Institute of Medicine (US) Council on Health Care Technology, Goodman C, Baratz SR. Improving consensus development for health technology assessment: An international perspective. Washington: National Academies Press (US); 1990.

    Google Scholar 

  61. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. The RAND/UCLA appropriateness method user's manual. Santa Monica: Rand Corp; 2001.

    Google Scholar 

  62. Brook RH. Assessing the appropriateness of care—its time has come. JAMA. 2009;302(9):997–8.

    Article  CAS  PubMed  Google Scholar 

  63. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67.

    Article  Google Scholar 

  64. Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang. 2006;73(5):467–82.

    Article  Google Scholar 

  65. Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48.

    Article  CAS  PubMed  Google Scholar 

  66. Banno M, Tsujimoto Y, Kataoka Y. The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines. J Clin Epidemiol. 2020;124:50–7.

    Article  PubMed  Google Scholar 

  67. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of Health Research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Hsu C-C, Sandford BA. The Delphi Technique: Making Sense of Consensus. Pract Assess Res Eval. 2007;12:10.

    Google Scholar 

  69. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12.

    Article  PubMed  Google Scholar 

  70. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  71. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    CAS  PubMed  Google Scholar 

  72. Brouwers MC, Kerkvliet K, Spithoff K. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016:i1152.

  73. Graham ID, Kothari A, McCutcheon C. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13(1):1.

    Article  Google Scholar 

  74. Brereton L, Ingleton C, Gardiner C, Goyder E, Mozygemba K, Lysdahl KB, et al. Lay and professional stakeholder involvement in scoping palliative care issues: methods used in seven European countries. Palliat Med. 2017;31(2):181–92.

    Article  PubMed  Google Scholar 

  75. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706.

    Article  PubMed  Google Scholar 

  76. Herodotou C, Aristeidou M, Sharples M, Scanlon E. Designing citizen science tools for learning: lessons learnt from the iterative development of nQuire. Res Pract Technol Enhanc Learn. 2018;13(1):1.

    Article  Google Scholar 

  77. Wald DM, Longo J, Dobell AR. Design principles for engaging and retaining virtual citizen scientists. Conserv Biol. 2016;30(3):562–70.

    Article  PubMed  Google Scholar 

  78. Boers M, Kirwan JR, Wells G, Beaton D, Gossec L, D'Agostino M-A, et al. Developing Core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67(7):745–53.

    Article  PubMed  Google Scholar 

  79. Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34.

    Article  CAS  PubMed  Google Scholar 

  80. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82.

    Article  PubMed  Google Scholar 

  81. Hussler C, Muller P, Rondé P. Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy. Technol Forecast Soc Chang. 2011;78(9):1642–53.

    Article  Google Scholar 

  82. Ioannidis JPA. Professional societies should abstain from authorship of guidelines and disease definition statements. Circulation. 2018;11(10):e004889.

    PubMed  Google Scholar 

  83. Hutchings A, Raine R. A systematic review of factors affecting the judgments produced by formal consensus development methods in health care. J Health Serv Res Policy. 2006;11(3):172–179H.

    Article  PubMed  Google Scholar 

  84. Marjanovic S, Harshfield A, Carpenter A, Bertscher A, Punch D, Ball S. Involving patients and the public in research. The Healthcare Improvement Studies (THIS) Institute 2019. Available at:

  85. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5(1):37.

    Article  PubMed  PubMed Central  Google Scholar 

  86. Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7(1):52.

    Article  PubMed  PubMed Central  Google Scholar 

  87. West S, Pateman R. Recruiting and retaining participants in citizen science: what can be learned from the volunteering literature? Citiz Sci. 2016;1(2):15.

    Google Scholar 

  88. Brouwer S, Hessels LK. Increasing research impact with citizen science: the influence of recruitment strategies on sample diversity. Public Underst Sci. 2019;28(5):606–21.

    Article  PubMed  Google Scholar 

  89. Fiske A, Prainsack B, Buyx A. Meeting the needs of underserved populations: setting the agenda for more inclusive citizen science of medicine. J Med Ethics. 2019;45(9):617–22.

    Article  PubMed  Google Scholar 

  90. De Las ND, Hacker K, DiGirolamo A, Hicks LS. A systematic review of community-based participatory research to enhance clinical trials in racial and ethnic minority groups. Health Serv Res. 2012;47(3pt2):1363–86.

    Article  Google Scholar 

  91. Hsu C-C, Sandford BA. Minimizing non-response in the Delphi process: How to respond to non-response. Pract Assess Res Eval. 2007;12(1):17.

    Google Scholar 

  92. Ogrinc G, Nelson WA, Adams SM, O'Hara AE. An instrument to differentiate between clinical research and quality improvement. Irb. 2013;35(5):1–8.

    PubMed  Google Scholar 

  93. Healthcare Quality Improvement Partnership (HQIP). Guide to managing ethical issues in quality improvement or clinical audit projects. 2021. Available at:

  94. Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decis Sci. 1997;28(3):763–74.

    Article  Google Scholar 

  95. Schmidt R, Lyytinen K, Keil M, Cule P. Identifying software project risks: an international Delphi study. J Manag Inf Syst. 2001;17(4):5–36.

    Article  Google Scholar 

  96. Brady SR. Utilizing and adapting the Delphi method for use in qualitative research. Int J Qual Methods. 2015;14(5):160940691562138.

    Article  Google Scholar 

  97. Hirschhorn F. Reflections on the application of the Delphi method: lessons from a case in public transport research. Int J Soc Res Methodol. 2019;22(3):309–22.

    Article  Google Scholar 

  98. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9.

    Article  PubMed  Google Scholar 

  99. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19(1):173–202.

    Article  CAS  PubMed  Google Scholar 

  100. Renfrew MJ, Cheyne H, Craig J, Duff E, Dykes F, Hunter B, et al. Sustaining quality midwifery care in a pandemic and beyond. Midwifery. 2020;88:102759.

    Article  PubMed  PubMed Central  Google Scholar 

  101. Coxon K, Turienzo CF, Kweekel L, Goodarzi B, Brigante L, Simon A, et al. The impact of the coronavirus (COVID-19) pandemic on maternity care in Europe. Midwifery. 2020;88:102779.

    Article  PubMed  PubMed Central  Google Scholar 

  102. Knight M, Bunch K, Vousden N, Morris E, Simpson N, Gale C, et al. Characteristics and outcomes of pregnant women admitted to hospital with confirmed SARS-CoV-2 infection in UK: national population based cohort study. BMJ. 2020:m2107.

  103. Royal College of Obstetricians & Gynaecologists, Royal College of Midwives, Royal College of Paediatrics and Child Health, Public Health England, Public Health Scotland. Coronavirus (COVID-19) infection in pregnancy. 2020. Available at:

  104. Rimmer M, Al Wattar B, Barlow C, Black N, Carpenter C, Conti-Ramsden F, et al. Provision of obstetrics and gynaecology services during the COVID-19 pandemic: a survey of junior doctors in the UK National Health Service. BJOG Int J Obstet Gynaecol. 2020;127(9):1123–8.

    Article  CAS  Google Scholar 

  105. Mehrabadi A, Hutcheon J, Lee L, Kramer M, Liston R, Joseph K. Epidemiological investigation of a temporal increase in atonic postpartum haemorrhage: a population-based retrospective cohort study. BJOG Int J Obstet Gynaecol. 2013;120(7):853–62.

    Article  CAS  Google Scholar 

  106. van Stralen G, von Schmidt auf Altenstadt JF, Bloemenkamp KW, van Roosmalen J, Hukkelhoven CW. Increasing incidence of postpartum hemorrhage: the Dutch piece of the puzzle. Acta Obstet Gynecol Scand. 2016;95(10):1104–10.

    Article  PubMed  Google Scholar 

  107. Knight M, Bunch K, Tuffnell D, Jayakody H, Shakespeare J, Kotnis R, et al. Saving Lives, Improving Mothers’ Care - Lessons learned to inform maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2014–16. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2018.

    Google Scholar 

  108. Dunning T, Harris JM, Sandall J. Women and their birth partners’ experiences following a primary postpartum haemorrhage: a qualitative study. BMC Pregnancy Childbirth. 2016;16(1):80.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  109. Say L, Chou D, Gemmill A, Tunçalp Ö, Moller A-B, Daniels J, et al. Global causes of maternal death: a WHO systematic analysis. Lancet Glob Health. 2014;2(6):e323–33.

    Article  PubMed  Google Scholar 

  110. Zwart JJ, Dupuis JRO, Richters A, Öry F, Van Roosmalen J. Obstetric intensive care unit admission: a 2-year nationwide population-based cohort study. Intensive Care Med. 2010;36(2):256–63.

    Article  PubMed  Google Scholar 

  111. Mavrides E, Allard S, Chandraharan E, Collins P, Green L, Hunt B, et al. Prevention and management of postpartum Haemorrhage. BJOG Int J Obstet Gynaecol. 2016;124(5):e106–49.

    Google Scholar 

  112. National Institute for Health and Care Excellence (NICE). Intrapartum care for healthy women and babies. 2017. Available at:

  113. Gayet-Ageron A, Prieto-Merino D, Ker K, Shakur H, Ageron F-X, Roberts I, et al. Effect of trekeratment delay on the effectiveness and safety of antifibrinolytics in acute severe haemorrhage: a meta-analysis of individual patient-level data from 40 138 bleeding patients. Lancet. 2018;391(10116):125–32.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  114. Shakur H, Roberts I, Fawole B, Chaudhri R, El-Sheikh M, Akintan A, et al. Effect of early tranexamic acid administration on mortality, hysterectomy, and other morbidities in women with post-partum haemorrhage (WOMAN): an international, randomised, double-blind, placebo-controlled trial. Lancet. 2017;389(10084):2105–16.

    Article  CAS  Google Scholar 

  115. Liberati EG, Tarrant C, Willars J, Draycott T, Winter C, Chew S, et al. How to be a very safe maternity unit: an ethnographic study. Soc Sci Med. 2019;223:64–72.

    Article  PubMed  PubMed Central  Google Scholar 

  116. Draycott T. Practical obstetric multi-professional training – PROMPT. 2013. Available at:

  117. Richmond H, Copsey B, Hall AM, Davies D, Lamb SE. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ. 2017;17(1):227.

    Article  PubMed  PubMed Central  Google Scholar 

  118. Mobasheri MH, King D, Johnston M, Gautama S, Purkayastha S, Darzi A. The ownership and clinical use of smartphones by doctors and nurses in the UK: a multicentre survey study. BMJ Innov. 2015;1(4):174–81.

    Article  Google Scholar 

  119. Gosbee J. Human factors engineering and patient safety. Qual Saf Health Care. 2002;11(4):352–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  120. Fricker RD. Sampling methods for web and e-mail surveys. In: Fielding N, Lee RM, Blank G, editors. The SAGE handbook of online research methods. London: SAGE Publications Ltd; 2008.

  121. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132.

    Article  PubMed  PubMed Central  Google Scholar 

  122. Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409.

    Article  PubMed  PubMed Central  Google Scholar 

  123. Maclennan S, Kirkham J, Lam TBL, Williamson PR. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. J Clin Epidemiol. 2018;93:1–8.

    Article  PubMed  Google Scholar 

  124. The Healthcare Improvement Studies (THIS) Institute. COVID-19: managing an obstetric emergency (video). 2020. Available at:

  125. The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (Infographic). 2020. Available at:

  126. The Healthcare Improvement Studies (THIS) Institute. COVID-19: Five key goals in managing an obstetric emergency (information sheet). 2020. Available at:

  127. Catchpole K. Improving performance through human-centred reconfiguration of existing designs. BMJ Qual Saf. 2013;22(1):5–7.

    Article  PubMed  Google Scholar 

  128. National Institute for Health and Care Excellence (NICE). Infection prevention and control. 2014. Available at:

  129. Brogaard L, Kierkegaard O, Hvidman L, Jensen K, Musaeus P, Uldbjerg N, et al. The importance of non-technical performance for teams managing postpartum haemorrhage: video review of 99 obstetric teams. BJOG. 2019;126:1015.

    CAS  PubMed  Google Scholar 

  130. Watson SI, Dixon-Woods M, Taylor CA, Wroe EB, Dunbar EL, Chilton PJ, et al. Revising ethical guidance for the evaluation of programmes and interventions not initiated by researchers. J Med Ethics. 2020;46(1):26–30.

    Article  PubMed  Google Scholar 

  131. Tredick CA, Lewison RL, Deutschman DH, Hunt TA, Gordon KL, Von Hendy P. A rubric to evaluate citizen-science programs for long-term ecological monitoring. BioScience. 2017;67(9):834–44.

    Article  Google Scholar 

  132. Kieslinger B, Schäfer T, Heigl F, Dörler D, Richter A, Bonn A. Evaluating citizen science-towards an open framework. In: Hecker S, Haklay M, Bowser A, Makuch Z, Vogel J, Bonn A, editors. Citizen science: innovation in open science, society and policy. London: UCL Press; 2018.

  133. Gordon T, Pease A. RT Delphi: an efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Chang. 2006;73(4):321–33.

  134. Gnatzy T, Warth J, von der Gracht H, Darkow I-L. Validating an innovative real-time Delphi approach - a methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Chang. 2011;78(9):1681–94.

    Article  Google Scholar 

  135. Gordon TJ. The real-time Delphi method. Futures Res Methodol Version. 2009;3:19.

    Google Scholar 

  136. Rowe G, Wright G. The Delphi technique: past, present, and future prospects — introduction to the special issue. Technol Forecast Soc Chang. 2011;78(9):1487–90.

    Article  Google Scholar 

  137. Ponti M, Hillman T, Kullenberg C, Kasperowski D. Getting it Right or Being Top Rank: Games in Citizen Science. Citizen Science. Theory and Pract. 2018;3(1):1–12.

  138. Veugelers R, Gaakeer MI, Patka P, Huijsman R. Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response. BMC Med Res Methodol. 2020;20(1):156.

    Article  PubMed  PubMed Central  Google Scholar 

  139. Hall L, Eccles M. Case study of an inter-professional and inter-organisational programme to adapt, implement and evaluate clinical guidelines in secondary care. Br J Clin Gov. 2000;5(2):72–82.

    Article  Google Scholar 

  140. Cook DJ, Pulido JN, Thompson JE, Dearani JA, Ritter MJ, Hanson AC, et al. Standardized practice design with electronic support mechanisms for surgical process improvement: reducing mechanical ventilation time. Ann Surg. 2014;260(6):1011–5.

    Article  PubMed  Google Scholar 

  141. Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implement Sci. 2018;13(1):72.

    Article  PubMed  PubMed Central  Google Scholar 

  142. Patey R, Flin R, Fletcher G, Maran N, Glavin R. Developing a Taxonomy of Anesthetists’ Nontechnical Skills (ANTS). In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005.

  143. Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13(Suppl 1):i85–90.

    Article  PubMed  PubMed Central  Google Scholar 

  144. Portela MC, Pronovost JP, Woodcock T, Carter P, Mary D-W. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24(5):325–36.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


Thiscovery Group: André Sartori, Andy Paterson, Doro Unger-Lee, Joann Leeding, Luke Steer.

Obstetric Emergency Consensus Group: Amanda Andrews, Rita Arya, Sarah F. Bell, Denise Chaffer, Andrew Cooney, Rachel Corry, Mair G.P. Davies, Lisa Duffy, Caroline Everden, Theresa Fitzpatrick, Courtney Grant, Mark Hellaby, Tracey A. Herlihey, Sue Hignett, Sarah Hookes, Fran R. Ives, Gyuchan T. Jun, Owen J. Marsh, Tanya R. Matthews, Celine McKeown, Alexandra Merriman, Giulia Miles, Susan Millward, Neil Muchatata, David Newton, Valerie G. Noble, Pamela Page, Vincent Pargade, Sharon P. Pickering, Laura Pickup, Dale Richards, Cerys Scarr, Jyoti Sidhu, James Stevenson, Ben Tipney, Stephen Tipper, Jo Wailling, Susan P. Whalley-Lloyd, Christian Wilhelm, Juliet J. Wood.

We thank the participants in this project. We thank the Infection Prevention Society, Each Baby Counts, and the Clinical Human Factors Group for help with recruitment. We thank the volunteers who participated in both shoots of the video, and North Bristol NHS Trust and the PROMPT Maternity Foundation for supporting the videos.

We thank the Royal College of Midwives, the Royal College of Obstetricians and Gynaecologists, Each Baby Counts, the Infection Prevention Society, the Association of Obstetric Anaesthetists, the PROMPT Maternity Foundation and the Health Foundation for their endorsement and support of the resources.

We thank Becky Kenny for help with project management. We thank the communications team at THIS Institute for their work on developing the communications resources.

We thank Jenny George, Emily Ryen Gloinson and Carolina Feijao of RAND Europe for their contributions to data analysis.


This project was supported by the Health Foundation’s grant to the University of Cambridge for The Healthcare Improvement Studies (THIS) Institute. The Health Foundation is an independent charity committed to bringing about better health and health care for people in the UK. Mary Dixon-Woods is a National Institute for Health Research (NIHR) Senior Investigator (NF-SI-0617-10026). The funders had no role in study design, data collection and analysis, interpretation of the data, and preparation of the manuscript.

Author information

Authors and Affiliations




MD-W and TD came up with the idea for the project. JWvdS led the design of the project, co-led data collection and analysis, and co-led drafting of the manuscript. MW contributed to the design of the consensus-building exercise, co-led data collection and analysis, and drafting of the manuscript. AA contributed to the design of the consensus-building exercise, data collection, and drafting of the manuscript. TD contributed to the design of the project and drafting of the manuscript. CW contributed to the design of the project and drafting of the manuscript. GM contributed to the design and oversight of the project, and drafting of the manuscript. KK contributed to the drafting of the manuscript. NR contributed to the data analysis and drafting of the manuscript. RK contributed to the oversight of the project and contributed to drafting of the manuscript, and led the Thiscovery Group. MD-W contributed to the design of the project, supervised the project and co-led drafting of the manuscript. The final version of the manuscript was read and approved by all authors. The final version of the manuscript was also read and approved by the members of the Thiscovery Authorship Group and the Obstetric Emergency Authorship Group (see acknowledgements).

Corresponding author

Correspondence to Jan W. van der Scheer.

Ethics declarations

Ethics approval and consent to participate

The UK’s Health Research Authority decision tool ( showed that ethics approval was not required for the consensus-building exercise, as the project was a consultation and engagement exercise classified as a quality improvement activity [92, 93], in which all of the participants were invited to join the authorship group and to be acknowledged in the project’s outputs. Participants provided written online consent to the terms and conditions regarding the Thiscovery platform’s privacy policy and terms of use ( Participants were provided with information about the project, and re-confirmed their consent before each new round of the consensus-building exercise.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: It has been found that Fig. 3B has an error of misplacement. In Fig. 3B, the images of Oil red O staining of liver sections from control (I) and palmitic acid (II) had been misplaced.

Supplementary Information

Additional file 1: Supplement 1.

Synthesis process of the 912 responses to the recommendation task.

Additional file 2: Supplement 2.

Interactive bar charts of ratings across the three stakeholder groups as presented to participants in the second Delphi round.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

van der Scheer, J.W., Woodward, M., Ansari, A. et al. How to specify healthcare process improvements collaboratively using rapid, remote consensus-building: a framework and a case study of its application. BMC Med Res Methodol 21, 103 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: