When complexity matters: a step-by-step guide to incorporating a complexity perspective in guideline development for public health and health system interventions
BMC Medical Research Methodology volume 20, Article number: 245 (2020)
Guidelines on public health and health system interventions often involve considerations beyond effectiveness and safety to account for the impact that these interventions have on the wider systems in which they are implemented. This paper describes how a complexity perspective may be adopted in guideline development to facilitate a more nuanced consideration of a range of factors pertinent to decisions regarding public health and health system interventions. These factors include acceptability and feasibility, and societal, economic, and equity and equality implications of interventions.
A 5-step process describes how to incorporate a complexity perspective in guideline development with examples to illustrate each step. The steps include: (i) guideline scoping, (ii) formulating questions, (iii) retrieving and synthesising evidence, (iv) assessing the evidence, and (v) developing recommendations. Guideline scoping using stakeholder consultations, complexity features, evidence mapping, logic modelling, and explicit decision criteria is emphasised as a key step that informs all subsequent steps.
Through explicit consideration of a range of factors and enhanced understanding of the specific circumstances in which interventions work, a complexity perspective can yield guidelines with better informed recommendations and facilitate local adaptation and implementation. Further work will need to look into the methods of collecting and assessing different types of evidence beyond effectiveness and develop procedural guidance for prioritising across a range of decision criteria.
Guidelines are a key instrument in clinical practice, public health, and health system decision-making and offer recommendations on how to choose among different interventions and policies to improve health. Development of clinical practice guidelines follows a systematic and transparent process primarily focusing on and prioritising questions about the health effects of interventions . However, decisions on public health and health system interventions often need to consider broader questions beyond effectiveness and safety . These interventions tackle a range of behavioural, social, commercial, political, and environmental determinants of health and are implemented in complex systems with specific contextual features [3, 4]. Here, guidelines usually need to consider more nuanced questions on why, how, and in what circumstances these interventions work and what their impact on the wider system might be.
The phrase complex intervention has often been used to describe public health and health system interventions, which: (i) have many interacting components; (ii) involve complex behaviours during their delivery and receipt; (iii) target different groups and levels; (iv) influence many health and non-health outcomes; or (v) require flexible implementation across different contexts [5, 16]. Complex systems, on the other hand, refer to the dynamic networks of social interactions in which interventions take place [17, 18]. In fact, interventions interact with and influence the wider systems in which they are delivered regardless of whether the interventions themselves may be simple or complex in design (e.g., a drug or a multi-component chronic disease management programme) . Smoke-free legislation provides an example of a simple intervention in design. However, the introduction of smoke-free legislation initiates complex system changes through its impact not only on the smoking-related health outcomes, but also on the patterns of socialising and drinking in the community [5, 19]. In this paper, we use the term complexity perspective to refer to the broad changes that interventions bring into the dynamic systems in which they are delivered regardless of their design features. To operationalise this perspective, we highlight key aspects of complexity derived from complex systems theory and illustrate them in the context of public health and health system interventions (see Table 1).
We believe that a complexity perspective facilitates a more nuanced consideration of a range of questions that are pertinent to decisions regarding public health and health system interventions (see Table 1). These include questions on the impact of the intervention on the broader system, including, for example, how an intervention interacts with a specific socio-economic and cultural context and what health and non-health outcomes it affects. Ultimately, this enables guideline recommendations that are driven not primarily by considerations of effectiveness and safety in relation to a narrow set of health outcomes, but equally by factors such as acceptability, feasibility, societal implications, and equity and equality . Taking a complexity perspective in guideline development therefore helps make better informed recommendations based on a comprehensive understanding of the intervention and its manifold implications for the system. Importantly, such a perspective helps avoid simplistic and misleading guideline recommendations, which may ignore critical contextual features affecting the benefits and harms, acceptability or feasibility of an intervention or running counter to relevant social or environmental considerations . The introduction of a levy on soft drinks to reduce obesity can serve as an illustrative case. A standard approach would be based on a linear model of cause and effect, where the introduction of the levy is expected to reduce the purchase and consumption of soft drinks, ultimately resulting in lower rates of obesity among children and the general population. On the other hand, a complexity perspective would encourage asking questions regarding the entire system, including access to and safety of healthier alternatives, notably water, and likely reactions from the concerned industries, such as changes to the composition, pricing, and marketing of various commodities . The recommendations of a guideline taking a complexity perspective on obesity may therefore be different from those reached by assuming and examining a linear cause-effect relationship .
The approach described in this paper builds upon and extends a series published in BMJ Global Health on Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions, as well as other selected methodological work on developing guidelines and considering complexity in evidence synthesis published in the last decade [2, 21,22,23,24]. The series was specifically commissioned by the World Health Organization (WHO) to strengthen its methods for guideline development. It involved convening working groups and consensus meetings with leading international experts in systematic review methodology, guideline development and complex systems thinking. The papers in the series focus on key concepts of complexity and their implications for guideline development and were developed using a range of methods, including systematic and non-systematic literature reviews .
In this paper we suggest a five-step process for guideline development when taking a complexity perspective (see Fig. 1): (i) guideline scoping, (ii) formulating questions, (iii) retrieving and synthesising evidence, (iv) assessing the evidence, and (v) developing recommendations. For each step, we explain specific methods to incorporate a complexity perspective and illustrate these by referring back to the above case of introducing a levy on soft drinks to reduce obesity. We then provide an example of how these methods have been applied in existing guidelines. Our aim is to help those involved in guideline development better understand and address the key aspects and challenges of taking a complexity perspective.
Step 1: guideline scoping
Whether to take a complexity perspective is a decision that should be made by considering the topic and aims of a guideline and the needs of guideline users. Taking a complexity perspective entails examination of a broader range of questions in guidelines, which may require additional time and resources. In this early phase, this can be facilitated by stakeholder consultations, examination of the relevant aspects of complexity, logic modelling, evidence mapping, and a review of Evidence to Decision (EtD) criteria. Guideline panels should choose the most suitable procedure or several among these. As illustrated in Fig. 1, using these procedures during scoping will inform all other steps in the guideline development process. In addition, some of these procedures can also be employed in subsequent steps.
While stakeholder consultation is commonly used in guideline development and increasingly in systematic reviews [25,26,27], it is particularly important when taking a complexity perspective and is relevant to all steps in the development process. Stakeholders can add valuable insights on the optimal scope, and input into setting priorities among a range of key questions regarding the interactions of the intervention with the system. For public health and health system interventions, key stakeholders include guideline end-users, such as providers and organisations delivering or financing the intervention and those directly affected by the guideline recommendations, such as specific population groups, industry, or the general public.
The means of involving stakeholders and their level of engagement in guideline development will depend on the topic, the stakeholders concerned, and feasibility considerations . For example, if a guideline includes children or adolescents as key stakeholders, it would be more appropriate to elicit their views using qualitative or participatory research than to invite them to join the guideline panels and sit through panel meetings . During guideline scoping, stakeholders can help define guideline priorities, relevant questions and contextual factors, and the key areas of uncertainty that need to be explored, for example, with new systematic reviews. The views of relevant stakeholders can be incorporated via their direct involvement in the guideline panel, through surveys or needs assessments or through primary or secondary research on their views. The TRANSFER Approach can be used to facilitate stakeholder input into contextual features from the beginning of the systematic review and – by extension – the guideline development process . It enables a stakeholder-driven systematic and transparent assessment of transferability factors using a structured process.
The aspects of complexity outlined in Table 1 provide guideline panels with key concepts that can help with identifying whether taking a complexity perspective is warranted, as well as with scoping the guideline and related decisions regarding relevant aspects of complexity. Consideration of these concepts in relation to the guideline topic and interventions can highlight key questions to prioritise in the guideline . This may be achieved by formulating and answering questions, such as (i) does the intervention of interest include many interacting components (e.g., different technologies and behaviour change activities as part of a sanitation intervention)? (ii) Does the intervention interact with and change the context into which it is introduced (e.g., changing social norms in addition to health outcomes in case of smoke-free legislation)? (iii) Does the intervention operate through system-level mechanisms (e.g., in order to change substance use outcomes in students, the entire school ethos might need to be transformed). Positive answers to these questions suggest that there is an added value in prioritising guideline questions that address aspects of complexity.
Logic models provide another useful approach to facilitate decisions on taking a complexity perspective [5, 29, 30]. They graphically display the intervention, different elements of the system and the relationships among them, including known or presumed pathways from the intervention to its various health and non-health outcomes. Two broad types of logic models are distinguished: system-based and process-oriented . System-based logic models, also referred to as conceptual frameworks and sometimes executed as causal loop diagrams, attempt to display the broad system in which the intervention is embedded, including contextual and implementation elements. Developing a system-based logic model can help guideline panels understand and prioritize aspects of complexity such as whether the guideline should only focus on intervention effects in relation to specific health outcomes or whether it should consider questions around intervention implementation. For example, when considering the effects of a levy on soft drinks to reduce obesity, developing a logic model which maps different elements of the system can help explicate how the industry might react to the levy by reformulating sugar in existing products (the intended impact) or by innovating and diversifying their product ranges (an unintended impact). This may inform the guideline panel that system adaptivity might be an important aspect of complexity to explore (see Table 1).
Process-oriented logic models, on the other hand, display the linear or non-linear pathways that lead from the intervention to multiple outcomes considering the temporal sequence of events . Such logic models can facilitate identification of key health and non-health outcomes, feedback loops and phase changes (see Table 1), however they are difficult to design and detailed evidence on pathways is often lacking. In general, logic models can be developed through a combination of searches of the literature and consultations with stakeholders, such as members of the guideline panel and content experts.
Evidence maps are also increasingly used in guideline scoping to obtain an overview of the existing evidence and thereby inform decisions on which aspects of complexity to consider in the guideline. Evidence maps involve systematic searches of a broad field to provide an overview and identify gaps in the evidence and/or future research needs . They often present the results in a user-friendly format, including visual graphs or searchable databases. Evidence maps can draw on a variety of synthesis products or individual studies, for example, existing systematic reviews of effectiveness, qualitative evidence syntheses of factors concerning intervention acceptability, or modelling studies of the various costs and societal benefits of the intervention . Through broad searches of the topic area, evidence maps can help panels decide on guideline priorities and questions and choose efficient methodological approaches to address them. Use of evidence maps can be particularly helpful in scoping when guidelines can draw on a well-developed evidence base – by summarising that evidence base and highlighting areas for further research.
EtD frameworks can also be used to help define guideline priorities, including important aspects of complexity, determine where systematic reviews are indicated, and add transparency in guideline development. They specify a set of criteria that inform the formulation of guideline recommendations, their direction and strength. For example, the EtD frameworks developed by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group highlight the priority of the problem, balance of benefits and harms, values and preferences, quality of evidence, resource implications, equity and human rights, acceptability and feasibility . However, currently these factors are usually considered at the end of the guideline development process as supplementary pieces of information to shape recommendations, which are largely driven by evidence on intervention effectiveness. WHO-INTEGRATE (Word Health Organization INTEGRATe Evidence) is another EtD framework that is well-suited for guidelines taking a complexity perspective. Adopting a societal perspective, it describes a broad set of criteria and specific sub-criteria to consider during guideline scoping, suggests an explicit and flexible approach towards weighting the importance of these criteria, and promotes a plurality of methods with different types of evidence collected and assessed for each criterion (see Table 2) . Reflecting upon and choosing among these criteria and sub-criteria for in-depth consideration at an early phase of guideline development will help explicate guideline priorities and identify types of evidence that should be consulted to develop recommendations (see Step 5).
Illustration from a guideline
To better address the sexual and reproductive health and human rights (SRHR) of women living with HIV, a World Health Organization (WHO) guideline was developed . To scope and structure the guideline from a complexity perspective, a preliminary literature review was conducted, which mapped the evidence on the consideration of human rights in sexual and reproductive health programmes. Based on this evidence map and a global survey to assess SRHR priorities of women living with HIV, a decision was made to structure the guideline explicitly following a woman-centred approach and to uphold the principles of human rights and gender equality. The findings from the survey were also used to draft the guideline questions and inform the recommendations and accompanying remarks in the guideline .
Step 2: formulating questions
Taking a complexity perspective will affect both the types of questions asked and how these questions are formulated. Typically, guidelines prioritise and/or are limited to questions of effectiveness. Often these are formulated as broad questions asking whether an intervention works compared with an alternative intervention, following the PICO (Population, Intervention, Comparison, Outcome) framework . However, public health and health system interventions affect the system in which they are implemented in multiple ways; how they operate and which effects they can achieve depend on a combination of geographical, epidemiological, socio-cultural, socio-economic, political, ethical, and legal factors [54, 55]. It is therefore important for guidelines taking a complexity perspective to more carefully attend to these aspects of complexity in formulating questions. For example, broadly defined PICO elements can be further dissected into sub-questions for description and quantitative examination, where possible  (see Step 3): What are the effects of the intervention across different population groups (dissecting the “P” element)? What is the independent effect of a given individual component of the intervention (dissecting the “I” element)? What are the effects of the intervention as assessed by different outcome measures (dissecting the “O” element)? Variation of intervention effects across different contextual characteristics would also be important to explore and assess through quantitative analyses, where data allow.
Questions that extend beyond intervention effectiveness can be formulated using frameworks other than PICO. For example, PerSPEcTiF (Perspective, Setting, Phenomenon of interest, Environment, Comparison, Time, and Findings) can be used to formulate guideline questions related to stakeholder experiences with the intervention in a specific context (see Table 3) . The criteria and sub-criteria of the WHO-INTEGRATE framework can suggest specific guideline questions (see Additional file 1) . This could include, for example: to what extent do stakeholders value different health outcomes (benefits and harms)? What are their views about the acceptability of and preferences regarding the intervention (socio-cultural acceptability)? How will the intervention impact health expenditures, equity, equality, and non-discrimination? What is the ecological impact of the intervention (societal implications)? What is the cost of the intervention (financial impact)? What aspects of the health system influence implementation of the intervention (feasibility and health system considerations)?
Stakeholder consultations, logic modelling, and evidence mapping can also help inform question formulation by identifying relevant aspects of the intervention, context, and outcomes (see Step 1) [6, 57]. As presented above, development of a system-based logic model on the potential effects of a levy on soft drinks could help identify important aspect of complexity, such as system adaptivity (see Table 1). This would then inform the following key question: how might the system change when a levy is imposed on the soft drink industry?
Illustration from a guideline
To address aspects of complexity, the guideline panel developing recommendations on antenatal care (ANC) for a positive pregnancy experience first conducted a scoping exercise to identify and map existing guidelines related to ANC [58, 59]. This highlighted the need to identify women-centred interventions and outcomes for ANC. To this end, a qualitative systematic review was conducted to explore women’s needs and values in pregnancy and ANC. This revealed positive pregnancy experience as the primary outcome. The scoping process and stakeholder consultation also led to identification of guideline priority questions and outcomes related to the effectiveness of interventions for a positive pregnancy experience. Examples of the specific questions include the following: for pregnant women (P), do diet and/or exercise interventions (I) compared with standard ANC (C) improve maternal and perinatal outcomes (O) (an effectiveness question regarding a nutritional intervention)? Should pregnant women carry their own ANC case notes to improve quality of care (a question regarding a delivery of a health system intervention)?
Step 3: retrieving and synthesising evidence
Standard approaches to evidence retrieval and synthesis can be used to explore the details of the guideline PICO questions and contextual variations of the effect . For example, subgroup analyses and meta-regression can be used to explore variation of intervention effects across different contexts and population groups . A component-level approach and network meta-analysis can be used to examine the effects of individual components of the intervention or their combinations . Having few primary studies and many sources of variation can, however, jeopardise the validity of these methods. It is therefore important that plausible sources of diversity are pre-specified in a guideline , and logic modelling and stakeholder consultations during guideline scoping may be helpful (see Step 1). A more iterative and flexible process may also be needed to identify and explore these sources. In this case, changes are made to the guideline development protocol as the panel identifies relevant aspects of complexity. These changes should be explicitly documented and the rationale provided. In the case of few primary studies, qualitative comparative analysis (QCA) can be used involving cross-tabulation of evidence to identify configurations of interventions and various contextual factors that may explain the effects . When studies are too diverse to combine in a meta-analysis, findings can be synthesised and reported in a narrative manner , and graphical displays (e.g., harvest, forest, albatross, or bubble plots) can be used to illustrate patterns in the data . When effect size estimates are not reported, other information from each study can be used for statistical inferences, such as the direction of effect .
Evidence on questions beyond intervention effectiveness can be synthesised using quantitative, qualitative, and mixed methods synthesis. Quantitative approaches include model-driven meta-analysis, which can be used to explore intervention mechanisms driving the overall effect . Model-based approaches can also be used to examine how the wider system changes with intervention implementation. These approaches can be viewed as mathematical representations of (often simplified) logic models and may include empirical data (e.g., from systematic reviews), computer simulation, direct computation, or a combination of these . Qualitative evidence synthesis (QES) refers to all methods involving synthesis of diverse types of qualitative evidence from primary studies . The choice of a QES method will depend on the guideline’s scope and the specific questions asked (see Table 2). For example, thematic synthesis can be used for questions relating to socio-cultural acceptability of an intervention, as it aims to develop descriptive or analytic themes . Meta-ethnography, on the other hand, would be more suitable for questions exploring why and how intervention components work together as it aims to develop new explanations [33, 64].
Synthesis of both qualitative and quantitative evidence (so called “mixed methods”) might be required to answer some guideline questions. For example, quantitative evidence can inform whether the effects of a levy on soft drinks differ for people from different socio-economic backgrounds . Qualitative evidence, on the other hand, can further help to understand the reasons behind these differences. Mixed methods syntheses can involve separate analysis and synthesis of qualitative and quantitative evidence (i.e., segregated design), or a cyclical approach can be taken when the findings from one synthesis inform the next synthesis (i.e., contingent design) . Qualitative and quantitative evidence can be integrated in a guideline in different ways. They may be analysed in a parallel or complementary way (i.e., convergent synthesis) or conducted with one synthesis following and informing the other (i.e., sequential synthesis) . This integration can occur in a single synthesis, or two or more stand-alone reviews may be conducted first and then the findings consolidated in a cross-study synthesis .
Illustration from a guideline
To inform development of a guideline on protecting, promoting, and supporting breastfeeding practices in healthy mothers with healthy full-term babies , a systematic review was conducted . In addition to estimating the overall effects, the review team conducted sub-group analyses to explore the variation of effects based on who was delivering the intervention. The findings showed that the effect on cessation of exclusive breastfeeding at up to 6 months was greater for lay support in comparison with health professionals or mixed support . In addition, QES was conducted on the values and preferences of mothers and the factors influencing acceptability among health workers and stakeholders. It showed that most mothers found that breastfeeding was not adequately taught and reported receiving inconsistent advice from different healthcare workers, which could help explain the observed differences in the effects based on who delivered the intervention. The evidence gleaned from these quantitative and qualitative evidence syntheses was used in structuring and formulating specific guideline recommendations, including those aiming to create an enabling environment through enhanced access to adequate breastfeeding support .
Step 4: assessing the evidence
The next step in guideline development is to assess the quality of each type of contributing evidence, including evidence on effectiveness, as well as evidence addressing broader guideline questions pertinent to complexity. Different approaches can be used based on the question and type of evidence synthesis (see Table 2). For many questions, particularly those relating to effectiveness, the GRADE approach is appropriate . GRADE is designed to rate the certainty of evidence for specific outcomes, which, in a guideline context, reflects the confidence in where the true intervention effect lies relative to a meaningful threshold . Identification of meaningful thresholds across a large number of health (and non-health) outcomes can be challenging for public health and health system guidelines, particularly for global guidelines, whose implementation contexts may vary greatly. Stakeholder consultations can be helpful in identifying meaningful thresholds (see Step 1). The non-null effect can also serve as a relevant threshold in guidelines on public health and health systems interventions, such as the aforementioned example of a levy on soft drinks, as even small effect sizes can be important given the manifold impacts of a levy on the population at large. Guideline panels need to pre-specify the thresholds, as these further inform judgements on specific domains of GRADE, such as inconsistency and imprecision .
Extensions to the GRADE approach can be used to assess the quality of other types of evidence considered in a guideline, such as for different criteria of the WHO-INTEGRATE framework (see Table 2) . For example, GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) has been developed for assessing confidence in findings from QES . Some useful approaches have been developed outside of the frameworks for systematic reviewing and guideline development. For example, Quality Standards for Ethics Analyses (Q-SEA) can be used as a tool to assess the quality of ethics analyses conducted for a guideline .
Illustration from a guideline
To develop guidelines on the best approaches for strengthening and sustaining Emergency Risk Communication (ERC) capacity, a broad approach to formulating questions, evidence synthesis, and evidence assessment was adopted . The guideline panel chose the SPICE (Setting, Perspective, phenomenon of Interest, Comparison, Evaluation of impact) format for key question development to facilitate identification and synthesis of quantitative, qualitative, and mixed methods evidence, which was expected to be highly relevant to this guideline. The team identified four main methodological streams in evidence synthesis: quantitative methods with comparison groups, quantitative methods using descriptive survey methods, qualitative methods, and mixed methods and case studies. The GRADE approach, a modified GRADE approach, GRADE-CERQual, and a modified GRADE approach combined with GRADE-CERQual were used to assess evidence from these four streams, respectively .
Step 5: developing recommendations
To develop recommendations, guideline panels should consider relevant criteria, such as those published in EtD frameworks [6, 52] along with the evidence collected, synthesised, and assessed for each criterion.
To illustrate how the criteria of the WHO-INTEGRATE framework may be used to develop specific recommendations (see Table 2), let us return to the case of introducing a levy on soft drinks to tackle obesity . Let us assume that during guideline scoping (see Step 1), in addition to focusing on the direct health benefits or harms associated with the intervention, the guideline panel chose the following criteria for in-depth consideration through evidence collection, synthesis, and assessment: the acceptability of the intervention among different groups of stakeholders (e.g., different Ministries, the food industry, the general public), its societal and ecological implications (e.g., changes in social norms in relation to soft drinks and reductions in aluminium and plastic waste), and its impact on health equity, equality, and non-discrimination (e.g., increase in consumption patterns among certain socio-economic groups). The collected and assessed evidence for these criteria will inform specific recommendations. For example, if there is evidence deemed reliable by the guideline panel that shows a positive impact on the environment associated with levy introduction, this may contribute towards making a recommendation in favour of this intervention – given that environmental sustainability is a priority for the guideline panel. Guideline panels will need to use judgement in weighing the importance of the criteria when developing recommendations (e.g., prioritising net societal benefits over intervention acceptability). This prioritisation process should be explicitly documented and reflect the perspectives from all relevant stakeholders (see above) .
Illustration from a guideline
To develop a guideline on how to safely design and implement sanitation services, the guideline panel conducted a survey of selected global sanitation actors in health, public sectors, sanitation financing, academic institutions, and international and not-for profit organisations to help define the guideline scope and priorities . The team considered evidence for each of the six substantive criteria of the WHO-INTEGRATE framework to formulate guideline recommendations assigning importance to each of these. For each criterion, the evidence was summarised and the rationale was given for making a judgment about how the criterion influenced the recommendation. The meta-criterion, quality of evidence, was applied only in relation to intervention effectiveness, as the panel did not find suitable methods to use it for the other criteria.
In this paper we describe the process and methods for developing guidelines from a complexity perspective. Public health and health systems interventions often interact with and adapt to the system in which they are implemented; thus in assessing their impact, it is important to consider this wider system. In this step-by-step guide, we particularly emphasise and recommend that guideline panels make investment in the early phase of guideline scoping. This will set the stage for subsequent steps, including timely collection, synthesis, and assessment of evidence. Through explicit consideration of a range of questions in addition to those about intervention effectiveness, taking a complexity perspective will produce guidelines with better informed recommendations and more transparent procedures. Furthermore, by providing an enhanced understanding of the specific circumstances in which interventions work, a complexity perspective can also facilitate local adaptation and implementation of guidelines. This can be achieved through the explicit addition of contextual specifications in the guideline, as well as documentation of the voices and potentially diverging perspectives of key stakeholders. This paper provides general guidance on when and how to take a complexity perspective in guideline development. Although it draws on examples of guidelines on public health and health system interventions and is likely to be of most relevance for guideline panels working on these types of interventions, the described steps can also be applied to guidelines on clinical interventions if a guideline panel thinks that taking a complexity perspective may add value. Indeed, interventions and services in clinical care practice often target complex health issues and are often delivered in complex healthcare systems. For example, it has been shown that use of emergency health services displays characteristics of complex systems, including heavy-tailed distribution and sequences of consultations clustered in time . These call for services that address the whole system rather than focusing on problematic individuals only.
While this paper largely draws on the series of papers from BMJ Global Health, which was developed using a consensus-based methodology and different systematic and other review methods, several areas will benefit from further methodological research and development. Specifically, further work will need to look into the methods of collecting and assessing different types of evidence, such as ethics, financial, and economic analyses (see Table 2). Standardised approaches similar to GRADE and GRADE-CERQual for rating these types of evidence may be helpful. However, such approaches should also be pragmatic to enable rapid application, given that guideline development tends to happen under significant time and resource constraints. There is also a need for further procedural guidance on how to prioritise across a range of EtD criteria in a guideline. Prioritisation can be a challenging process, because of many potentially divergent perspectives . Finally, while there are many published guidelines that have used some of the procedures we describe in this paper, the overall 5-step process has not yet been used and systematically tested in a single guideline. We therefore lack real-world confirmation of the value and feasibility of this approach. However, we are aware of at least one WHO guideline that is currently using this process. As complexity is an evolving topic in public health and health systems research, more examples are needed of guidelines taking such a perspective.
Availability of data and materials
Confidence in the Evidence from Reviews of Qualitative Research
Emergency Risk Communication
Grading of Recommendations Assessment, Development, and Evaluation
Perspective, Setting, Phenomenon of interest, Environment, Comparison, Time, and Findings
Population, Intervention, Comparison, Outcome
Qualitative Comparative Analysis
Qualitative Evidence Synthesis
Quality Standards for Ethics Analyses
Setting, Perspective, phenomenon of Interest, Comparison, Evaluation of impact
Word Health Organization INTEGRATe Evidence
Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.
Norris SL, Rehfuess EA, Smith H, Tuncalp O, Grimshaw J, Ford N, et al. Complex health interventions in complex systems: improving the process and methods for evidene-informed health decisions. BMJ Glob Health. 2019;4:e000963.
Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci. 2017;12(1):21.
Kickbusch I. Addressing the interface of the political and commercial determinants of health. Health Promot Int. 2012;27(4):427–8.
Petticrew M, Knai C, Thomas J, Rehfuess EA, Noyes J, Gerhardus A, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision-making. BMJ Glob Health. 2019;4:e000899.
Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris S, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: Intergrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4:e000844.
Mikton C, Butchart A. Child maltreatment prevention: a systematic review of reviews. Bull World Health Organ. 2009;87(5):353–61.
Azad K, Costello A. Extreme caution is needed before scale-up of antenatal corticosteroids to reduce preterm deaths in low-income settings. Lancet Glob Health. 2014;2(4):e191–e2.
Moore G, Evans R, Hawkins J, Littlecott H, Melendez-Torres GJ, Bonell C, et al. From complex social interventions to interventions in complex social systems: future directions and unresolved questions for intervention development and evaluation. Evaluation. 2018;25(1):23–45.
Huang LL, Baker HM, Meernik C, Ranney LM, Richardson A, Goldstein AO. Impact of non-menthol flavours in tobacco products on perceptions and use among youth, young adults and adults: a systematic review. Tob Control. 2017;26(6):709–19.
Brisson M, Benard E, Drolet M, Bogaards JA, Baussano I, Vanska S, et al. Population-level impact, herd immunity, and elimination after human papillomavirus vaccination: a systematic review and meta-analysis of predictions from transmission-dynamic models. Lancet Public Health. 2016;1(1):e8–e17.
Cronin AA, Gnilo ME, Odagiri M, Wijesekera S. Equity implications for sanitation from recent health and nutrition evidence. Int J Equity Health. 2017;16(1):211.
Penney TL, Brown HE, Maguire ER, Kuhn I, Monsivais P. Local food environment interventions to improve healthy food choice in adults: a systematic review and realist synthesis protocol. BMJ Open. 2015;5(4):e007161.
Petticrew M, Shemilt I, Lorenc T, Marteau TM, Melendez-Torres GJ, O'Mara-Eves A, et al. Alcohol advertising and public health: systems perspectives versus narrow perspectives. J Epidemiol Community Health. 2017;71(3):308–12.
Siegfried N, Pienaar DC, Ataguba JE, Volmink J, Kredo T, Jere M, et al. Restricting or banning alcohol advertising to reduce alcohol consumption in adults and adolescents. Cochrane Database Syst Rev. 2014;11:CD010704.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: new guidance Medical Research Council; 2008. [cited 2020 May 28]. Available from: https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/.
Galea S, Riddle M, Kaplan GA. Causal thinking and complex system approaches in epidemiology. Int J Epidemiol. 2010;39(1):97–106.
Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3.
Gruer L, Tursan d'Espaignet E, Haw S, Fernandez E, Mackay J. Smoke-free legislation: global reach, impact and remaining challenges. Public Health. 2012;126(3):227–9.
White M. Evaluation of the health impacts of the UK Treasury soft drinks industry levy (SDIL)Protocol. ISRCTN: 18042742 Funded by NIHR Public Health Research Programme Study number: 16/130/01 [cited 2020 Jun 2]. Available from: https://njl-admin.nihr.ac.uk/document/download/2010886; 2017.
Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76.
Guise JM, Chang C, Butler M, Viswanathan M, Tugwell P. AHRQ series on complex intervention systematic reviews-paper 1: an introduction to a series of articles that provide guidance and tools for reviews of complex interventions. J Clin Epidemiol. 2017;90:6–10.
Anderson LM, Petticrew M, Chandler J, Grimshaw J, Tugwell P, O'Neill J, et al. Introducing a series of methodological articles on considering complexity in systematic reviews of interventions. J Clin Epidemiol. 2013;66(11):1205–8.
WHO. Handbook for guideline development (2nd ed). Geneva: World Health Organization; 2014. [cited 2020 Jul 19]. Available from: https://apps.who.int/iris/bitstream/handle/10665/75146/9789241548441_eng.pdf;jsessionid=0B1B3B6FDC8E02D80D1486701EB1AD4F?sequence=1.
Concannon TW, Grant S, Welch V, Petkovic J, Selby J, Crowe S, et al. Practical guidance for involving stakeholders in Health Research. J Gen Intern Med. 2019;34(3):458–63.
Cottrell E, Whitlock E, Kato E, Uhl S, Belinson S, Chang C, et al. Defining the benefits of stakeholder engagement in systematic reviews. RockvilleReport No.: 14-EHC006-EF: AHRQ Methods for Effective Health Care; 2014.
Munthe-Kaas H, Nøkleby H, Lewin S, Glenton C. The TRANSFER approach for assessing the transferability of systematic review findings. BMC Med Res Methodol. 2020;20:11.
Larsson I, Staland-Nyman C, Svedberg P, Nygren JM, Carlsson IM. Children and young people's participation in developing interventions in health and well-being: a scoping review. BMC Health Serv Res. 2018;18(1):507.
Allender S, Owen B, Kuhlberg J, Lowe J, Nagorcka-Smith P, Whelan J, et al. A community based systems diagram of obesity causes. PLoS One. 2015;10(7):e0129683.
Rehfuess EA, Booth A, Brereton L, Burns J, Gerhardus A, Mozygemba K, et al. Towards a taxonomy of logic models in systematic reviews and health technology assessments: a priori, staged, and iterative approaches. Res Synth Methods. 2018;9(1):13–24.
Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5:28.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions. 2nd ed. Chichester: Wiley; 2019.
Flemming K, Booth A, Garside R, Tuncalp O, Noyes J. Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Glob Health. 2019;4:e000882.
Booth A, Noyes J, Flemming K, Gerhardus A, Wahlster P, van der Wilt GJ, et al. Structured methodology review identified seven (RETREAT) criteria for selecting qualitative evidence synthesis approaches. J Clin Epidemiol. 2018;99:41–52.
Noyes J, Booth A, Moore G, Flemming K, Tuncalp O, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4:e000893.
Siegfried N, Narasimhan M, Kennedy CE, Welbourn A, Yuvraj A. Using GRADE as a framework to guide research on the sexual and reproductive health and rights (SRHR) of women living with HIV - methodological opportunities and challenges. AIDS Care. 2017;29(9):1088–93.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strenght of recommendations. BMJ. 2008;336:924–6.
Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1):2.
Droste S, Dintsios CM, Gerber A. Information on ethical issues in health technology assessment: how and where to find them. Int J Technol Assess Health Care. 2010;26(4):441–9.
Mertz M, Kahrass H, Strech D. Current state of ethics literature synthesis: a systematic review of reviews. BMC Med. 2016;14(1):152.
Booth A, Noyes J, Flemming K, Moore G, Tuncalp O, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative eviedence synthesis. BMJ Glob Health. 2019;4:e001107.
Scott AM, Hofmann B, Gutierrez-Ibarluzea I, Bakke Lysdahl K, Sandman L, Bombard Y. Q-SEA - a tool for quality assessment of ethics analyses conducted as part of health technology assessments. GMS Health Technol Assess. 2017;13:Doc02.
Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
O'Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014;67(1):56–64.
Campbell and Cochrane Equity Methods Group. Progress- plus; 2017. [cited 2020 May 25]. Available from: https://methods.cochrane.org/equity/projects/evidence-equity/progress-plus.
Welch VA, Akl EA, Pottie K, Ansari MT, Briel M, Christensen R, et al. GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence. J Clin Epidemiol. 2017;90:76–83.
EUnetHTA Joint Action 2 WP. HTA Core Model ® version 3.0; 2016.
Drummond M, Sculpher M, Kea C. Methods for the economic evaluation of health care programmes. 4th ed. Oxford: Oxford University Press; 2015.
Shemilt I, McDaid D, Marsh K, Henderson C, Bertranou E, Mallander J, et al. Issues in the incorporation of economic perspectives and evidence into Cochrane reviews. Syst Rev. 2013;2:83.
Brunetti M, Shemilt I, Pregno S, Vale L, Oxman AD, Lord J, et al. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. J Clin Epidemiol. 2013;66(2):140–50.
Alonso-Coello P, Schunemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE evidence to decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction. BMJ. 2016;353:i2016.
World Health Organization. Consolidated guideline on sexual and reproductive health and rights of women living with HIV. Geneva: World Health Organization; 2017. Contract No.: Licence: CC BY-NC-SA 3.0 IGO.
Booth A, Moore G, Flemming K, Garside R, Rollings N, Tuncalp O, et al. Taking account of context in systematic reviews and guidelines considering a complexity perspective. BMJ Glob Health. 2019;4:e000840.
Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, et al. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes. 2015;109(2):103–14.
Higgins JPT, Lopez-Lopez JA, Becker BJ, Davies SR, Dawson S, Grimshaw J, et al. Synthesising quantitative evidence in systematic reviews of complex health interventions. BMJ Glob Health. 2019;4:e000858.
Kneale D, Thomas J, Harris K. Developing and Optimising the use of logic models in systematic reviews: exploring practice and good practice in the use of Programme theory in reviews. PLoS One. 2015;10(11):e0142187.
Downe S, Finlayson K. Tuncalp, Metin Gulmezoglu a. what matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.
World Health Organization. WHO recommendations on antenatal care for a positive pregnancy experience. Geneva: World Health Organization; 2016.
Melendez-Torres GJ, Bonell C, Thomas J. Emergent approaches to the meta-analysis of multiple heterogeneous complex interventions. BMC Med Res Methodol. 2015;15:47.
Thomas J, O'Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.
Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368:l6890.
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.
Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: SAGE; 1988.
Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29.
Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017;6(1):61.
World Health Organization. Guideline: protecting, promoting and supporting breastfeeding in facilities providing maternity and newborn services. Geneva: World Health Organization; 2017. Contract No.: Licence: CC BY-NC-SA 3.0 IGO.
McFadden A, Gavine A, Renfrew MJ, Wade A, Buchanan P, Taylor JL, et al. Support for healthy breastfeeding mothers with healthy term babies. Cochrane Database Syst Rev. 2017;2:CD001141.
Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the journal of clinical epidemiology. J Clin Epidemiol. 2011;64(4):380–2.
Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE working group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13.
Montgomery P, Movsisyan A, Grant S, Macdonald G, Rehfuess E. Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health. BMJ Glob Health. 2019;4:e000848.
World Health Organization. Communicating risk in public health emergencies: a WHO guideline for emergency risk communication (ERC) policy and practice. Geneva: World Health Organization; 2017. Contract No.: Licence: CC BY-NC-SA 3.0 IGO.
Martin D, Singer P. A strategy to improve priority setting in health care institutions. Health Care Anal. 2003;11(1):59–68.
World Health Organization. Guidelines on sanitation and health. Geneva: World Health Organization; 2018. Contract No.: Licence: CC BY-NC-SA 3.0 IGO.
Burton C, Elliott A, Cochran A, Love T. Do healthcare services behave as complex systems? Analysis of patterns of attendance and implications for service delivery. BMC Med. 2018;16(1):138.
Daniels N. Accountability for reasonableness. BMJ. 2000;321(7272):1300–1.
This paper draws on the series of papers published in BMJ Global Health entitled “Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions”. We are indebted to all the authors and the many thought-provoking discussions held in preparing this series.
This paper has been commissioned by WHO. Specifically, AM received funding from WHO to prepare this paper.
Ethics approval and consent to participate
Consent for publication
All authors are members of the GRADE Working Group and have been involved in the group’s methodological work supporting a complexity perspective in evidence synthesis and guidelines. ER is a methods editor with Cochrane Public Health and a founding member of Cochrane Public Health Europe. SLN is an employee of WHO and oversees the quality of its guidelines. AM, SLN, and ER have published methods articles as part of the BMJ Global Health series on improving the process and methods for evidence-informed health decisions through a complexity perspective commissioned by WHO. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Movsisyan, A., Rehfuess, E. & Norris, S.L. When complexity matters: a step-by-step guide to incorporating a complexity perspective in guideline development for public health and health system interventions. BMC Med Res Methodol 20, 245 (2020). https://doi.org/10.1186/s12874-020-01132-6
- Systematic reviews
- Complexity perspective
- Systems thinking
- Logic model
- Decision criteria