Skip to content

Advertisement

You're viewing the new version of our site. Please leave us feedback.

Learn more

BMC Medical Research Methodology

Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Using the scenario method in the context of health and health care – a scoping review

  • Horst Christian Vollmar1, 2Email author,
  • Thomas Ostermann3 and
  • Marcus Redaèlli1, 4
BMC Medical Research Methodology201515:89

https://doi.org/10.1186/s12874-015-0083-1

Received: 16 January 2015

Accepted: 12 October 2015

Published: 16 October 2015

Abstract

Background

The scenario technique is a method for future research and for strategic planning. Today, it includes both qualitative and quantitative elements. The aims of this scoping review are to give an overview of the application of the scenario method in the fields of health care and to make suggestions for better reporting in future scenario projects.

Methods

Between January 2013 and October 2013 we conducted a systematic search in the databases Medline, Embase, PsycInfo, Eric, The Cochrane Library, Scopus, Web of Science, and Cinahl since inception for the term ‘scenario(s)’ in combination with other terms, e.g. method, model, and technique. Our search was not restricted by date or language. In addition, we screened the reference lists of the included articles.

Results

A total of 576 bibliographical records were screened. After removing duplicates and three rounds of screening, 41 articles covering 38 different scenario projects were included for the final analysis. Nine of the included articles addressed disease related issues, led by mental health and dementia (n = 4), and followed by cancer (n = 3). Five scenario projects focused on public health issues at an organizational level and five focused on the labor market for different health care professionals. In addition, four projects dealt with health care ‘in general’, four with the field of biotechnology and personalized medicine, and additional four with other technology developments. Some of the scenario projects suffered from poor reporting of methodological aspects.

Conclusions

Despite its potential, use of the scenario method seems to be published rarely in comparison to other methods such as the Delphi-technique, at least in the field of health care. This might be due to the complexity of the methodological approach. Individual project methods and activities vary widely and are poorly reported. Improved criteria are required for reporting of scenario project methods. With improved standards and greater transparency, the scenario method will be a good tool for scientific health care planning and strategic decision-making in public health.

Keywords

Scenario methodHealth planningPublic health administrationPlanning techniquesDecision-makingPolicy makingForesightForecasting

Background

Strategic decision-making processes in the field of health care and public health have always been a point of critical discussion between the stakeholders involved. In particular, prospective planning of financial resources for epidemiologically relevant and cost intensive diseases, like dementia, is often challenging. In such cases tools to support stakeholders in the field of evidence-based decision-making have become quite important [1]. Tools often used in strategic decision-making in public health are consensus processes e.g. the Delphi technique [2, 3]. These approaches use rounds of questionnaire surveys where information and results are fed back to panel members between each round [4]. According to a recent review by Diamond and colleagues, 98 % of Delphi studies claimed to assess consensus [5]. Other methods used by policy makers in the provision of health care are based on simulation modeling. In their systematic review of the “use and value of computer simulation modeling in population health and health care delivery”, Fone et al. found 182 papers using simulation techniques in the field of public health [6]. Both the Delphi technique and simulation modeling are widely used for health care issues, but were originally developed to support forecast and foresight processes [7]. These methods are not without critique and leave room for improvement [5, 8].

Over the last decades, the scenario method has become an additional tool in foresight activities and research. To a certain extent, it makes use of both qualitative (e.g. expert opinion and discussion) and quantitative elements (e.g. scenario calculations) [911].

Glenn gave the following definition: “a scenario is a story with plausible cause and effect links that connects a future condition with the present, while illustrating key decisions, events, and consequences throughout the narrative” [9]. Scenarios are often described as outlines of possible futures, but they do not describe comprehensive pictures of the future and do not claim to be complete or correct [9, 10, 12, 13]. Although, scenarios are always hypothetical, they are not arbitrary [10]. Additionally, the creation of scenarios presents an interdisciplinary approach to explore future issues while offering several advantages, e.g. the support of a future-oriented way of thinking by taking alternative developments into consideration [911]. Furthermore, it fosters systematic and structured discussion of uncertain alternative futures by the incorporation of expert knowledge. Proceeding step-by-step reduces the perceived complexity of the correlations examined, generates findings that are comprehensible [14], and should improve strategic decision-making [1517]. It may be combined with other foresight methods such as the Delphi technique or road-mapping [18, 19]. Since its first appearance in the 1950s, and after a decline during the 1980s, the number of published articles using this method is again increasing [20].

Although less commonly used in the context of health and health care than either the Delphi technique or simulation modeling, the scenario method has also been used to support strategic decision-making in the field [13, 1517, 2125]. Unfortunately, information about the different scenario projects in the context of public health or health policy seems to be disparate and often not known to researchers in this field. Thus, there is a basic need to provide an overview of published scenario projects. The first aim of this review is to give such an overview of the application of the scenario method in the context of health and health care. The second aim is to make first recommendations for improved reporting in future scenario projects.

Methods

We conducted our scoping review [26] on the basis of the enhanced recent recommendations of Arksey and O’Malley [27] by Levac and colleagues [28] and presented the results of our search strategy in a flowchart (Fig. 1). Between January 2013 and October 2013 we conducted a systematic search in the following: Medline, Embase, PsycInfo, Eric, The Cochrane Library, Scopus, Web of Science, and CINAHL. An initial search was carried out in January 2013 and an additional search was made in October 2013. The word ‘scenario’ was searched in combination with other terms, e.g. method, model, technique, etc. (see Additional files 1 and 2). In an additional step, the reference lists were tracked backwards for further relevant publications not listed in the databases mentioned above (Fig. 1). We also included in our review manuscripts which were recommended by authors or experts in the field [16, 29]. We also scanned articles suggested by the ´related citations in PubMed’ option for the three most recently published articles [16, 21, 30]. Our search was not restricted by date or language. After screening the title, and, if available, the abstract, all articles that both dealt with the scenario method and addressed issues related to health or health care were included for full text screening. This full screening was performed by two reviewers (HCV, MR) with the following exclusion criteria. Discrepancies were discussed between the two reviewers to achieve consensus. In the case of a possible disagreement a third author (TO) was designated (not required).
Fig. 1

Flow chart of the review process

Articles were excluded in which the term ‘scenario’ was used only to refer to a possible (future) event [31, 32]. Other exclusion criteria were:
  • ‘scenarios’ in epidemiology when used only as projections (e.g. ‘population ageing’, defined as an increase in the percentage of elderly persons in the population) [33],

  • ‘scenarios‘which were ‘pure’ simulation modeling [34, 35],

  • ‘scenarios’ which were used only to support shared decision-making (e.g. determination of patient preferences) [36, 37],

  • ‘scenarios’ in microbiology or genetics [38, 39],

  • description of the scenario method itself (without a concrete project) [4042],

  • publications unrelated to health or health care [43],

  • abstracts only (no full-text available or full-text already included in our review) [4446],

  • grey literature (e.g. reports) without publication in a scientific journal [25, 47].

Results

Our search resulted in the identification of a total of 576 bibliographical records. Charting of the data was undertaken independently by two authors (HCV, MR). After removing duplicates, 379 references remained, which, after thorough title and abstract screening, left 67 selected references for possible inclusion (Fig. 1). After full-text screening, 21 publications were excluded because they did not fit the criteria previously determined. From these 46 articles describing the scenario method in use, a total of 41 remained after a final step of exclusion (Fig. 1). Reasons for exclusion were as follows: description of a whole national scenario program (see below) [48], only an abstract of an included project [46], only a simulation modeling project without the description of the scenarios [49], one background paper of an included project with no direct link to the scenarios [50], and one article with no scenario project [51]. The final 41 publications described a total number of 38 different scenario projects. As the studies were quite heterogeneous and included a variety of perspectives, it was decided to classify them by using the following categories (according to Schnaars): year, institution, country, focus, time horizon, number of developed scenarios (Table 1) [52]. We also had discussed to include details of the used methods (e.g. qualitative, quantitative, or both), however in many studies the methods were not adequately described, so we decided not to expand this point. Table 1 gives an overview of the subjects of the included scenario projects, most of them with a time horizon from 10 to 20 years (n = 28), two with a 5 year- [13, 53] and one with a 50 year-perspective [54]. In seven projects the adopted time horizon was not mentioned [24, 5560]. The background of the participating experts was not always adequately reported. The reported background of the experts ranged from “leading health futurists” [61], “members of scientific expert societies and/or staff associations” [55, 62], “experienced managers” [56], “RAND health researchers” [63], “genomic experts and breast cancer specialists” [64], “younger citizens” [65], to “secondary school pupils and university students” [29] or “community members” [30]. Only two projects explicitly stated the background of the experts in a table [16, 66]. Five projects reported a combination of the scenario method with a Delphi technique to reach consensus among participating experts [18, 64, 6771]. One project was published in Dutch [70] and one in German [55]. All others were published in English. The number of developed scenarios ranged from one [72] to 19 [73] scenarios, but most frequently the numbers of scenarios were three (n = 10), four (n = 9) or five (n = 6) (Table 1).
Table 1

Subjects of the scenario projects

Year of publication

Reference

Institution

Country

Focus/title of the project

Time horizon [approx. in years]

Number of scenarios

Before 1995

      

2 x 1988

Becker [81], also Schreuder [73]

STGb

Netherlands

Aging in the Netherlands

10

3

1991, 1992

Bijl [69] also Bijl & Ketting [70]

STGb

Netherlands

Dementia in the Netherlands

10

3

1994

Leufkens et al. [79]

STGb

Netherlands

Future of Medicine

10 (15)

4

1988, 1989

Schaapveld & Cleton [74] also Schreuder [73]

STGb

Netherlands

Cancer diseases in the Netherlands

15

>5

1988

Schreuder [73]

STGb

Netherlands

Cardio-vascular diseases in the Netherlands

15

19

1989, 1997

Van Beeck et al.[68] also van Beeck & Mackenbach [67]

STGb

Netherlands

Accident mortality and unintentional injuries in the Netherlands

15

9

1992

Bezold [61]

IAFa

US

Leadership practices and organizational demands

10

5

1993

Venable et al. [53]

University of Alabama

US

Local public health departments

5

2

1991

Zentner [76]

Institute for Health Care Marketing

US

Health care organization (case example: future opportunities of American Transitional Care, Inc.)

10

3

1995-2004

      

2001

Harmsen et al. [80]

Aarhus School of Business

Denmark

Danish food industry

10

3

1999

Islei et al. [56]

Various universities

UK

Pharmaceutical industry

n.s.c

7

1997

Leufkens et al. [60]

Department of Pharmaco-epidemiology & Pharmacotherapy

Netherlands

Clinical pharmacy

n.s.c

3

2000

Ling & Hadridge [78]

Cambridge Foresight

UK

Health care

(15-) 20

2

2004

Neiner et al. [24]

National Center for Chronic Disease Prevention and Health Promotion

US

Public health (specifically to illustrate a health department’s desire to address chronic disease prevention and control)

n.s.c

3

1998

Nielsen [72]

Allen Memorial Hospital

US

Healthcare delivery

10

1

2001

Sager [59]

Life Science Strategy Consulting

US

Biotechnology

n.s.c

4

2003

Van Lente et al. [71]

University of Utrecht

Netherlands

Biotechnology in Europe

10-15

4

Since 2005

      

2005

Bezold [22]

IAFa/Picker Institute

US

Patient-centered care

10

4

2005

Bezold & Beck [54]

IAFa

US

Drug regulation

50

3

2011

Bierbooms et al. [13]

Tilburg University

Netherlands

What types of residence should be organized for people with mental health problems?

5

4

2012

Buchan & Seccombe [23]

Queen Margaret University

UK

Future supply of registered nursing staff, midwives and health visitors in the National Health Service (NHS)

10

8

2013

Carlsen et al. [84]

Defense Research Agency

Sweden

Local adaptation to climate change (health aspects)

20

2

2 x 2005

Clark et al. [83] also Awasthi et al. [82]

International Campaign to Revitalise Academic Medicine (ICRAM)

International

Academic medicine

20

5

2006

Eberl & Schnepp [55]

The German Nurses Association

Germany

Family health nursing in Germany

n.s.c

7 reduced to 5

2011

Enzmann et al. [62]

Society of Chairs of Academic Radiology Departments (SCARD)

US

Field of radiology

10

3

2012

Gnatzy & Moser [18]

EBS Business School, Deutschland

India (Germany)

Evolving health insurance industry in rural India

10

4

2014

Gregório et al. [16]

WHO Collaborating Centre for Health Workforce Policy and Planning

Portugal

Community pharmacists

10

3

2013

Karger [29]

Forschungs-zentrum Jülich

Germany

Personalized medicine on the example of dementia

20

4

2006

Ma & Seid [63]

RAND cooperation

US

Disease management in the US

15

8

2009

Meristö [57]

My wellbeing project

Finland

Life control especially related to health and personal wellbeing using ICT-tools

n.s.c

3

2014

Nguyen et al. [30]

Centre for Public Health and Ecosystem Research

Vietnam

Community development

10

2d

2005

Niewöhner et al. [65]

Max-Delbrueck-Center for Molecular Medicine (MDC)

Germany

Relationship between biomedicine and economy in Germany

10

4

2012

Retèl et al. [64]

Netherlands Cancer Institute

Netherlands

Developments in technology assessment (e.g. clinical implementation of the 70-gene signature for breast cancer)

15

10 reduced to 5

2012

Rhea & Bettles [77]

Academy of Nutrition & Dietetics

US

Dietetics workforce supply and demand

10

4

2006

Rydström & Törnberg [58]

Karolinska Institute

Sweden

External influences on cervical cancer incidence and mortality

n.s.c

8

2011

Suk & Semenza [75]

European Centre for Disease Prevention and Control (ECDC)

Europe (Sweden)

Future infectious disease threats to Europe

10

8

2014

Vollmar et al. [21]

German Center for Neuro-degenerative Diseases (DZNE)

Germany

Health care for people with dementia in Germany

20

5

2008

Wiek et al. [66]

Institute for Environmental Decisions (IED)

Switzerland

Possible future developments of nanotechnology in Switzerland

10

5

a IAF Institute for Alternative Futures, b STG Steering Committee on Future Health Scenarios, c n.s. not specified d“the outputs were limited to the best and worst case scenarios” [30]

Main topics

The main topics of the scenario projects differed in many ways. Most of them addressed disease related issues (n = 9), led by mental health and dementia (n = 4) [13, 21, 29, 69, 70] and cancer (n = 3) [58, 64, 74]. Only one scenario project each reported on cardio-vascular diseases [73] and infectious diseases [75].

Five scenario projects dealt with public health issues on an organizational level [24, 30, 53, 61, 76] and five on the labor market of different health care professionals [16, 23, 55, 60, 77], with two of them focusing on the pharmacy profession [16, 60].

In addition, four projects dealt with health care ‘in general’ [22, 63, 72, 78], four with other technology developments [57, 62, 64, 66], and an additional four with the field of biotechnology and personalized medicine [29, 59, 65, 71].

Three projects were concerned with the pharmaceutical industry and drug development [54, 56, 79]. We could identify only one scenario project each about the food industry [80], aging issues [73, 81], ‘recommendations’ for a developing country [18], academic medicine [82, 83], and the influence of climate change (on health) [84].

Eleven scenario projects (Table 1) were from North America, 25 from Europe (one of them with a topic about India [18]), one from Vietnam [30] and one had an international focus [82, 83]. More than half of the projects (n = 21) were published in the last ten years, eight between 1995 and 2004 and nine before 1995. Of these nine, six projects [6770, 73, 74, 79, 81] were part of a national program in the Netherlands. During this program the study group for future scenarios in health care (STG/STC) developed several scenarios for certain health issues from the mid-1980s to the mid-1990s for the Dutch government [48, 73, 85]. In this review we considered (as mentioned in our inclusion/exclusion criteria) only those program projects which were published in scientific journals (n = 8) [6770, 73, 74, 79, 81]. A full list of all topics covered by the STG/STC program can be found in the program description of Schreuder [48].

Discussion

The scenario method has been used for a wide spectrum of strategic issues and different applications, starting with military planning in the 1960s [911, 52, 86, 87]. Despite its potential, use of the scenario method seems to have been published rarely in comparison to other methods such as the Delphi-technique, at least in the field of health care since the 1980s [40, 48, 88]. Our scoping review could identify 41 relevant publications in scientific journals representing 38 scenario projects. There were a lot of different perspectives as indicated by the wide range of participating institutions and experts (Table 1). The scenario projects addressed not only public health problems, but also strategic issues for business decisions (e.g. the future of the Danish food industry [80], the opportunities of nanotechnology in Switzerland [66], or the relationship between biomedicine and the economy in Germany [65]). One project even came from the Swedish Defense Research Agency (FOI) with tailor-made scenarios for local adaptation to climate change [84]. Nevertheless, this project addressed (among other things) the effects of a heat wave on the health care sector, which is definitely an important topic for public health researchers. One project described three fictive scenarios as examples for the scenario method itself [24]. We decided to include this “project” because it addressed a relevant public health issue (“using a health department’s desire to address chronic disease prevention and control”) [24]. Most of the projects were developed after the year 2000 and addressed a wide range of topics, from regional institutional perspectives (e.g. local public health departments [53]) to global challenges (e.g. future infectious disease threats to Europe [75]). Many of the scenario projects in this review provide a framework for determining actions in research, as well as in public policy-making, e.g. it could be the basis for discussing a national dementia plan [21, 69, 70] or for developing a strategy to ‘revitalize’ academic medicine [82, 83]. None of the projects has been designated as unsuccessful by the authors, which could be either a sign of the method’s strength or of publication bias. In fact, Gregório and colleagues stated: “The use of scenario analysis in a strategic thinking process has demonstrated to be of value while planning for future resources and other policy issues” [16]. Several of these scenario projects were classified as helpful for strategic planning and also for enabling the incorporation of expert knowledge (the qualitative ‘human factor’) [13, 48, 53]. Additionally, several projects used quantitative approaches to calculate the scenarios [21, 23, 56]. The resulting scenarios were illustrated in many different ways or combinations (e.g. tables [82], text descriptions [78], pictures [83], or short stories called storylines [21]). Although there is no definite response to the question of how many scenarios are optimal in the scenario planning literature [87], three to five scenarios are considered appropriate by most of the researchers [86, 87]. This number also occurred most frequently in our review. Scenarios can be described as outlines of possible variations of the future [88], but do not describe comprehensive pictures of the future and do not claim to be complete or correct [10]. Sometimes doubts with respect to the reliability of the scenarios may arise because the methods are not clearly described [15, 54, 78, 82, 83, 89]. Compared to conventional methodological reporting, i.e. in clinical studies, the method in scenario projects should be described as precisely as possible due to the process-oriented character of scenario development. This includes the selection of the experts, the applied software tools, the use of additional literature sources, and also the method’s use in combination with other methods, like the Delphi technique [6771].

Limitations

Although this review is quite comprehensive with respect to the scenario method in the field of health or health care, there are some limitations (partly inherent to the scenario method, see below) which should be mentioned. First of all, despite our detailed search strategy, it was difficult to accurately identify appropriate scenario projects. This issue is summarized by Glenn’s statement: “scenario is the most abused term in futures research” [9]. Bishop and colleagues added: “even the most basic vocabulary is used every which way in this field [89]. We provided some examples of this issue in the Methods section (reasons for exclusion). Even when an article’s title seems to be clear, one cannot be certain that the article deals with a scenario project [31, 90]. Furthermore, there is no clear-cut scenario method. Unlike classical epidemiological research, many variants of the method exist and, according to our findings, are applied in various projects [9, 10, 48, 87, 88]. An additional difficulty is that these different variants of the method have not remained stable. For example, there has been a notable shift from quantitative to more qualitative or mixed method approaches, as evidenced by statements like “Scenario-projects are primarily simulation” [81], “scenario analysis is essentially a qualitative technique” [53], “we have used […], the qualitative scenario method, in order to assess and rank possible influencing factors” [58]. According to Glenn, “often projections are confused with scenarios” [9]. Another limitation is that only 11 publications were identified from searches in the databases used in this review. The remaining 30 publications were found by screening the reference lists, on the internet, and through recommendations from experts in the field. It was also apparent that only a few scenario projects have been published in scientific journals (by researchers), whereas a substantial number of project reports have been published as grey literature by government institutions [25, 91], non-government organizations [47, 92] or private (commercial/consulting) firms [92, 93]. So, it should be taken into consideration that some of the scenario projects have not been developed by scientific experts, but rather came from non-scientific institutions or clinical organizations (with little awareness of scientific research techniques) [55, 62, 72]). Additionally, some included scenario projects [21, 29] have also been published in more detailed reports [94] or books [95].

Decision against reviewing grey literature

The “Pisa Declaration on Policy Development for Grey Literature Resources” [96], a Cochrane report [97], and the enhanced Arksey and O´Malley framework [27, 28] all recommend including grey literature to validate the results of a research-based literature search. However, Levac and colleagues also point out the cost-to-benefit ratio consideration: “Balancing breadth and comprehensiveness of the scoping study with feasibility of resources can be challenging” [28].

We decided not to include grey literature because of the following reasons:
  • Firstly, although non-peer reviewed publications have the potential to provide valuable insights in this area, the quality of methods applied to data collection, analysis and interpretation may vary substantially [98]. We conducted an exploratory search on the internet before we used a systematic search approach. We randomly surveyed the reports and found a very heterogeneous quality. Although there were a few reports reflecting high scientific quality (e.g. [25, 99]) we also identified reports of lesser quality and reports with minimal to non-existent descriptions of the methods used [93]. Notably, one can be assumed that reports from commercial/consulting firms [92, 93] try to avoid detailed and transparent description of the scenario preparation process in order to protect proprietary data and the nature of their business model.

  • Secondly, in order to confirm that our results could be replicated, we wanted to ensure that all studies had been subjected to some form of peer-review. In order to still allow a maximal amount of comprehensiveness, we conducted a systematic search for literature in the mentioned databases and also in the reference lists of the identified literature (backward tracking).

  • Thirdly, there is still a lack of persistent identifiers and open standards of metadata for grey literature, which complicates the identification of relevant publications.

  • Fourthly, the large number of existing scenario projects addressed in reports (e.g. [25, 91, 93, 94, 99101]) made it impossible to handle a comprehensive search with our limited resources. Because of that, the effort associated with a comprehensive search for grey literature would be disproportionate to the resulting benefits. For example, the study group for future scenarios in health care (STG/STC) listed 29 published books for their program alone [48]. For this program we identified eight scenario project articles published in scientific journals [6770, 73, 74, 79, 81].

For these reasons, we think a subsequent integration of grey literature would not have led to new and stronger results for our research aims.

Limitations of the scenario method itself

The following limitations are inherent to the scenario method itself and should be considered also in health planning [9, 10, 48, 87, 88]. Firstly, creating explorative scenarios can be time-consuming and therefore cost-intensive, in particular because they tie up personnel resources [21]. However, the processes are scalable; a small group might be able to develop consistent scenarios, for example [58, 61]. Secondly, the quality of the scenarios depends greatly on the imagination, information basis and competency of the experts taking part [21, 102]. Thus, there is a potential risk of biased scenarios if experts are inclined to give preference to well-known developments and to reject any that seem too unorthodox; or in other words: opinion leaders who try to dominate a scenario group are counterproductive [64]. As a result, the selection of the experts is of considerable importance and should depend on the criteria applied to consensus processes and Delphi methods [64, 102]. But, as shown in the included scenario projects, it is possible to get usable conclusions with ‘ordinary’ persons acting as experts [29, 30, 65]. Thirdly, if the scenario developing process is not only narrative like some included projects [55, 58, 72], but also includes quantitative aspects by means of calculations [64, 81], the mathematical processes used to generate the scenarios may be plausible for scenario-natives but incomprehensible for non-experts [21]. Thus, unlike other methods (e.g. Monte-Carlo simulation), there is no standard gateway for the researcher to use common tools or packages. Fourthly, the selection of the key factors is the crucial point of each scenario analysis [9]. In principle, a systematic search for each selected key factor is desirable to generate evidence, but available resources would hardly allow this. Fifthly, scenarios are not forecasting the future as each step always entails subjective assessments and evaluations of abstract and complex facts. So, another threat might be the overestimation of the exactness of explorative scenarios. The sixth and final point seems to be critical for the acceptance of the scenario method as a scientific tool particularly in the field of public health. Even though the reporting of the more common Delphi technique has room for further improvement [5, 8], it seems that the variability in using and reporting the scenario method is much higher [10, 15, 87, 103]. The authors strongly believe that there is a need to improve the reporting of scenario projects, along the lines of a GRAMMS-like guideline which is used for mixed methods studies and recommended by the equator-network (www.equator-network.org) [104]. Proposed indicators as a result from this review are listed in Table 2. Only if the transparency required to reproduce the underlying evidence exists, will the scenario method be a useful tool for future health care planning and strategic public health decision-making [103].
Table 2

Proposed key methodological criteria to report in scenario projects

Criteria

Aim of the scenario project

 • Does the word “scenario project (planning, approach…)” appear in the title along with the topic of primary interest?

 • Is the topic of interest clearly described?

 • What are the proposed implications?

 • Are the target groups and/or stakeholders specified?

 • Is there a clear time horizon?

Framework of the scenario project

 • Are the preconditions and presuppositions well described?

 • Is the process of developing the areas of influence, key factors, and future projections adequately described?

 • Who is involved (description of scenario development team and participants/experts)?

 • Is the background of participants/experts clear?

 • How will participants/experts be selected or excluded?

Methodological approach of the scenario project

 • Is the specific scenario technique used (e.g. only narrative, consistence analysis, cross-impact analysis) well described?

 • If any, is the mathematical approach well described?

 • How is the mathematical approach transformed/implemented in software (if applicable)?

 • Is there any combination with other methods like the Delphi technique?

 • Is the presentation of the scenario development process adequate?

 • Are the scenarios presented in a sound manner (to the specified target groups/stakeholders)?

Impact of the scenario project

 • Are there any recommendations for different target groups/stakeholders?

 • What are the next steps after the scenario project?

Conclusions

In recent years, more scenario projects relating to health and health care have been published in scientific journals than ever before. This review provides a comprehensive overview of the use of the scenario method in the field of public health and health policy research. The scenario method has been classified as most helpful for strategic issues by several authors of the projects. However, there is no ‘one” scenario method. There is a wide spectrum of strategic aims covered by heterogeneous variants of the scenario method.

To establish the scientific use of scenario methods, uniform qualitative reporting would be useful, based on the GRAMMS criteria, for example [104].

Declarations

Acknowledgements

We thank Karla Bergerhoff, MD, Cochrane Metabolic and Endocrine Disorders Group (CMED), Medical Faculty, Heinrich-Heine-University Düsseldorf for her helpful support during research. We also thank Nancy Myers, BA, for her language support.

Funding

Research reported in this publication was supported only by the institutions of the authors.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Institute of General Practice, Medical Faculty, Heinrich-Heine-University Düsseldorf
(2)
Institute of General Practice and Family Medicine, Faculty of Health, Witten/Herdecke University
(3)
Institute of Integrative Medicine, Faculty of Health, Witten/Herdecke University
(4)
Institute of Health Economics and Clinical Epidemiology, University Hospital Cologne

References

  1. Yost J, Dobbins M, Traynor R, DeCorby K, Workentine S, Greco L. Tools to support evidence-informed public health decision making. BMC Public Health. 2014;14:728.PubMedPubMed CentralView ArticleGoogle Scholar
  2. Devenish G, Pollard C, Kerr D. The Delphi Process for Public Health Policy Development: Five things you need to know. In. Western Australia: Curtin University; 2012:14.Google Scholar
  3. Ferri CP, Prince M, Brayne C, Brodaty H, Fratiglioni L, Ganguli M, et al. Global prevalence of dementia: a Delphi consensus study. Lancet. 2005;366(9503):2112–7.PubMedPubMed CentralView ArticleGoogle Scholar
  4. Gensichen J, Vollmar HC, Sönnichsen A, Waldmann UM, Sandars J. E-learning for education in primary healthcare - turning the hype into reality: A Delphi study. Eur J Gen Pract. 2009;15(1):11–4.PubMedView ArticleGoogle Scholar
  5. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9.PubMedView ArticleGoogle Scholar
  6. Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, et al. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health Med. 2003;25(4):325–35.PubMedView ArticleGoogle Scholar
  7. Cuhls K. From forecasting to foresight processes—new participative foresight activities in Germany. Journal of Forecasting. 2003;22(2–3):93–111.View ArticleGoogle Scholar
  8. Landeta J. Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change. 2006;73(5):467–82.View ArticleGoogle Scholar
  9. Glenn JC, The Futures Group International. Scenarios. In: Glenn JC, Gordon TJ, editors. Futures Research Methodology Version 30. The Millenium Project; 2011. p. 52.Google Scholar
  10. Kosow H, Gaßner R. Methods of future and scenario analysis: overview, assessment, and selection criteria, vol. Studies 39. Bonn: Deutsches Institut für Entwicklungspolitik/German Development Institute; 2008.Google Scholar
  11. Schwartz P. The Art of the Long View. Paths to Strategic Insight for Yourself and your Company. New York, London, Toronto, Sydney, Auckland: Doubleday; 1996.Google Scholar
  12. Wright G, Van der Heijden K, Burt G, Bradfield R, Cairns G. Scenario planning interventions in organizations: an analysis of the causes of success and failure. Futures. 2008;40(3):218–36.View ArticleGoogle Scholar
  13. Bierbooms JJ, Bongers IM, van Oers HA. A scenario analysis of the future residential requirements for people with mental health problems in Eindhoven. BMC Med Inform Decis Mak. 2011;11:1.PubMedPubMed CentralView ArticleGoogle Scholar
  14. Wilkinson A, Kupers R, Mangalagiu D. How plausibility-based scenario practices are grappling with complexity to appreciate and address 21st century challenges. Technological Forecasting and Social Change. 2013;80(4):699–710.View ArticleGoogle Scholar
  15. Postma TJBM, Liebl F. How to improve scenario analysis as a strategic management tool? Technological Forecasting and Social Change. 2005;72(2):161–73.View ArticleGoogle Scholar
  16. Gregório J, Cavaco A, Velez Lapão L. A scenario-planning approach to human resources for health: the case of community pharmacists in Portugal. Human Resources for Health. 2014;12:58.PubMedPubMed CentralView ArticleGoogle Scholar
  17. Moore S, Mawji A, Shiell A, Noseworthy T. Public health preparedness: a systems-level approach. J Epidemiol Community Health. 2007;61(4):282–6.PubMedPubMed CentralView ArticleGoogle Scholar
  18. Gnatzy T, Moser R. Scenario development for an evolving health insurance industry in rural India: INPUT for business model innovation. Technological Forecasting and Social Change. 2012;79:688–99.View ArticleGoogle Scholar
  19. Anderson LA, McConnell SR. The healthy brain and our aging population: translating science to public health practice. Alzheimer’s and Dementia: the Journal of the Alzheimer’s Association. 2007;3(2 Suppl):S1–2.View ArticleGoogle Scholar
  20. Varum CA, Melo C. Directions in scenario planning literature – A review of the past decades. Futures. 2010;42(4):355–69.View ArticleGoogle Scholar
  21. Vollmar HC, Goluchowicz K, Beckert B, Donitz E, Bartholomeyczik S, Ostermann T, et al. Health care for people with dementia in 2030 - results of a multidisciplinary scenario process. Health Policy. 2014;114(2–3):254–62.PubMedView ArticleGoogle Scholar
  22. Bezold C. The future of patient-centered care: scenarios, visions, and audacious goals. J Altern Complement Med. 2005;11 Suppl 1:S77–84.PubMedGoogle Scholar
  23. Buchan J, Seccombe I. Using scenarios to assess the future supply of NHS nursing staff in England. Human Resources for Health. 2012;10(1):16.PubMedPubMed CentralView ArticleGoogle Scholar
  24. Neiner JA, Howze EH, Greaney ML. Using scenario planning in public health: anticipating alternative futures. Health Promot Pract. 2004;5(1):69–79.PubMedView ArticleGoogle Scholar
  25. Butler-Jones D, Trumble Waddell J. Children and Physical Activity Scenarios Project: Evidence-Based Visions of the Future. In. Team PHAaS, editors. Ottawa: Public Health Agency of Canada; 2011.Google Scholar
  26. Armstrong R, Hal lBJ, Doyle J, Waters E. Cochrane Update. ‘Scoping the scope’ of a cochrane review. J Public Health. 2011;33(1):147–50.View ArticleGoogle Scholar
  27. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19–32.View ArticleGoogle Scholar
  28. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.PubMedPubMed CentralView ArticleGoogle Scholar
  29. Karger CR. Citizen scenarios for the future of personalized medicine: a participatory scenario process in Germany. The International Journal of Interdisciplinary Social and Community Studies. 2013;7(2):1–16.View ArticleGoogle Scholar
  30. Nguyen V, Nguyen-Viet H, Pham-Duc P, Wiese M. Scenario planning for community development in Vietnam: a new tool for integrated health approaches? Glob Health Action. 2014;7:24482.PubMedGoogle Scholar
  31. Ashton J. Future Scenarios for Public Health in Europe. Eur J Pub Health. 1992;2(1):54–9.View ArticleGoogle Scholar
  32. Norgaard LS, Colberg L, Niemann MR. The role of the Danish community pharmacist: perceptions and future scenarios. Pharm World Sci. 2001;23(4):159–64.PubMedView ArticleGoogle Scholar
  33. Olshansky SJ, Goldman DP, Zheng Y, Rowe JW. Aging in America in the Twenty-first Century: Demographic Forecasts from the MacArthur Foundation Research Network on an Aging Society. Milbank Quarterly. 2009;87(4):842–62.PubMedPubMed CentralView ArticleGoogle Scholar
  34. Kalogirou S, Foley R. Health, Place and Hanly: Modelling accessibility to hospitals in Ireland. Irish Geography. 2006;39:52–68.View ArticleGoogle Scholar
  35. Comas-Herrera A, Northey S, Wittenberg R, Knapp M, Bhattacharyya S, Burns A. Future costs of dementia-related long-term care: exploring future scenarios. Int Psychogeriatr. 2011;23(1):20–30.PubMedView ArticleGoogle Scholar
  36. Ryynanen OP, Myllykangas M, Kinnunen J, Halonen P, Takala J. Prioritization attitudes among doctors and nurses examined by a scenario method. Int J Technol Assess Health Care. 2000;16(1):92–9.PubMedView ArticleGoogle Scholar
  37. Boakes E. Using foresight in safe nursing care. J Nurs Manag. 2009;17(2):212–7.PubMedView ArticleGoogle Scholar
  38. Persky S, Kaphingst KA, Condit CM, McBride CM. Assessing hypothetical scenario methodology in genetic susceptibility testing analog studies: a quantitative review. Genet Med. 2007;9(11):727–38.PubMedView ArticleGoogle Scholar
  39. Kossovsky N, Brandegee B. Why compliance is not good enough. J Biomed Mater Res. 1999;48(1):1–4.PubMedView ArticleGoogle Scholar
  40. Pesut DJ. Scenarios: Stories about the future. Nursing Outlook. 1998;46:55.View ArticleGoogle Scholar
  41. Edgar B, Abouzeedan A, Hedner T, Maack K, Lundqvist M. Using scenario planning in regional development context: the challenges and opportunities. World Journal of Science, Technology and Sustainable Development. 2013;10(2):103–22.View ArticleGoogle Scholar
  42. Hilderink H. Scenario development methodology. In: McKee M, Ricciardi W, Zeegers Paget D, editors. 6th European Public Health Conference Eur J Public Health, vol. 23. Brussels: Eur J Public Health; 2013. p. 15.Google Scholar
  43. Reed MS, Kenter J, Bonn A, Broad K, Burt TP, Fazey IR, et al. Participatory scenario development for environmental management: A methodological framework illustrated with experience from the UK uplands. J Environ Manage. 2013;128:345–62.PubMedView ArticleGoogle Scholar
  44. Batenburg R. The Dutch model of health human resources planning and the new challenges of an integrative European perspective. In: McKee M, Ricciardi W, Zeegers Paget D, editors. 6th European Public Health Conference Eur J Public Health, vol. 23. Brussels: Eur J Public Health; 2013. p. 82.Google Scholar
  45. Hicks C. Sustainable and healthy lifestyles towards 2050; lessons learned from SPREAD 2050. In: McKee M, Ricciardi W, Zeegers Paget D, editors. 6th European Public Health Conference Eur J Public Health, vol. 23. Brussels: Eur J Public Health; 2013. p. 92.Google Scholar
  46. Vollmar HC, Buscher I, Goluchowicz K, Donitz E, Wilm S, Beckert B, et al. Health care of people with dementia in germany in 2030 - A multidisciplinary scenario approach (SCE-DEM). In: Alzheimer’s Association International Conference: 2011. Paris: Alzheimer’s and Dementia; 2011. p. S447–8.Google Scholar
  47. Braun A, Rijkers-Defrasne S, Pechmann A, Amanatidou E, Psarra F. Future of public health systems in Europe. EFMN Issue Analysis 2008 Final Report. In. Düsseldorf: VDI Technologiezentrum; 2008: 113.Google Scholar
  48. Schreuder RF. Health scenarios and policy making: Lessons from the Netherlands. Futures. 1995;27(9–10):953–8.View ArticleGoogle Scholar
  49. Stephan JC. The scenario method in health planning. Cahiers De Sociologie et De Demographie Medicales. 1983;23:139–48.PubMedGoogle Scholar
  50. Buchan J, Seccombe I. The end of growth? Analysing NHS nurse staffing. J Adv Nurs. 2013;69(9):2123–30.PubMedView ArticleGoogle Scholar
  51. Schwarz JO. Assessing future disorders in organizations: implications for diagnosing and treating schizophrenic, depressed or paranoid organizations. Foresight. 2007;9(2):15–26.View ArticleGoogle Scholar
  52. Schnaars SP. How to develop and use scenarios. Longe Range Planning. 1987;20(1):105–14.View ArticleGoogle Scholar
  53. Venable JM, Ma QL, Ginter PM, Duncan WJ. The use of scenario analysis in local public health departments: alternative futures for strategic planning. Public Health Rep. 1993;108(6):701–10.PubMedPubMed CentralGoogle Scholar
  54. Bezold C, Peck J. Drug regulation 2056. Food Drug Law J. 2005;60(2):127–36.PubMedGoogle Scholar
  55. Eberl I, Schnepp W. Family health nursing in Germany: a process of consensus finding as basis. Pflege. 2006;19(4):234–43.PubMedView ArticleGoogle Scholar
  56. Islei G, Lockett G, Naudé P. Judgemental modelling as an aid to scenario planning and analysis. Omega, Int J Mgmt Sci. 1999;27(1):61–73.View ArticleGoogle Scholar
  57. Meristo T, Tuohimaa H, Leppimaki S, Laitinen J. Alternative futures of proactive tools for a citizen’s own wellbeing. Stud Health Technol Inform. 2009;146:144–8.PubMedGoogle Scholar
  58. Rydstrom C, Tornberg S. Cervical cancer incidence and mortality in the best and worst of worlds. Scand J Public Health. 2006;34(3):295–303.PubMedView ArticleGoogle Scholar
  59. Sager B. Scenarios on the future of biotechnology. Technological Forecasting and Social Change. 2001;68:109–29.View ArticleGoogle Scholar
  60. Leufkens H, Hekster Y, Hudson S. Scenario analysis of the future of clinical pharmacy. Pharm World Sci. 1997;19(4):182–5.PubMedView ArticleGoogle Scholar
  61. Bezold C. Five futures. Healthc Forum J. 1992;35(3):28–42.PubMedGoogle Scholar
  62. Enzmann DR, Beauchamp Jr NJ, Norbash A. Scenario planning. J Am Coll Radiol. 2011;8(3):175–9.PubMedView ArticleGoogle Scholar
  63. Ma S, Seid M. Using foresight methods to anticipate future threats: the case of disease management. Health Care Manage Rev. 2006;31(4):270–9.PubMedView ArticleGoogle Scholar
  64. Retel VP, Joore MA, Linn SC, Rutgers EJ, van Harten WH. Scenario drafting to anticipate future developments in technology assessment. BMC Research Notes. 2012;5:442.PubMedPubMed CentralView ArticleGoogle Scholar
  65. Niewoehner J, Wiedemann P, Karger C, Schicktanz S, Tannert C. Participatory prognostics in Germany - developing citizen scenarios for the relationship between biomedicine and the economy in. Technological Forecasting and Social Change 2005. 2014;72(2):195–211.View ArticleGoogle Scholar
  66. Wiek A, Gasser L, Siegrist M. Systemic scenarios of nanotechnology: Sustainable governance of emerging technologies. Futures. 2009;41(5):284–300.View ArticleGoogle Scholar
  67. van Beeck EF, Mackenbach JP. Future health scenarios as a tool in the surveillance of unintentional injuries. Health Policy. 1997;40(1):13–28.PubMedView ArticleGoogle Scholar
  68. van Beeck EF, Mackenbach JP, van Oortmarssen GJ, Barendregt JJM, Habbema JDF, van der Maas PJ. Scenarios for the future development of accident mortality in The Netherlands. Health Policy. 1989;11:1–17.View ArticleGoogle Scholar
  69. Bijl R. Delphi in an future scenario study on mental health and mental health care. Futures. 1992;24(3):232–50Google Scholar
  70. Bijl RV, Ketting E. Future scenario’s Dementia 1990-2010--various major points from the scenario report ‘Care for mental health in the future’. Tijdschr Gerontol Geriatr. 1991;22(3):110–4. discussion 114–116.PubMedGoogle Scholar
  71. van Lente H, Willemse J, Vorstman C, Modder JF. Scenario planning as policy instrument: Four scenarios for biotechnology in Europe. Innovation: Management, Policy and Practice. 2003;5:4–14.View ArticleGoogle Scholar
  72. Nielsen GA. Preparing for change: strategic foresight scenarios. Radiol Manage. 1996;18(2):43–7.PubMedGoogle Scholar
  73. Schreuder RF. Scenarios for health planning and management: The dutch experience. Int J Health Plann Manage. 1988;3(2):73–87.PubMedView ArticleGoogle Scholar
  74. Schaapveld K, Cleton FJ. Cancer in The Netherlands. From scenarios to health policy. European Journal of Cancer and Clinical Oncology. 1989;25(4):767–71.PubMedView ArticleGoogle Scholar
  75. Suk JE, Semenza JC. Future infectious disease threats to Europe. Am J Public Health. 2011;101(11):2068–79.PubMedPubMed CentralView ArticleGoogle Scholar
  76. Zentner RD. Scenarios: a planning tool for health care organizations. Hospital & Health Services Administration. 1991;36(2):211–22.Google Scholar
  77. Rhea M, Bettles C. Four futures for dietetics workforce supply and demand: 2012–2022 scenarios. Journal of the Academy of Nutrition and Dietetics. 2012;112(3 Suppl):S25–34.PubMedView ArticleGoogle Scholar
  78. Ling T, Hadridge P. The Madingley scenarios for the future context of healthcare: understanding today by imagining tomorrow. Br Homeopath J. 2000;89 Suppl 1:S3-7. discussion S8-9.Google Scholar
  79. Leufkens H, Haaijer-Ruskamp F, Bakker A, Dukes G. Scenario analysis of the future of medicines. BMJ. 1994;309(6962):1137–40.PubMedPubMed CentralView ArticleGoogle Scholar
  80. Harmsen H, Jensen BB, Sonnie AM. Scenarios for the food industry in 2010. Agrarwirtschaft. 2001;50(6):314–22.Google Scholar
  81. Becker HA. Simulating the Aging of the Netherlands. Simulation and Games. 1988;19:186–209.View ArticleGoogle Scholar
  82. Awasthi S, Beardmore J, Clark J, Hadridge P, Madani H, Marusic A, et al. Five futures for academic medicine. PLoS Med. 2005;2(7):e207.PubMedPubMed CentralView ArticleGoogle Scholar
  83. Clark J. Five futures for academic medicine: the ICRAM scenarios. BMJ. 2005;331(7508):101–4.PubMedPubMed CentralView ArticleGoogle Scholar
  84. Carlsen H, Dreborg KH, W-S P. Tailor-made scenario planning for local adaptation to climate change. Mitigation and Adaptation Strategies for Global Change. 2013;18(8):1239–55.View ArticleGoogle Scholar
  85. Folb PI. Scenario analysis of the future of medicines. Lancet. 1993;342(8866):294.PubMedView ArticleGoogle Scholar
  86. Pillkahn U. Trends und Szenarien als Werkzeuge zur Strategieentwicklung. Wie Sie die unternehmerische und gesellschaftliche Zukunft planen und gestalten. Erlangen: Publicis Kommunikations Agentur GmbH, GWA; 2007.Google Scholar
  87. Amer M, Daim TU, Jetter A. A review of scenario planning. Futures. 2013;46:23–40.View ArticleGoogle Scholar
  88. Lexa FJ, Chan S. Scenario analysis and strategic planning: practical applications for radiology practices. J Am Coll Radiol. 2010;7(5):369–73.PubMedView ArticleGoogle Scholar
  89. Bishop P, Hines A, Collins T. The current state of scenario development: an overview of techniques. Foresight. 2007;9(1):5–25.View ArticleGoogle Scholar
  90. Longley M, Warner M. Future Health Scenarios - Strategic Iusses for the British Health Service. Long Range Planning. 1995;28(4):22–32.View ArticleGoogle Scholar
  91. Butland B, Jebb S, Kopelman P, McPherson K, Thomas S, Mardell J, Parry V: Tackling obesities: future choices. Project report. vol. 2: Government Office for Science; 2007: 155. https://www.gov.uk/government/publications/reducing-obesity-futurechoices
  92. Amara R, Bodenhorn K, Cain M, Carlson R, Chambers J, Cypress D, et al. Health and health care 2010: the forecast, the challenge, vol. 2. San Francisco: Jossey-Bass; 2003.Google Scholar
  93. Scenario Management International AG (ScMI). Germany’s health care system in the year 2020. In. Hannover; 2011: 51.Google Scholar
  94. Karger CR, Robertz N, Hüsing B. Personalisierte Medizin im Gesundheitssystem der Zukunft. In. Edited by Forschungszentrum J. Jülich: Forschungszentrum Jülich; 2009: 245.Google Scholar
  95. Vollmar HC, editor. Leben mit Demenz im Jahr 2030: Ein interdisziplinäres Szenario-Projekt zur Zukunftsgestaltung. Weinheim und Basel: Beltz Juventa; 2014.Google Scholar
  96. GreyNet. Pisa Declaration on Policy Development for Grey Literature Resources.; 2014: 2.Google Scholar
  97. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;2:MR000010.PubMedGoogle Scholar
  98. Unterkalmsteiner M, Gorschek T, Islam AKMM, Cheng CK, Permadi RB, Feldt R. Evaluation and Measurement of Software Process Improvement — A Systematic Literature Review. Software Engineering, IEEE Transactions on. 2012;38(2):398–424.View ArticleGoogle Scholar
  99. Butler-Jones D, Trumble Waddell J. Public Health Scenario: Canada without the Polio Vaccine. In. Edited by Team PHAaS. Ottawa: Public Health Agency of Canada; 2012.Google Scholar
  100. Greeuw SCH, van Asselt MBA, Grosskurth J, Storms CAMH, Rijkens-Klomp N, Rothman DS, et al. Cloudy crystal balls. An assessment of recent European and global scenario studies and models. In: European Environment Agency (EEA), editor. Environmental issues series, vol. 17. Copenhagen: International Centre for Integrative Studies (ICIS); 2000.Google Scholar
  101. Manning J. Health,humanity and justice: Emerging technologies and health policy in the 21st Century. In. London: 2020health.org; 2010.Google Scholar
  102. Baker J, Lovell K, Harris N. How expert are the experts? An exploration of the concept of “expert" within Delphi panel techniques. Nurse Res. 2006;14(1):59–70.PubMedView ArticleGoogle Scholar
  103. Gerhold L, Holtmannspötter D, Neuhaus C, Schüll E, Schulz-Montag B, Steinmüller K, et al., editors. Standards und Gütekriterien der Zukunftsforschung. Ein Handbuch für Wissenschaft und Praxis. Wiesbaden: Springer VS; 2015.Google Scholar
  104. O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy. 2008;13(2):92–8.PubMedView ArticleGoogle Scholar

Copyright

© Vollmar et al. 2015

Advertisement