Skip to main content
  • Research article
  • Open access
  • Published:

Adapting the nominal group technique for priority setting of evidence-practice gaps in implementation science



There are a variety of methods for priority setting in health research but few studies have addressed how to prioritise the gaps that exist between research evidence and clinical practice. This study aimed to build a suite of robust, evidence based techniques and tools for use in implementation science projects. We applied the priority setting methodology in lung cancer care as an example.


We reviewed existing techniques and tools for priority setting in health research and the criteria used to prioritise items. An expert interdisciplinary consensus group comprised of health service, cancer and nursing researchers iteratively reviewed and adapted the techniques and tools. We tested these on evidence-practice gaps identified for lung cancer. The tools were pilot tested and finalised. A brief process evaluation was conducted.


We based our priority setting on the Nominal Group Technique (NGT). The adapted tools included a matrix for individuals to privately rate priority gaps; the same matrix was used for group discussion and reaching consensus. An investment exercise was used to validate allocation of priorities across the gaps. We describe the NGT process, criteria and tool adaptations and process evaluation results.


The modified NGT process, criteria and tools contribute to building a suite of methods that can be applied in prioritising evidence-practice gaps. These methods could be adapted for other health settings within the broader context of implementation science projects.

Peer Review reports


Setting priorities in health services delivery and research is highly relevant in an era of scarce resources. In many settings, health researchers and clinicians participate in collaborative efforts to determine how best to implement evidence into routine clinical practice. Priority setting offers an important opportunity for collaborators to determine what evidence is relevant to implement, how best to do this, and where the greatest gains can be made in changing clinical practice [1]. There is a need to determine the most useful methods for eliciting priorities. Methods for priority setting in service delivery are already well developed [2, 3] but little attention has focused on how these methods could be usefully adapted for implementation science projects.

Few studies have addressed methods that specifically prioritise existing gaps between research evidence and clinical practice [1]. To date, only one cluster randomised controlled trial (RCT) in setting priorities for healthcare improvement in community settings has been published [4, 5]. The patient involvement intervention for chronic diseases was based on the Nominal Group Technique (NGT) and trialled in community workshops to set priorities with participants (professionals alone in the control group or professional and patients in the intervention group) [5]. The study found that the NGT intervention was effective and patient involvement significantly shifted priorities towards quality of care factors.

The NGT is an interpretive approach that engages stakeholders in interactive discussion to generate priorities [6, 7]. NGT has demonstrated validity, emphasizes considering all participants’ views equally and enables consensus on highly complex issues [8]. Consensus methods such as the NGT seek to overcome group or committee decision making that can be dominated by individuals or coalitions who have a vested interest in a specific outcome [9]. NGT is frequently used for problem identification; in helping to generate appropriate research questions; for development of solutions; and establishing priorities for action [8]. In cancer research, the NGT and modified Delphi techniques have been used to determine priorities in pancreatic cancer [10], haematological cancers [11, 12], colorectal cancer [13], colorectal cancer care coordination [14], psycho-oncology [15], cancer genomics [16] and with consumers in UK treatment centres [17]. To our knowledge, no studies have focused on methods for setting priorities about identified evidence-practice gaps in lung cancer care. Furthermore, there are few guides or resources specific to priority setting in ‘evidence to practice’ research [2, 18].

Our group had separately developed a set of seven evidence-practice gaps in lung cancer on the basis of a scoping review of the literature [19] and examining local data sources such as the Clinical Cancer Registry of New South Wales (NSW), Australia. Our primary aim was to test the relevance of these identified gaps with lung cancer health professionals using existing evidence-based approaches to priority setting (Additional file 1). Our second aim was to systematically develop a suite of robust, evidence based techniques, criteria and tools for use in implementation science projects. This paper describes the outcomes of the second aim. We show how the selected technique is relevant to implementation science, as most theoretical models and frameworks include pre-implementation engagement of stakeholders as a fundamental step in change processes [2022]. The resulting tools and process were subsequently used in three focus groups with multidisciplinary teams (n = 42 participants) to fulfil our primary aim. Data outcomes from the focus groups are reported in a separate paper [23].


Selecting priority setting techniques

We conducted a literature search of journal articles and reports that specifically described priority-setting techniques in health research. We conducted three search strategies in Medline, PsychInfo and PubMed databases to:

  1. a)

    identify relevant health priority setting approaches or techniques that had been used in focus or small group settings;

  2. b)

    identify original research studies conducted in oncology settings, where oncology health professionals and/or consumers were the target audience for priority setting; and,

  3. c)

    locate relevant documents in the grey literature (i.e. reports and publications that may not be peer-reviewed) and web-based sources about group decision making tools used in health. This last search included checking the Cochrane Library (including the publication list of the Cochrane Agenda and Priority Setting Methods Group).

For all three search strategies, the inclusion criteria were those articles or sources published between January 2008 and December 2012 and published in English. The exclusion criteria were any studies that had a primary aim of conducting a comparative benefit or cost analyses of health services; we also excluded commentaries or letters to editors.

Selecting criteria

The team conducted a literature search to identify and review existing criteria for rating evidence-practice gaps. We used the same databases and inclusion and exclusion criteria as described in the previous section. There are many and varied criteria used to assess priorities, which are generally categorised into three dimensions: public health benefit, feasibility and cost [18]. We focused on the first two dimensions and criteria options were collated from a range of sources [24, 25]. Cost criteria were not included as we had already excluded comparative benefit or cost analyses of health services in the previous section.

Consensus approach to selecting relevant technique and criteria

A consensus process was undertaken to select: a) the most relevant technique; and, b) criteria to use in the rating tools. We formed an expert interdisciplinary consensus group comprising of cancer clinician-researchers and academic experts from multiple disciplines including nursing, psychology, health services research, epidemiology and clinical oncology. Results from the literature reviews were presented. To select the most relevant technique, the consensus group considered the relative advantages, disadvantages and/or the applicability of each one. They considered how optimal and practical each technique was in terms of equity of participation encouraged, time required to enact and the group size, stakeholder mix of disciplines and level of expertise required. To select the criteria for use in the rating tools, the consensus group focused on whether there was a clear definition for each criteria and how easily participants would be able to rate each gap against the criteria. The group discussed the criteria options until consensus was reached.

Pilot testing of the process and tools

A pilot was conducted with key stakeholders (n = 7, including two medical oncologists and 5 health service researchers) to test the tools and the priority setting process. Minor adjustments were made to the content and formatting of the tools based on participant feedback. Participants indicated a maximum time of two hours should be allocated for the priority setting process. The pilot test confirmed that the priority setting technique and criteria were acceptable and feasible within this time-frame.

Process and consensus evaluation

We asked the three lung cancer multidisciplinary teams (n = 42) to complete a brief 8-item survey to evaluate the process and provide feedback about whether consensus had been achieved.


The results are described in the following sections: selecting the technique; selecting the criteria; the resulting tools and the modified NGT process.

Selecting the technique: outcomes of the literature search

A total of 155 journal articles were identified from the first two search strategies; the abstract for each article was checked for relevance. Twenty-two articles relevant to the review were identified, which included 14 describing frameworks or techniques and eight that were original research articles in oncology. In the third strategy, five reports and two e-bulletins/newsletters from website sources were judged as relevant (see Fig. 1).

Fig. 1
figure 1

Literature review strategy for priority setting techniques

The following techniques were identified: the Nominal Group Technique (NGT) [2]; the Delphi technique [10]; the Six-Steps of Listening for Direction [18]; the Combined Matrix Approach (3D CAM) [26] and Dotmocracy [27]. Through consensus, it was agreed that the NGT would be most relevant to modify for the following reasons. It can engage clinicians and researchers in face-to-face structured priority setting groups. This approach allows information to be collected from health professionals working in ‘front line’ clinical areas [8]. NGT is time efficient and cost effective, as priorities can be elicited in a short time period (e.g. 2 h) and requires few resources, which appeals to busy clinicians and resource-limited research budgets [28]. The technique requires no preparation on the part of participants and yet their expertise is of fundamental importance in considering the issues for prioritization. Another advantage is that the results generated can be immediately shared with the group. This provides ample opportunities for further reflection and clarification before the priority setting group session concludes [28]. This is particularly relevant in settings where membership of clinical teams frequently changes.

Selecting criteria: outcomes of the literature review and consensus

Using this same strategy described above, we identified seven journal articles and reports that listed relevant criteria options and three guides to consensus processes for selecting criteria [2, 18, 22]. Criteria options considered by the investigator team included those listed in: a) the Institute of Medicine (IOM) quality criteria (safety, effectiveness, patient-centered, timely, efficiency and equity) [29]; b) the 3D CAM criteria (disease burden, determinants of health, present level of knowledge, effectiveness and resource flows) [26], and c) the behavioural matrix in the Precede-Proceed model criteria (importance and changeable) [30, 31].

Through the consensus process, four criteria were selected against which the gaps could be rated. These were: 1) relevance to the local setting; 2) magnitude of the issue; 3) burden of suffering; and 4); amenable to change. Criteria 1-3 (relevance, magnitude and burden) were grouped under the heading of ‘significance’, so that results could later be collated and summarised more efficiently.

The resulting tools

Tool 1: the priority setting matrix

This tool is used to elicit individual responses from participants, rating the gaps against the criteria using a matrix scoring sheet (Table 1). We adapted this tool from the 3D CAM [26]. The priority setting matrix places the evidence practice gaps in the left hand column and the criteria on the right. The matrix tool enables participants to privately rate each gap on a Likert scale ranging from 5 (most) to 1 (least) according to the four criteria. Likert scales have been used successfully in the past to rate priority items [32]. It should be noted that evidence practice gaps statements should be independent from each other and without multiple clauses.

Table 1 Evidence-practice gaps priority setting matrix

The same matrix tool can be used to elicit group responses for the purpose of discussing each gap and reaching consensus, and subsequently giving a rating to each one [23]. A ‘Dotmocracy’ approach is used to highlight common themes and reach consensus [27]. Gold and silver stars are allocated by each participant to reflect their highest priorities. Through facilitated discussion, the group is asked to discuss their individual ratings in a small group with each person nominating their top two (first ‘gold’ and second ‘silver’) gaps per criteria. This component is the small group trigger for discussion between participants of justifications for their ratings. Using the Dotmocracy approach can help small groups to determine if participant priorities align and to respond as appropriate until consensus is achieved.

Tool 2: validating priority selection through an investment exercise

Each participant receives $100 fake ‘dollars’ to spend across the gaps. This seeks to allow participants to make a global decision on resource allocation and also confirms whether the gaps are reflected in how participants would most like to invest in each gap. This acts as a cross check with the priorities selected in the matrix.

The modified NGT process

Table 2 shows the original NGT steps and the modified NGT steps and a brief description for each.

Table 2 Comparison of original NGT steps and the modified process and brief descriptions

The modified NGT process developed for this project was subsequently used in focus groups and is described here as an exemplar. (Please note that the results of the focus groups are described in a separate paper [23]):

  • Step 1: Describe identified evidence practice gaps

    In this step, an investigator team member (facilitator) gave a presentation about the approach used to review evidence about gaps in lung cancer care, read out a brief summary for each gaps and answered any participant questions.

  • Step 2: Present local data/information about gaps

    This step included a presentation about national, jurisdictional and local data gathered to support the gaps. Where available, this was tailored to the local site to include analysis of local health district or hospital data collected in registries or administrative databases. This step was an adaptation of Lomas’ second step in the ‘listening model’ [6].

  • Step 3: Elicit feedback and include additional gaps identified by stakeholders

    Participants were asked to provide feedback about the seven gaps and whether they were relevant in the local hospital setting and surrounding community. We provided an opportunity for participants to nominate any additional gaps and this was considered as elicitation and inclusion of ‘practice-based evidence’. Additional gaps were only included if consensus was reached within the group that the gap should be added.

  • Step 4: Individuals vote privately to prioritise gaps, using moderator-created criteria

    The criteria were described by the facilitator and participants had an opportunity to seek any clarification before privately rating each gap. Participants were then asked to rate each gap for across the criteria listed above without sharing these with others. This was considered an important step as this private rating of items allowed participants to make their own considered judgements prior to reaching group consensus. This step also ensured that every member of the group had read and considered each gap in detail.

  • Step 5: Each participant selects the two most important gaps from the prioritised list.

    At the conclusion of step 4, each participant was asked to select the two most important gaps from on the matrix tool for the two criteria of ‘significance’ and ‘amenable to change’, according to their highest scores. This step replicates the original NGT step 5 but is modified by selecting only two most important items instead of five.

  • Step 6: Focus group participants discuss ratings and moderator uses matrix tool as a tally

    In this step, participants broke into small groups (maximum of seven participants in each) and shared their ratings for each gap. We pre-assigned participants into small groups that consisted of a mix of professional groups. Each group was asked to nominate a spokesperson to report back to the larger group and provide feedback about the choices made. To achieve equity within small groups, each person spoke in turn about their top two gaps and provided any clarification for their choices. All responses were recorded by a group facilitator on an A4/US letter-sized printed sheet of the matrix tool in ‘Dotmocracy’ style [27], assigning a gold and silver star to the highest priority gaps. Priorities were not weighted across the chosen top two gaps.

  • Step 7: Whole group consensus

    At the conclusion of discussion, small groups reformed back into a larger group, compared ratings for differences and anomalies and resolved these to reach group consensus. The moderator led discussion about the gaps against the criteria of significance and amenability to change.

  • Step 8. Investment exercise

    In the final step, participants were asked to spend $100 fake dollars across each gap, configured as five $20 notes. Eight A3/US Ledger-sized envelopes were placed on a table (one for each gap, including the eighth local gap) and participants were asked to spend their money in any way they might choose. Participants were able to configure the dollars in any way so long as all $100 was allocated (e.g. all $100 in one envelope or any configuration of $20 allocations across the envelopes). At the conclusion of the task, the investigator team tallied the dollars in each envelope and then reported back to the group about the proportions (total dollar values and per cent) spent on each gap.

Participants’ feedback in the lung cancer exemplar

As a result of testing the methodology with the three lung cancer teams, we observed the following.

  • Tools: The matrix tool was easy to complete and the Likert scale enabled participants to examine how they had comparatively rated each gap. By first completing the matrix tool privately, individual participants had an opportunity to focus on the evidence gaps and consider their responses according to the criteria. Using the same matrix for the group consensus process facilitated discussion in the small group setting. The investment tool of $100 fake dollars allocated across the gaps was a useful way to double check priorities after reaching consensus. The research team also observed that the investment exercise was a fun and engaging way to end the group session, with participants interested in the outcome and whether it was reflective of whole group consensus.

  • Criteria: We found that the ‘amenable to change’ criteria was particularly meaningful for participants, as it allowed them to consider whether their current health service was capable of making change in the ‘gap’. Participation was a useful exercise for reflecting on current practice and what would be desirable to change in the future.

  • Process evaluation: Thirty (of 42) (71 %) participants completed the evaluation survey. Results are shown in Table 3. There was strong agreement from participants that the priority setting process enabled full participation (93 %); was clearly communicated by the project team (86 %) and had adequate resources and time allocated (83 %). We asked participants if consensus of priorities had been reached via the process: 73 % responded ‘definitely yes’. Three participants provided a comment about the process: ‘fantastic brainstorming session; a very useful exercise; this was very novel and engaging’.

    Table 3 Process and consensus evaluation for three focus group participants (n = 30)


This study aimed to describe the methods for selecting and adapting a priority setting technique, selecting criteria and developing tools for implementation science projects. We sought to modify the NGT and relevant tools to allow the identified gaps to be rated according to agreed criteria. We believe this is the first study to demonstrate how to modify the NGT in setting priorities for evidence practice gaps within the context of implementation science projects. The selected technique, criteria, tools and process have drawn on evidence based approaches and we believe these will be highly relevant in other health care settings. A similar process was reported by Buckley and colleagues that assessed the effects of a research prioritization process to influence the research agenda in urinary incontinence [1]. Their team also noted that consensus methods for priority settings are still being developed and questions remain about whether this will impact on research priorities.

We consider that there is significant value in prioritising gaps in research evidence and where change should be targeted in clinical settings. Gaining stakeholder engagement is key in building effective relationships for adoption of intervention strategies [21]. The advantages of the NGT made it the most relevant choice for the context of this study and we noted many similar advantages to those reported by Harvey and Holmes in priority setting about pregnancy triaging in emergency departments [8]. We selected it over other techniques including a modified Delphi approach, which has been used in previous cancer research [10, 14, 16], as we sought to reach consensus about priority gaps in a workshop setting. The selected criteria enabled our team to seek participants’ views about what gaps were significant and amenable to change in their local setting, drawing on Green and Kreutzer’s behaviour matrix from the Precede-Proceed model [30].

It is worth reflecting on the modifications made to the NGT process. We modified the first two steps of NGT (generate and recording ideas) to become a presentation of an evidence-practice gaps summary, as the investigator team had already undertaken a significant amount of work to identify relevant gaps. By doing so, we sought to address the documented barrier of clinicians having insufficient time to read and assimilate research evidence [33]. Step 2 focused on presenting data at the jurisdictional (NSW) and local levels (local health district or hospital level) to support the selection of gaps. This adaptation draws on Lomas’ step of identifying and assembling data needed to help inform participants’ discussion [6] and we consider that this is an important component in appealing to a clinical audience whose priorities are likely to focus on the technical aspects of disease management [4] and epidemiological data. Step 3 provided participants with an opportunity to discuss the gaps and elicit any additional gaps for the local setting. This step allowed for group discussion and generation of new gaps of relevance. We replicated Steps 4 and 5 of the original NGT model to be a private ranking test of items, however, in Step 5 we reduced the number of top gaps from five to two. This reduction was due to time constraints, and with having eight gaps to assess, we considered this a reasonable adjustment to make to the process. An alternative is to maintain original NGT step to rank the 5 top items or to rank items on a Likert scale.

Our modifications include additional opportunities for group discussion (Steps 6 and 7). In a subsequent study to test success factors about coordinated cancer in care [34], we found in Step 6 that rating items using the top three priorities as scores (3 = top rated item, 2 = second highest item, 1 = third item) was an easier method for reporting results where there were multiple break out groups and reaching whole group consensus. The addition of the investment exercise in Step 8 was helpful way of reflecting on the group’s priorities. In the event that this step reveals different priorities, we recommend considering a return to the group consensus process (Step 7) to determine with the large group how best to resolve divergent responses. Facilitators who are faced with a lack of consensus amongst participants should focus on the provision of an ‘equal voice’ for all participants, rather than expecting a specific outcome. While reaching consensus may be a lengthy process and contentious, there is limited evidence to suggest that gaining stakeholder buy-in results in increased likelihood of success. This is preferable in health settings to ‘majority rules’ where dominant views overrule the minority, creating ‘winners and losers’ thus potential disenfranchising participants and leading to poorer outcomes.

Study limitations and future directions

The limitations of this study are that we modified the NGT process and tools with a specific purpose of testing evidence-practice gaps in lung cancer with multidisciplinary teams of clinicians working across three hospital settings. Whilst we have used a robust and evidence-based approach to reviewing existing priority setting techniques, it is by no means the only option. Researchers will seek to adapt techniques, criteria, tools and processes to suit their own research aims and local context. However, given the paucity of tools available to researchers working in implementation science [35], we consider that the methods described in this paper make a significant enhancement to the knowledge base. Since conducting this study, we have been approached by three research teams to use this methodology in different health settings. We have utilised this methodology in a project to define success factors for coordinated care in cancer [34] on behalf of the Cancer Institute NSW. Future directions include opportunities to replicate the process with consumers, or with combined groups of health professionals and consumers, similar to the cluster RCT conducted by Boivin and colleagues [4].


In conclusion, this study shows that existing priority setting techniques and tools can be successfully adapted for use in implementation science projects. The modified NGT process and tools can be used by collaborations of researchers and clinicians who seek to identify priorities for research within a particular health condition. The modified NGT approach is practical, user-friendly and easy to conduct in small groups of clinicians. The study provides a methodology for how to prioritize evidence-practice gaps in other health settings, within the broader context of implementation science projects.



Randomised controlled trial


Nominal group technique


New South Wales


Combined matrix approach


Institute of medicine


  1. Buckley BS, Grant AM, Glazener CMA. Case study: A patient–clinician collaboration that identified and prioritized evidence gaps and stimulated research development. J Clin Epidemiol. 2013;66(5):483–9. doi:10.1016/j.jclinepi.2011.03.016.

    Article  PubMed  Google Scholar 

  2. Campbell S. Deliberative Priority Setting: a CIHR KT module; Ottawa, ON. Canadian Institutes of Health Research;2010.

  3. Smith N, Mitton C, Cornelissen E, Gibson J, Peacock S. Using evaluation theory in priority setting and resource allocation. J Health Organ Manag. 2012;26(4-5):655–71.

    Article  PubMed  Google Scholar 

  4. Boivin A, Lehoux P, Lacombe R, Burgers J, Grol R. Involving patients in setting priorities for healthcare improvement: a cluster randomized trial. Implement Sci. 2014;9:24.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Boivin A, Lehoux P, Lacombe R, Lacasse A, Burgers J, Grol R. Target for improvement: a cluster randomised trial of public involvement in quality-indicator prioritisation (intervention development and study protocol). Implement Sci. 2011;6:45.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q. 2003;81(3):363–88.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: a Guide to Nominal Group and Delphi Processes. Dallas: Scott-Foresman; 1975.

    Google Scholar 

  8. Harvey N, Holmes CA. Nominal group technique: An effective method for obtaining group consensus. Int J Nurs Pract. 2012;18:188–94.

    Article  PubMed  Google Scholar 

  9. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Robotin MC, Jones SC, Biankin AV, et al. Defining research priorities for pancreatic cancer in Australia: results of a consensus development process. Cancer Causes Control. 2010;21(5):729–36. doi:10.1007/s10552-010-9501-1.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Clinton-McHarg T, Paul C, Sanson-Fisher R, D’Este C, Williamson A. Determining research priorities for young people with haematological cancer: a value-weighting approach. Eur J Cancer. 2010;46(18):3263–70.

    Article  PubMed  Google Scholar 

  12. Paul C, Sanson-Fisher R, Douglas H, Clinton-McHarg T, Williamson A. Cutting the research pie: a value-weighting approach to explore perceptions about psychosocial research priorities for adults with haematological cancers. Eur J Cancer Care. 2010;20(3):345–53.

    Article  Google Scholar 

  13. Bajracharya SM. An assessment of the perceived barriers and strategies to promoting early detection of colorectal cancer: a practitioners’ perspective. Int Q Community Health Educ. 2006;26(1):23–44.

    Article  PubMed  Google Scholar 

  14. Young JM, Masya LM, Solomon M, Shepherd HL. Identifying indicators of colorectal cancer care coordination: a Delphi study. Color Dis. 2013;16:17–27. doi:10.1111/codi.12399.

    Article  Google Scholar 

  15. Rankin NM, Butow PN, Price MA, Evans A. Views of psycho-oncology health professionals on priority psycho-oncology research questions. Support Care Cancer. 2011;19:1133–41. doi:10.1007/s00520-010-0922-x.

    Article  PubMed  Google Scholar 

  16. Esmail LC, Roth J, Rangarao S, et al. Getting our priorities straight: a novel framework for stakeholder-informed prioritization of cancer genomics research. Genet Med. 2013;15(2):115–22.

    Article  PubMed  Google Scholar 

  17. Corner J, Wright D, Hopkinson J, Gunaratnam Y, McDonald JW, Foster C. The research priorities of patients attending UK cancer treatment centres: findings from a modified nominal group study. Br J Cancer. 2007;96(6):875–81.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Viergever RF, Olifson S, Ghaffar A, Terry RF. A checklist for health research priority setting: nine common themes of good practice. Health Res Policy Syst. 2010;8:9. doi:10.1186/1478-4505-8-36.

    Article  Google Scholar 

  19. Rankin N, Shaw T, McGregor D et al. The relevance of implementation science to psycho-oncology: Evidence practice gaps in lung cancer. Psycho-Oncology. 2013;22(S3): 46.

  20. Grol R, Wensing M, Eccles M, Davis D. Improving patient care: The implementation of change in health care. West Sussex, UK: Wiley Blackwell; 2013.

  21. Graham ID, Logan J, Harrison MB, et al. Lost in Knowledge Translation: Time for a Map? J Contin Educ Health Prof. 2006;26:13–24.

    Article  PubMed  Google Scholar 

  22. Smith N, Mitton C, Peacock S, Cornelissen E, MacLeod S. Identifying research priorities for health care priority setting: a collaborative effort between managers and researchers. BMC Health Serv Res. 2009;9:9. doi:10.1186/1472-6963-9-165.

    Article  Google Scholar 

  23. McGregor D, Rankin N, Butow P, et al. Closing evidence-practice gaps in lung cancer: Results from priority setting in the clinical context. Asia Pac J Clin Oncol. 2016. doi:10.1111/ajco.12499.

    PubMed  Google Scholar 

  24. Berra S, Sanchez E, Pons JMV, Tebe C, Alonso J, Aymerich M. Setting priorities in clinical and health services research: Properties of an adapted and updated method. Int J Technol Assess Health Care. 2010;26(2):217–24. doi:10.1017/s0266462310000012.

    Article  PubMed  Google Scholar 

  25. Ghaffar A. Setting research priorities by applying the combined approach matrix. Indian J Med Res. 2009;129(4):368–75.

    PubMed  Google Scholar 

  26. Global Forum for Health Research. The 3D Combined Approach Matrix: An improved tool for setting priorities in research for health. Geneva, SUI: Global Forum for Health Research; 2009.

  27. Public Health Ontario, Health Nexus Canada. Priority Setting – Four Methods for Getting to What’s Important. Ottawa, ON: Ontario Health Promotion E-Bulletin; 2010.

  28. Potter M, Gordon S, Hamer P. The Nominal Group Technique: A useful consensus methodology in physiotherapy research. N Z J Physiother. 2004;32(3):126–30.

    Google Scholar 

  29. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the TwentyfirstCentury. Washington DC. National Academies Press; 2001.

  30. Green LW, Kreuter MW. Health Program Planning: An Educational and Ecological Approach. 4th Edition ed. McGraw-Hill Higher Education: New York; 2005.

    Google Scholar 

  31. Phillips JL, Rolley JX, Davidson PM. Developing Targeted Health Service Interventions Using the PRECEDE-PROCEED Model: Two Australian Case Studies. Nurs Res Pract. 2012;2012:279431. doi:10.1155/2012/279431.

    PubMed  PubMed Central  Google Scholar 

  32. Hermens R, Ouwens M, Vonk-Okhuijsen SY, et al. Development of quality indicators for diagnosis and treatment of patients with non-small cell lung cancer: A first step toward implementing a multidisciplinary, evidence-based guideline. Lung Cancer. 2006;54(1):117–24. doi:10.1016/j.lungcan.2006.07.001.

    Article  CAS  PubMed  Google Scholar 

  33. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362(9391):1225–30. doi:10.1016/S0140-6736(03)14546-1.

    Article  PubMed  Google Scholar 

  34. Shaw T, York S, Rankin NM, McGregor D, Hawkey A. Success Factors in Coordinated Cancer Care. A report commissioned by the Cancer Institute NSW. Sydney: The University of Sydney; 2014.

    Google Scholar 

  35. Grimshaw J, Eccles M, Thomas R, et al. Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. J Gen Intern Med. 2006;21 Suppl 2:S14–20. doi:10.1111/j.1525-1497.2006.00357.x.

    PubMed  PubMed Central  Google Scholar 

Download references


We would like to thank the lung cancer health professionals who participated in this project and the lead clinicians and managers at clinical sites, Professor David Barnes, Dr Emily Stone, Dr Rob Zielinski and Ms Ruth Jones, for facilitating the study and enabling access to multidisciplinary lung cancer teams.


This project was funded by Sydney Catalyst Translational Cancer Research Centre flagship funding grant (2012_CIA_Shaw). Sydney Catalyst is funded by the Cancer Institute NSW through a Translational Cancer Research Centre Program Grant.

Availability of data and materials

The evaluation dataset generated and analysed during the current study is not publicly available due to the ethics approval condition to confidentially store data. However, these data are available from the corresponding author on request.

Authors’ contributions

NR, DM and TS drafted the methodological approach and all authors participated in its refinement. DM and NR conducted the literature reviews. NR prepared the manuscript. DM, PB and TS provided critical review of manuscript drafts. All authors and participated in revising the submitted manuscript and approved the final version.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

All participants gave written informed consent prior to participating in the evaluation component of this study.

Ethics approval and consent to participate

Ethics approval was obtained from the University of Sydney Human Research Ethics Committee prior to study commencement (Protocol No. 2013/528).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Nicole M. Rankin.

Additional file

Additional file 1:

An exemplar of evidence-practice gaps in lung cancer. (DOCX 14 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rankin, N.M., McGregor, D., Butow, P.N. et al. Adapting the nominal group technique for priority setting of evidence-practice gaps in implementation science. BMC Med Res Methodol 16, 110 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: