- Research article
- Open Access
- Open Peer Review
A multidisciplinary systematic review of the use of diagrams as a means of collecting data from research subjects: application, benefits and recommendations
BMC Medical Research Methodologyvolume 11, Article number: 11 (2011)
In research, diagrams are most commonly used in the analysis of data and visual presentation of results. However there has been a substantial growth in the use of diagrams in earlier stages of the research process to collect data. Despite this growth, guidance on this technique is often isolated within disciplines.
A multidisciplinary systematic review was performed, which included 13 traditional healthcare and non-health-focused indexes, non-indexed searches and contacting experts in the field. English-language articles that used diagrams as a data collection tool and reflected on the process were included in the review, with no restriction on publication date.
The search identified 2690 documents, of which 80 were included in the final analysis. The choice to use diagrams for data collection is often determined by requirements of the research topic, such as the need to understand research subjects' knowledge or cognitive structure, to overcome cultural and linguistic differences, or to understand highly complex subject matter. How diagrams were used for data collection varied by the degrees of instruction for, and freedom in, diagram creation, the number of diagrams created or edited and the use of diagrams in conjunction with other data collection methods. Depending on how data collection is structured, a variety of options for qualitative and quantitative analysis are available to the researcher. The review identified a number of benefits to using diagrams in data collection, including the ease with which the method can be adapted to complement other data collection methods and its ability to focus discussion. However it is clear that the benefits and challenges of diagramming depend on the nature of its application and the type of diagrams used.
The results of this multidisciplinary systematic review examine the application of diagrams in data collection and the methods for analyzing the unique datasets elicited. Three recommendations are presented. Firstly, the diagrammatic approach should be chosen based on the type of data needed. Secondly, appropriate instructions will depend on the approach chosen. And thirdly, the final results should present examples of original or recreated diagrams. This review also highlighted the need for a standardized terminology of the method and a supporting theoretical framework.
Diagrams are graphic representations used to explain the relationships and connections between the parts it illustrates. There are many subcategories of the broader term 'diagram', which are distinguished by the elements they incorporate or their overall topic. Two dominant subcategories include 'concept maps' and 'mind maps'. Diagrams are typically brought into the research process in later stages of data analysis or when summarizing and presenting final results. It is commonplace to see a diagram illustrating how concepts or themes relate to each other or to explain how the research data relates to an underlying theory. These diagrams can be developed through the researchers' inductive reasoning of the data collected or may be assisted by computer software.
The use of diagrams in earlier stages of the research process (i.e. to collect data) is a relatively new method and is not a common data collection approach at present. However, their use is developing in multiple disciplines, including healthcare research. Diagrams have been used to collect data from research subjects by asking them to either draw a diagram themselves or modify a prototypic diagram supplied by the researcher. The use of diagrams in data collection has been viewed favorably in helping to gather rich data on healthcare topics. These research topics are widely varied and include collecting information to improve patient safety with medication, understanding neighborhood characteristics related to mental well-being, mapping out healthcare networks, evaluating patient educational programs[6, 7], understanding how different populations view microbial illnesses, diagramming as part of nursing education that is evidence-based and involves critical thinking[10, 11], to engage youth in healthcare consultations, and to gain insights on physician professional growth and their accountability relationships.
Despite the increasing use of diagrams in data collection, there lacks a strong "supportive structure" (pg. 343) for researchers choosing this method. The use of diagrams in data collection has developed independently in multiple disciplines under a number of different names, making knowledge transfer regarding this technique difficult. For example, little has been published on process mapping outside of the organizational literature until fairly recently[5, 16, 17]. This has limited the exchange of best practices between disciplines. Researchers are often starting from scratch when designing their diagramming data collection approaches and their analysis of the unique data collected.
By conducting a multidisciplinary systematic review, as defined in the PRISMA statement, we hope to consolidate lessons learned and offer recommendations for researchers in healthcare and other disciplines about how diagrams may be incorporated into their data collection process. The questions that guided our search for relevant studies were:
(1) What drives the selection of a diagramming approach for data collection?
(2) What are the different approaches to diagramming for data collection?
(3) What are the different approaches to analyzing data collected with diagramming?
(4) What are the benefits and challenges of using diagramming for data collection?
Diagramming techniques used for data collection in the research process is a challenging area to review, given the variable terminology across, and even within, fields. A preliminary survey of the literature helped identify some key terminology used in different disciplines (e.g. "graphic elicitation" or "participatory diagramming"). The terms used in the titles and abstracts of the preliminary articles identified, as well as the keywords used to index them in databases, formed the basis of our multidisciplinary search strategy. We combined these specific terms with general 'diagram' terms and with general 'data collection' and/or 'analysis' terms.
In December 2009, we electronically searched 13 indexed sources, including traditional health care related indexes and non-health focused indexes (EMBASE; HealthSTAR; Medline; Cumulative Index to Nursing and Allied Health Literature; GEOBASE; InfoTrac Environmental Issues & Policy eCollection; ProQUEST Dissertations; Design and Applied Arts Index; Education Resources Information Center; International Bibliography of the Social Sciences; PsychINFO; Public Affairs Information Services; and Social Science Citation Index). To ensure that all appropriate references were identified and to limit publication bias, non-indexed sources were also searched via general search tools (i.e. Google Scholar and Google Books) to uncover any additional publications. To supplement the search, 35 experts, identified by our searches, were contacted and asked to identify additional relevant articles and grey literature.
Reference Manager 11 was used to support the review. Following the removal of duplicates, articles were screened based on their title and abstract. The full-text was then screened for articles not excluded based on their title/abstract. Articles were excluded if they were not written in English, did not use diagramming techniques in the data collection process (i.e. research subjects did not create or edit diagrams) or were not evaluative or reflective about the data collection process and/or analysis of data collected from diagramming methods. No publication date or publication type restrictions were imposed; research studies, theoretical articles, method articles and opinion pieces were included if they met the above criteria.
The screening was undertaken by two authors (MJU, PT). Double screening was done at regular intervals to ensure inter-rater reliability. Further, the two researchers met weekly during the screening and data extraction phases to discuss the nuances of the articles and to resolve differences by deliberation until consensus was reached.
A total of 2690 references were identified, after the removal of duplicates. Given our search had no publication date restrictions and included dissertations, full-text articles were sometimes difficult to retrieve. Authors were contacted when the article could not be found online or through the University of Toronto's library system. While 4 articles were retrieved in this manner, 27 full-text articles still could not be found and were ultimately excluded. In total, 233 full-text articles were screened and a total of 80 articles were included in the study's review. Figure 1 presents a flow diagram of our search and screening. Data was extracted on the general characteristics of the articles and the four objectives detailed earlier (see Table 1).
Of the 80 articles included in our review, 53 were published studies[1, 4–15, 19–58], 19 were dissertations[59–77], 2 were books[78, 79] and 6 represented grey literature[3, 80–84], including unpublished working papers submitted by key experts and reports available on the internet. These articles were published between 1986 and 2010, with the majority published after 2000 and a substantial increase after 2006. This suggests that interest in these techniques has been increasing in recent years.
The most common discipline, determined by the lead author's affiliation and/or publication title, was from the education field. Other disciplines included healthcare, engineering, environmental science, geography, industrial design, psychology, and social science. The majority of articles clearly specified the study sample size, which averaged 36 research subjects, with a range of 2 to 243. Diagramming methods were used with a wide variety of research subjects, including students (elementary to graduate school), farmers, nurses, physicians, engineers, administrators and graphic designers.
What drives the selection of a diagramming approach for data collection?
The majority of articles specified at least one explicit reason why a form of diagramming was selected for data collection. These reasons fall into two broad categories: requirements or challenges of the research topic and the unique dataset that results from using diagrams.
The specific research topic examined was the most common reason for researchers choosing a diagramming technique for data collection. For some research topics, past studies have validated diagramming data collection techniques as a useful way to collect data. For example, research has established the usefulness of diagrams in collecting data about research subjects' knowledge or cognitive structures[19–22, 59, 80]. Diagrams in data collection have also been validated as a means of measuring changes over time[6, 10, 20, 22–24, 60, 85] and differences between participant groups[20, 25, 61, 81]. Diagramming methods were also sought out when research topics were not conducive to the more common qualitative data collection methods, such as interviews alone. These reasons include a research topic that deals with a population with linguistic, cultural, social or intuitional barriers the researcher wants to overcome[12, 14, 15, 26–29, 79] or with highly complex subject matter[12, 14, 25, 30–32, 85]. Examples of highly complex subject matters include the abstract nature of the research topic of 'pedagogical constructs' and the multifaceted and diverse nature of 'clinical accountability relationships'.
Secondly, researchers sought out diagramming data collection methods because of the benefits previous studies found regarding the quality and uniqueness of the collected dataset. When research subjects drew diagrams without prompts, previous studies concluded that it minimized the influence of the researcher on the participant and their responses[1, 33–35, 61, 62, 81, 82]. Studies have also found that diagramming is a reflective tool for the research subjects[28, 29, 36, 63]. Since diagrams can represent both concrete and theoretical notions, diagramming offers a more holistic coverage of the topic[29, 38, 61], with more uncensored and unique data gathered[1, 24, 28, 35, 39, 40, 58] than more traditional qualitative data collection methods.
What are the different approaches to diagramming for data collection?
A range of applications were identified, which varied widely based on the degree of instruction, degree of freedom in diagram creation, the number of diagrams created or edited and the use of diagram in conjunction with other data collection methods.
Half of the studies did not report the details of the instructions provided to the research subjects, except for describing the basic request to create a diagram. One study explicitly observed that specific instructions are needed to ensure the research participants create a diagram and not another form of written material, such as a drawing or table. Simple or short instructions were often given to research subjects when the diagram sought by the researchers did not have to conform to a rigid structure, such as life-cycle and professional practice diagrams.
When the researcher sought highly structured diagrams, the degree of instruction provided to the research subjects ranged from the preferred method of giving specific and detailed instructions on what elements should be included (e.g. hierarchies, arrows) to showing an example of the type of diagram the researcher would like the participant to create. For example, in comparison to other diagramming techniques, concept maps have a fairly rigid definition and a very specific set of elements that the end diagram should contain. Such diagrams may require more detailed instructions. One study had associate nursing degree students draw their own concept maps after a 20-minute introductory tutorial, presentation of a sample diagram, discussion and question period, and instructions listing all elements to be included (e.g. arrangement of items, hierarchal order, linking concepts with arrow, labeling propositions, identifying cross-links/relationships). In some instances, research subjects were given the opportunity to practice the diagramming method and receive corrective feedback prior to data collection[19, 23, 41, 64].
It was most common for research subjects to create an original diagram on their own, in groups or a combination of both. Alternatively, some studies had the participant edit either a designated diagram provided by the researcher[3, 14, 15, 42] or a researcher-created diagram generated by the researcher during the interview with participant input[4, 5, 7, 43]. A few studies also chose the middle ground between original and prepared diagrams. For example, some provided a central concept or word to create the diagram around[14, 44], or included some words or shapes to fill in on a prepared diagram and others gave research subjects a list of words to use in the creation of their diagram[65, 80].
Half of the studies used diagrams at multiple times within the data collection process as a means of comparison. A subset used a pre-/post- (or time series) approach to data collection, allowing researchers to track changes before, after, and sometimes during an intervention. This was found primarily within the discipline of education. For example, Rios asked a sample of teachers to create concept maps at multiple time intervals in order to identify their conceptual models and examine the impact of student interactions on the teachers' subject matter structure or vice versa.
Some researchers explicitly expressed the idea that diagrams alone would not capture complete perspectives from the research subjects[57, 77], suggesting that diagrams should be used in combination with other data collection methods. The majority of the studies did use diagramming data collection techniques in addition to other methods. In some cases the additional data collection methods were used explicitly in conjunction with the diagramming techniques, such as creating the diagram within interviews or discussion of the diagrams in later focus groups or interviews[8, 23, 39, 59, 60, 67].
What are the different approaches to analyzing data collected with diagramming?
The majority of the articles reported details about the analysis of diagram data. The use of only quantitative or qualitative analysis, or a mixture of both types of analysis was fairly equally distributed among the articles. Within each of these three categories of analysis, there were a variety of different techniques used that are briefly outlined below.
The majority of studies comparing diagrams across time or across research subjects chose either quantitative techniques only or a mix of qualitative and quantitative techniques for their analysis. Quantitative analysis techniques included counting (e.g. number of concepts identified, number of links between concepts, number of examples given, levels in hierarchy) and scoring. The two most common scoring methods were structural scoring and relational scoring. Structural scoring refers to when weights are assigned to hierarchical structures, links between concepts and other elements. For example, a link between two concepts may be 10 points, an example 1 point and invalid examples and cross links 0 points. Relational scoring reflects the quality or importance of each concept or link as determined by the researcher, by comparison to a similar diagram created by an expert or by other research subjects[36, 38, 60, 62, 68]. Some studies using quantitative analysis showed diagrams to illustrate how the final counting and scoring of a diagram was completed in the presentation of final study results[35, 40, 41, 46–49, 54, 62]. For example, Kesby had focus groups in Zimbabwe create scored diagrams with local materials of rocks, string and bottle caps. These diagrams were photographed and also reproduced on computer for legibility. Including an original diagram in the presentation of results also helps to orient the reader to the type of diagram that was used.
Studies which had diagrams completed or edited in the presence of the researcher, included additional data collection methods, and/or studies using less structured diagrams to collect data were likely to use either qualitative techniques only or a mix of qualitative and quantitative techniques for their analysis. For example, Haidet et al. gave medical school research subjects less structure and encouraged creativity in their diagramming exercise within interviews. The diagram was used as a prompt to stimulate discussion, which was then analyzed through the interview transcripts. The diagram itself was then displayed in the final results to visually summarize the verbal exchange. Qualitative analysis included thematic and content analysis of the diagrams and additional data sources, such as transcripts, to identify prominent topics, themes and patterns in the diagrams[13, 25, 26, 34, 63, 66, 69–71].
In some studies that used mixed-methods, the collected diagrams were the primary source of data and guided the analysis of additional data sources, for example, by providing the core themes for transcript analysis. In other cases the additional data sources played the dominant role in analysis and the diagrams were used almost as verification or visual representation to illustrate conclusions.
What are the benefits and challenges of using diagramming for data collection?
Some of the benefits to using diagrams for data collection have already been discussed in the section on why researchers chose diagramming data collection approaches. In addition to these, diagramming approaches that were seen as complementary to other data collection approaches were commonly used in interviews and focus groups[60, 73]. They were found to help focus discussions on particular themes and enabled research subjects to more easily reflect on a topic or their beliefs by helping them to express thoughts in a more structured and organized manner[50, 51]. The use of diagrams was also seen to increase recall and self-reflectiveness[53, 54]. In 1992, Powell found that interviews which made use of diagramming approaches were more introspective and tended to be more theoretical and philosophical than those that did not use diagramming methods.
Over half of the articles discussed at least one challenge of using a diagramming method for data collection. Interestingly, these challenges were often contradicted by other articles. All studies completed their data collection with diagrams, with some reporting that the diagramming allowed research subjects to overcome challenges of verbal communication[12, 14, 15, 26–29, 75, 79]. However, many studies found that at least some of the research subjects expressed difficulty or discomfort with the diagramming task[1, 9, 10, 14, 28, 35, 55, 56, 58, 62, 74, 76, 83]. Some identified the ease and speed of data collection as benefits of using diagramming approaches[24, 31, 54], while others saw it as being time-intensive, particularly for analysis[6, 11, 26, 62]. Related to the visual organization and structure of knowledge that the diagrams presented, an advantage to using diagramming approaches for data collection is their ability to obtain unique and unsolicited data[7, 14, 15, 26, 29, 33, 42, 43, 58]. In contrast, there were also concerns regarding the data it did not collect, such as non-verbal communication, that require the discretion and experience of researchers to identify and interpret[15, 24, 84]. These contradictions illustrate that the benefits and challenges to using diagramming approaches for data collection depend on the application and type of diagram used in each research study.
This systematic review represents the first overview of diagrams being used as a data collection approach in multiple disciplines. In 2006, Nesbit & Adesope conducted a widely cited meta-analysis looking at peer-reviewed articles focusing on learning with concept and knowledge maps and found that the interest in using diagrams appeared to be on the rise. While our systematic review concurs that interest is growing, it differs from their meta-analysis in two ways. Firstly our definition of a diagram is much broader, encompassing a variety of diagrams that extend beyond concept and knowledge maps. As well as the structured diagrams that Nesbit & Adescope focused on, our review also includes less structured diagrams, such as the life-cycle and professional practice discussed in our results section. Secondly, our focus is solely on diagrams being used as a data collection approach, whereas their meta-analysis included diagrams used as analysis techniques as well.
Given our broad definition of a diagram, we have reviewed approaches for collecting data through a wide spectrum of diagrams, from highly structured concept maps to less defined diagrams. We have provided an overview of the instruction options for research subjects, the creation and analysis of diagramming as a data collection approach, as well as highlighted some of the benefits and challenges. While there is variation regarding the guidance in instructions and approaches to the construction of different diagrams, use of diagrams as a data collection tool as a whole is clearly increasing in healthcare and in other disciplines. This systematic review is the first step in consolidating this information to assist in the refinement of this approach. For those considering using diagramming as a data collection approach, we offer three recommendations. Firstly, the diagrammatic approach should be chosen based on the type of data needed to answer the research question(s). Secondly, based on the diagrammatic approach chosen, it is important to select the appropriate instructions needed. Finally, presentation of final results should include examples of the original or recreated diagrams.
Choice of Diagramming Approach
The most important considerations for choosing the diagramming approach is the type of data needed to answer the research question (e.g., examining change over time, exploring people's experiences or views) and the type of analysis preferred by the researcher. For example, highly structured diagrams allowed for valid quantitative analysis, such as counting and ranking, which could be compared across research subjects. It should be noted that there is some controversy whether items should even be counted and that both an over- or under-reliance may be dangerous to the final research conclusions. In comparison, other approaches that used a less structured diagrams relied heavily on qualitative analysis.
Instruction and Creation
The appropriateness of different approaches to instructions to guide diagram creation is an important consideration in ensuring the validity of data. It is clear from our review that the initial instructions given to research subjects varied in structure but had a great impact on the resulting diagrams and their potential for different analysis techniques. If researchers require highly structured diagrams it may be useful to give research subjects more detailed instructions and the opportunity for practice and feedback[30, 64].
Presentation of results
The last recommendation is that studies using diagramming data collection approaches should include visual presentations of the findings in their results sections. The use of diagrams often results in the collection of unique and unsolicited data through a visual component, which can then be displayed along with the final analysis. Just as diagrams can provide data not easily obtained through verbal data collection techniques, visual presentation of the collected diagrams may provide insights not as easily grasped through verbal communication of study results. While providing a scanned image of the original diagram can be difficult at times, it is also possible to present a photograph of the original diagram or a computer-generated recreation of the diagram. This is especially important given the variation in terminology between disciplines, as it relays to the reader the type and structure of diagrams created or used.
In addition to the recommendations for researchers considering the use of diagramming data collection approaches, this multidisciplinary review also identifies areas where future research is needed. This review required a substantial amount of preliminary work to understand the terminology used to describe diagrams and this data collection approach across different disciplines and fields. The intent was to devise a sensitive search, so as to cast a wide net in order to capture articles in a range of disciplines where the terminology is not standardized. Cole et al. illustrate this issue by identifying over a dozen different terms that are used to describe concept maps. Thus far, development of terminology has focused on the end result, i.e. what type of diagram is created based on the elements it contains, rather than focusing on the actual data collection approach itself. This has created different data collection approaches that use diagrams separately, isolating research done across disciplines and even within disciplines. Therefore, it is our recommendation that efforts are directed towards standardizing the terminology for this data collection method. This would allow researchers to maintain the work they have done regarding specific types of diagrams, whilst providing an umbrella term to help with the sharing of best practices. Future research should also be directed at identification of the underpinning theory of the method as the review demonstrated a gap in this area[15, 32, 35]. Such a theory may help to further inform researchers regarding the appropriate use and applications for diagramming data collection approaches.
A limitation of our review is that the database search strategy did not capture general articles on visual data collection methods, which may include specific information on diagramming data collection approaches. However the non-indexed searches and articles identified by experts did pick up some of these articles. While efforts were made to contact authors to retrieve articles from our search, twenty-seven full-text articles were irretrievable. Ten of these were dissertations and five were books. This may have contributed to an incomplete representation of what the literature has to offer about diagramming data collection approaches.
There has been a growing interest in the use of diagrams for data collection in the research process over recent years, as shown by the increase in publications and the wide range of approaches developed for diagramming data collection and diagram data analysis. As noted earlier, diagrams have been used to collect rich data on a variety of healthcare topics and it is expected that the use of this method will continue to grow. The results of this multidisciplinary systematic review provide an overview of the application of diagrams in research data collection and the methods for analyzing the unique datasets elicited. Recommendations are presented to assist researchers considering the use of diagrams in their data collection process. This review also highlighted the need for a standardized terminology of the method and a supporting theoretical framework.
Wheeldon J, Faubert J: Framing experience: concept maps, mind maps, and data collection in qualitative research. International Journal of Qualitative Methods. 2009, 8: 68-83.
Forbes MA: Hope in the older adult with chronic illness: A comparison of two research methods in theory building. Advances in Nursing Science. 1999, 22: 74-87.
Jafri T, Lyons MN, Clarkson PJ: The supply of medicines in the NHS. 2008, Improving Patient Safety: Cambridge, UK
O'Campo P, Salmon C, Burke J: Neighbourhoods and mental well-being: what are the pathways?. Health and Place. 2009, 15: 56-68.
Pluto DM, Hirshorn BA: Process mapping as a tool for home health network analysis. Home Health Care Services Quarterly. 2003, 22: 1-16. 10.1300/J027v22n02_01.
Franca S, d'Ivernois JF, Marchand C, Haenni C, Ybarra J, Golay A: Evaluation of nutritional education using concept mapping. Patient Education and Counseling. 2004, 52: 183-192. 10.1016/S0738-3991(03)00037-5.
Marchand C, d'Ivernois JF, Assal JP, Slama G, Hivon R: An analysis, using concept mapping, of diabetic patients' knowledge, before and after patient education. Medical Teacher. 2002, 24: 90-99. 10.1080/01421590120091087.
Jones MG, Rua MJ: Conceptual representations of flu and microbial illness held by students, teachers, and medical professionals. Social Science and Mathematics. 2008, 108: 263-278.
Abel WM, Freeze M: Evaluation of concept mapping in an associate degree nursing program. Journal of Nursing Education. 2006, 45: 356-364.
Daley BJ, Shaw CR, Balistrieri T, Glasenapp K, Piacentine L: Concept maps: a strategy to teach and evaluate critical thinking. Journal of Nursing Education. 1999, 38: 42-47.
Hicks-Moore SL, Pastirik PJ: Evaluating critical thinking in clinical concept maps: A pilot study. International Journal of Nursing Education Scholarship. 2006, 3: 1-15. 10.2202/1548-923X.1314.
Coad J: Using art-based techniques in engaging children and young people in health care consultations and/or research... including commentary by Bishop V and Picard C. Journal of Research in Nursing. 2007, 12: 487-497. 10.1177/1744987107081250.
Haidet P, Hatem DS, Fecile ML, Stein HF, Haley H-L, Kimmel B, et al: The role of relationships in the professional formation of physicians: Case report and illustration of an elicitation technique. Patient Education and Counseling. 2008, 72: 382-387. 10.1016/j.pec.2008.05.016.
Umoquit MJ, Dobrow MJ, Lemieux-Charles L, Ritvo PG, Urbach DR, Wodchis WP: The efficiency and effectiveness of utilizing diagrams in interviews: An assessment of participatory diagramming and graphic elicitation. BMC Medical Research Methodology. 2008, 8-
Crilly N, Blackwell A, Clarkson P: Graphic elicitation: using research diagrams as interview stimuli. Qualitative Research. 2006, 6: 341-366. 10.1177/1468794106065007.
Jun GT, Ward JR, Morris ZS, Clarkson PJ: Health care process modelling: which method when?. International Journal for Quality in Health Care. 2009, 21: 214-224. 10.1093/intqhc/mzp016.
Jun GT, Ward JR, Clarkson PJ: Systems modelling approaches to the design of safe healthcare delivery:ease of use and usefulness percived by healthcare works. Ergonomics. 2010, 53: 829-847. 10.1080/00140139.2010.489653.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine. 2009, 6: e100100-10.1371/journal.pmed.1000100.
Zhang L, Wang Y, Dong B, Zhou Z: The comparison study of Chinese and American secondary school students' knowledge structure--An experimental research based on concept map assessment technique. Frontiers of Education in China. 2009, 4: 286-297. 10.1007/s11516-009-0015-0.
Hay DB: Using concept maps to measure deep, surface and non-learning outcomes. Studies in Higher Education. 2007, 32: 39-57. 10.1080/03075070601099432.
Lavigne NC, Salkind SJ, Yan J: Exploring college students' mental representations of inferential statistics. The Journal of Mathematical Behavior. 2008, 27: 11-32. 10.1016/j.jmathb.2007.10.003.
West DC, Park JK, Pomeroy JR, Sandoval J: Concept mapping assessment in medical education: A comparison of two scoring systems. Medical Education. 2002, 36: 820-826. 10.1046/j.1365-2923.2002.01292.x.
Carter G, Jones MG, Rua M: Effects of partner's ability on the achievement and conceptual organization of high-achieving fifth-grade students. Science Education. 2002, 87: 94-111. 10.1002/sce.10031.
Hay DB, Wells H, Kinchin IM: Quantitative and qualitative measures of student learning at university level. Higher Education. 2008, 56: 221-240. 10.1007/s10734-007-9099-8.
Powell RR: The influence of prior experiences on pedagogical constructs of traditional and nontraditional preservice teachers. Teaching and Teacher Education. 1992, 8: 225-238. 10.1016/0742-051X(92)90022-U.
Kesby M: Participatory diagramming as a means to improve communication about sex in rural Zimbabwe: a pilot study. Social Science & Medicine. 2000, 50: 1723-1741.
Van der Riet M: Diagramming as mediational means: Vygotskian theory and participatory research. South African Journal of Psychology. 2008, 38: 455-465.
Wheeldon J: Is a picture worth a thousand words? Using mind maps to facilitate participant recall in qualitative research. Qualitative Report.
Bagnoli A: Beyond the standard interview: The use of graphic elicitation and arts-based methods. Qualitative Research. 2009, 9: 547-571. 10.1177/1468794109343625.
Lim SE, Chan Cheng PW, Lam MS, Ngan SF: Developing reflective and thinking skills by means of semantic mapping strategies in kindergarten teacher education. Early Child Development and Care. 2003, 173 (1): 55-72. 10.1080/0300443022000022422.
Ozesmi U, Ozesmi SL: Ecological models based on people's knowledge: A multi-step fuzzy cognitive mapping approach. Ecological Modeling. 2004, 176: 43-64. 10.1016/j.ecolmodel.2003.10.027.
Varga-Atkins T, O'Brien M: From drawings to diagrams: Maintaining researcher control during graphic elicitation in qualitative interviews. International Journal of Research & Methods in Education. 2009, 32: 53-67.
Williams CG: Using concept maps to assess conceptual knowledge of function. Journal for Research in Mathematics Education. 1998, 29: 414-421. 10.2307/749858.
Strean WB, Holt NL: Coaches', athletes', and parents' perceptions of fun in youth sports: assumptions about learning and implications for practice. AVANTE. 2000, 6: 83-98.
Wheeldon J: Mapping mixed methods research: Methods, measures, and meaning. Journal of Mixed Methods Research. 2010, 48: 87-102. 10.1177/1558689809358755.
Rye JA, Rubba PA: An exploration of the concept map as an interview tool to facilitate the externalization of students' understandings about global atmospheric change. Journal of Research in Science Teaching. 1998, 35: 521-546. 10.1002/(SICI)1098-2736(199805)35:5<521::AID-TEA4>3.0.CO;2-R.
Lourdel N, Gondran N, Laforest V, Debray B, Brodhag C: Sustainable development cognitive map: A new method of evaluating student understanding. International Journal of Sustainability in Higher Education. 2007, 8: 170-182. 10.1108/14676370710726634.
Koul R, Clariana RB, Salehi R: Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research. 2005, 32: 227-239. 10.2190/5X9Y-0ETN-213U-8FV7.
Shymansky JA, Yore LD, Treagust DF, Thiele RB, Harrison A, Waldrip BG, et al: Examining the construction process: A study of changes in level 10 students' understanding of classical mechanics. Journal of Research in Science Teaching. 1997, 34: 571-593. 10.1002/(SICI)1098-2736(199708)34:6<571::AID-TEA3>3.0.CO;2-K.
West DC, Pomeroy JR, Park JK, Gerstenberger EA, Sandoval J: Critical thinking in graduate medical education: A role for concept mapping assessment?. Journal of the American Medical Association. 2000, 284: 1105-1110. 10.1001/jama.284.9.1105.
Lian MWS: An investigation into high-achiever and low-achiever knowledge organisation and knowledge processing in concept mapping: A case study. Research in Science Education. 1998, 28: 337-352. 10.1007/BF02461567.
Crilly N, Clarkson P, Blackwell A: Using research diagrams for member validation in qualitative research. Diagrams. 2006, 4045: 258-262.
Cox P, Niewoehner J, Pidgeon N, Gerrard S, Fischhoff B, Riley D: The use of mental models in chemical risk protection: Developing a generic workplace methodology. Risk Analysis. 2003, 23: 311-324. 10.1111/1539-6924.00311.
Koury K, Hollingsead C, Fitzgerald G, Miller K, Mitchem K, Tsai HH, et al: Case-based instruction in different delivery contexts: The impact of time in cases. Journal of Interactive Learning Research. 2009, 20: 445-467.
Gill PE, Persson M: On using concept-maps to study school-children's understanding of leisure-time. Leisure Studies. 2008, 27: 213-220. 10.1080/02614360802048795.
Wheeldon J: To guide or provoke? Maps, pedagogy, and the value(s) of teaching criminal justice ethics. Criminal Justice Education.
Mouratiadou I, Moran D: Mapping public participation in the Water Framework Directive: A case study of the Pinios River Basin, Greece. Ecological Economics. 2007, 62: 66-76. 10.1016/j.ecolecon.2007.01.009.
Kimber K, Pillay H, Richards C: Technoliteracy and learning: An analysis of the quality of knowledge in electronic representations of understanding. Computers & Education. 2007, 48: 59-79.
Artiles AJ, Barreto RM, Pena L, McClafferty K: Pathways to teacher learning in multicultural contexts - A longitudinal case study of two novice bilingual teachers in urban schools. Remedial and Special Education. 1998, 19: 70-90. 10.1177/074193259801900203.
Farrell TSC: Critical reflection in a TESL Course: Mapping conceptual change. ELT Journal. 2008, 63: 221-229. 10.1093/elt/ccn058.
Lourdel N, Gondran N, Laforest V, Brodhag C: Introduction of sustainable development in engineers' curricula: Problematic and evaluation methods. International Journal of Sustainability in Higher Education. 2005, 6: 254-264. 10.1108/14676370510607223.
Naykki P, Jarvela S: How pictorial knowledge representations mediate collaborative knowledge construction in groups. Journal of Research on Technology in Education. 2008, 40: 359-
Conceicao SC, Taylor LD: Using a constructivist approach with online concept maps: relationship between theory and nursing education. Nursing Education Perspectives. 2007, 28: 268-275.
Kesby M: Participatory diagramming: Deploying qualitative methods through an action research epistemology. Area. 2000, 32: 423-435. 10.1111/j.1475-4762.2000.tb00158.x.
Ahmed R, Ali NA: Performance appraisal decision in Malaysian public service. International Journal of Public Sector Management. 17: 48-64. 10.1108/09513550410515565. 4 A.D
Cole C, Lin Y, Leide J, Large A, Beheshti J: A classification of mental models of undergraduates seeking information for a course essay in history and psychology: preliminary investigations into aligning their mental models with online thesauri. Journal of the American Society for Information Science & Technology. 2007, 58: 2092-2104.
Rowe AL, Cooke NJ: Measuring mental models: Choosing the right tools for the job. Human Resources Development Quarterly. 1995, 6: 243-255. 10.1002/hrdq.3920060303.
Wheeldon J: Learning from Latvia: Adoption, adaptation, and evidence based justice reform. Journal of Baltic Studies.
Garegae KGM: . Teachers, beliefs about mathematics, its teaching and learning and the communication of these beliefs to students: A case study in Botswana. Ph.D. 2002, Canada: The University of Manitoba
McWhirter LJ: Conceptual development and retention within the learning cycle. Ph.D. 1998, United States -- Oklahoma: The University of Oklahoma
Williams CG: . Using concepts maps to determine differences in the concept image of function held by students in reform and traditional calculus classes. Ph.D. 1994, United States -- California: University of California
Puukari S: . Video programmes as learning tools: Teaching the gas laws and behaviour of gases in Finnish and Canadian senior high schools. Ed.D. 2003, Finland: Jyvaskylan Yliopisto
Mathews S: An ethnographic examination of perspective consciousness and intercultural competence among social studies student-teachers in Kenya, East Africa. Ph.D. 2008, United States -- Indiana: Indiana University
Skidmore L: Concept mapping to promote meaningful learning at the community college level. 2008, United States -- Minnesota: Walden University, Ed.D
Yin Y, Vanides J, Ruiz-Primo MA, Ayala CC, Shavelson R: A Comparison of Two Construct-a-Concept-Map Science Assessments: Created Linking Phrases and Selected Linking Phrases. 2004, The Regents of the University of California, 1-28. unknown CSE Report 624
Rios JM: Studies of contrast: Relationships between subject matter structures and teaching. 1995, United States -- Wisconsin: The University of Wisconsin, Ph.D
Feldsine JE: The construction of concept maps facilitates the learning of general college chemistry: A case study. 1987, United States -- New York: Cornell University, Ph.D
Butler KA: Scaffolding software: How does it influence student conceptual understanding and motivation?. 2004, United States -- Illinois: Southern Illinois University at Carbondale, Ph.D
Alexander LM: Variations of professional identity over time: A study of physician assistants. 2003, United States -- District of Columbia: The George Washington University, Ed.D
Cantu DA: A naturalistic investigation of the relationship between secondary social studies teachers' beliefs and practice. 1997, United States -- Illinois: Southern Illinois University at Carbondale, Ph.D
Cozza B: Concept mapping through logs and metacognitive reflection during third graders' scientific problem-solving. 1996, United States -- New York: Fordham University, Ph.D
Ballard BG: Preservice teachers' beliefs about classroom management before and after student teaching. 2002, United States -- Nevada: University of Nevada, Las Vegas, Ed.D
Lowe M: Continuing medical education for physicians: Key factors contributing to learning and change. 2007, United States -- Connecticut: University of Connecticut, Ph.D
Yue H: Concept maps as assessment tools in mathematics: Comparison with clinical interviews. 2008, United States -- Texas: The University of Texas at El Paso, MAT
White E: Institutional effectiveness: The integration of program review, strategic planning and budgeting processes in two California community colleges. 2007, Andrews University, Berrien Springs, MI
Grunow JEM: Using concept maps in a professional development program to assess and enhance teachers' understanding of rational number. 1998, United States -- Wisconsin: The University of Wisconsin, Ph.D
Rowe AL: Mental models of physical systems: Examining the relationship between knowing and doing. 1995, Rice University
Kane M, Trochim WMK: Concept mapping for planning and evaluation. 2007, Thousand Oaks, CA, US: Sage Publications, Inc
Kesby M: Participatory diagramming and the ethical and practical challenges of helping Africans themselves to move HIV work 'beyond epidemiology'. HIV/AIDS in Africa: Beyond Epidemiology. Edited by: Kalipendi E, Craddock S, Oppong J, Ghosh J. 2004, Blackwell, 217-228.
Schreiber DA, Abegg GL: Scoring Student-Generated Concept Maps in Introductory College Chemistry. 1991, The National Association for Research in Science Teaching, 1-19. Annual meeting
Williams CG: Concept Maps as Research Tools in Mathematics. 18 April, 1995. 1995, 1-65.
Varga-Atkins T, O'Brien M: Graphic elicitation as a research technique: reflecting on drawings and diagrams as complementary methods. 2008, 1-22. Ref Type: Report
Emmel N: Participatory Mapping: An innovative sociological method. Toolkit #3. 2008, ESRC National Centre for Research Methods, 1-8. Ref Type: Report
Beyerbach BA: Concept Mapping in Assessing Prospective Teachers' Concept Development. n/a, 1-16. 1986, US Department of Education, Ref Type: Report
Kane M, Trochim WMK: Concept Mapping for Planning and Evaluation. 2007, Thousand Oaks, CA, US: Sage Publications, Inc
Nesbit JC, Adescope O: Learning with concept and knowledge maps: A meta-analysis. Review of Education Research. 2006, 76: 413-448. 10.3102/00346543076003413.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/11/prepub
This study was supported by funding from a grant from the Canadian Institutes of Health Research.
The authors declare that they have no competing interests.
MJU, MJD, HEDB participated in the conception and design of the study. Data collection, abstraction and analysis was carried out by MJU and PT. MJU and PT prepared the original draft of the manuscript, and all authors reviewed and critically revised the original and subsequent manuscript drafts and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.