Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews
BMC Medical Research Methodology volume 7, Article number: 4 (2007)
The inclusion of qualitative studies in systematic reviews poses methodological challenges. This paper presents worked examples of two methods of data synthesis (textual narrative and thematic), used in relation to one review, with the aim of enabling researchers to consider the strength of different approaches.
A systematic review of lay perspectives of infant size and growth was conducted, locating 19 studies (including both qualitative and quantitative). The data extracted from these were synthesised using both a textual narrative and a thematic synthesis.
The processes of both methods are presented, showing a stepwise progression to the final synthesis. Both methods led us to similar conclusions about lay views toward infant size and growth. Differences between methods lie in the way they dealt with study quality and heterogeneity.
On the basis of the work reported here, we consider textual narrative and thematic synthesis have strengths and weaknesses in relation to different research questions. Thematic synthesis holds most potential for hypothesis generation, but may obscure heterogeneity and quality appraisal. Textual narrative synthesis is better able to describe the scope of existing research and account for the strength of evidence, but is less good at identifying commonality.
The inclusion of qualitative data in systematic reviews is an area of ongoing methodological development [1–3], with particular problems arising for reviews attempting to synthesise quantitative with qualitative data. The Cochrane qualitative methods group  suggests four areas in which development is needed; (1) searching, (2) critical appraisal, (3) synthesis/summary, and (4) loss of research context. This paper aims to contribute to development in the synthesis of qualitative and quantitative data. Alternative models and vocabularies of synthesis are emerging [3–9], but standard methods for combining different data types from the qualitative and quantitative research traditions have not yet been agreed .
Innovative methods are often developed during the course of research, but in general, papers report methods only briefly. As a result, the material that could inform learning is more often to be found in filing cabinets than in journals. In this paper we aim to distinguish between "the trivial and non-trivial points of divergence" p.31  by providing worked examples of two methods of evidence synthesis (thematic and textual narrative) tested in one systematic review.
A systematic review of lay views about infant size and growth was undertaken as part of a series of interlinked reviews examining the evidence for associations between early growth and a number of later outcomes. The systematic review of views included both qualitative and quantitative studies.
Study methods and findings are reported in greater detail elsewhere [10–13]. Standard systematic review methods were employed, following guidance from the Centre for Reviews and Dissemination  and from an advisory group with backgrounds in public health, paediatrics, infant nutrition, qualitative and quantitative methods, systematic reviewing, and including representatives from user groups. Twelve databases were searched using terms for growth, height, weight and infancy as well as appropriate methodological terms. 2,694 abstracts were retrieved, from which 19 studies met the inclusion criteria for the review.
Two researchers independently extracted findings by interrogating each study using the following questions developed from the aims of the review:
What is healthy growth/size?
How important is growth/size to participants?
What concepts are used to define healthy growth/size?
How do participants assess growth/size?
Where does growth lie among priorities for child health?
What information influences views/behaviour?
Who influences views/behaviour?
Directly reported participant data (e.g. verbatim quotations or scores on attitudinal scales) and author interpretations were recorded separately, to retain the richness or 'thickness' of the contributing data. 'Thickness' in this context refers to the kinds of relatively detailed descriptions and contextual material which help the reader to make judgements about the trustworthiness of the data, particularly when applying it to different contexts [15, 16]. Study characteristics and quality assessment were summarised (for examples see Table 3). There is vigorous debate on whether qualitative research can be assessed using standard quality criteria, or whether this process is contrary to the nature of qualitative enquiry . While the controversy on the use of critical appraisal in systematic reviews including qualitative data lies beyond the scope of this article, with views ranging from those who believe that critical appraisal is core to qualitative synthesis  to those who, like Barbour  consider that critical appraisal of qualitative research can be reductionist, it is notable that there is general agreement that a checklist approach to critical appraisal can bring its own problems, particularly in relation to transparency in assessing interpretative work. We took the view that applying quality criteria rigidly would be likely to exclude relevant studies that had failed to comply with a particular reporting regime. Thus, all studies meeting our inclusion criteria listed were included and quality appraisal was used at the data synthesis stage contributing to strength of evidence.
Two methods were proposed for synthesis of findings, textual narrative and thematic, both of which the advisory group agreed were appropriate to our needs. The first, the textual narrative approach, involves a commentary reporting on study characteristics, context, quality, and findings, using the scope, differences and similarities among studies were used to draw conclusions across the studies, whilst the second, the thematic approach, groups data into the themes. Given the relatively small number of studies located, it was feasible to test both methods. Findings from the review are provided briefly for illustration, but the focus of this paper is on the process of synthesis and a comparison of methods used. The two reviews ran in tandem, as the thematic review needed time for response and comparison between reviewers.
Worked Example 1 – Textual Narrative Synthesis
Factors identified by the research team from the research literature as likely to affect views on infant growth were used to define a number of sub-groups. These were:
Relationship between participant and infant (e.g. mothers, other family members, health professionals, unrelated others)
Weight status of participant
Ethnicity of participant
Age of infant
Views about infants considered 'high risk' at birth i.e. those born too small or too early, or who were placed in a neonatal intensive care unit (NICU)
Weight/growth status of infant after birth
Mode of infant feeding (breast fed, bottle fed, weaned)
Using agreed versions of quality appraisal and extracted data a textual narrative synthesis was undertaken by a single researcher (PL). Each study within a sub-group was described in a commentary reporting on study characteristics, context, quality, and findings. The scope, differences and similarities among studies were used to draw conclusions across the studies (the synthesis). Drawing conclusions across studies was not always possible due to study heterogeneity and lack of data. A worked example of the process is shown in Table 1.
Findings – Textual Narrative Synthesis
We noted that unrelated members of the public tended to prefer infants of mid-range body sizes, but the evidence to support this observation was thin. Families of children with poor growth were acutely aware of growth as a problem; they monitored growth and discussed it with others. They desired "normal" growth in their child, and looked for ways that they could interpret the infant's growth as normal (for example finding members of the extended family who were of similar body shape). The most common method of assessing size in all sub-groups was by comparison with others, although the use of growth charts and physical measurement were also important for those with children with poor growth including babies born too small or too early. However, growth and size in themselves were low among concerns about such 'high risk' babies. The predominance of those with 'high risk' infants may explain our conclusion that growth was low among priorities for mothers of younger infants (aged 0–3 and 3–6 months). Among older children (more than 12 months) with poor growth there was concern among parents. Parents wanted to see good growth in their children, but they also considered love, attention, good health and good diet as important.
We judged that we had insufficient data to draw conclusions about the views of family members other than mothers, health professionals, or to compare the views of participants of different weight, ethnicity, or toward breast versus bottle fed infants.
Worked Example 2 – Thematic Synthesis
Thematic synthesis was undertaken by two researchers, LA and PL. Findings from all studies were collated under the 7 questions used in data extraction. Each researcher independently conducted a thematic analysis using these findings. On initial discussion of themes, researchers judged that there was repetition between the data extraction questions, and that data referred to four broad areas of enquiry:
Understanding healthy growth/size
Assessment of growth/size
Concerns about growth/size
Influences on views, behaviour, interpretations of growth/size
Data and themes were grouped into these areas and emerging themes were then considered for relevance, presence across studies, 'thickness' and duplication. This process was repeated until researchers were satisfied that all data could be interpreted within these themes and an agreed version reached. A worked example of the process is shown in Table 2.
Findings – Thematic Synthesis
Across the thematic synthesis the predominant concern of participants was normality. This was seen through the creation of norms of growth and models to explain difference. This was conducted across physical, observable characteristics, but included physical unobservable (such as underlying health status) and non physical (such as emotional care) dimensions. Where growth differed from the norm and a plausible explanation could not be found, for example among families of those with faltering growth , growth became an important concern for parents.
Data from across studies could be usefully combined in this method, for example in listing all the sources of influence on behaviour or views found. Family, other parents and friends, information from the infant themselves, health professionals, clothing sizes, magazines, books, radio, TV and their religious beliefs were all important to some, but the relative importance of these could not be explored.
Strengths and limitations of our study
While the data extraction and thematic synthesis was undertaken by two researchers working independently, only one of these researchers (employed to work on the qualitative aspect of the review) worked on the narrative synthesis with a second researcher discussing the work as it progressed. Whether the findings might be different with more than one researcher working on both syntheses, or researchers not involved in the data extraction doing the syntheses, or the syntheses being carried out in a different order, are themselves research-able (if rather expensive) questions, as is the issue of whether the immersion of one researcher in the data at every stage a strength (as we believe it to be) or a source of bias.
Reassuringly, the conclusions to which these analyses led us about lay perspectives were largely similar across the thematic and textual narrative synthesis. Whether using a different research team, or a larger number of reviewers, would have produced different results is itself a researchable question. However, in this case conclusions from both analyses were dominated by importance of having babies that were a 'normal' size, leading to interest in monitoring of growth in a number of ways and, sometimes, to concern that there was an underlying problem leading to 'abnormal' growth. While the general conclusions were the same, the process and the implications of the two types of synthesis differed.
Strengths and Weaknesses of Textual Narrative Synthesis Methods
A textual narrative approach typically groups studies into more homogenous groups. This technique has been particularly successful in synthesising different types of research evidence (e.g. qualitative, quantitative, economic). Examples include a number of reviews carried out by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) [21–23], reviews of tobacco use and exposure to tobacco smoke , reviews of ultrasound in pregnancy  and of communication between health care professionals and patients about prescribing .
In our review, the textual synthesis proved a useful way to describe difference in the included studies, making explicit the diversity in study designs and contexts. The textual narrative review also described gaps in the literature, both by showing where evidence was absent and by making an evaluation of the strength of evidence in different areas. Using this method enabled us to comment on, for example, the ethnic uniformity of participants, and the lack of evidence collected regarding mode of feeding.
However, transparency remained a problem. For example, decisions about which sub-groups to use for synthesis of individual studies rely on judgements, albeit ones which can be informed by the scientific literature and by lay views. While we sought to make the decision making process clear, interpretation and judgement, which are not fully susceptible to external scrutiny, lie at the heart of the process.
Strengths and Weaknesses of Thematic Synthesis
The strengths of the thematic synthesis lie in its potential to draw conclusions based on common elements across otherwise heterogeneous studies. This synthesis is potentially more accessible for the reader than a textual synthesis. Conclusions from this thematic synthesis fulfil an important research aim of qualitative research in generating hypotheses, an area to which traditional systematic reviews are poorly suited .
However, pooling findings in the thematic synthesis risks masking the shortcomings of the individual studies that make up the review. Although descriptions of study characteristics and quality appraisal were presented alongside synthesised findings, the synthesis process obscured these in the conclusions. We believe that further debate about the reliability of this approach would be useful. On the one hand, the hypotheses that emerge from this synthesis draw on a broader body of views than any single study (as in a meta-analysis) and may therefore increase reliability; on the other, we risk making strong conclusions based on a group of studies none of which is in itself reliable on the grounds of quality or diversity of context. This method may also be poor at examining contradictions, as well as commonalities, in the data and at highlighting gaps in the evidence.
The selection of synthesis method for systematic reviews such as this may depend on the aims of the synthesis. For the purpose of generating future research hypotheses, the thematic synthesis appears to hold the greatest potential; describing common themes and providing a possible structure for new research. In contrast, the textual narrative synthesis might be better suited to reviews which aim to describe the existing body of literature; identifying the scope of what has been studied, the strength of evidence available, and gaps that need to be filled.
Petticrew M, Roberts H: Systematic Reviews in the Social Sciences. A Practical Guide. 2006, Oxford, UK, Blackwell Publishing
CQMG: Cochrane Qualitative Methods Group. 2007, [http://www.joannabriggs.edu.au/cqrmg/index.html]
Popay J, Roberts H, Sowden A, Pettticrew M, Arai L, Rodgers M, Britten N: Guidance on the conduct of narrative synthesis in systematic reviews. http://www lancs ac uk/fass/projects/nssr/. 2007
Dixon-Woods M, Agarwal S, Young B, Jones DR, Sutton AJ: Integrative approaches to qualitative and quantitative evidence. 2004, London, Health Development Agency
Harden A: Extending the boundaries of systematic reviews to integrate different types of study: examples of methods developed within reviews of young people's health. Moving beyond effectiveness in evidence synthesis. Edited by: Popay J. 2006, London, National Institute for Health & Clinical Excellence, 15-30.
Campbell R, Britten N, Pound P, Donovan J, Morgan M, Pill R, Pope C: Using meta-ethnography to synthesise qualitative research. Moving beyond effectiveness in evidence synthesis. Edited by: Popay J. 2006, London, National Institute for Health & Clinical Excellence, 75-82.
Dixon-Woods M, Bonas S, Booth A, R JD, Miller T, Sutton AJ, Shaw RL, Smith J, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research. 2005, 6: 27-44. 10.1177/1468794106058867.
CQMG: Methodological Issues Arising from the Inclusion of Qualitative Evidence in Systematic Reviews. http://www lancs ac uk/fass/ihr/research/public/cochrane htm. 2007
Shaw RL, Booth A, Sutton AJ, Miller T, Smith J, Young B, Jones DR, Dixon-Woods M: Finding qualitative research: an evaluation of search strategies. BMC Medical Research Methodology. 2004, 4:
Baird J, Fisher D, Lucas P, Kleijnen J, Roberts H, Law C: Being Big or Growing Fast; a systematic review of size and growth in infancy and later obesity. British Medical Journal. 2005, 331: 929-934. 10.1136/bmj.38586.411273.E0.
Baird J, Lucas P, Kleijnen J, Fisher D, Roberts H, Law C: Defining optimal infant growth for lifetime health: a systematic review of lay and scientific literature. 2005, [http://www.mrc.soton.ac.uk/index.asp?page=176]
Fisher D, Baird J, Payne L, Lucas P, Kleijnen J, Roberts H, Law C: Are infant size and growth related to burden of disease in adulthood? A systematic review of literature. International Journal of Epidemiology. 2006, 35: 1196-1210. 10.1093/ije/dyl130.
Lucas P, Arai L, Baird J, Kleijnen J, Law C, Roberts H: A systematic review of lay views about infant size and growth. Archives of Disease in Childhood. 2007
NHS Centre for Reviews and Dissemination: Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. 2001, CRD, 4 (2nd Edition):
Arai L, Popay J, Roen K, Roberts H: It might work in Oklahoma but will it work in Oakhampton? What does the effectiveness literature on domestic smoke detectors tell us about context and implementation?. Injury Prevention. 2005, 11: 148-151. 10.1136/ip.2004.007336.
Popay J, Rogers A, Williams G: Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research. 1998, 8: 341-351.
Dixon-Woods M, Bonas S, Booth A, R JD, Miller T, Sutton AJ, Shaw RL, Smith J, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research. 2006, 6: 27-44. 10.1177/1468794106058867.
Attree P, Milton B: Critically appraising qualitative research for systematic reviews: defusing the methodological cluster bombs. Evidence & Policy. 2006, 2: 109-126.
Barbour RS: Checklists for improving rigour in qualitative research: a case of the tail wagging the dog?. British Medical Journal. 2001, 322: 1115-1117. 10.1136/bmj.322.7294.1115.
Thomlinson EH: The lived experience of families of children who are failing to thrive. Journal of Advanced Nursing. 2002, 39: 537-545. 10.1046/j.1365-2648.2002.02322.x.
Shepherd J, Garcia J, Oliver S, Harden A, Rees R, Brunton G, Oakley A: Barriers to, and facilitators of the health of young people: A systematic review of evidence on young people's views and on intervention in mental health, physical activity and health eating. Volume 2: Complete Report. 2002, London, EPPI-Centre, Social Science Research Unit, Institute of Education, [http://eppi.ioe.ac.uk]
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health research. . 2004, J Epidemiol Community Health, 58: 794-800. 10.1136/jech.2003.014829.
Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A: An emerging framework for including different types of evidence in systematic reviews for public policy. . 2005, Evaluation, 11: 446-10.1177/1356389005059383.
Hopkins DP, Briss PA, Ricard CJ, Husten CG, Carande-Kulis VG, Fielding JE, McKenna JW, Sharp DJ, Harris JR, Wollery TA, Harris KW: Reviews of evidence regarding interventions to reduce tobacco use and exposure to environmental tobacco smoke. American Journal of Preventive Medicine. 2001, 20: 16-66. 10.1016/S0749-3797(00)00297-X.
Garcia J, Bricker L, Henderson J, Martin M, Mugford M, Nielson J, Roberts T: Women's views of pregnancy ultrasound: A systematic review. Birth. 2002, 29: 225-250. 10.1046/j.1523-536X.2002.00198.x.
Cox K, Stevenson F, Britten N, Dundar Y: A Systematic review of communication between patients and health care professionals about medicine-taking and prescribing. 2003, King's College London, GKT Concordance Unit, Guys' King's and St Thomas' School of Medicine
Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Medical Research Methodology. 2006, 6:
Baughcum AE, Burklow KA, Deeks CM, Powers SW, Whitaker RC: Maternal feeding practices and childhood obesity: a focus group study of low-income mothers. Archives of Pediatrics & Adolescent Medicine. 1998, 152: 1010-1014.
Baughcum AE, Powers SW, Johnson SB, Chamberlin LA, Deeks CM, Jain A, Whitaker RC: Maternal feeding practices and beliefs and their relationships to overweight in early childhood. . 2001, J Dev Behav Pediatr, 22: 391-408.
Hewat RJ, Ellis DJ: Similarities and differences between women who breastfeed for short and long duration. . 1986, Midwifery, 2: 37-43.
May KM: Searching for normalcy: mothers' caregiving for low birth weight infants. . 1997, Pediatr Nurs, 23: 17-20.
McCann JB, Stein A, Fairburn CG, Dunger DB: Eating habits and attitudes of mothers of children with non-organic failure to thrive. Archives of Disease in Childhood. 1994, 234-236.
Pridham KF: Information needs and problem solving behavior of parents of infants. Birth Defects: Original Article Series. 1984, 20: 125-165.
Sturm LA, Drotar D, Laing K, Zimet GD: Mothers' beliefs about the causes of infant growth deficiency: is there attributional bias?. Journal of Pediatric Psychology. 1997, 22: 329-344. 10.1093/jpepsy/22.3.329.
Hall WA, Shearer K, Mogan J, Berkowitz J: Weighing preterm infants before & after breastfeeding: does it increase maternal confidence and competence?. . 2002, MCN Am J Matern Child Nurs, 27: 318-326. 10.1097/00005721-200211000-00004.
Rajan L, Oakley A: Low birth weight babies: the mother's point of view. . 1990, Midwifery, 6: 73-85.
Reifsnider E, Allan J, Percy M: Mothers' explanatory models of lack of child growth. Public Health Nursing. 2000, 17: 434-442. 10.1046/j.1525-1446.2000.00434.x.
Sherratt F, Johnson A, Holmes S: Responding to parental concerns at the six-month stage. Health Visitor. 1991, 64: 84-86.
Smith MP: Postnatal concerns of mothers: an update. Midwifery. 1989, 5: 182-188. 10.1016/S0266-6138(89)80005-1.
Kramer MS, Barr RG, Leduc DG, Boisjoly C, Pless IB: Maternal psychological determinants of infant obesity. Development and testing of two new instruments. Journal of Chronic Diseases. 1983, 36: 329-335. 10.1016/0021-9681(83)90118-2.
Brown MM: An exploration of parental concerns about preterms and full term infants during the first nine months of life. 1981, University of California, Los Angeles, USA, 1-122.
Vehvilainen-Julkunen K: The function of home visits in maternal and child welfare as evaluated by service providers and users. Journal of Advanced Nursing. 1994, 20: 672-678. 10.1046/j.1365-2648.1994.20040672.x.
Rand CSW, Wright BA: Continuity and change in the evaluation of ideal and acceptable body sizes across a wide age span. International Journal of Eating Disorders. 2000, 28: 90-100. 10.1002/(SICI)1098-108X(200007)28:1<90::AID-EAT11>3.0.CO;2-P.
Rand CSW, Wright BA: Thinner females and heavier males: Who says? A comparison of female to male ideal body sizes across a wide age span. International Journal of Eating Disorders. 2001, 29: 45-50. 10.1002/1098-108X(200101)29:1<45::AID-EAT7>3.0.CO;2-I.
Birgenaeu CT: Body Image in Infancy. Adult body weight-related biases applied to infants. 2001, Univeristy of Massachesetts Boston, USA, 1-127.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/7/4/prepub
We would like to thank our advisory group for their input to the project, especially Paul Dieppe for chairing it, Sandy Oliver and David Jones for methodological advice and Phyll Buchanan for the additional lay input. Jos Kleijnen assisted CL, JB and HR in obtaining funding for the study and provided methodological advice. This project was funded by the Department of Health in the UK, and we thank them for their support. The views expressed in this report are those of the authors and not necessarily those of the Department of Health.
The authors declare no competing interests.
CL, JB, HR, obtained funding for the study. All authors were responsible for the concept and design of the study. PL, and HR carried out the review work with assistance from all other authors. PL, LA & HR were responsible for the interpretation of findings. PL and HR produced the first and subsequent drafts of the paper, all authors were responsible for critical revision of the manuscript.
About this article
Cite this article
Lucas, P.J., Baird, J., Arai, L. et al. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Methodol 7, 4 (2007). https://doi.org/10.1186/1471-2288-7-4
- Textual Synthesis
- Quality Appraisal
- Thematic Synthesis
- Systematic Review Method
- Commentary Reporting