Finding qualitative research: an evaluation of search strategies

Background

Qualitative research makes an important contribution to our understanding of health and healthcare. However, qualitative evidence can be difficult to search for and identify, and the effectiveness of different types of search strategies is unknown.

Methods

Three search strategies for qualitative research in the example area of support for breast-feeding were evaluated using six electronic bibliographic databases. The strategies were based on using thesaurus terms, free-text terms and broad-based terms. These strategies were combined with recognised search terms for support for breast-feeding previously used in a Cochrane review. For each strategy, we evaluated the recall (potentially relevant records found) and precision (actually relevant records found).

Results

A total yield of 7420 potentially relevant records was retrieved by the three strategies combined. Of these, 262 were judged relevant. Using one strategy alone would miss relevant records. The broad-based strategy had the highest recall and the thesaurus strategy the highest precision. Precision was generally poor: 96% of records initially identified as potentially relevant were deemed irrelevant. Searching for qualitative research involves trade-offs between recall and precision.

Conclusions

These findings confirm that strategies that attempt to maximise the number of potentially relevant records found are likely to result in a large number of false positives. The findings also suggest that a range of search terms is required to optimise searching for qualitative evidence. This underlines the problems of current methods for indexing qualitative research in bibliographic databases and indicates where improvements need to be made.

The important contribution that qualitative research can make to our understanding of health and healthcare is increasingly well-recognised [1]. However, it is often difficult to find [2]. The progress that has been made on the development of indexing systems on bibliographic databases for quantitative study designs, including randomised controlled trials and case control studies, has not been as rapid for qualitative research [3, 4]. The issue of searching for qualitative research is becoming increasingly important as interest in incorporating qualitative research into systematic reviews grows [5]. Conducting a thorough search is a distinguishing characteristic of systematic reviews, and there is a need for reviewers to be able to demonstrate the comprehensiveness and reproducibility of their searches [6].

The aim of literature searching is to optimise the ability of the search to identify relevant articles, and to exclude irrelevant articles, for a specifically defined research question. Search strategies can be evaluated in terms of their comprehensiveness in identifying relevant literature, known as 'recall', a property which can be likened to the sensitivity of a screening test. Search strategies can also be evaluated in terms of the extent to which the records identified are found to be truly relevant, known as 'precision', a property which can be likened to the positive predictive value of a screening test [7, 8]. A range of strategies is available for searching for qualitative research, but few have been formally evaluated, and little is known about the effectiveness of different search strategies, particularly across different bibliographic databases.

We selected the example area of support for breast-feeding as an area where qualitative research is likely to be of particular value, where there was likely to be a substantial body of qualitative research, and where there was an existing systematic review which had used recognised search terms for the subject area [9]. The search strategy for this existing review had been approved by the Cochrane Collaboration, and had used search terms that were designed to identify research on support for breast-feeding but were neutral to methodology; our aim was to identify research on support breast-feeding which had used a qualitative methodology. We evaluated the recall and precision of three electronic strategies for searching for qualitative research on support for breast-feeding across six bibliographic databases.

Using recognised search terms for support for breast-feeding derived from a previous Cochrane systematic review in the area, we searched for qualitative research on support for breast-feeding using six electronic bibliographic databases: MEDLINE, EMBASE, CINAHL, British Nursing Index, ASSIA and Social Sciences Citation Index. These databases represented the disciplines of medicine, nursing, and social sciences. We evaluated three electronic search strategies: using thesaurus terms; using free-text terms; and using broad-based terms.

Strategy 1: Using thesaurus terms

Indexing systems on electronic literature databases involve systems of controlled keywords (known as thesaurus terms or subject headings) that are used to categorise each record stored. Medline, for instance, uses Medical Subject Headings (MeSH), a highly structured thesaurus. The thesaurus terms that we used varied for each database according to their indexing system. For example, qualitative research is indexed on MEDLINE as "Qualitative Research" or "Nursing Methodology Research", while on CINAHL their subject heading "Qualitative Studies" is complemented by more detailed terms, including "Phenomenological Research" and "Grounded Theory".

Strategy 2: Using free-text terms

This strategy was based on using free-text terms that might specifically identify qualitative research. This strategy searches the titles, abstracts and keywords of records held in the databases. We used over 40 commonly used qualitative research methodology terms, including "ethnograph$", "lived experience$", "narrative analysis", "grounded theory" and "glaser adj2 strauss$". These terms were derived through reference to existing methodology filters for qualitative research [4] and through our own expertise in information retrieval.

Strategy 3: Using broad-based terms

This strategy was based on using three broad free-text terms, "qualitative", "findings" and "interview$" and the thesaurus term "Interviews". These terms were selected as a result of unpublished research shared with the project team [10].

The terms used in all three strategies were purposively chosen in order to maximise the comprehensiveness or sensitivity of searching for qualitative methodologies. The thesaurus and free-text strategies included terms that are often associated with quantitative rather than qualitative research, including "questionnaire" and "attitude". It was necessary to include these terms because pilot work suggested that they may also be used by bibliographic databases to classify qualitative studies, even though many qualitative researchers might not choose to use such terms to describe their work.

Relevance

The "yield" of the searching strategies was assessed as the total number of records identified by each strategy. The total initial yield of all three strategies across the six databases was used as a proxy for the population of qualitative studies in the area of support for breast-feeding. It was not within the scope of our study to determine the "true" population: establishing the "true" population would have required a "gold standard" method of retrieving qualitative evidence, which currently does not exist, and would also have required additional techniques including hand-searching of relevant core journals. Initial analyses indicated that the references were likely to be so widely scattered across journals as to make this a logistically impractical task.

"Relevance" of records identified by the searches was defined by whether the records were relevant to the topic of breast-feeding support and whether they used a recognised qualitative methodology. The judgements of relevance were made by experts in qualitative research and the topic area (RS, TM, JS, BY, MDW and SB). All judgements of relevance were based on the abstracts for the records, or, where these were unavailable (in 23% of cases), full-text articles were obtained. Any ambiguities or difficult cases were settled by consensus.

Recall: potentially relevant studies ("tested positive")

The total initial yield across the three strategies was 7420 records (see Table 1). This figure was calculated following elimination of duplicate records that were found in more than one database or by more than one strategy, and also records that referred to non-human research. The total initial yield contained all potentially relevant records. We assessed recall by calculating the proportion of records from each strategy that were potentially relevant, or, in an analogy with screening tests, the proportion of records for each strategy that "tested positive" for being both about support for breast-feeding and using a qualitative methodology.

The broad-based strategy identified 3912 potentially relevant records and had a recall rate of 52.7%, suggesting that if a record was both potentially qualitative and potentially about breast-feeding, its chances of being identified by this strategy were 52.7% (3912/7420). The thesaurus and free-text strategies had recall rates of 47.6% (3537/7420) and 46.5% (3451/7420), respectively. No single strategy was sufficiently comprehensive to identify all of the total initial yield (7420 records). This suggests that relying on one strategy alone risks missing potentially relevant records.

Precision: actually relevant studies ("diagnosed positive")

The total initial yield of 7420 potentially relevant records was subject to abstract screening by subject matter and methodology, with the aim of identifying a total number of records, as well as the number of records for each strategy, judged relevant to both breast-feeding support and qualitative research (Table 2). The search strategies had identified 2608 records that were relevant to the content area – breast-feeding support – but were judged not relevant to qualitative methodology. On the other hand, records reporting qualitative research and judged not relevant to breast-feeding support totalled 26. The 262 records that were judged relevant to the criteria both of content area and methodology were published between 1976 and 2002 in over 100 journals.

We were able to use the judgements of relevance to assess precision, calculated as the proportion of actually relevant records ("diagnosed positive", in screening terms) divided by the number of records that "tested positive" for relevance for that strategy. The precision of all strategies was poor: the thesaurus strategy precision rate was highest at 5.4% (191/3537), suggesting that if a record was initially identified as being about support for breast-feeding and using a qualitative methodology, its chances of being subsequently found to meet these criteria were 5.4%. The precision of the free-text strategy was 4.9% (172/3451) and of the broad-based strategy was 4.7% (187/3912). Overall, 96% ((7420-262)/7420) of the potentially relevant records retrieved were judged irrelevant to breast-feeding support and qualitative methodology. Furthermore, the most successful strategy, using thesaurus terms, only identified 72.9% (191/262) of the total actually relevant records found by all three strategies combined. This suggests that relying on one strategy alone would miss actually relevant records.

Our study has evaluated three strategies for searching for qualitative research in a specific topic area known to be rich in qualitative research across six databases. A limitation of the study is that it was not possible for practical reasons to identify the "true" population of qualitative research in the area of breast-feeding. For example, to conduct hand-searching of journals, it would have been necessary to search almost 30 years of more than 20 journals to retrieve just half of relevant records identified through our electronic searches. Nonetheless, our findings do have important implications for those seeking to produce documented and transparent searching strategies for qualitative research. Our findings suggest that any of the strategies we have used can identify potentially relevant qualitative studies. However, relying on any one strategy is likely to miss relevant records. This suggests that a combination of searching strategies (using thesaurus terms and free-text terms) is required in order to maximise recall – the number of potentially relevant articles retrieved. The strategies used in this study have poor precision, meaning that few articles initially identified will prove relevant on grounds of methodology.

Clearly a high recall, high precision search strategy would constitute the optimal literature-searching technique, but trade-offs between precision and recall are unavoidable [11]. Given that we did attempt to maximise recall by using a very wide range of search terms, it was perhaps not surprising that the precision of all three strategies was poor, with 96% of the total initial yield of records identified being judged not relevant. These findings suggest that the task of screening abstracts to identify qualitative research is likely to remain daunting.

The precision of searching for qualitative research could be improved in a number of ways. It is particularly disappointing that current thesaurus terms used by bibliographic database indexing systems are of limited value when searching for qualitative literature [3, 12]. If indexing systems designed specifically to identify qualitative research were available to bibliographic database providers, optimal search strategies would be enabled. Authors of qualitative studies could assist this process by making explicit their study design, and by using structured abstracts (where instructions to authors allow) [13].

The problem of poor indexing is compounded by the absence of abstracts for many qualitative studies: 23% of the records screened for our study did not have abstracts stored in the databases. The complex task of determining whether a study is qualitative is made almost impossible when no abstract is provided. The only way in this instance of confirming whether a study uses qualitative methodology is to obtain the full-text article, adding an expensive and time consuming extra layer to the review process.

Better indexing by databases and explicit qualitative methodology descriptors from authors are required to make searching for qualitative evidence more efficient and effective. Until improvements are made, those searching for qualitative research must be aware that the price of designing a high recall search strategy is poor precision. Search strategies may need to be over-inclusive so as not to miss any potentially relevant records but this necessitates a time consuming and costly process of screening for very small return. These findings are especially significant for those engaged in the increasingly widespread practice of incorporating qualitative research in systematic reviews.

We would like to thank the ESRC Research Methods Programme for funding the work on which this paper is based (project number: H333250043). We would like to thank Sheila Bonas for help with screening abstracts.

Affiliations

1. Department of Health Sciences, University of Leicester, 22-28 Princess Road West, Leicester, LE1 6TP, UK
   • Rachel L Shaw
   • , Alex J Sutton
   • , David R Jones
   •  & Mary Dixon-Woods
2. ScHARR, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK
   • Andrew Booth
3. Department of Sociology and Social Policy, Oxford Brookes University, Gipsy Lane Campus, Headington, Oxford, OX3 0BP, UK
   • Tina Miller
4. Department of Psychology, Birkbeck College, University of London, Malet Street, London, WC1E 7HX, UK
   • Jonathan A Smith
5. Department of Clinical Psychology, University of Liverpool, The Whelan Building, Quadrangle, Brownlow Hill, Liverpool, L69 3GB, UK
   • Bridget Young

Corresponding author

Correspondence to Rachel L Shaw.

Competing interests

None declared.

Authors' contributions

AB and AS designed the study and AB wrote the initial drafts of the search strategies. All authors were involved in the development of the strategies and RS conducted the literature searches using these search strategies. RS, TM, JS, BY, DJ, MDW and SB carried out the abstract screening. The analysis of recall and precision was performed by RS. RS wrote the initial draft of the paper and all authors contributed to and approved the final manuscript.