Development and validation of a search filter to identify equity-focused studies: reducing the number needed to screen

Fewer than fifteen per cent of intervention studies report any outcome by a social determinant of health [1]. We label the notion of unfair and avoidable differences in health by socioeconomic group as ‘equity’ throughout this paper, noting that the terms health inequalities, inequities, and disparities are increasingly used interchangeably [2]. Locating this minority of studies within the literature is a challenge that is hampered by the absence of validated equity search filters [3]. The current Preferred Reporting Items for Systematic Reviews and Meta-Analyses equity extension guidance (PRISMA-E) recommends that systematic reviewers do not attempt to filter searches due to poor indexing of equity terms which would lead to relevant studies being missed [4]. It is unclear how many systematic reviewers follow this advice. An obvious disadvantage to the advised sensitive (non-filtered) approach to study identification is that increased resources are needed at the screening stage, which can place a considerable burden on the research team [5].

The motivation to design and test the performance of an equity filter arose from plans to conduct a multi-disciplinary systematic review. This review aimed to examine the effect of any medical, screening or health service intervention, or any social or policy intervention that reported the prevention or reduction in the burden of mental health inequalities in UK adults. Filters were applied for study design and participant type (adult, humans); but not for the type of intervention and outcome measures as these needed to be as broad as possible. Scoping searches revealed that upwards of fifty thousand records would be retrieved, too vast for the limited resource available to independently screen studies. A crude unevaluated filter for health inequalities was observed to reduce the number of records retrieved in MEDLINE and Embase tenfold. As a result, it was decided to attempt to develop a filter to identify equity-focused studies. The sensitivity of the filter, i.e. how many relevant records it retrieved, was of paramount interest.

We therefore aimed to build and test a search strategy to identify studies that reported health inequalities, and assess the sensitivity of the filter against a gold-standard set of equity-focused studies. Secondary aims were to (1) report the reduction in the number needed to screen as a result of adding the filter to a systematic review search strategy, and (2) report on the proportion of published equity-focused systematic reviews that used an equity filter.

We aimed to develop and test a sensitive search filter to locate studies reporting equity outcomes in MEDLINE and Embase. A comprehensive strategy comprised of specific terms related to the social determinants of health filtered out 93-95% of all records, and had a sensitivity of 0.84 in MEDLINE, and 0.87 in Embase against a gold standard set of records. The sensitivity improved to 0.92 in MEDLINE (0.94 in Embase) when combined with a strategy comprising phrases often used to report equity study aims or findings. This combination met our criteria for a ‘good’ filter. The number needed to screen for the motivating systematic review was reduced by 77% by applying the specific terms strategy, and by 59.7% (MEDLINE) to 63.5% (Embase) by applying the combined strategy. Despite our study being the first to report the development and testing of an equity filter, we found that eighty-one per cent of systematic reviews published between 2014 and 2017 applied apparently unevaluated filters as part of their search strategy.

We used a combination of methods to collect gold standard papers; references from systematic reviews, personal files and hand searching journals. While combining these methods is recommended [9], , it may not eliminate bias caused by any one method. Accordingly, ur gold standard sample may not be fully representative of those in the corpus of literature, particularly studies where the inequality is not indexed and reported only in the body of the text. The majority of citations were identified from papers included in equity-focused systematic reviews that did not apply a filter; i.e. some of these studies may have been identified as reporting an equity outcome only after reading the whole article. Given that the most challenging papers to identify are those that provide few clues in the title, abstract, or keywords that they report equity outcomes, the over-inclusion of such studies improves the rigour of our gold standard set. However, studies contained in the reviews we examined may not be representative of the full corpus of literature and important studies may be missed [5]. Further, the reviews we located may not themselves fully represent the sample universe of equity-focused reviews [9]. We hand-searched only a limited number of issues in two journals, and records contained in personal files are unlikely to have broad coverage of the topic area. We used the same method to develop and measure the performance of our sample (internal validation). Validation of an external set, located by different methods than those used for the evaluation, would have strengthened our study [9] but this was beyond current resources. Although we did not formally assess reliability, two authors experienced in social epidemiology independently identified gold standard studies. There were few duplicates in the studies they identified. The selection and randomisation of gold standard studies into evaluation and validation sets was carried out independent of the author who compiled specific terms strategy, and prior to this stage being carried out. Gold standard study selection was also carried out blind to the search term compilation, minimising the risk of purposive sampling.

We used PROGRESS to help structure the specific terms strategy. Terms were mapped onto what we considered to be the most relevant concepts, but this process is subjective. Some terms cover more than one concept; for example ‘Social determinants of health’ apply to all categories and there is overlap between terms relating to SES and social capital, and differences in these between MEDLINE and Embase. We caution authors interested in filtering for a single concept to scrutinise other concepts for related terms, and emphasise that we did not estimate performance for single concepts. No -Plus terms were used for the specific terms strategy, which means that this filter may perform poorly if determinants such as age-related inequalities, disabilities or other personal characteristics are of interest. Similarly, terms such as ‘insurance’ are missing, which may disproportionately bias against US research.

Further reductions in the number needed to screen could be achieved by using text mining. Text mining techniques have been found to be useful for helping devise search strategies for complex topics and for helping to rank order records to help with screening large libraries of records. Text mining encompasses a range of statistical approaches to textual analysis and much of its value can lie in its automation and objectivity. Procedures to develop search strategies routinely using text mining approaches are available [42–44] and these tend to focus on frequency analysis of words and phrases within records. Another approach is to generate large libraries of records and then use text mining for study identification [45]. Text mining technologies can prioritise title-abstract records for manual screening, use active learning to improve the prioritisation as more records are screened and stopping rules. Given the complexity of this topic and the potential for large numbers of records, text mining could prove particularly useful in this area.

To our knowledge, our filters are the first to be formally assessed for performance; but consequent to this first attempt, they are basic and lengthy. Further development could include refining to improve sensitivity and specificity, screening all retrieved records for equity studies in order to calculate accuracy and precision, and validating against an external standard [9].

We surmise that there is a need for validated equity filters. We found the vast majority of equity-focused systematic reviews applied a filter, presumably to reduce the burden of screening for inclusion. There were, however, no reports that any equity strategy had been formally tested. Most of the search strategies in these reviews would have been compiled prior to guidance (not to filter) issued in 2015 [4], but it is unclear whether researchers will follow this guidance given the large, and ever increasing, numbers needed to screen. This is perhaps particularly true for reviews without substantial topic focus such as the effect of social determinants on a particular condition, or range of conditions, where a vast literature might be retrieved.

We chose to evaluate a broad social determinants of health filter, and consequently our study was not powered to determine the performance of the filter on, for example, studies that reported outcomes by gender, or employment status. We evaluated two subject heading terms that could be used by indexers to categorise equity studies more broadly, and not just on the determinant measured, but found very low levels of indexing. We suspect that this is due to a combination of two factors; the relatively recent introduction of these terms and a lack of familiarity of indexers with the concept of the social determinants of health, and, consequently, what equity studies look like. In particular, this under-indexing may be amplified in the many studies where equity is not the focus, but reported as a secondary outcome. Authors of studies that report equity outcomes (whether primarily or secondarily) could assist in this process by consistently suggesting the keywords ‘Social determinants of health’ and/or ‘Health inequality’ in their manuscript, along with the specific determinants measured. The Consolidated Standards of Reporting Trials (CONSORT) equity extension for randomised trials recommends trialists suggest their papers are indexed under the heading ‘Health equity’ [46]. The usefulness of indexing on any of these will only be realised for the inclusion of current and ongoing primary research in future reviews as all three of them have only recently been added as subject headings (2016 in the case of Health equity) and existing studies are not retrospectively re-indexed. The term ‘health equity’ is recommended to be included in titles of trials where appropriate [46]; we suggest that the word ‘equity’ is sufficient for this purpose and any of these phrases could be usefully incorporated into abstracts which would then be picked up with key word searches using the multi-purpose (.mp.) field.

The highest sensitivity in retrieving studies reporting outcomes by equity was a combined approach of a comprehensive list of specific terms and subject headings related to social and economic factors that produce health inequality, plus a list of non-specific terms related to the reporting of equity studies. At this early stage of equity filter development, if systematic reviewers wish to filter, this combined approach is recommended. Further efforts should concentrate on the standardisation of indexing for equity studies and additional development and testing of both specific and non-specific terms for accurate study retrieval.