Development and validation of a search filter to identify equity-focused studies: reducing the number needed to screen

Abstract

Background

Health inequalities, worse health associated with social and economic disadvantage, are reported by a minority of research articles. Locating these studies when conducting an equity-focused systematic review is challenging due to a deficit in standardised terminology, indexing, and lack of validated search filters. Current reporting guidelines recommend not applying filters, meaning that increased resources are needed at the screening stage.

Methods

We aimed to design and test search filters to locate studies that reported outcomes by a social determinant of health. We developed and expanded a ‘specific terms strategy’ using keywords and subject headings compiled from recent systematic reviews that applied an equity filter. A ‘non-specific strategy’ was compiled from phrases used to describe equity analyses that were reported in titles and abstracts, and related subject headings. Gold standard evaluation and validation sets were compiled. The filters were developed in MEDLINE, adapted for Embase and tested in both. We set a target of 0.90 sensitivity (95% CI; 0.84, 0.94) in retrieving 150 gold standard validation papers. We noted the reduction in the number needed to screen in a proposed equity-focused systematic review and the proportion of equity-focused reviews we assessed in the project that applied an equity filter to their search strategy.

Results

The specific terms strategy filtered out 93-95% of all records, and retrieved a validation set of articles with a sensitivity of 0.84 in MEDLINE (0.77, 0.89), and 0.87 (0.81, 0.92) in Embase. When combined (Boolean ‘OR’) with the non-specific strategy sensitivity was 0.92 (0.86, 0.96) in MEDLINE (Embase 0.94; 0.89, 0.97). The number needed to screen was reduced by 77% by applying the specific terms strategy, and by 59.7% (MEDLINE) and 63.5% (Embase) by applying the combined strategy. Eighty-one per cent of systematic reviews filtered studies by equity.

Conclusions

A combined approach of using specific and non-specific terms is recommended if systematic reviewers wish to filter studies for reporting outcomes by social determinants. Future research should concentrate on the indexing standardisation for equity studies and further development and testing of both specific and non-specific terms for accurate study retrieval.

Background

Fewer than fifteen per cent of intervention studies report any outcome by a social determinant of health [1]. We label the notion of unfair and avoidable differences in health by socioeconomic group as ‘equity’ throughout this paper, noting that the terms health inequalities, inequities, and disparities are increasingly used interchangeably [2]. Locating this minority of studies within the literature is a challenge that is hampered by the absence of validated equity search filters [3]. The current Preferred Reporting Items for Systematic Reviews and Meta-Analyses equity extension guidance (PRISMA-E) recommends that systematic reviewers do not attempt to filter searches due to poor indexing of equity terms which would lead to relevant studies being missed [4]. It is unclear how many systematic reviewers follow this advice. An obvious disadvantage to the advised sensitive (non-filtered) approach to study identification is that increased resources are needed at the screening stage, which can place a considerable burden on the research team [5].

The motivation to design and test the performance of an equity filter arose from plans to conduct a multi-disciplinary systematic review. This review aimed to examine the effect of any medical, screening or health service intervention, or any social or policy intervention that reported the prevention or reduction in the burden of mental health inequalities in UK adults. Filters were applied for study design and participant type (adult, humans); but not for the type of intervention and outcome measures as these needed to be as broad as possible. Scoping searches revealed that upwards of fifty thousand records would be retrieved, too vast for the limited resource available to independently screen studies. A crude unevaluated filter for health inequalities was observed to reduce the number of records retrieved in MEDLINE and Embase tenfold. As a result, it was decided to attempt to develop a filter to identify equity-focused studies. The sensitivity of the filter, i.e. how many relevant records it retrieved, was of paramount interest.

We therefore aimed to build and test a search strategy to identify studies that reported health inequalities, and assess the sensitivity of the filter against a gold-standard set of equity-focused studies. Secondary aims were to (1) report the reduction in the number needed to screen as a result of adding the filter to a systematic review search strategy, and (2) report on the proportion of published equity-focused systematic reviews that used an equity filter.

Methods

There are four stages of search filter development [6]:

• 1) Search term selection

• 2) Identification of a ‘gold standard’

• 3) Evaluation of the filter

• 4) Validation of the filter.

Search term selection

We sought to evaluate two complementary sets of terms; one ‘specific’ set of terms relating to words used to describe the factors of interest (the social determinants of health) and a ‘non-specific’ set of terms relating to words used to describe the aims, methods or results of equity analyses.

Specific terms

The goal of this stage was to populate the categories of interest with search terms. We adapted a pragmatic approach of using terms reported in published search strategies [7] into an iterative strategy of identifying terms included in systematic reviews.

First, the search strategies of equity-focused systematic reviews published within the most recent year were examined. These reviews were identified via Embase and MEDLINE (OVID) on 3.10.2017 using the search terms:

(equit* or inequal* or inequit* or socioeconomic or socio$economic or dispar*).ti. AND (systematic adj review).ti., limited to English language, excluding conference abstracts, and published in 2017.

Data extracted by one author (SLP) included; the citation, whether an equity filter was applied, and whether an example strategy for the equity filter was made available. Terms in English relating to the social determinants of health used in the published example search strategy were extracted. If a strategy for more than one database was provided then terms for only one was extracted (in all cases this was MEDLINE). Terms relating to other exposures examined in the reviews were ignored, unless this was also a term of interest.

Inequality terms were classified into concepts and measures where possible and classified into PROGRESS categories by one author (SLP). ‘PROGRESS’ is an acronym for common situations and circumstances associated with health inequality that can be used by authors of equity-focused systematic reviews to categorise social determinants of health [4]. The acronym refers to Place of residence, Race /ethnicity, Occupation, Gender, Religion, Education, Socio-economic status and Social capital. In ‘Place of residence’ we included terms such as marital or cohabitation status and the size of the household, and included income under ‘Socio-economic status’. The three further categories, the –Plus in PROGRESS-Plus, refers to factors that may put individuals at disadvantage. These are listed as; (1) personal characteristics such as age or disability, (2) situations resulting from interdependent relationships and (3) time-limited disadvantage resulting from a temporary change in circumstance [8]. As a pragmatic step, all the -Plus groupings were dropped because they could encompass almost limitless situations and demographics. Terms that did not map to PROGRESS categories (and were not related to -Plus) were classified under an additional ‘Other’ category.

A second search was undertaken to identify further equity-focused reviews to provide terms for unpopulated or sparsely populated (≤3 studies contributing terms) PROGRESS categories. This search (carried out on 19.10.2017) was based on the strategy above but for reviews published in 2016.

A third set of category-specific searches were undertaken to identify key terms for categories that remained un- or sparsely populated. These searches aimed to identify any systematic reviews, not just those with an equity-focus, which included basic terms relating to the desired category in the title. The search was carried out in Embase and MEDLINE (OVID) on 19.10.2017 using the search terms:

([PROGRESS keyword]ti. AND (systematic adj review).ti., limited to English language, excluding conference abstracts, and published between 2012 and 2015.

In the event that very large numbers of reviews were retrieved from the targeted searches, reviews were randomly ordered using a list randomiser (https://www.random.org/lists/) and sequentially inspected until five reviews presenting the terms of interest were included.

The strategy was developed in MEDLINE. Included Medical Subject Headings (MeSH) terms were verified in the MeSH browser (meshb.nlm.nih.gov) and related tree structures inspected to determine if the term should be exploded i.e. all nested concepts were relevant, or not. When examining the tree structures, we noted related subject headings and added those that were missing from our included compilation. We examined each term for face validity and removed those that we did not consider to be a social determinant of health, and those that were too general. We adapted the strategy to be compatible with both American English and English spelling, and truncated terms where appropriate to capture both singular and plural endings and for similar stem terms.

Categorised terms were compiled into a search strategy for MEDLINE by SLP which was checked for errors by SG. The search was carried out in OVID MEDLINE by SLP without knowledge of which articles were in the gold standard.

Non-specific terms

We compiled terms and phrases reported in titles and abstracts, and MeSH descriptors, which were associated with the reporting of differential effects from papers included in the two evaluation sets (see below). Examples of these phrases were ‘…risk factors for…’, ‘…differed by…’, and ‘…significant among…’.

Gold standard sample size

The overall aim of the study was to calculate the sensitivity of the filter against the gold standard set of articles [6]:

$$ \frac{No. of\ relevant\ records\ retrieved\ by\ the\ filter}{No. of\ records\ in\ the\ gold\ standard}\ \mathrm{x}\ 100 $$

We considered that the retrieval of 90% of relevant records would be a good result for this filter. Ninety-five per cent binomial confidence intervals around a sensitivity of 0.90 were 0.84 to 0.94 for a sample size of 150 gold standard papers.

Identification of a ‘gold standard’ set of papers

We used similar methods to identify the gold standard papers for use in both the development and validation of the filter (internal standard). Identification of the gold standard papers was carried out independently by two reviewers (MP and EPU) blind to SLP who compiled the search terms, and who were themselves blinded to the compilation of the search terms. MP and EU are experienced researchers who have published on social epidemiology. Reliability was not formally assessed.

Three strategies were employed. First, the ‘relative recall’ method was used, identifying relevant studies from a range of previously published systematic reviews [5]. This method works well in comparison to standard methods for filter validation that use hand searching of journals to identify a gold standard, has major advantages in terms of time and resources, and can result in a wider spread of journal titles being included in the gold standard [5]. The following search strategy was applied in MEDLINE and Embase to identify equity-focused systematic reviews.

(equit* or inequal* or inequit* or socioeconomic or socio$economic or dispar*).ti. AND (systematic adj review).ti., limited to English language, excluding conference abstracts

One reviewer searched for reviews published in 2014 and one for reviews published in 2015. Articles were read in full and those that did not filter for equity terms were included. Studies of any design that were included in these reviews were considered for inclusion in the gold standard. Equity-focused studies that were excluded in the last screening stage of a systematic review for reasons irrelevant to this current study (e.g. wrong outcome measures, study design not meeting inclusion criteria) were also included.

Second, each reviewer was asked to review their ‘personal files’ (both physical and electronic reference manager databases) to identify all studies of any design reporting an equity outcome.

Third, each reviewer was asked to review at least two issues in any one year (2005 onwards) of the journals Social Science in Medicine and International Journal for Equity in Health and include all studies reporting an equity outcome.

There were no publication date limits on the gold standard articles. For the first and second strategies the reviewers checked that an identified article was indexed by MEDLINE and/or Embase and excluded articles that were not indexed in either database [5]. Each reviewer was asked to retrieve a minimum of 150 papers.

The MEDLINE citation for each study identified by any of these three methods of retrieval were combined into a single Endnote library and duplicates removed (n=5). Duplicates resulted from overlap with the personal files method. Studies were randomly sampled into a validation set (N=150), with the remaining randomly split into two evaluation sets (set 1 N=101, set 2 N=102). This stage was conducted by SG, blind to SLP who compiled the search terms. The sampling process was repeated for the Embase citations; duplicates removed n=4; evaluation set 1 N=103, set 2 N=102; validation set N=150.

Evaluation of the filters

The aim of the evaluation stage was to calculate the sensitivity of the filters against the gold standard evaluation sets [6], examine text and indexing terms for the gold standard studies that were not retrieved, and make any appropriate adjustments to the filters based on these results. The evaluation was first conducted in MEDLINE, and the final strategy subsequently adapted for Embase.

Validation of the filter

The aim of the validation was to calculate the sensitivity of the evaluated filters against the gold standard validation set. Our validation method is considered an internal gold standard [9]. In order to examine indexing, we also calculated sensitivity for selected subject headings we considered to be general to the social determinants of health.

Reduction in the number needed to screen

The reduction in the number needed to screen was calculated using search results from the motivating example systematic review in both the MEDLINE and Embase databases. This was defined as the percentage difference in numbers of un-deduplicated records retrieved when conducting the full search with, and without, the equity filter.

Use of equity filters

We calculated the proportion of all systematic reviews examined in this study that did not use equity filters. We estimated whether the use of filters changed over time with a non-parametric test for trend (‘nptrend’) using Stata V15 (Statacorp LLC).

RESULTS

Specific terms

Equity search term compilation

We located forty-nine equity-focused systematic reviews published in 2016 and 2017, of which the full text could be retrieved for forty-six (N=18 in 2017; N=28 in 2016). One of the 2017 reviews and five of the 2016 reviews did not filter for equity, leaving a pool of forty reviews potentially containing details of equity search strategies (N=17 in 2017; N=23 in 2016).

Example search strategies

Thirteen out of the seventeen 2017 reviews (76.4%) that used an equity filter provided an example search strategy for a specific database. A further two listed the concepts searched in the text, and a further two listed examples of keywords.

PROGRESS categories populated

Most of the reviews in the initial (2017) search (N=12, 70.6%) listed terms that mapped onto the category of socio-economic status; including, for example, the concepts of Social welfare, Socio-economic factors, deprivation, and the measures of poverty, social class, and income. Five reviews used terms or concepts that did not map onto PROGRESS; insurance, concepts of health disparities, the medically underserved, access and utilisation and we created a separate category for these. Unpopulated or sparsely populated categories were topped up with terms from nine 2016 reviews which contained such terms, and targeted searches located reviews to populate the sparsely populated categories of religion, and social capital; six of which provided terms. Potential equity terms were extracted from strategies reported in a total of 32 different reviews, with between three and 12 different strategies contributing to each PROGRESS category (Table 1).

Table 1 Categories populated by terms included in detailed search strategies

Modification of the compiled terms

After inspecting the MeSH browser we noted that the following subject headings were missing from our compiled terms and were added: Social Determinants of Health/; Health Equity/; Psychosocial Deprivation/; Social Stigma/; Working Poor/; Unemployment/; Sociological Factors/; Hierarchy, Social/; exp Educational Status/.

The following keywords and subject headings were considered too general to be of practical use in this filter and were removed: black; asian; Sex/; sex, Women/; women*; woman*; female*; gender; male; men; education; access*; unequal; poor; insurance; gross domestic product; gross national product; gdp; gnp; employment; occupation; profession; and exp Health (nested headings we deemed relevant were added). Terms related to spirituality were deemed irrelevant to disadvantage and removed (spiritual; spirituality; spiritu*). We removed the explosion term from Socioeconomic Factors/ as not all of the nested terms were relevant.

Each search term was entered one per line with terms combined under concepts, and concepts combined overall, by the Boolean ‘OR’ (Additional file 1).

Non-specific terms

A list of 30 terms and phrases (Additional file 1) relating to the reporting of equity papers were compiled from the abstracts, titles and keywords of the second evaluation set of papers and checked in the first. In the interest of specificity, terms or phrases that retrieved more than 200,000 results with the resulting loss of only 1-2% of evaluation papers were removed. These were: Risk Factors/; associat* with; were high* in; were low* in. Terms that retrieved fewer than 100 results from MEDLINE were also removed. Concepts were combined with the Boolean ‘OR’.

Gold standard sets

Most of the papers (66.9%) used in the deduplicated gold standard sets were those retrieved from 16 systematic reviews. The personal files method accounted for 20.9% of papers and the hand searching journal method for 12.2%. Publication dates for the articles in the validation sets ranged from 1985 to 2017 with a median publication date of 2009 in the MEDLINE validation set and 2010 in the Embase set.

Strategies in OVID

Following testing and modification in the first and then second evaluation sets, the final strategies were run on 02.02.2018 in OVID (MEDLINE, and then Embase). The strategies filtered out between 88-94% of all records (Table 2).

Table 2 Percentage of records filtered out

Validation results

The combined strategy had the highest sensitivity; over 0.90 in both databases (Table 3). The specific terms strategy had a sensitivity of 0.84 in MEDLINE and 0.87 in Embase. The non-specific terms strategy used alone had relatively low sensitivity.

Table 3 Sensitivity of finalised strategy against validation set

For the specific terms strategy, we sought to evaluate the extent to which studies were indexed on subject headings. As the denominator we counted only those papers published in the year following the introduction of the heading, or subsequent years. The subject heading most obviously related to the social determinants of health were established in MEDLINE and Embase in only 2014. We were unable to evaluate these headings because only six studies in our evaluation sets were published in or after 2015. None were indexed on the heading (Table 4). The second heading was related to health inequalities, established in Embase in 2009 and MEDLINE in 2008. Sensitivity was low; 0.18 in MEDLINE and 0.13 in Embase. We could not evaluate the effect of the subject heading ‘Health equity’ as it was only introduced in 2016 in both databases.

Table 4 Sensitivity of selected subject headings against validation set

Reduction in the number needed to screen

We added the filters (Boolean ‘AND’) to the search strategy developed for motivating example systematic review. This aimed to locate interventions reporting effects on mental health inequalities. The percentage reduction in the number needed to screen (NNS) are presented in Table 5. The specific terms strategy reduced the NNS by around 77% in both databases. The combined strategy reduced the NNS by 59.7% in MEDLINE, and 63.5% in Embase.

Table 5 Reduction in the number needed to screen

Use of equity filters

One hundred and eighteen equity-focused systematic reviews, located in MEDLINE and Embase, were examined in this study. Seventy-two (published in years 2014 & 2015) were located as a strategy to find gold standard records, and 46 (2016 & 2017) were located in order to compile search terms. Out of the 118 reviews, only 22 (18.6%) did not use an equity filter (Table 6). There was little evidence of change in the use of filters over the four publication years; z = -1.48, p=0.139.

Table 6 Use of equity filters in systematic reviews

Discussion

We aimed to develop and test a sensitive search filter to locate studies reporting equity outcomes in MEDLINE and Embase. A comprehensive strategy comprised of specific terms related to the social determinants of health filtered out 93-95% of all records, and had a sensitivity of 0.84 in MEDLINE, and 0.87 in Embase against a gold standard set of records. The sensitivity improved to 0.92 in MEDLINE (0.94 in Embase) when combined with a strategy comprising phrases often used to report equity study aims or findings. This combination met our criteria for a ‘good’ filter. The number needed to screen for the motivating systematic review was reduced by 77% by applying the specific terms strategy, and by 59.7% (MEDLINE) to 63.5% (Embase) by applying the combined strategy. Despite our study being the first to report the development and testing of an equity filter, we found that eighty-one per cent of systematic reviews published between 2014 and 2017 applied apparently unevaluated filters as part of their search strategy.

We used a combination of methods to collect gold standard papers; references from systematic reviews, personal files and hand searching journals. While combining these methods is recommended [9], , it may not eliminate bias caused by any one method. Accordingly, ur gold standard sample may not be fully representative of those in the corpus of literature, particularly studies where the inequality is not indexed and reported only in the body of the text. The majority of citations were identified from papers included in equity-focused systematic reviews that did not apply a filter; i.e. some of these studies may have been identified as reporting an equity outcome only after reading the whole article. Given that the most challenging papers to identify are those that provide few clues in the title, abstract, or keywords that they report equity outcomes, the over-inclusion of such studies improves the rigour of our gold standard set. However, studies contained in the reviews we examined may not be representative of the full corpus of literature and important studies may be missed [5]. Further, the reviews we located may not themselves fully represent the sample universe of equity-focused reviews [9]. We hand-searched only a limited number of issues in two journals, and records contained in personal files are unlikely to have broad coverage of the topic area. We used the same method to develop and measure the performance of our sample (internal validation). Validation of an external set, located by different methods than those used for the evaluation, would have strengthened our study [9] but this was beyond current resources. Although we did not formally assess reliability, two authors experienced in social epidemiology independently identified gold standard studies. There were few duplicates in the studies they identified. The selection and randomisation of gold standard studies into evaluation and validation sets was carried out independent of the author who compiled specific terms strategy, and prior to this stage being carried out. Gold standard study selection was also carried out blind to the search term compilation, minimising the risk of purposive sampling.

We used PROGRESS to help structure the specific terms strategy. Terms were mapped onto what we considered to be the most relevant concepts, but this process is subjective. Some terms cover more than one concept; for example ‘Social determinants of health’ apply to all categories and there is overlap between terms relating to SES and social capital, and differences in these between MEDLINE and Embase. We caution authors interested in filtering for a single concept to scrutinise other concepts for related terms, and emphasise that we did not estimate performance for single concepts. No -Plus terms were used for the specific terms strategy, which means that this filter may perform poorly if determinants such as age-related inequalities, disabilities or other personal characteristics are of interest. Similarly, terms such as ‘insurance’ are missing, which may disproportionately bias against US research.

Further reductions in the number needed to screen could be achieved by using text mining. Text mining techniques have been found to be useful for helping devise search strategies for complex topics and for helping to rank order records to help with screening large libraries of records. Text mining encompasses a range of statistical approaches to textual analysis and much of its value can lie in its automation and objectivity. Procedures to develop search strategies routinely using text mining approaches are available [42,43,44] and these tend to focus on frequency analysis of words and phrases within records. Another approach is to generate large libraries of records and then use text mining for study identification [45]. Text mining technologies can prioritise title-abstract records for manual screening, use active learning to improve the prioritisation as more records are screened and stopping rules. Given the complexity of this topic and the potential for large numbers of records, text mining could prove particularly useful in this area.

To our knowledge, our filters are the first to be formally assessed for performance; but consequent to this first attempt, they are basic and lengthy. Further development could include refining to improve sensitivity and specificity, screening all retrieved records for equity studies in order to calculate accuracy and precision, and validating against an external standard [9].

We surmise that there is a need for validated equity filters. We found the vast majority of equity-focused systematic reviews applied a filter, presumably to reduce the burden of screening for inclusion. There were, however, no reports that any equity strategy had been formally tested. Most of the search strategies in these reviews would have been compiled prior to guidance (not to filter) issued in 2015 [4], but it is unclear whether researchers will follow this guidance given the large, and ever increasing, numbers needed to screen. This is perhaps particularly true for reviews without substantial topic focus such as the effect of social determinants on a particular condition, or range of conditions, where a vast literature might be retrieved.

We chose to evaluate a broad social determinants of health filter, and consequently our study was not powered to determine the performance of the filter on, for example, studies that reported outcomes by gender, or employment status. We evaluated two subject heading terms that could be used by indexers to categorise equity studies more broadly, and not just on the determinant measured, but found very low levels of indexing. We suspect that this is due to a combination of two factors; the relatively recent introduction of these terms and a lack of familiarity of indexers with the concept of the social determinants of health, and, consequently, what equity studies look like. In particular, this under-indexing may be amplified in the many studies where equity is not the focus, but reported as a secondary outcome. Authors of studies that report equity outcomes (whether primarily or secondarily) could assist in this process by consistently suggesting the keywords ‘Social determinants of health’ and/or ‘Health inequality’ in their manuscript, along with the specific determinants measured. The Consolidated Standards of Reporting Trials (CONSORT) equity extension for randomised trials recommends trialists suggest their papers are indexed under the heading ‘Health equity’ [46]. The usefulness of indexing on any of these will only be realised for the inclusion of current and ongoing primary research in future reviews as all three of them have only recently been added as subject headings (2016 in the case of Health equity) and existing studies are not retrospectively re-indexed. The term ‘health equity’ is recommended to be included in titles of trials where appropriate [46]; we suggest that the word ‘equity’ is sufficient for this purpose and any of these phrases could be usefully incorporated into abstracts which would then be picked up with key word searches using the multi-purpose (.mp.) field.

Conclusions

The highest sensitivity in retrieving studies reporting outcomes by equity was a combined approach of a comprehensive list of specific terms and subject headings related to social and economic factors that produce health inequality, plus a list of non-specific terms related to the reporting of equity studies. At this early stage of equity filter development, if systematic reviewers wish to filter, this combined approach is recommended. Further efforts should concentrate on the standardisation of indexing for equity studies and additional development and testing of both specific and non-specific terms for accurate study retrieval.