Skip to main content
Download PDF

Restrictions and their reporting in systematic reviews of effectiveness: an observational study

BMC Medical Research Methodology volume 22, Article number: 230 (2022) Cite this article

Abstract

Background

Restrictions in systematic reviews (SRs) can lead to bias and may affect conclusions. Therefore, it is important to report whether and which restrictions were used. This study aims to examine the use of restrictions regarding language, publication period, and study type, as well as the transparency of reporting in SRs of effectiveness.

Methods

A retrospective observational study was conducted with a random sample of 535 SRs of effectiveness indexed in PubMed between 2000 and 2019. The use of restrictions and their reporting were analysed using descriptive statistics.

Results

Of the total 535 SRs included, four out of every ten (41.3%) lacked information on at least one of the three restrictions considered (language, publication period, or study type). Overall, 14.6% of SRs did not provide information on restrictions regarding publication period, 19.1% regarding study type, and 18.3% regarding language. Of all included SRs, language was restricted in 46.4%, and in more than half of the SRs with restricted language (130/248), it was unclear whether the restriction was applied during either the search or the screening process, or both. The restrictions were justified for publication period in 22.2% of the respective SRs (33/149), study type in 6.5% (28/433), and language in 3.2% (8/248). Differences in reporting were found between countries as well as between Cochrane and non-Cochrane reviews.

Conclusions

This study suggests that there is a lack of transparency in reporting on restrictions in SRs. Authors as well as editors and reviewers should be encouraged to improve the reporting and justification of restrictions to increase the transparency of SRs.

Peer Review reports

Background

Systematic reviews (SRs) are becoming increasingly popular in the health sector [1, 2] and are considered the gold standard for summarising the available evidence in clinical research [3]. Traditionally, SRs examine the effectiveness of interventions or therapies [4]. Despite the fact that review methodology has evolved over the years with regard to address a broader range of questions, SRs on effectiveness are still the most common type of SRs [1, 4]. Another development over recent years has been the rapid increase in the number of SRs [1, 2, 5], which may have led to research waste [5, 6]. To avoid this, it is important for published SRs to be of high methodological quality.

Since the aim of SRs is to summarise all available evidence on a topic, it is recommended that they do so regardless of year of publication, study type, language, or other restrictions [7,8,9,10]. However, due to lack of time or financial resources, this is not always possible. Language restrictions are particularly frequent [1, 2, 11] and are thus of great relevance and frequently discussed [12, 13]. Furthermore, insufficient language specification in the database and technical limitations regarding language restriction functions in some of the databases can lead to problems when restricting the language during the search process in SRs [7]. In addition, restrictions regarding publication date and study type may increase the potential for systematic bias [7].

Clear reporting is a prerequisite for assessing the methodological quality of SRs and can reduce the risk of bias [14]. Thus, high reporting quality is necessary to identify high quality SRs. To support authors of SRs focusing on randomised controlled trials, the QUOROM statement was published in 1999 [15]. In 2009, it was updated and extended, resulting in the PRISMA statement [16]. Advances in recent years led to a further update of the PRISMA statement in 2020 [17]. Both the original QUOROM and the PRISMA statements explicitly describe the importance of reporting whether SRs apply any restrictions and why. For language restrictions, it is of further importance to know how and where they were used as this can have an impact on search results as well as on reproducibility [7, 12]. Therefore, Pieper and Puljak [12] suggest that the search step in which the language restriction was applied should always be reported, with preference given to restrictions implemented at the full-text screening level. While some literature addresses the transparency of reporting in SRs [1, 2, 5], little research considering restrictions in SRs has been conducted so far.

Therefore, the aim of this research was to examine the use of restrictions regarding language, publication period, and study type as well as transparency of reporting in SRs of effectiveness.

Methods

This retrospective observational research is based on a previous study [1] for which a protocol is available from the Open Science Framework (https://osf.io/pxjrv/).

In brief, the aim of the previous study was to examine trends in the epidemiology and reporting of published SRs over the past 20 years. SRs indexed between 2000 and 2019 were included. SRs were defined according to the PRISMA-P guideline [18]. To be included, SRs had to provide a clearly stated set of objectives, a systematic search that attempts to identify all studies meeting the eligibility criteria, methods that state the search strategy (or keywords used) and study selection (e.g., eligibility criteria and selection process) as well as a systematic presentation and synthesis of the characteristics and findings of the included studies. Due to limited resources, SRs published in languages other than English were excluded. Using a search strategy from Montori et al. [19], which was adapted according to Moher et al. [20], PubMed was searched on 1 January 2020. The search returned 572,871 results, from which a random sample of 4,000 articles was selected. After titles and abstracts were screened by two independent reviewers (JH, KA, TR, AH), 1,687 articles, one of which was not available, were included in the full-text screening. This led to the exclusion of 555 articles that were not classified as SRs. Finally, 1,132 SRs were included in the original study [1], of which 535 were SRs of effectiveness.

Data extraction

For the purposes of the current analysis, these 535 SRs on effectiveness were included. Besides the data already extracted in the previous study [1] (journal, publication year, country of corresponding author, number of authors, number of database searched, number of articles included in SRs, meta-analysis performed, number of studies included in the meta-analysis, named article as SR, date of search, assessment of validity, and language restriction), three additional reporting characteristics (reference to PRISMA, full search strategy available, and flow chart available) and six items on the restrictions used (publication period reported, publication period justified, study type reported, study type justified, language restriction justified, and point of language restriction) were considered. Regarding the justification of restrictions, it was recorded which justification the author provided, if any; reasonableness was not assessed in this context. More detailed information on the items can be found in Table 1.

Table 1 Operationalisation of the items extracted
Full size table

As the aim of this research was to examine transparent reporting in SRs, authors were not contacted for additional information. Appendices were checked for the search strategy or the flow chart only if they were explicitly referred to in the text.

Data extraction was done by one reviewer (JH). Plausibility checks were performed, and a random sample of 20% of all records was verified by one of the co-authors (DP, TM, TR, KA, and FH).

Data analysis

Reporting characteristics were analysed and presented using descriptive statistics with a focus on restrictions applied to SRs, including publication period, study type, and language. Frequencies and proportions were calculated for categorical data, and means with standard definitions (SDs) and medians with interquartile ranges (IQR) were calculated for continuous data. Based on the findings from the predecessor study [1], which indicate that the epidemiology and reporting of SRs differ by 5-year period (2000–2004, 2005–2009, 2010–2014, and 2015–2019), country of the corresponding author, and depending on whether the SR was a Cochrane review (defined as a publication in the Cochrane Database of Systematic Review), data were analysed using the same stratifications. Since most of the items considered were explicitly mentioned in the PRISMA checklist [16], which was first published in 2009, a sensitivity analysis was performed for the years 2010–2019. All statistical analyses were conducted with SAS for Windows version 9.4 (SAS Institute INC, Cary, North Carolina).

Results

Baseline and reporting characteristics

A total of 535 SRs from 43 different countries were included, with the highest proportion of corresponding authors located in the USA (16.6%), China (16.5%), and the UK (13.5%). On average, 5.2 authors were involved, and 78.9% of the included SRs searched at least three databases. The mean number of included studies was 20.9. Most SRs (65.8%) performed a meta-analysis, of which 95.7% reported the number of included studies, with a mean of 9.0 (Table 2).

Table 2 Total frequencies of epidemiological and reporting characteristics in SRs of effectiveness indexed in MEDLINE (via PubMed), 2000–2019
Full size table

Of all included SRs, 68.8% described themselves as SR in the title, abstract, or methodology, 31.1% referred to the PRISMA guideline, and half of the SRs (55.0%) reported a full search strategy (Table 3). The search date was provided by 89.4% of SRs. An assessment of validity was reported in 77.1%. A flow chart was provided in 77.3% of the SRs.

Table 3 Differences in reporting characteristics and the use of restrictions between Cochrane and Non-Cochrane reviews indexed in MEDLINE (via PubMed), 2000–2019
Full size table

Restrictions used in SRs

In terms of the restrictions used, 14.6% of the SRs did not specify the publication period, 57.6% included all studies regardless of publication date, and 27.9% applied a restriction (Table 3). Of those SRs restricting their publication periods (149/535), a justification was provided in 22.2% (33/149). The most common reasons for a restricted search period were the attempt to include only recent evidence (27.3%, 9/33), to update a review (21.2%, 7/33), or to reflect the start of scientific relevance (21.2%, 7/33).

Whether restrictions regarding the included study types were applied was not reported in 19.1% of all SRs. In 40.0% of SRs, only randomised controlled trials (RCTs), and in 40.9%, both RCTs and nonrandomized studies of interventions (NRSIs) were included. Where reported (433/535), the selection of study types was justified in 6.5% of SRs (28/433). The main reasons for restrictions were an attempt to include only the highest level of evidence (32.1%, 9/28) or a lack of evidence (32.1% 9/28).

Regarding the language included, 18.3% did not report whether a language restriction was used, 35.3% included all studies regardless of language, and in 46.4%, a restriction was used. Of all SRs that restricted language (248/535), 3.2% provided a justification. All but one SR (87.5%, 7/8) justified language restrictions by limited resources or linguistic capabilities. In one SR (12.5%, 1/8), the restriction was justified by the fact that the main journals in the field are published in English. In 52.4% of all SRs with language restriction (130/248), it was unclear when this step was applied, 20.6% restricted the search strategy, and 27.0% did this during screening. Of all SRs restricting the language in the screening process (67/248), 43.3% (29/67) reported the number of studies excluded due to language, with a mean of 4.6 (SD 5.7, median 2 with a mean of 25.8 included articles, SD 32.3, median 15).

When considering all three restrictions (publication period, study type, or language), 41.3% of all SRs lack information on at least one of them (Table 3).

Comparisons of cochrane and non-cochrane reviews

When comparing Cochrane (n = 81) and non-Cochrane reviews (n = 454), differences in reporting and use of restrictions were found (Table 3). Cochrane reviews described themselves less frequently as an SR (23.5% vs. 76.9%), never referred to the PRISMA guideline (0% vs. 36.6%) but were more likely to report the full search strategy (76.5% vs. 51.0%) and search date (97.5% vs 87.9%). Cochrane reviews were less likely to present a flow chart (43.2% vs. 83.4%) but more often provided an assessment of validity (97.4% vs. 73.4%).

Considering the restrictions used, 22.2% of Cochrane reviews and 13.2% of non-Cochrane reviews did not provide any information about the publication period. Overall, 9.9% of Cochrane reviews and 31.1% of non-Cochrane reviews restricted the publication period. While most Cochrane reviews (62.5%, 5/8) justified this restriction, only 19.9% (28/141) of non-Cochrane reviews did so. The type of studies included was reported in all Cochrane reviews versus 77.5% of non-Cochrane reviews. Most Cochrane reviews (56.8%) included only RCTs (37.2% of non-Cochrane reviews).

About every fifth Cochrane review (21.0%) and non-Cochrane review (17.8%) provided no information on whether a language restriction was used. While 4.9% of Cochrane reviews were restricted by language, this proportion was much higher in non-Cochrane reviews (53.7%). In the SRs that were restricted (4/81 Cochrane reviews; 244/454 non-Cochrane reviews), 25.0% of Cochrane reviews and 2.9% of non-Cochrane reviews justified the selected language. More Cochrane than non-Cochrane reviews reported on the step in the research process at which the restriction was used.

Considering all three restrictions, 35.8% (29/81) of Cochrane reviews and 42.3% (192/454) of non-Cochrane reviews lack information on at least one of the restrictions (publication period, study type, or language).

Differences by country of corresponding author

When stratifying by country of corresponding author, some differences in the reporting of restrictions become apparent (see Additional File 1). Articles with corresponding authors from China restricted the publication period in 15.9% of SRs (14/88), whereas corresponding authors from all other countries restricted the publication period more often, ranging from 26.5% (9/34) in the Netherlands to 38.2% (34/89) in the USA. While SRs from the USA and Australia often did not specify which study type was included (30.3%, 27/89 and 27.8%, 15/54, respectively), 13.6% (12/88) SRs from China lacked information on study type. The highest prevalence of language restrictions was found in SRs from the USA (67.4%, 60/89) and the lowest in SRs from the UK (30.6%, 22/72). Overall, the proportion of articles not reporting at least one of the restrictions considered ranged between 35.2% (19/54) for SRs from Australia to 48.3% (43/89) for those from the USA. A total of 37.5% (33/88) of SRs with a corresponding author from China failed to report at least one restriction (publication period, study type, or language).

Comparison over time

Looking at the results over time, some trends in the reporting can be identified (Table 4). An increasing number of SRs refer to the PRISMA checklist, named the article as a SR, and presented a flow chart. Some trends are also apparent in the reporting of restrictions. While between 2000 and 2004, one in four SRs (28.0%, 7/25) lack information on the publication period, between 2015 and 2019, only 12.0% (39/324) do so. The proportion of SRs only including RCTs decreased from 60% (15/25) between 2000 and 2004 to 35.5% (115/324) between 2015 and 2019. Regarding language restrictions, on the one hand, the number of SRs reporting no information decreased over time, while the number of SRs reporting a language restriction increased from 24.0% (6/25) between 2000 and 2004 to 52.8% (171/324) between 2015 and 2019. Overall, the number of SRs failing to report at least one of the restrictions considered (publication period, study type, or language) decreased over the years, from 60% (15/25) in 2000–2004 to 40.4% (131/324) in 2015–2019. The proportion of SRs with justifications did not change over the years for any of the three characteristics.

Table 4 Reporting characteristics and the use of restrictions in SRs of effectiveness indexed in MEDLINE (via PubMed) by years
Full size table

The sensitivity analysis for the years 2010 to 2019 supports the trend of improvements in reporting. Overall, 25.0% (13/52) of Cochrane reviews and 41.7% (170/408) of non-Cochrane reviews lack information on at least one of the restrictions (see Additional File 2).

Discussion

In this study, which analysed a random sample of 535 SRs of effectiveness, we found that 41.3% of all SRs failed to report at least one of the considered restrictions (publication period, study type, or language). Specifically, 14.6% did not report the publication period, 19.1% gave no information about the study type included, and 18.3% did not report the language of studies included. Almost half of the SRs restricted the language of studies included, and for 52.4% of those, it was not clear at which step of the research process the restriction was applied. Only very few SRs provided a justification for the restrictions used. Cochrane reviews more often reported whether restrictions by publication period and study type were done, but in 20% of both Cochrane reviews and non-Cochrane reviews, information on language restrictions was lacking. Furthermore, Cochrane reviews less often presented a flow chart and described their article as SR.

Almost half of all considered SRs restricted the language of studies included. This is a slightly higher proportion than indicated by a range of older studies [2, 5] but quite similar when stratified by year. At first glance, it appears that more SRs restrict their language today than in the past. At the same time, however, reporting has improved over the years, and the proportion of SRs not providing information on language restrictions has decreased. Presumably, rather than reflecting changes in the frequency of applied language restrictions, the observed changes reflect improved reporting. There is conflicting evidence on whether language restrictions increase the risk of language bias [12, 13, 21, 22] and thus might hamper conclusions. Nevertheless, studies show that the likelihood of being published, read, or cited is associated with the language of publication [23,24,25] and that the influence of language restrictions on the pooled effect estimates can vary between different areas of healthcare research [26, 27]. This complex relationship underlines the need for clear reporting. Pieper and Puljak [12] agreed with the general recommendation [8,9,10, 16] not to use any language restriction while emphasising that if it is necessary for a good reason (e.g., time or financial resources), the restriction should be applied in the screening process rather than, e.g., during the search. Doing so not only increases transparency but also gives an impression of the relevance of non-English language studies in a research area. Furthermore, it enables other researchers to take up and expand on the study results [7, 12]. However, half of SRs restricting the language did not clearly report at which step of the research process the restriction was applied, and only 27.0% explicitly stated that they restricted the language in the screening process. Where reported, about 5 full text articles were excluded due to language, indicating that non-English language studies might be of importance. For transparency, it is important to not only justify the language restriction, but also to provide the number of excluded studies.

As our results suggest, the reporting of language restrictions needs further improvement. While, the QUOROM statement from 1999 [15] and the PRISMA guideline from 2009 [16] both recommend to report language restrictions [28], in PRISMA 2020 [17], restrictions can only be found in the expanded checklist. This document might be used less frequently because only the core PRISMA 2020 checklist has been translated into other languages, is available as a fillable word document and might be a requirement for submission by journals. Authors should also specify at which step of the research process the language was restricted and how many articles were excluded based on language, as this is not yet included in PRISMA.

While PRISMA 2009 [16] additionally recommended presenting both a flow chart and the full search strategy, we found that 77.1% of included SRs provided a flow chart, and only 55.0% presented a full search strategy for at least one database. It should be noted that PRISMA was introduced in 2009 [16], but the recommendation to show a flow chart was already given in the QUOROM statement from 1999 [15]. On the one hand, most of the reporting-related aspects we assessed were recommended during the whole study period. On the other hand, we found some trends over time, especially for the presentation of a flow chart after PRISMA was introduced. Interestingly, 36.1% of SRs included between 2010 and 2019 referred to the PRISMA statement, while a closer look at the SRs included between 2000 and 2009 reveals that hardly any of them referred to the QUOROM statement. This is supported by previous research indicating that journals have more frequently included the PRISMA guideline in their instructions to authors [29, 30] and that author compliance has increased [31, 32].

Furthermore, QUOROM [15] as well as PRISMA 2009 [16] requires that restrictions regarding publication period and study type must also be reported and justified. Nevertheless, almost 15% of SRs did not report the publication period, and few of them justified the restriction used. As the SRs included in our study showed a high degree of heterogeneity in terms of where and how they stated the publication period, it was not always possible to identify whether a restriction was applied. In addition, the date of intercept for one database varied between SRs. Looking at the type of studies included in SRs, it is notable that most SRs included only RCTs, which are generally considered to be of higher internal validity [33, 34]. However, we found that over the years, the diversity of study types included in SRs has increased, i.e. that fewer SRs include only RCTs. Some studies suggest that NRSIs may be of increasing importance for some areas of research [35, 36].

In accordance with other studies [2, 37,38,39], we found that aspects considered determinants of methodological quality were more frequently fulfilled by Cochrane reviews than by non-Cochrane reviews. While almost every Cochrane review reported the date of search (97.5%) and provided an assessment of validity (97.4%), only 87.9% and 73.4%, respectively, of non-Cochrane reviews did so. In addition, a full search strategy is available in three quarters of all Cochrane reviews compared to half of non-Cochrane reviews. This is in line with the strict requirements of the Cochrane Handbook [10] and the Methodological Expectations of Cochrane Intervention Reviews (MECIR) [40]. However, MECIR is strongly oriented towards the PRISMA guidelines [16] and is consistent with it in many aspects. While the Cochrane Handbook [10] encourages authors to illustrate the flow of studies through the selection process according to PRISMA [16], less than half of all Cochrane reviews (vs. 83.4%) present a flow chart. This proportion however increased after 2010 for Cochrane reviews (see Additional File 2). Furthermore, it was striking that in some Cochrane reviews (12.3%), the search strategy was not stated and instead, reference was made to a Cochrane search group or a Cochrane register. This approach complicates reproducibility and is inconsistent with the Cochrane Handbook, which requires the presentation of a full search strategy for at least one database. Some other studies indicate that in Cochrane reviews, the authors’ approach deviates from the recommendations of the Cochrane Handbook [41, 42]. Overall, however, few studies are available which compare the methodological approach and reporting of Cochrane reviews versus non-Cochrane reviews. This is important because peculiarities of Cochrane reviews might lead to deficient retrieval [38, 43].

Strengths and limitations

To our knowledge, few studies focusing on restrictions in SRs exist. General strengths and limitations of the methodological approach have been described in previous research [1]. Including only SRs of effectiveness on the one hand increased homogeneity of the sample. On the other hand, it was sometimes difficult to clearly distinguish these SRs from other types of reviews, which may have led to misclassification. Also, due to limited resources, only English-language articles were included, which may have had an impact on the use of restrictions, especially regarding language. Data were extracted by one reviewer only. However, 20% of the extracted data were checked by a second independent reviewer to ensure methodological quality and to reduce the risk of bias. Additionally, plausibility checks were executed. Despite various efforts, a clear assignment of the restrictions used was difficult for some SRs. When extracting the type of studies included in SRs, a lack of consistency was found regarding the operationalisation of clinical trials. Further, it was not always possible to clearly determine whether a language restriction was used because the supplementary material (such as the search strategy or flow chart) was not accessible for some articles. In most cases, this was due to a missing or incorrect link, inconsistency between journals (some offer only a free full text, others offer only a free appendix, and some always have both available for free), or the problem that the appendix was not available on interlibrary loan. Although most of the recommendations we assessed were already mentioned in the QUOROM statement from 1999, some were first introduced in 2009 with PRISMA. However, more than 85% of included SRs were published between 2010 and 2019, and we also presented analyses stratified by years (see Table 4 and Additional File 2).

Conclusion

We found that there is a lack of transparency in the reporting of restrictions in SRs of effectiveness. Four out of ten SRs lack information on at least one of the three restrictions considered (publication period, study type, and language). Overall, 20% do not provide information on language restrictions, and among the remaining SRs, many fail to report the step of the research process at which the restriction was applied. Only a small number of SRs justified the restriction used. Furthermore, we also found weaknesses in the reporting of Cochrane reviews. Authors as well as editors and reviewers of all SRs should be encouraged to improve the reporting of restrictions. Explicitly addressing the reporting of restrictions in the main PRISMA checklist in the future may assist authors in doing so.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Hoffmann F, Allers K, Rombey T, Helbach J, Hoffmann A, Mathes T, et al. Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000–2019. J Clin Epidemiol. 2021;138:1–11. https://doi.org/10.1016/j.jclinepi.2021.05.022.

    Article  PubMed  Google Scholar 

  2. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016;13(5):1–30. https://doi.org/10.1371/journal.pmed.1002028.

    Article  Google Scholar 

  3. Aromataris E, Pearson A. The systematic review: An overview. Am J Nurs. 2014;114(3):53–8. https://doi.org/10.1097/01.NAJ.0000444496.24228.2c.

    Article  PubMed  Google Scholar 

  4. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should i conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):1–9. https://doi.org/10.1186/s12874-017-0468-4.

    Article  Google Scholar 

  5. Ioannidis JPA. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016;94(3):485–514. https://doi.org/10.1111/1468-0009.12210.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Page MJ, Moher D. Mass Production of Systematic Reviews and Meta-analyses: An Exercise in Mega-silliness? Milbank Q. 2016;94(3):515–9. https://doi.org/10.1111/1468-0009.12211.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Aali G, Shokraneh F. No limitations to language, date, publication type, and publication status in search step of systematic reviews. J Clin Epidemiol. 2021;133:165–7. https://doi.org/10.1016/j.jclinepi.2021.02.002.

    Article  PubMed  Google Scholar 

  8. Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AK, Hammerstrøm K, et al. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Campbell Syst Rev. 2017;13(1):1–73. https://doi.org/10.4073/cmg.2016.1.

    Article  Google Scholar 

  9. Aromataris E, Munn Z. JBI Manual for Evidence Synthesise. JBI. 2020. Available from: https://synthesismanual.jbi.global.

  10. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). 2021. Available from: www.training.cochrane.org/handbook.

  11. Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Syst Rev. 2017;6(1):1–14. https://doi.org/10.1186/s13643-017-0527-2.

    Article  Google Scholar 

  12. Pieper D, Puljak L. Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. J Clin Epidemiol. 2021;132:146–7. https://doi.org/10.1016/j.jclinepi.2020.12.027.

    Article  PubMed  Google Scholar 

  13. Nussbaumer-Streit B, Klerings I, Dobrescu AI, Persad E, Stevens A, Garritty C, et al. Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study. J Clin Epidemiol. 2020;118:42–54. https://doi.org/10.1016/j.jclinepi.2019.10.011.

    Article  CAS  PubMed  Google Scholar 

  14. Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34. https://doi.org/10.1016/j.jclinepi.2015.06.005.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.

    Article  CAS  Google Scholar 

  16. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ. 2009;339(7716):332–6. https://doi.org/10.1136/bmj.b2535.

    Article  Google Scholar 

  17. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):1–15. https://doi.org/10.1371/JOURNAL.PMED.1003583.

    Article  Google Scholar 

  18. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. https://doi.org/10.1186/2046-4053-4-1.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: Analytical survey. Br Med J. 2005;330(7482):68–71. https://doi.org/10.1136/bmj.38336.804167.47.

    Article  Google Scholar 

  20. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):447–55. https://doi.org/10.1371/journal.pmed.0040078.

    Article  Google Scholar 

  21. Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of english-language restriction on systematic review-based meta-analyses: A systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44. https://doi.org/10.1017/S0266462312000086.

    Article  PubMed  Google Scholar 

  22. Neimann Rasmussen L, Montgomery P. The prevalence of and factors associated with inclusion of non-English language studies in Campbell systematic reviews: A survey and meta-epidemiological study. Syst Rev. 2018;7(1):1–12. https://doi.org/10.1186/s13643-018-0786-6.

    Article  Google Scholar 

  23. Walpole SC. Including papers in languages other than English in systematic reviews: important, feasible, yet often omitted. J Clin Epidemiol. 2019;111:127–34. https://doi.org/10.1016/j.jclinepi.2019.03.004.

    Article  PubMed  Google Scholar 

  24. Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350(9074):326–9. https://doi.org/10.1016/S0140-6736(97)02419-7.

    Article  CAS  PubMed  Google Scholar 

  25. Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19(2):159–66. https://doi.org/10.1016/S0197-2456(97)00150-5.

    Article  CAS  PubMed  Google Scholar 

  26. Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: Empirical study. Int J Epidemiol. 2002;31(1):115–23. https://doi.org/10.1093/ije/31.1.115.

    Article  PubMed  Google Scholar 

  27. Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769-776.e2. https://doi.org/10.1016/j.jclinepi.2004.08.021.

    Article  PubMed  Google Scholar 

  28. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339. doi:https://doi.org/10.1136/bmj.b2700

  29. Tao K, Li X, Zhou Q, Moher D, Ling C, Yu W. From QUOROM to PRISMA: A Survey of High-Impact Medical Journals’ Instructions to Authors and a Review of Systematic Reviews in Anesthesia Literature. PLoS ONE. 2011;6(11): e27611. https://doi.org/10.1371/JOURNAL.PONE.0027611.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Tam W, Lo K, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2). doi:https://doi.org/10.1136/BMJOPEN-2016-013905

  31. Page MJ, Moher D. Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review. Syst Rev. 2017;6(1). doi:https://doi.org/10.1186/S13643-017-0663-8

  32. Gundogan B, Dowlut N, Rajmohan S, et al. Assessing the compliance of systematic review articles published in leading dermatology journals with the PRISMA statement guidelines: A systematic review. JAAD. 2020;1(2):157–74. https://doi.org/10.1016/J.JDIN.2020.07.007.

    Article  Google Scholar 

  33. McKenzie J, Brennan S, Ryan R, Thomson H, Johnston R, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis.le. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 62 (updated February 2021). Cochrane; 2021. Available from: www.training.cochrane.org/handbook

  34. Canadian Task Force on the Periodic Health Examination. The periodic health examination. Can Med Assoc J. 1979;121(9):1193–254.

    Google Scholar 

  35. Reeves BC, Deeks J, Higgins J, Shea B, Tugwell P, Wells G. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 62 (updated February 2021). Cochrane; 2021. Available from: https://training.cochrane.org/handbook.

  36. Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, et al. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4(1):49–62. https://doi.org/10.1002/jrsm.1078.

    Article  PubMed  Google Scholar 

  37. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: A quality paradigm? Eur J Orthod. 2013;35(2):244–8. https://doi.org/10.1093/ejo/cjs016.

    Article  PubMed  Google Scholar 

  38. Goldkuhle M, Narayan VM, Weigl A, Dahm P, Skoetz N. A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer. BMJ Open. 2018;8(3). doi:https://doi.org/10.1136/bmjopen-2017-020869

  39. Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30. https://doi.org/10.1016/j.jclinepi.2008.09.018.

    Article  PubMed  Google Scholar 

  40. Higgins J, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, et al. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. 2021.

  41. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009;97(1):21–9. https://doi.org/10.3163/1536-5050.97.1.004.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Propadalo I, Tranfic M, Vuka I, Barcot O, Pericic TP, Puljak L. In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane’s Handbook guidance. J Clin Epidemiol. 2019;106:10–7. https://doi.org/10.1016/j.jclinepi.2018.10.002.

    Article  PubMed  Google Scholar 

  43. Pieper D, Hoffmann F. Retrieving Cochrane reviews is sometimes challenging and their reporting is not always optimal. Res Synth Methods.2022;1–4.doi: https://doi.org/10.1002/jrsm.1564

Download references

Acknowledgements

We would like to thank Amrei Hoffmann (AH) for her help in screening and data extraction.

Funding

Open Access funding enabled and organized by Projekt DEAL. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Health Services Research, School of Medicine and Health Sciences, Carl Von Ossietzky University Oldenburg, Oldenburg, Germany

    Jasmin Helbach, Katharina Allers & Falk Hoffmann

  2. Institute for Research in Operative Medicine, Faculty of Health, School of Medicine, Witten/Herdecke University, Cologne, Germany

    Dawid Pieper, Tim Mathes & Tanja Rombey

  3. Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Institute for Health Services and Health System Research, Rüdersdorf, Germany

    Dawid Pieper

  4. Center for Health Services Research, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany

    Dawid Pieper

  5. Institute for Medical Statistics, University Medical Centre Göttingen, Göttingen, Germany

    Tim Mathes

  6. Department of Health Care Management, Technische Universität Berlin, Berlin, Germany

    Tanja Rombey

  7. Leibniz Institute for Prevention Research and Epidemiology – BIPS, Bremen, Germany

    Hajo Zeeb

  8. Health Sciences Bremen, University of Bremen, Bremen, Germany

    Hajo Zeeb

Authors
  1. Jasmin Helbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dawid Pieper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tim Mathes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tanja Rombey
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hajo Zeeb
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Katharina Allers
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Falk Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JH, FH, KA, and DP designed the study. JH screened and extracted the data. DP, TM, TR, KA, and FH verified the data. JH and FH analysed and interpreted the data. JH wrote the first version of the manuscript. All authors have critically revised the article and have read and approved the final version of the article to be submitted.

Corresponding author

Correspondence to Jasmin Helbach.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no potential conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

 eTable 1. Reporting characteristics and the use of restrictions in SRs of effectiveness indexed in MEDLINE (via PubMed) by country of corresponding author, 2000-2019

Additional file 2:

 eTable 2. Differences in reporting characteristics and the use of restrictions after the publication of PRISMA in 2009 (via PubMed)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Helbach, J., Pieper, D., Mathes, T. et al. Restrictions and their reporting in systematic reviews of effectiveness: an observational study. BMC Med Res Methodol 22, 230 (2022). https://doi.org/10.1186/s12874-022-01710-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-022-01710-w

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us