The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004
© Galandi et al; licensee BioMed Central Ltd. 2006
Received: 05 March 2006
Accepted: 06 July 2006
Published: 06 July 2006
In order to reduce systematic errors (such as language bias) and increase the precision of the summary treatment effect estimate, a comprehensive identification of randomised controlled trials (RCT), irrespective of publication language, is crucial in systematic reviews and meta-analyses. We identified trials in the German general health care literature.
Eight German language general health care journals were searched for randomised controlled trials and analysed with respect to the number of published RCTs each year and the size of trials.
A total of 1618 trials were identified with a median total number of 43 patients per trial. Between 1970 and 2004 a small but constant rise in sample size from a median number of 30 to 60 patients per trial can be observed. The number of published trials was very low between 1948 and 1970, but increased between 1970 and 1986 to a maximum of 11.2 RCTs per journal and year. In the following time period a striking decline of the number of RCTs was observed. Between 1999 and 2001 only 0.8 RCTs per journal and year were published, in the next three years, the number of published trials increased to 1.7 RCTs per journal and year.
German language general health care journals no longer have a role in the dissemination of trial results. The slight rise in the number of published RCTs in the last three years can be explained by a change of publication language from German to English of three of the analysed journals.
Approximately more than 10,000 randomised controlled trials (RCTs) – widely acknowledged as the gold standard for the evaluation of medical interventions – are published each year. Systematic reviews of RCTs, such as those disseminated through the Cochrane Collaboration, synthesise the results of individual trials to facilitate the use of external evidence in clinical decision making. The process of systematic reviewing is designed to maximise validity. This is accomplished with a series of steps to minimise bias, such as a comprehensive literature search. Indeed, the exhaustive identification of trials, irrespective of the language of publication, seems important to reduce systematic errors like language bias. Studies investigating the influence of language bias reveal controversial findings. For the comparison of German and English-language trials, a significant trend towards publication of statistically significant results in English-language journals was identified . However, reanalysis of published meta-analysis including both English and non-English language trials found no consistent change in the summary estimates of meta-analyses when excluding non-English language trials. These findings, however, seem to depending on the medical specialty. In complementary medicine several meta-analyses resulted in significantly smaller summary result estimates when excluding non-English language trials [2–4]. Therefore, without consistent empirical evidence supporting the exclusion of non-English language trials, the set of eligible trials for systematic reviews and meta-analysis should be as comprehensive as possible and should therefore not preclude the inclusion of trials because of language reasons.
In order to support reviewers the Cochrane Collaboration organised an international project to manually search the health care literature from many countries for RCTs. The trials identified are added to the Cochrane Controlled Trials Register and thus made available to reviewers.
Eight German language general health care journals (and their predecessors) were manually searched for the years 1948 to 2004: Deutsche Medizinische Wochenschrift, Journal of Molecular Medicine (formerly Klinische Wochenschrift), Medizinische Klinik, Medizinische Welt, Münchener Medizinische Wochenschrift, Swiss Medical Weekly (formerly Schweizerische Medizinische Wochenschrift), Wiener Medizinische Wochenschrift, Zeitschrift für Allgemeinmedizin.
The sample sizes of the identified trials were extracted and analysed in groups of three consecutive calendar years.
1618 RCTs were published as original articles with a median total number of patients of 43 (quartiles 20 and 87). Only 19.3% of the identified trials included more than 100 patients. The number of published trials was very low between 1948 and 1970 and increased between 1970 and 1986 to a maximum of 11.2 RCTs per journal and year (figure 1). In the following time period a striking decline of the number of RCTs was observed. Between 1999 and 2001 only 0.8 RCTs per journal and year were published on average. In the last analysed time period (2002–2004) an increase in the number of published trials can be observed: 1.7 trials per journal and year were published.
Personal communication with editors of the journals resulted in a fairly uniform explanation of this trend: The widespread introduction of using journal impact factors for the allocation of resources has strongly motivated researchers to publish in higher impact journals. As these are almost entirely in English, German general health care journals now rarely receive RCTs for publication.
Concerning the trial size, a slight but steady increase of the number of patients per trial can be observed. The median number of patients per trial rose from 30 in the years 1969 to 1971 to 60 between 2002 and 2004.
The number of RCTs published in the German language general health care literature has declined dramatically between 1986 and 2001. Between 1999 and 2001 eight major German language general health care journals published only 20 RCTs.
This observation confirms developments in other general health care journals, but is in clear contrast to high impact international general and specialist journals where a constant number of RCTs has been observed over past decades [5–7].
The slight increase of the number of published trials in the years 2002 to 2004 can (partially) be explained by changes of publication language from German to English in three of the analysed journals (more than 50% of the 39 RCTs published between 2002 and 2004 were in English language).
The sample size of trials identified in German language general health care journals was small with an increase in the more recent volumes. Although the trial size increased, it remains even in recent years rather small with a median number of 60 patients per trial. The average sample size of RCTs published in German appears to be comparable to other general health care journals and specialist journals, however, trials published in international journals of high impact are considerably larger [5, 7]. The proportion of RCTs with 100 or more patients published in the BMJ and the Annals of Internal Medicine in the years 1995–1997 was 76% compared to 19.3% in the investigated German language journals .
The observed decline of RCTs published in German language general health care journals raises the question whether the number of trials is declining or whether trials nowadays are more frequently not published or published elsewhere. Currently German ethics committees do not know the exact number of approved RCTs because their records do not distinguish between different study designs. We can only speculate whether an increasing proportion of these trials is never published, or where these trials have been published in recent years. What can be concluded is that the leading general health care journals in the German language area have abandoned their role in the dissemination of RCTs. However, the change of the publication language from German to English (as in the Schweizerische Medizinische Wochenschrift – now swiss Medical Weekly) resulted in a contrasting development with an increase of the number of published RCTs.
Allocation of funding resources and the scientific career depend to a great extent on the value of the scientific output (measured by impact factor points) of an institution or an individual researcher. The result is an increasingly fierce competition for publication in high impact journals, with high rejection rates, influencing the chance to get manuscripts published in due time. German-language RCTs, and those from comparable non-English language countries, face far more obstacles to publication than a decade ago when they were easily submitted to German-language or other non-English language journals, with a good chance of being accepted for publication. It is unclear how these changes in dissemination patterns affect the validity of systematic reviews and meta-analyses, and, generally, the transfer of knowledge from research into practice. Prospective trial registration, frequently stipulated but not implemented universally, would allow tracking trials from ethics committee approval to publication and help prevent under- and overreporting . This study adds another piece of evidence to the urgent need for more transparency in the dissemination process, as it is intended by the present initiative of WHO . In this initiative WHO has taken the lead to establish a standardized, global framework for the prospective registration of clinical trials as e.g. required by the International Committee of Medical Journal Editors (ICMJE) to reduce biases in the reporting and publication process of clinical trials and thus improve the base for decision making in health care .
randomized controlled trial
We would like to thank our students and colleagues who have contributed to the handsearching of the journals.
EU-program BIOMED I, contract: BMH4-CT98-3803; Federal Ministry of Health and Social Security; Federal Ministry of Education and Research
- Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G: Language bias in randomised controlled trials published in English and German. Lancet. 1997, 350: 326-9. 10.1016/S0140-6736(97)02419-7.View ArticlePubMedGoogle Scholar
- Juni P, Holenstein F, Sterne J, Bartlett C, Egger M: Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002, 31: 115-23. 10.1093/ije/31.1.115.View ArticlePubMedGoogle Scholar
- Pham B, Klassen TP, Lawson ML, Moher D: Language of publication restrictions in systematic reviews gave different results depending on wether the intervention was conventional or complementary. J Clin Epidemiol. 2005, 58: 769-76. 10.1016/j.jclinepi.2004.08.021.View ArticlePubMedGoogle Scholar
- Moher D, Pham B, Klassen TP, Schulz KF, Berlin JA, Jadad AR, Liberati A: What contributions do languages other than English make on the results of meta-analyses?. J Clin Epidemiol. 2000, 53: 964-72. 10.1016/S0895-4356(00)00188-8.View ArticlePubMedGoogle Scholar
- McDonald S, Westby M, Clarke M, Lefebvre C, Cochrane Centres' Working Group on 50 Years of Randomized Trials: Number and size of randomized trials reported in general health care journals from 1948 to 1997. Int J Epidemiol. 2002, 31: 125-7. 10.1093/ije/31.1.125.View ArticlePubMedGoogle Scholar
- McDonald S, Lefebvre C, Antes G, Galandi D, Gotzsche P, Hammarquist C, Haugh M, Jensen KL, Kleijnen J, Loep M, Pistotti V, Ruther A: The Contribution of Handsearching European General Health Care Journals to The Cochrane Controlled Trials Register. Eval Health Prof. 2002, 25: 65-75. 10.1177/0163278702025001005.View ArticlePubMedGoogle Scholar
- Thornley B, Adams C: Content and quality of 2000 controlled trials in schizophrenia over 50 years. BMJ. 1998, 317: 1181-4.View ArticlePubMedPubMed CentralGoogle Scholar
- Dickersin K, Rennie D: Registering clinical trials. JAMA. 2003, 290: 516-23. 10.1001/jama.290.4.516.View ArticlePubMedGoogle Scholar
- Evans T, Gülmezoglu M, Pan T: Registering clinical trials: an essential role for WHO. Lancet. 2004, 363: 1413-4. 10.1016/S0140-6736(04)16136-9.View ArticlePubMedGoogle Scholar
- De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van der Weyden MB: Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004, 351: 1250-1. 10.1056/NEJMe048225.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/6/30/prepub