- Research article
- Open Access
- Open Peer Review
Are the Cochrane group registers comprehensive? A case study of Japanese psychiatry trials
BMC Medical Research Methodologyvolume 2, Article number: 6 (2002)
Language bias is a form of publication bias and constitutes a serious threat to meta-analyses. The Cochrane Controlled Trials Register is one attempt to remedy this and now contains more than 300,000 citations. However we are still unsure if it provides comprehensive coverage, particularly for non-English trials.
We have recently established a comprehensive register of Japanese trials of psychotropic drugs through extensive personal contacts, electronic searches and handsearches. We examined two Cochrane psychiatry group registers against this Japanese database.
The Japanese register contained 56 reports of randomized controlled trials (RCTs) of antidepressants for depression but the Cochrane Depression, Anxiety and Neurosis group register contained 18, with an overlap of only nine. The Japanese register contained 61 reports of RCTs of neuroleptics for schizophrenia and the Cochrane Schizophrenia group register contained 36, with an overlap of only six. Taking account of some duplicate publications, only a quarter to a third of all relevant Japanese RCTs were retrievable from the Cochrane group registers.
Similar, or worse, yields may be expected with RCTs conducted in other East Asian countries, and in other fields of medicine. What evidence there is suggests that this situation may lead to a systematic over estimate of treatment effect.
Publication bias represents an Achilles' heel in meta-analyses. A particular form of publication bias is the so-called language bias, by which randomised controlled trials (RCTs) with greater estimates of effect size tend to be published in English rather than in the original authors' native languages . This bias is compounded by Anglophone databases and journals considerably under representing the totality of RCTs. For example, more than half of trials published in a Hungarian journal, identified through a cover-to-cover hand search, were not reported in MEDLINE . Electronic searches of non-English databases may yet not suffice: a mono-lingual English searcher would be able to identify only 73% of the trials in the Spanish/Portuguese LILACS  and 70% of the German PSYNDEX .
Attempts are underway to remedy this situation. Two notable examples are the Medical Editors Trial Amnesty  and the Cochrane Controlled Trials Register (CCTR). The latter now contains more than 300,000 citations. Despite these enormous efforts we still are unsure if any database provides comprehensive coverage, particularly for non-English trials. We have recently established a comprehensive register of Japanese trials of psychotropic drugs. We therefore examined how successful the Cochrane psychiatry groups have been in identifying trials conducted in Japan and reported in Japanese by comparing the yields of the groups' specialized registers against this recently created "gold standard."
Three of the authors (TI, AI, SN) have established a comprehensive register of Japanese RCTs of psychotropic drugs. First, we requested all the pharmaceutical companies marketing a psychotropic drug in Japan (n = 20) to provide them with the references concerning their RCTs with Japanese participants. Second, we carefully searched the on-line bibliographic database, JMEDICINE (Jan 1981–June 2000). This contains reports of medical literature in Japanese. Thirdly, we also collected references to RCTs through the "reference search" of the identified reports. Since on-line JMEDICINE is not available before 1980, and even the best electronic search is likely to miss a considerable proportion of reports of trials, these authors made considerable efforts to hand search psychiatric and medical journals likely to contain reports of psychotropics trials (n = 35). Abstracts of academic meetings have not been included; neither have RCTs undertaken in Japan but published in the English language journals.
We then asked two Cochrane groups in mental health, namely the Cochrane Schizophrenia Group and the Depression, Anxiety and Neurosis Group, to search their specialized registers for reports of trials possibly conducted in Japan. We employed two ways to identify such trials. The first strategy was to use the search term "Japan*" (*is a wild card). Because we were not sure if all candidates contained a word "Japan*," we employed the second strategy in which we searched for all the authors' names appearing in the Japanese register.
The Japanese register contained 56 reports of RCTs of antidepressants for depressed people but the CCDAN register contained 18, with an overlap of only nine. It was possible to retrieve five more studies as the CCDAN register contained six English language duplicate publications of the five original Japanese RCTs. The remaining three studies found in the CCDAN register but not in the Japanese database were English reports of RCTs conducted in Japan but reported only in the English language. Searching the CCDAN register would therefore have identified a maximum of only 25% (=14/56, 95%CI: 15 to 38%) of all Japanese reports of relevant RCTs.
The CSG register contained 36 reports of RCTs of neuroleptics given to Japanese people with schizophrenia. The Japanese register contained 61. Of these, only six had been registered in the CSG register; 13 more studies were possible to retrieve as the CSG register contained 18 English language duplicate publications of original Japanese RCTs. The remaining 12 reports found in the CSG register but not in the Japanese register were reports of RCTs conducted in Japan but published only in the English journals. Searching the CSG register only would have identified 19 or only 31% ( = 19/61, 95%CI: 21 to 44%) of all relevant reports in the Japanese language.
Discussion and conclusion
The comprehensiveness of the collaborative databases is now deemed all the more important because doubts have been expressed as to effectiveness of traditional statistical tools to examine publication bias such as funnel plots . However, searching the Cochrane group registers would have missed 69% to 75% of all Japanese reports of RCTs of antidepressants or neuroleptics. These results clearly demonstrated the limitations of the current Cochrane group registers, despite their extensive searching (information available from the CCDAN and CSG modules in the Cochrane Library). Similar, or worse, yields may be expected with RCTs conducted in other East Asian countries (Korea, China), and in other fields of medicine.
Many systematic reviews, at least in the field of mental health, are therefore likely to present results grossly underestimating research output from Asia. What evidence there is suggests that this will result in a systematic over estimate of effect . Efforts must be expended for non-Anglophone countries to prepare their own register of RCTs for all health interventions in humans. The Japanese database of psychiatry trials has now been merged with the Cochrane group registers.
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G: Language bias in randomised controlled trials published in English and German. Lancet. 1997, 350: 326-329. 10.1016/S0140-6736(97)02419-7.
Bereczki D, Gesztelyi G: A Hungarian example for handsearching specialized national healthcare journals of small countries for controlled trials. Is it worth the trouble?. Health Libr Rev. 2000, 17: 144-147. 10.1046/j.1365-2532.2000.00280.x.
Soares KVS, Gill D, Adams CE: Searching non-Anglophone databases II: LILACS. In. IVth Cochrane Colloquium. 1996
Adams CE, Langer M, Neumann D: Searching non-Anglophone databases I: developing a search strategy for PSYNDEX. In. IVth Cochrane Colloquium. 1996
Horton R: Medical editors trial amnesty. Lancet. 1997, 350: 756-
Tang JL, Liu JL: Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol. 2000, 53: 477-484. 10.1016/S0895-4356(99)00204-8.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/2/6/prepub
TAF, TI and CEA participated in the design of the study and in its coordination. TI, CEA, HM, AI and SN participated in the collection of the datasets and in their analyses. TAF and CEA drafted the manuscript and all the authors participated in the rewriting. All authors read and approved the final manuscript.