BMC Medical Research Methodology BioMed Central Correspondence

Background It is often stated that external validity is not sufficiently considered in the assessment of clinical studies. Although tools for its evaluation have been established, there is a lack of awareness of their significance and application. In this article, a comprehensive checklist is presented addressing these relevant criteria. Methods The checklist was developed by listing the most commonly used assessment criteria for clinical studies. Additionally, specific lists for individual applications were included. The categories of biases of internal validity (selection, performance, attrition and detection bias) correspond to structural, treatment-related and observational differences between the test and control groups. Analogously, we have extended these categories to address external validity and model validity, regarding similarity between the study population/conditions and the general population/conditions related to structure, treatment and observation. Results A checklist is presented, in which the evaluation criteria concerning external validity and model validity are systemised and transformed into a questionnaire format. Conclusion The checklist presented in this article can be applied to both planning and evaluating of clinical studies. We encourage the prospective user to modify the checklists according to the respective application and research question. The higher expenditure needed for the evaluation of clinical studies in systematic reviews is justified, particularly in the light of the influential nature of their conclusions on therapeutic decisions and the creation of clinical guidelines.


Background
Publication bias is defined as "the tendency to publish research results based on the strength and direction of findings" [1]. It is well documented that studies with nonsignificant or negative results are substantially less likely to be submitted for publication [2,3]. In addition, the proportion of published articles from low income countries is lower than more developed nations in many research fields, including psychiatry [4], cardiovascular disease [5] and epidemiology [6]. A recent survey concluded that researchers from less-developed countries believe that one likely reason is a substantial editorial bias against their work [7]. The tendency of editors to publish research results based on study characteristics other than quality indicators is called generally manuscript selection bias in the current study, which is broader than the publication bias definition. The purpose of present study was to compare the methodological quality and statistical appeal of published trials from countries with different developmental status and to determine their association with the journal impact factor and language of publication.

Selection criteria
Based on the World Bank income criteria countries were divided into low, lower-middle, upper-middle and high income groups [8]. All records of clinical trials conducted in each income group during 1993 and 2003 were included if they contained abstract and the number of study participants (study sample size) could be calculated from the abstract content.

Search strategy
Cochrane Central Register of Controlled Trials (2005, issue1) was searched for all trials including the names of countries in each income group in the Institution field of the records. Fifty articles in each income group were selected using a systematic random sampling method in years 1993 and 2003 separately, which comprised of the total of 400 citations. All the citations identified by the above searches were screened and animal studies, non interventional studies or conference abstracts were excluded by consensus; then the sampling was performed once more to replace the excluded citations.

Data extraction
Data were extracted by two reviewers on the language of publication, use of randomization, blinding (masking, placebo or sham), intent(ion) to treat analysis, study sample size and statistical significance. An article was classified as non-English if it was described in the bibliographic details. Quality assessment was restricted to the statement of randomization, blinding and intention to treat analysis. The results of a trial was called significant when there was at least a p < 0.05, the 95% confidence interval excluded no effect, or the statistical significance was stated qualitatively in the results section for any finding of the study. Disagreement was dealt with by discussion. Information on journal impact factor was obtained from the 1994 and 2003 editions of the Science Citation Index Journal Citation Reports.

Statistical analysis
The Chi square for trend test was applied to the cross-tab of randomization (or blinding, significance) and country income to assess the differences between levels of country income.
Forward stepwise linear regression was performed to model the relation of country income to journal impact factor. We included the following potential confounders as covariates: language of publication, study sample size, randomization, blinding, statistical significance and publication year.
As the journal impact factor was not normally distributed, we performed logarithmic transformation. We dichotomized the country income (High income = 1, others = 0), publication year (2003 = 1, 1993 = 0), language of publication (English = 1, others = 0), use of randomization, use of blinding and statistical significance. All analyses were undertaken in SPSS version 11.5 (SPSS, Inc., Chicago, IL). A p value of less than 0.01 was considered statistically significant.

Results
Four hundred records were explored. There were 377(94.3%) English language articles. According to the title, abstract or keywords the number of Controlled clinical trials which refer to the studies that compare one or more intervention groups to one or more comparison (control) groups was 384(96%). The number of randomized controlled trials was 298(74.5%), before and after studies was 14(3.5%) and the frequency of crossed-over studies was 39(9.7%). Double-blind and single-blind methods were mentioned in 141(35.3%) and 39(9.8%) of records respectively. The loss to follow-up percent was stated in 29 abstracts and 5 studies reported an intentionto-treat analysis.

Country income
In 1993 the results of 68% of high income trials and 64.7% of other groups were statistically significant; but in 2003 they were 66% and 82% respectively (table 1). Whilst in low income countries the probability of blinding was increased from 36% in 1993 to 46% in 2003, in high income countries there was about fourteen percent decrease. Country income had a non-significant inverse association with the presence of randomization (Chi square for trend = 5.6, p = 0.02) and a direct significant association with the use of blinding (Chi square for trend = 6.9, p = 0.008). As shown in table 2 in 1993 the articles from high income countries had an odds ratio of 1.17 for having statistically significant results compared with the other countries, after adjustment for the language of publication. The odds ratio decreased to 0.41 in the year 2003. There were 2 studies in high-middle income group and 3 in low-middle income group reported an intention to treat analysis.

Journal impact factor
In forward stepwise linear regression model, fitted on the log impact factor the country income, the language of publication, study sample size and the publication year were independent predictors of journal impact factor which remained in the model (table 3). The final model accounted for 14.7% of the variance in journal impact factor and was statistically significant (F = 12.2, p < .0001).

Discussion and conclusion
The frequencies of blinding and statistical appealing indicators in the published articles from low income nations have increased more than developed countries during the past ten years and the presence of randomization had a negative association with country development in both occasions. On the one hand, this finding may be due to an enhancement of exclusive and biased use of critical appraisal checklists by the editors of western medical journals. On the other hand, it may refer to the developing world authors, who might be more keen to selectively report the studies that are larger, have less serious limitations, and containing positive and significant findings in international english language journals, because of the presumption that editors and reviewers are biased against their nationality [9,10].
The poorer countries are underrepresented both on the editorial boards and in the pages of medical journals, which stated as a systematic bias against the diseases of poverty [11].
Some aspects of this bias is demonstrated in the studies, that the majority of the editorial board members of international medical journals come from nations with a high human development index [12,13] and the studies enrolling some participants from the United States are more likely to be published [14]. In our study the effect of country development on the odds of the statistical significance of published studies was considerable after adjustment for the language.
We didn't find a significant relationship between methodological quality indicators of the research and the impact factor of the medical journal that cited it. Conversely, study sample size and country income were more impor-  tant predictors of journal impact factor, despite the generally accepted belief that the journal impact factor could be used as a measure of journal quality [15,16].
Consequently we can interpret the result as an alarm sign that the impact of country development on the manuscript selection bias is increasing by time. Publication bias is a preventable problem [17] and we suggest that "Developing country bias" should be intentionally stated as an independent factor which can contribute to biased inclusion of research manuscripts for publication; Publication bias awareness raising programs should be planned for developing world researchers; In addition, current standards for reporting trials should be enforced more equally on the articles from different nations [18].
We used The Cochrane Central Register of Controlled Trials (CENTRAL) as the source of studies which serves as the most comprehensive source of records related to controlled trials. As of January 2003, CENTRAL contained just over 350,000 citations to reports of trials and other studies potentially relevant to Cochrane Reviews. CENTRAL includes citations to reports of controlled trials that might not indexed in MEDLINE, EMBASE or other bibliographic databases; citations published in many languages; and other sources that are difficult to access [19].
Present study has several limitations. First, the main problem with our study is the lack of a denominator i.e. the study presents no data on the total number of papers submitted only on those papers published. If only very small numbers of papers are published then this introduces a bias so that it is not possible to make assumptions about the quality of papers as a whole. Second, all assessments are done based on the data provided in title, abstract and keywords and obviously several useful information might be omitted. Abstracts from higher impact journals might contain more information and therefore yield higher quality scores than those from other types of journal. If so, the real differences between low and high income countries may be more than what is reported in our study. Poor (or even fraudulent) documentation and adoption of CONSORT statement from journals may affect the results. Another important constraint of relying on the abstract is that the failure to report a feature, e.g. blinding in the abstract does not necessarily equate with failure to carry it out. Third, we limit our investigation to clinical trials and may underestimate the real heterogeneity among different income groups. Fourth, data extractors were not blinded to country and journal. We reduced its effects by considering the most objective characteristics of the studies that could be extracted by the abstracts. Fifth, the country income has been presumed as a surrogate for developmental status. Finally, all studies included in the present study were cited in Cochrane Central Register of Controlled Trials which serves as the most comprehensive source of records related to controlled trials, but mainly consists of the articles indexed in MEDLINE and EMBASE. There is a proven language bias in these databases [20] and may limit the external validity of our results.
Because of the above mentioned drawbacks we suggest to interpret the results of the present study with more caution.