Table 1 Characteristics of included methodological studies investigating publication bias in HSR

Ammenwerth, 2007 [16] (Health Informatics)

To determine:

- what percentage of IT evaluation studies are not published in international journals or proceedings

- what are typical reasons for not publishing the results of an IT evaluation study

Written, e-mail-based survey of academics. Survey sample included members of several mailing lists and first authors of IT evaluation papers that were published between 2001 and 2006 and Medline indexed

Only half of the evaluation studies reported by responders were published. Common reasons for non-publication included ‘not of interest for others’, ‘no time for writing’, ‘limited scientific quality’, ‘political and legal reasons’ and ‘only meant for internal use’

Low response rate (19%, 136/722). Study could be influenced by sampling, response and recall bias

Costa-Font, 2013 [18] (Health Policy)

To examine the winner’s curse phenomenon (studies needing to have more extreme results to be published in high-impact journals) and publication selection bias using quantitative findings on income and price elasticities as reported in health economics research

Funnel plot and multivariate analysis to examine the association between estimated effect sizes (and their statistical significance) and the impact factors of the journals in which they are published

Meta-regression analysis demonstrated that both publication bias (reflected by positive correlation between effect size and standard error) and the winner’s curse (reflected by an independent association between effect size and journal impact factor) influence the estimated income/price elasticity

Alternative explanations for the observed associations cannot be excluded.

Literature in the field concerned are often reported in grey literature rather than academic journals

Machan, 2006 [17] (Health Informatics)

To determine:

- the percentage of evaluation studies describing positive, mixed or negative results

-the possibility of statistical assessment of publication bias in health informatics

- the quality of reviews and meta-analysis in health informatics with regard to publication bias

Descriptive analysis of random sample of 86 evaluation studies and planned to construct funnel plot

Examined characteristics and quality of reviews and meta-analyses (n = 54) in medical informatics

For the primary studies, 69.8% had positive results, 14% negative and 16.3% unclassified

For the reviews 36.6% had positive conclusion, 61.5% were inconclusive, and only one review came to a negative conclusion

Small number of comparable studies prevented the quantitative analysis of potentiation publication bias

Proportion of studies/reviews with a positive conclusion may not be good indicators for the existence of publication bias

Vawdrey, 2013 [19] (Health Informatics)

To measure the rate of non-publication and assess possible publication bias in clinical trials of electronic health records

Follow-up of health informatics trials registered in ClinicalTrials.gov (2000–2008)

Trials with positive results were more likely to be published compared with trials with null results (92% of trials with positive results [35/38] vs 75% of trials with neutral or negative results [12/16], but the difference was not statistically significant (p = 0.177^a)

Sample size relatively small; no information could be obtained for 8 unpublished trials; completeness of trial registration and representativeness of registered uncertain