- Research
- Open access
- Published:
Studies with statistically significant effect estimates are more frequently published compared to non-significant estimates in oral health journals
BMC Medical Research Methodology volume 23, Article number: 6 (2023)
Abstract
Background
Studies reporting statistically significant effect estimates tend to be more frequently published compared to studies reporting non-significant or equivalent estimates. Consequently, this may lead to distortion of the literature. The aim of this study is to assess the prevalence of reporting statistically significant effect estimates in leading oral health journals and to explore associations between the effect estimates and record characteristics.
Methods
An electronic database search was undertaken of a selection of leading oral health journals including general oral health journals to identify primary oral health records published in 2019. Descriptive statistics and population average GEE logistic regression model was used to assess associations between articles reporting a statistically significant effect estimate and the record characteristics.
Results
In 1335 records, 82.4% records reported a statistically significant effect estimate. All speciality journals compared to general oral health journals were less likely to publish a record with significant effect estimates. Authors based in Asia or other (OR 1.49; 95% CI :1.02,2.19; p = 0.037) were more likely to report significant effect estimates compared to those based in Europe. Interventional (OR 0.35; 0.22,0.58; p < 0.001) and observational (OR 0.56; 0.36, 0.89; p = 0.013) records were less likely to report significant effect estimates compared to in-vitro studies. Registered records were less likely to report significant effect estimates when compared to non-registered studies (OR 0.22; 95% CI :0.14,0.32; p < 0.001).
Conclusion
The publishing of records with significant effect estimates is prevalent within the oral health literature. To reduce dissemination bias and overestimation of effect sizes in systematic reviews, the publishing of studies with non-significant or equivalent effect estimates should be encouraged.
Introduction
It is well established that studies reporting statistically significant findings are more likely to be published compared to studies reporting non-significant or equivalent findings [1, 2]. Systematic non-dissemination of studies has been reported to distort the evidence base for decision making, lead to wastage of resources and potentially have major consequences on healthcare [3]. Dissemination bias is described as when the “dissemination profile of a study’s results depends on the direction or strength of its findings” [4]. Dissemination profile is defined as the “accessibility of research results, or the possibility of research findings being identified by potential users” [4]. Publication bias which to an extent contributes to dissemination bias occurs as a result of researchers failing to write up and submit their research findings [4]. However, the decision to submit for publication can also be influenced by sponsors, journal editors’ preference and both internal and external driven factors to fulfil academic goals [4]. Ultimately, the dissemination of research findings can be viewed as a biased process. However, its impact will depend on other variables [4]. This form of bias is not uncommon with indirect evidence suggesting publication bias occurs prior to dissemination of results to the scientific community [5]. Furthermore, increasing concerns have been raised that, most research results may represent false findings and not be reflective of the “truth” [6].
It has been suggested that the desire to report significant p-values is not the main driver of publication bias within the literature but it merely reflects the incentives to report positive effects [7]. Due to competition for funding and career progression, authors may avoid publishing studies with non-significant findings. Additionally, authors may assume their findings will be deemed to be not interesting and this could influence their willingness to publish [8]. Rejection rates and bias towards studies reporting negative or non-significant results is increasing [9, 10]. Conversely, journal editors may favour accepting studies with positive results as this could increase the journal citation metrics [11]. Furthermore, the drive to publish positive findings in journals with a high Impact Factor could be seen as favourable when to both publishers and advertisers and could strengthen the application for additional funding and income [11].
Within the medical literature the trend for reporting studies with positive results does not appear to be abating with yearly increases observed [12]. Previous studies have shown the reporting of significant results in dental speciality journals to range between 71.3 and 90% [13, 14]. However, these assessments were undertaken 5–10 years ago, leaving the question if a preponderance to publish studies with positive effects is still active given the increased awareness of the problem. Therefore, the aim of this study is to assess the prevalence of reporting statistically significant effect estimates in leading oral health journals and to explore associations between the effect estimates and record characteristics.
Methods
Eligibility criteria
A selection of leading oral health journals (general and speciality) with the highest impact factor as published in 2019 were included in this study. No other selection criteria were used. Records published in English were included. Case reports, review articles, editorials and systematic reviews were excluded. Similar to previous investigations, studies were categorized as (1) in-vitro, (2) interventional and (3) observational [13].
Search of oral health studies
An electronic database search was undertaken using Medline via PubMed (www.pubmed.ncbi.nlm.nih.gov) by one author (HK) in August 2020. Primary oral health records (in-vitro and in-vivo) published between 1st January 2019 and 31st December 2019 in the following journals were sourced: Journal of Dental research (JDR), Journal of American Dental Association (JADA), European Journal of Orthodontics (EJO), American Journal of Orthodontics and Dentofacial Orthopaedics (AJODO), Journal of Clinical Periodontology (JCP), Journal of Periodontology(JOP), Journal of Endodontics (JOE), International Journal of Oral and maxillofacial surgery (IJOMS), Journal of Oral and Maxillofacial Surgery (JOMS), Pediatric dentistry (PD), European Journal of Paediatric dentistry (EJPD), Journal of Prosthetic Dentistry (JPD) and Journal of Prosthodontics (JOPR). Field tags were not employed. The full record titles were searched without any language filters. The date limit function was used to identify records published within the study timeframe. The titles and abstracts of records (case reports, reviews, editorials, and systematic reviews) not meeting eligibility criteria were excluded during the screening process. All titles and abstracts were screened independently by 2 authors (HK and JS). Full-text records of abstracts fulfilling the inclusion criteria were retrieved and further analysed for eligibility independently by two authors (HK and JS). Any disagreements in the final records were resolved by discussion among the authors.
Data extraction
A pilot assessment of 10 random records was undertaken between two authors (HK and JS) to ensure consistency in data extraction variables. All record characteristics were extracted by a single author (HK) and entered into a pre-piloted Microsoft Excel® (Microsoft, Redmond, WA) data collection sheet. A second author (JS) independently cross-checked the collected data. Any discrepancies were resolved by discussion. At the level of each record the following characteristics was extracted: journal title, continent of corresponding author, journal impact factor, speciality of journal, study type (in-vitro, interventional, observational), ethical approval (approval obtained or not reported). When the relevant information was not reported it was assumed that no approval was required, or the project was exempt from approval), involvement of statistician (yes or no; inferred from author affiliations and materials and methods section), significance of effect estimate (based on primary outcome. In the absence of no clear primary outcome, the first outcome was analysed: significant or non-significant.), study registration (yes or no) and conflict of interest (yes conflict of interest is present/declared, or no conflict of interest is present/declared).
Statistical analysis
Descriptive statistics on the characteristics of the records were calculated. A population average univariable GEE logistic regression model with an exchangeable correlation structure was fit to assess associations between records reporting a statistically significant effect estimate and the record characteristics (independent variables). Estimates, corresponding 95% CIs and p-values were calculated. Significant predictors identified during the univariable analysis were entered in the multivariable model. The variable journal was used as the clustering unit. Statistical significance was set at 0.05 (2-sided). All statistical analyses were performed using STATA software version 16.1 (Stata Corporation, College Station, Texas, USA) and R Software version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).
Results
A total of 1335 records were included in this study (Fig. 1).
Study identification flow diagram
Over 82% (n = 1101) of the records reported a statistically significant effect estimate. Within this cohort, the highest number of records reporting a significant effect estimate were published in JDR (97.8%) with the lowest number published in EJO (67.9%). The range of reporting significant effect estimates (highest-lowest) for the following variables were: specialty journal (general oral health journals (96.8%)-orthodontics (73.7%), continent of corresponding author (Asia or other (85.3%)-Europe (76.5%) and study type (in-vitro (90.8%) - interventional (67.1%). In records reporting significant effect estimates, ethical approval was not reported (87.0%), did not involve a statistician (82.8%), were not registered (85.2%) and reported a conflict of interest (84.0%) (Table 1).
Distribution of articles by specialty of journal, and significance of effect estimate
In the multivariate analysis, all speciality records compared to general oral health types were less likely to publish a study with significant effect estimates. The continent of the corresponding author was associated with the odds of publishing an article with significant effect estimates, with authors based in Asia or other (OR 1.49; 95% CI :1.02,2.19; p = 0.037) being more likely to publish significant effect estimates compared to those based in Europe. In addition, interventional (OR 0.35; 0.22,0.58; p < 0.001) and observational (OR 0.56; 0.36, 0.89; p = 0.013) studies were less likely to report significant effect estimates compared to in-vitro studies. Finally, studies that were registered were less likely to report significant effect estimates when compared to non-registered studies (OR 0.22; 95% CI :0.14,0.32; p < 0.001) (Table 2).
Discussion
This study assessed the prevalence of reporting statistically significant effect estimates in leading oral health journals and examined associations between the direction of the results and record characteristics. In the study timeframe, 82.4% full-text records reported a statistically significant effect estimate confirming the apparent preponderance towards publishing positive results within the oral health literature. The odds of publishing significant effect estimate were also associated with the continent of the corresponding author, the record type and registration of the record. This result is comparable to previous investigations which reported the reporting of significant results in dental speciality journals to range between 47 and 86% [13] and 75–90% [14]. However, the same trend is not evident in the publishing of dental abstracts where the significance of the results does not predict the likelihood of publication [15]. Although, the current study was conducted after a significant time lag between previous investigation [13, 14], it appears the publishing of records with significant effect estimates still dominates and may have increased over this time period. Indeed, investigations within medical speciality journals have found a highly significant trend of reporting positive results increasing on a yearly basis by 6% [12].
There was variation in the percentage reporting of significant effect estimates between the leading oral health journals included in this study. In addition, all speciality journals compared to general oral health types were less likely to publish positive results. As previously postulated, this could be a reflection of differences in the study types published in each journal [13]. Articles published by authors based in Asia or other were more likely to publish records with positive findings compared to those based in Europe which mirrors the findings of previous studies [12, 13, 16]. This maybe reflective of the fact that trials carried out in developing countries are reported to show more positive findings compared to trials performed in developed countries [17]. Registration of trials is encouraged to improve transparency in the conduct of the study but also to eliminate publication and selective reporting bias [18, 19]. Interestingly, registration of records was less likely to be associated with the reporting of significant effect estimates when compared to non-registered records which may suggest that registration is having the desired impact. This is also supported by the finding that at the study level, interventional type studies which are encouraged to be registered are less likely of reporting positive findings compared to in-vitro studies. This corroborates the findings of a similar studies [13]. This really highlights the importance of correct interpretation of studies with a perceived weaker design as they are more likely to report exaggerated treatment effects whereas interventional studies such as Randomised Clinical Trials can contradict the findings reported by observational studies [20, 21].
The reproducibility of research study design is reported to be poor [22]. Conversely, if replication of study design can be achieved, the results of such studies are more likely to contradict the reported initial stronger results over time, independent of the study design [23]. If an improvement in research methodology is excluded, then reporting of a significant result could be dependent on other factors [24]. First of all, it could be the fact that hypotheses tested are true. However, this needs to be balanced with the fact that authors may be confirming known hypothesis in order to get “publishable’ results [6]. Authors who detect non-significant results decline to submit for publication or these results are turned into a positive direction through post-hoc analyses, selective reporting and reinterpretation [12]. The consequences of selective reporting or “p-hacking” where investigators carry out multiple statistical tests and then report only those which produce significant results has been highlighted [7].
Although every attempt was made to elicit the primary outcome from each record, when it was not obvious, the first outcome was analysed which introduces a degree of subjectivity and potential interpretation of misleading outcome results. The presence SPIN, where beneficial effects of an intervention are highlighted despite a non-significant difference detected between treatment interventions has been established in dental speciality trials [25,26,27]. Indeed, SPIN related to the focusing on significant within-group comparisons, focusing on a significant primary outcome when there are several co-primary outcomes and focusing on significant secondary outcomes has been reported in orthodontic trial abstracts [25]. To avoid any subjectivity, future assessments could review the registration record, published protocol, or duplicate publication, when the primary outcome is not specifically reported.
The selected in-vitro records also include animal studies and the only articles excluded were case reports, review articles, editorials and systematic reviews. The aim of our study was to assess the prevalence of reporting significance results in oral health journals and to see if this known problem still persists. The justification for the inclusion of in-vitro records is that they include an experiment, record an outcome and commonly include the results of statistical tests. Statistically significant effect estimates in in-vitro studies can influence decisions in conducting other similar or higher-level studies. In the analysis, it is also interesting to see that statistically significant records are more prevalent in in-vitro records compared to clinical studies. For example, there is evidence that RCTs which are usually the most rigorous studies have the lowest prevalence of significant results which may imply that in vitro studies may pass under the radar and thus more attention should be paid when interpreting the in vitro studies results apart from whether the results are generalizable due to the in-vitro setting.
Reasons for non-publication of records despite the strength of the findings include lack of time, incomplete study status, low priority and issues with co-authors [15]. However, researcher related factors are primarily cited [15, 28]. In the investigation of publication of abstracts following presentation at a biomedical conference, the most frequently cited reason by authors to not publish was a lack of time [29]. Methods to encourage publishing of non-significant findings could be suggested at the study ethical approval stage. For instance, ethics committees could suggest the reporting of the results of clinical studies regardless of the direction of the effect [30]. Mandatory registration of clinical trials, enforcing guidelines for accurate reporting and creating journals of negative results have also been suggested [4, 31,32,33,34]. Regarding the latter, such journals could be funded by public or charitable support [11]. Furthermore, trial funding agencies could make it a pre-requisite to publish both significant and non-significant findings of primary hypotheses tested [11]. The disclosure of funders or reporting of funding sources was not collected as a variable in this study. Future studies, should consider this characteristic and its relationship on the publication of studies with positive results, as trials with a high or unclear risk of sponsorship bias are reported to be associated with larger treatment effect size estimates [35].
Limitations
Searching of relevant records was conducted electronically rather than hand searching of journal issues. This decision was influenced by limitations regarding the access of hand copy journals within library institutions. Furthermore, records were only sourced from high impact oral health journals. Both these factors may have resulted in potential non-identification of potentially relevant records and selection bias. Although, the study timeframe was limited to twelve months only, we feel the total number of records included in this study represents a large enough sample to allow us to gauge the current issue of publication bias within the oral health literature [14]. Data extraction of the whole sample was primarily undertaken by a single author, but an initial calibration and cross-check for any discrepancies of the collected data by a second author has reduced errors in reporting and classification.
Conclusion
The publishing of records with significant effect estimates is prevalent within the oral health literature. To reduce dissemination bias and overestimation of effect sizes in systematic reviews, the publishing of records with non-significant or equivalent effect estimates should be encouraged. Methods to facilitate this include ethics committees or funding agencies insisting on the reporting of the results of records regardless of the direction of the effect, mandatory registration of clinical trials, enforcing guidelines for accurate reporting and creating journals with the remit of publishing non-significant effect estimates.
Availability of data and materials
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.
References
Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS ONE. 2013;8:e66844.
Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, et al. Publication bias in editorial decision making. JAMA. 2002;287:2825–8.
Joober R, Schmitz N, Annable L, Boksa P. Publication bias: what are the challenges and can they be overcome? J Psychiatry Neurosci JPN. 2012;37:149–52.
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess Winch Engl. 2010;14:1–193 iii, ix–xi.
Song F, Parekh-Bhurke S, Hooper L, Loke YK, Ryder JJ, Sutton AJ, et al. Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies. BMC Med Res Methodol. 2009;9:79.
Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:e124.
Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol. 2015;13:e1002106.
Calnan M, Smith GD, Sterne JAC. The publication process itself was the major cause of publication bias in genetic epidemiology. J Clin Epidemiol. 2006;59:1312–8.
Statzner B, Resh VH. Negative changes in the scientific publication process in ecology: potential causes and consequences. Freshw Biol. 2010;55:2639–53.
Lortie CJ, Dyer AR. Over-interpretation: avoiding the stigma of non-significant results. Oikos. 1999;87:183–4.
Joober R, Schmitz N, Annable L, Boksa P. Publication bias: what are the challenges and can they be overcome? J Psychiatry Neurosci JPN. 2012;37:149.
Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012;90:891–904.
Papageorgiou SN, Kloukos D, Petridis H, Pandis N. Publication of statistically significant research findings in prosthodontics & implant dentistry in the context of other dental specialties. J Dent. 2015;43:1195–202.
Polychronopoulou A, Pandis N, Eliades T. Assessment of publication bias in dental specialty journals. J Evid-Based Dent Pract. 2010;10:207–11.
Hua F, Walsh T, Glenny A-M, Worthington H. 30% of abstracts presented at dental conferences are published in full: a systematic review. J Clin Epidemiol. 2016;75:16–28.
Yuan JC-C, Shyamsunder N, Barao VAR, Lee DJ, Sukotjo C. Publication bias in five dental implant journals: an observation from 2005 to 2009. Int J Oral Maxillofac Implants. 2011;26:1024–32.
Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JPA. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ. 2013;346:f707.
Zarin DA, Keselman A. Registering a clinical trial in ClinicalTrials.gov. Chest. 2007;131:909–12.
Viergever RF, Ghersi D. The quality of registration of clinical trials. PLoS ONE. 2011;6:e14701.
Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286:821–30.
Tatsioni A, Bonitsis NG, Ioannidis JPA. Persistence of contradicted claims in the literature. JAMA. 2007;298:2517–26.
Ioannidis JPA. An epidemic of false claims. Competition and conflicts of interest distort too many medical findings. Sci Am. 2011;304:16.
Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294:218–28.
Fanelli D. “Positive” results increase down the Hierarchy of the Sciences. PLoS ONE. 2010;5:e10068.
Guo F, Fang X, Li C, Qin D, Hua F, He H. The presence and characteristics of “spin” among randomized controlled trial abstracts in orthodontics. Eur J Orthod. 2021;43:576–82.
Eleftheriadi I, Ioannou T, Pandis N. Extent and prevalence of spin in randomized controlled trials in dentistry. J Dent. 2020;100:103433.
Roszhart JI, Kumar SS, Allareddy V, Childs CA, Elangovan S. Spin in abstracts of randomized controlled trials in dentistry: A cross-sectional analysis. J Am Dent Assoc. 2019:26–32.e3.
Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992;267:374–8.
Scherer RW, Ugarte-Gil C, Schmucker C, Meerpohl JJ. Authors report lack of time as main reason for unpublished research presented at biomedical conferences: a systematic review. J Clin Epidemiol. 2015;68:803–10.
Scholey JM, Harrison JE. Delay and failure to publish dental research. Evid Based Dent. 2005;6:58–61.
Bian Z-X, Wu T-X. Legislation for trial registration and data transparency. Trials. 2010;11:64.
Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010;8:1–6.
Kundoor V, Mueen Ahmed KK. Uncovering negative results: introducing an open access journal “Journal of Pharmaceutical negative results. J Pharmacol Pharmacother. 2010;1:118–9.
Knight J. Negative results: null and void. Nature. 2003;422:554–5.
Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, Major PW, da Costa BR, et al. Influence of sponsorship bias on treatment effect size estimates in randomized trials of oral health interventions: a meta-epidemiological study. J Evid-Based Dent Pract. 2021;21:101544.
Acknowledgements
Not applicable.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Author information
Authors and Affiliations
Contributions
Conceptualization: Nikolaos Pandis and Jadbinder Seehra. Methodology: Nikolaos Pandis and Jadbinder Seehra. Validation: Jadbinder Seehra, Hadil Khraishi and Nikolaos Pandis. Formal analysis: Nikolaos Pandis. Investigation: Jadbinder Seehra and Hadil Khraishi. Writing— original draft preparation: Jadbinder Seehra, Hadil Khraishi and Nikolaos Pandis. Writing—review and editing: Jadbinder Seehra, Hadil Khraishi and Nikolaos Pandis. Visualization: Nikolaos Pandis and Jadbinder Seehra. Supervision: Nikolaos Pandis. The author(s) read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Seehra, J., Khraishi, H. & Pandis, N. Studies with statistically significant effect estimates are more frequently published compared to non-significant estimates in oral health journals. BMC Med Res Methodol 23, 6 (2023). https://doi.org/10.1186/s12874-022-01795-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12874-022-01795-3