Quality of reporting according to the CONSORT, STROBE and Timmer instrument at the American Burn Association (ABA) annual meetings 2000 and 2008

Inclusion criteria

A total number of 511 abstracts presented at the American Burn Association annual meeting in 2000 (259 abstracts) and 2008 (252 abstracts) were screened according to the quality of reporting. According to evidence-based medicine, the study design (randomized controlled trial, observational study, case series, experimental study) was determined. Clinical studies with the known study design according to the level of evidence were discriminated from experimental studies. In abstracts without mentioned study design the study design was evaluated based on the method part, prospective, retrospective, follow-up method, intervention, data gathering method and description of the results.

A RCT was defined if the abstract described a prospective study in which individuals are allocated at random to an intervention or to a controll group. A porspective or retrospective study that involves identification of two groups (cohorts) of patients, one which did receive the exposure of interest, and one which did not, and following these cohorts forward for the outcome of interest wa s defined as cohort study. A retrospective study that compared the exposure of individual in two groups, one of which had the condition under consideration and one which did not" was definined of a case control study. Video studies was defined as those which descibed a evaluation of a video. Case report was defined as reporting of an individual or on a series of patients with no control group, intervention or statistical analysis. Systematical review was defined as a literature review focused on a single question which tries to identify, appraise, select and synthesis all high quality research evidence relevant to that question. If the study describes purely about a surgery technique without control group it was defined as a describing op technique. If the study was conducted with any kind of animals, tissue or cells it was defined as an experimental study. In the study desing was still unlcer or did not match with the definition above, abstracts were re-analysed by the researchers (UY and KK) together. The type and method of intervention was discussed and it was searched for words like placebo, blinded, random, placebo, questionare, lab techniques (PCR, western/southern/northern-blot), odds ratio, relative risk to analyse the study design.

For each RCT or observational study conference abstract, the score for CONSORT abstract or STROBE was calculated by two independent researchers (UY and KK). The scores of each abstract were compared and the mean were used when the data varies. Additionally the timmer instrument was calculated for each abstract and by inconsistencis again the mean was used. Experimental studies were not analysed for reporting quality.

Both researchers were blinded for journal year, oral/poster, author and institution. The mean score of both investigators was taken. Interpreter variability was calculated by t-test (p = 0.873) and Pearson correlation (0.761).

CONSORT criteria for abstract [7] (additional file 1) were applied for randomised-controlled trial abstracts and STROBE criteria (additional file 2) for observational studies. Scores were calculated for 17 CONSORT and 22 STROBE criteria. One point was given if the abstract meets the items proposed in the criteria and zero points if not met (CONSORT score range: 0-17, STROBE score range: 0-22). All individual criteria were weighted equally.

In addition to the CONSORT and STROBE score the Timmer quality scoring instrument (additional file 3) was applied for all abstracts. The Timmer instrument consists of 19 items as a quality index which was developed for the evaluation of scientific meeting abstracts in a standardized way. We modified the score calculation to focus on the reporting quality of the abstracts. Therefore we excluded the study design in the scoring system and 0-1 points were given for each item (1 if met, 0 if not met). For the item which is not applicable, such as controll group (item 5), randomisation (item 7) or blinding (item 8, 9) of subjects in observational studies, the points were subtracted from the score. Therefore for randomized controlled trials the score range was 0-19 points and for observational studies 0-15 points.

Outcome measures

The study design (randomized controlled trial, non-randomized trial, experimental, prospective, retrospective), 17 CONSORT and 22 STROBE criterias for reporting in conference abstracts, and the Timmer instrument for abstract quality were determined.

Statistics

Descriptive statistics consisted of the calculation of frequencies and percentages. T-test was used to calculate and compare oral vs poster according to CONSORT and STROBE score in 2000 and 2008 respectively. The Chi-square tests were applied for assessment of the improvement between each CONSORT and STROBE criteria in 2000 and 2008. Fisher exact tests were used in the analysis of categorical data where sample sizes were small. Data were analyzed using the SPSS statistical software package Version 14.0.

Evidence-based study design of American Burn Association annual meeting abstracts

CONSORT criteria and Timmer quality instrument for reporting randomised-controlled trials in annual meeting abstracts

Reporting quality of RCT was for oral vs. poster abstracts in 2000 (CONSORT; oral: 7.2 ± 0.8 vs. poster: 7 ± 0) and 2008 (CONSORT; oral: 7.2 ± 1.4 vs. poster: 6.5 ± 1, Timmer; 9.7 ± 1.1 vs. 9.5 ± 0.7) Figure 1). The Timmer instrument for RCTs in ABA 2000 meeting were 7.8 ± 0.7 (oral) and 7.5 ± 0.6 (poster,), respectively. In 2008 the Timmer instrument were 9.7 ± 1.1 for oral and 9.5 ± 0.7 for poster presentations (p = 0.819, CI -1.79 to 2.19, Figure 2). There were no significant reporting quality differences between oral and poster presented abstracts. Details of numbers and percentages for each item are shown on table 2, 3.

STROBE criteria and Timmer quality instrument for reporting observational studies in annual meeting abstracts

According to STROBE criteria, the 2000 annual meeting displayed no significant differences in reporting quality of observational studies in oral and poster presentations [oral: 8.3 ± 1.7 (score range: 4-12) vs. poster: 8.9 ± 1.6 (score range: 5-12), p = 0.977, CI -37.3 to 36.3]. Likewise, the Timmer instrument revealed no significant differences for oral: [8.6 ± 1.5 (score range: 6-12)] vs. poster abstracts [8.5 ± 1.4 (score range: 5-12), p = 0.712, CI -0.44 to 0.64]. In 2008, oral presented observational abstracts had a significantly better reporting quality than poster presented abstracts [oral: 9.4 ± 1.9 (score range: 4-13), vs. poster: 8.5 ± 2 (score range: 5-14), p = 0.005, CI 0.28 to 1.54, Figure 3]. Also the Timmer instrument was significantly better for oral vs. poster observational studies in 2008 (oral: 9.4 ± 1.4 (score range: 5-12), vs. poster: 8.6 ± 1.7 (score range: 6-11), p = 0.013, CI 0.32 to 1.28, Figure 4).

Limitations

Several limitations of our study should be noted. First, we evaluated the quality of reporting, which is not the same as the methodologic quality of the study. It is possible that a poorly reported study is well designed and executed, and a well-reported one may have several shortcomings. However, empirical evidence exists that indicates poorly reported studies are associated with larger estimates of intervention effect, i.e. poor reporting reflects poor methodology, which in turn is associated with biased results [20].

Second, given the suggestions of the conference organizing committee regarding the abstract format and the restricted word count, it has to take into account that the implementation of a more comprehensive reporting in abstracts might be hampered by the abstract requirements stated by the conference organizers. Also, submission and selection process of journals, including author request of presentation (oral, poster) could influence the type of presentation. And also it should be acknowledged that abstract are submitted and judged by category. Therefore request of presentation by author or categorisation by journal could be a potential confounder in this study.

Furthermore, we analysed only studies of evidence level 1++ to 2- according to the Scottish Intercollegiate Guidelines Network (SIGN). Because of the low evidence, experimental studies, new techniques and describing new concepts were excluded from the analysis. In this context, laboratory studies and animal studies were classified as experimental studies. Furthermore, it is possible that the analysed study designs in abstracts are misinterpertated. In addition, the CONSORT criteria were suggested primarily for randomized-controlled trials which took only about 2-7% of study designs in conference abstracts. We also acknowledge that STROBE is currently limited to three main observational study designs: cohort, case-control, and cross-sectional studies. Furthermore the STROBE criteria were not primary designed for abstracts. But no other better instrument was available and because of its specificity for observational studies, we think it was the best way to evaluate the reporting quality. While no such statements or checklists for experimental or other study designs were available, we applied the Timmer instrument for abstract quality in addition to the aforementioned measures. The Timmer instrument is a valid quality assessment tool for conference abstracts and comparing these three instruments for analysing the abstract quality leads to an increased internal validity and reliability of the study. In addition, it should be kept in mind that in this study misclassification in approaching to CONSORT and STROBE criteria could occur [21]. To minimize this effect, criteria for reporting in conference abstracts were analysed by two researchers independently. Finally, we compared just two different years, with two different sets of abstract. This limited amout of sampling gives some limitation in external validity of these trials. Therefore further analysis of more data, especially multiple year comparisons and comparing trends over time, is needed.