Negative pressure wound therapy: Potential publication bias caused by lack of access to unpublished study results data

Abstract

Background

Negative pressure wound therapy (NPWT) is widely applied, although the evidence base is weak. Previous reviews on medical interventions have shown that conclusions based on published data alone may no longer hold after consideration of unpublished data. The main objective of this study was to identify unpublished randomised controlled trials (RCTs) on NPWT within the framework of a systematic review.

Methods

RCTs comparing NPWT with conventional wound therapy were identified using MEDLINE, EMBASE, CINAHL and The Cochrane Library. Every database was searched from inception to May 2005. The search was updated in December 2006. Reference lists of original articles and systematic reviews, as well as congress proceedings and online trial registers, were screened for clues to unpublished RCTs. Manufacturers of NPWT devices and authors of conference abstracts were contacted and asked to provide study information. Trials were considered nonrandomised if concealment of allocation to treatment groups was classified as "inadequate". The study status was classified as "completed", "discontinued", "ongoing" or "unclear". The publication status of completed or discontinued RCTs was classified as "published" if a full-text paper on final study results (completed trials) or interim results (discontinued trials) was available, and "unpublished" if this was not the case. The type of sponsorship was also noted for all trials.

Results

A total of 28 RCTs referring to at least 2755 planned or analysed patients met the inclusion criteria: 13 RCTs had been completed, 6 had been discontinued, 6 were ongoing, and the status of 3 RCTs was unclear. Full-text papers were available on 30% of patients in the 19 completed or discontinued RCTs (495 analysed patients in 10 published RCTs vs. 1154 planned patients in 9 unpublished RCTs). Most information about conference abstracts and unpublished study information referring to trials that were unpublished at the time these documents were generated was obtained from the manufacturer Kinetic Concepts Inc. (KCI) (19 RCTs), followed by The Cochrane Library (18) and a systematic review (15). We were able to obtain some information on the methods of unpublished RCTs, but results data were either not available or requests for results data were not answered; the results of unpublished RCTs could therefore not be considered in the review. One manufacturer, KCI, sponsored the majority of RCTs (19/28; 68%). The sponsorship of the remaining trials was unclear.

Conclusion

Multi-source comprehensive searches identify unpublished RCTs. However, lack of access to unpublished study results data raises doubts about the completeness of the evidence base on NPWT.

Background

Conservative estimates report that about 1% of the population in Western countries is affected by chronic wounds [1], with a much higher prevalence rate in inpatient and nursing facilities [2]. Besides presenting a significant risk factor for complications such as infection and amputation, chronic wounds also lead to marked impairment of patients' quality of life [3, 4].

Negative pressure wound therapy (NPWT) consists of an open-cell foam dressing covered with an adhesive drape. The dressing is connected to a vacuum pump that creates and maintains a subatmospheric pressure [5]. The most commonly used NPWT device is the Vacuum Assisted Closure (V.A.C.^®) device.

The Federal Joint Committee (Gemeinsame Bundesausschuss), the decision-making body of the self-administration of the German health care service, commissioned the Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWiG) to conduct a systematic review on the efficacy and safety of NPWT versus conventional wound therapy in patients with acute or chronic wounds. The full review was published in German in March 2006, and an English-language article has recently been published [6, 7]. An update of the systematic review, a rapid report, was published in German in January 2007 [8].

Although NPWT is widely applied, particularly for chronic wounds, at the time of the review only a few relevant randomised controlled trials (RCTs) were available; these were, moreover, of poor quality [9]. Previous reviews on NPWT have not provided clear evidence of the superiority of NPWT over conventional wound therapy. For example, a review published by the Canadian Medical Advisory Secretariat in 2006 concluded that "Based on the evidence to date, the clinical effectiveness of NPWT to heal wounds is unclear" [10].

Previous reviews on medical interventions have shown that conclusions on efficacy and safety based on published data alone may no longer hold after consideration of unpublished data, which may reverse favourable risk-benefit profiles and attenuate treatment effects of an apparently superior intervention [11–13]. Within the framework of the systematic review on NPWT, our main objective was therefore not only to identify published, but also unpublished completed or discontinued RCTs in order to gain as complete an overview as possible of the evidence on NPWT. Further aims were to locate ongoing RCTs for potential consideration in future updates, and to determine the type of sponsorship of all eligible trials.

Methods

The literature search comprised a total of 7 steps. Table 1 shows an overview of the sources used.

Table 1 Literature sources and search steps*

We searched 4 bibliographic databases (MEDLINE, EMBASE, CINAHL, and The Cochrane Library) for RCTs on NPWT versus conventional wound therapy in patients with acute or chronic wounds. All databases were searched from inception to May 2005. The search was updated in December 2006 within the framework of the preparation of the rapid report on NPWT [8]. The search strategy in MEDLINE (Table 2) was applied according to the Cochrane Highly Sensitive Search Strategy described in the Cochrane Handbook for Systematic Reviews of Interventions [14], and tailored to the requirements of each database. Other search strategies are available upon request.

Table 2 Search strategy applied in MEDLINE

We then screened online trial registers (Table 1) and the US Food and Drug Administration website, and contacted German authorities, requesting information on all available RCTs on NPWT. Prior to the start of the review, the Federal Joint Committee had asked experts to provide relevant literature; these citations were also obtained by IQWiG as part of the commission. Moreover, the reference lists of retrieved original articles and systematic reviews, as well as congress proceedings, were handsearched.

Two manufacturers of NPWT devices were identified: Kinetic Concepts Inc. (KCI, San Antonio, Texas, USA) markets the V.A.C.^® device, and BlueSky Medical Group Inc. (BSM, Carlsbad, California, USA) markets the Versatile 1 Wound Vacuum System^®. Both were asked to provide information on the study status and publication status of sponsored trials, as well as on methods and results. Finally, we contacted the authors of conference abstracts by e-mail and by post. All contact details of the relevant institutions were checked beforehand on the corresponding websites. Manufacturers and authors were informed that IQWiG does not accept "commercial in confidence" data and publishes all data contributing to a systematic review [15].

All languages were included, as long as a title was available in English. If the title indicated a potential relevance of the study, the corresponding article was obtained and translated. Only RCTs comparing NPWT versus conventional therapy were eligible. Trials were considered nonrandomised if concealment of allocation to treatment groups was classified as "inadequate" [16]. The intervention was classified as NPWT if a medical device system identical or comparable to the V.A.C.^® Therapy System was used.

Results of trials that were not available as full-text publications were only considered in the review if detailed data were available (e.g. from a clinical study report or a manuscript in press). Results data from conference abstracts were not considered.

All steps of the literature screening were performed by two reviewers independently of one another. Any disagreements were resolved by discussion. The titles and abstracts of the retrieved documents were screened to exclude citations that were clearly irrelevant. The full texts of the remaining potentially relevant articles were then screened to identify RCTs that fulfilled the inclusion criteria stated above.

With the help of the information obtained, the study status of the RCTs identified was classified as "completed", "discontinued", "ongoing" or "unclear". The terminology used in the literature to classify unpublished data or so-called "grey literature" is inconsistent [12]. We classified the publication status of completed or discontinued trials as "published" if a full-text paper on final study results (completed trials) or interim results (discontinued trials) was identified in the literature search, and "unpublished" if this was not the case. In addition, for ongoing trials it was indicated whether interim results were available or not. The type of sponsorship was also noted for all trials.

Only a summary of results on the quality assessment and outcomes of the RCTs included in the IQWiG systematic review and rapid report is presented here, as the main focus of this paper was to identify unpublished RCTs.

Results

An overview of the search results is presented in Table 3. Detailed information on identified published and unpublished RCTs, as well as on the RCTs included in the systematic review and rapid report, is presented in Table 4.

Table 3 Overview of study status, publication status, and sponsorship (Status: December 2006)

Table 4 Search results for randomised controlled trials on negative pressure wound therapy* (Status: December 2006)

Search results

Of 2675 potentially relevant publications, 317 full-text papers were obtained for further assessment. Of these, 289 were excluded as not relevant (Figure 1).

Figure 1

Flow chart of the literature search for RCTs.

A total of 28 RCTs on NPWT referring to at least 2755 planned or analysed patients were identified: 13 RCTs (801 patients; 29%) had been completed [17–29], 6 RCTs (848 patients; 31%) had been discontinued [30–35], 6 RCTs (1106 patients; 40%) were ongoing [36–41], and the status of 3 RCTs was unclear [42–44] (Table 3). According to KCI, the main reasons for discontinuation were slow enrolment, high attrition rates, changes in clinical practice, or design flaws [6].

Full-text publications were available on 30% of the 1649 patients in the 19 completed or discontinued RCTs (495 analysed patients in 10 published RCTs vs. 1154 planned patients in 9 unpublished RCTs). Of the 14 conference abstracts on these 19 trials, 6 abstracts (43%) were later published as full-text articles.

Response behaviour of manufacturers and authors

A total of 17 authors and 2 manufacturers were contacted, of whom 10 (59%) and 2 (100%) responded respectively. KCI readily provided information on study and publication status and on methodological issues. BSM did not sponsor relevant RCTs and provided information only on case reports.

One author provided information on an RCT that had been classified as "ongoing", and stated that this trial had recently been completed and the manuscript submitted to The Lancet [22]. As this trial included 162 analysed patients and was the largest RCT on NPWT conducted so far, IQWiG postponed the publication of its review by 2 months until the final results of this RCT were available, in order to include the results in the body of evidence.

Literature sources

The sources of the conference abstracts and unpublished study information referring to 28 RCTs on NPWT are shown in Tables 5, 6: most information was obtained from KCI (19), followed by Cochrane (18), a systematic review (15) [9], and congress proceedings (14) [45].

Table 5 Sources of conference abstracts and unpublished study information on 28 RCTs: databases, trial registers, and systematic reviews

Table 6 Sources of conference abstracts and unpublished study information on 28 RCTs: manufacturers, authors, congress proceedings, and authorities

The sources of all 11 published full-text articles (referring to 12 completed, discontinued or ongoing RCTs) are shown in Tables 7, 8, 9. All 11 articles were obtained from the Cochrane Library followed by KCI (10), MEDLINE (10), and EMBASE (9).

Table 7 Sources of 11 published full-text articles on 12 RCTs: databases and trial registers

Table 8 Sources of 11 published full-text articles on 12 RCTs: systematic reviews

Table 9 Sources of 11 published full-text articles on 12 RCTs: manufacturers, authors, congress proceedings, and authorities

Summary of quality assessment and outcomes

Published trials

The overall methodological quality of the 12 published RCTs included in the IQWiG systematic review and rapid report was poor. Methodological flaws included the lack of blinding, the lack of intention-to-treat analyses, inadequate allocation concealment, the unclear definition of primary outcomes, and high drop-out rates. Significant differences in favour of NPWT for wound healing parameters, such as time to wound closure or the incidence of wound closure, were shown in 3 of 7 RCTs analysing these outcomes. Data on other patient-relevant outcomes, such as reoperation rates or pain scores, were scarce or not interpretable.

Unpublished trials

Because of insufficient available information on the 9 completed or discontinued unpublished RCTs, an assessment of their quality could not be conducted. Regarding the 5 discontinued industry-sponsored trials, as stated, the reasons for discontinuation were provided by KCI, which also reported the number of enrolled patients for one trial [33]. However, there was no response to a further request from IQWIG asking for more detailed information on all discontinued trials, such as the number of enrolled patients and other results data. Regarding the 4 completed RCTs with unclear sponsorship, the authors of publications did not respond to IQWiG's request to provide results data (Table 4). Consequently, the results of unpublished RCTs could not be considered in the review.

Industrial sponsorship

The manufacturer KCI sponsored the majority (19/28) of RCTs (68% of trials referring to 85% of patients planned or analysed). The sponsorship of the remaining trials was unclear.

Discussion

The main objective of this paper was to identify unpublished RCTs on NPWT within the framework of a systematic review. An RCT was classified as "unpublished" if no full-text paper on final study results (completed trials) or interim results (discontinued trials) was available.

An extensive search strategy was employed that included handsearching of retrievals, as well as contacting manufacturers and authors of publications. The sensitivity of bibliographic database searching for RCTs is reported to be unsatisfactory [46], and multi-source searching has been recommended to retrieve all available RCTs [47]. In this context, the usefulness of handsearching and contacting experts has been demonstrated [48, 49]. The extensive search strategy was also useful to detect ongoing RCTs, of which one [22] was subsequently added to the evidence base.

A total of 28 RCTs referring to at least 2755 patients were identified. The publications on completed or discontinued RCTs reported data on less than a third of patients planned or analysed. Less than half of the conference abstracts on these trials were later published as full-text publications; similar results were found in a review by Scherer et al, who found a full-text publication rate for abstracts describing RCTs or controlled clinical trials of about 60% [50].

On the basis of results of published data alone, the IQWiG review and rapid report concluded that although there was some indication that NPWT may improve wound healing, the body of evidence available was insufficient to clearly prove an additional clinical benefit of NPWT. This finding is in line with findings from previous reviews.

Regarding the discontinued unpublished trials, according to the manufacturer the main reason for discontinuation was insufficient patient numbers. No results data for these trials were available. Regarding the completed unpublished trials, the authors of conference abstracts did not respond to IQWiG's request for information. Therefore, unpublished results data could not be considered in the IQWiG review and consequently the impact of unpublished data on the conclusions of reviews based on published data could not be assessed.

Publication bias caused by unpublished data

Previous research has shown that inclusion of unpublished data in systematic reviews may affect the review's prior conclusions [12, 13]. The most prominent example in recent years has been the evidence on serotonin reuptake inhibitors (SSRIs) in paediatric depression: a systematic review of published versus unpublished data showed that whereas published trials indicated a favourable risk-benefit profile, the addition of unpublished data reversed this profile for some SSRIs [11].

Klassen 2002 assessed the proportion of RCTs presented at a major paediatric meeting that were subsequently published as full publications; about 60% of abstracts were subsequently published, and RCTs were more likely to be published if they favoured the new therapy. The author concluded that "publication bias is a serious threat to assessing the effectiveness of interventions in child health" and called for a registry of RCTs in children so that the totality of evidence could be assessed [51].

As unpublished results data on NPWT were not available, no statement can be made on the impact of unpublished data on the validity of published evidence. However, if one assumes a "worst case scenario" (all unpublished trials show results in favour of conventional therapy), the current conclusion would no longer hold that there is some indication of a potential advantage of NPWT over conventional therapy.

Besides possible problems of statistical evaluation due to insufficient sample sizes, the reasons for the existence of unpublished RCTs on NPWT remain unclear and one can only speculate. For example, it may be possible that some of these RCTs remained unpublished because they were of poor quality or produced negative results. Previous research has shown that studies with statistically significant or positive results are more likely to be published than those with non-significant or negative results, and are also published earlier [52–54]. These studies also lead to a greater number of publications, and are more likely to be published in high impact factor journals [55]. As for discontinued trials, even if small sample sizes make statistical evaluation difficult, in our opinion the full datasets of such trials should be publicly accessible to ensure that the evidence base is complete.

The inclusion of unpublished data in systematic reviews is controversial. Although less than a third of meta-analyses include unpublished data, nearly 80% of meta-analysts and methodologists believe that unpublished data should definitely or probably be included in systematic reviews; in contrast, less than half of journal editors agree [56]. The identification of unpublished data has been recommended to minimise the risk of bias in systematic reviews [12], but there are variations in policy regarding their inclusion [57]. This may be explained by incomplete or inaccurate reporting and the subsequent difficulties in assessing the methodological quality of trials [57, 58].

In general, the quality of reporting of systematic reviews on both drug and non-drug interventions is inconsistent [59]. Moreover, systematic reviews on non-drug interventions (e.g. surgical techniques) face specific problems, such as the limited quality and quantity of RCTs [60]. Searching for unpublished data and conducting sensitivity analyses to assess their impact is recommended for topics with little evidence and new or evolving interventions [58]. In their commentary on the problem of conducting systematic reviews of new health technologies, Moher and Schachter concluded that the inclusion of unpublished data was of "paramount importance" in assessing the usefulness of an intervention [61].

The role of industrial sponsorship

Approximately two thirds of the RCTs on NPWT were sponsored by one manufacturer (KCI). Industrial sponsorship of clinical trials is common. The percentage of industry-sponsored clinical trials has increased to over 60% in recent years and the number of industry employees named as co-authors of clinical trial publications is rising [62]. However, the role of industrial sponsorship in clinical research is controversial. Industry sponsorship is discussed as being associated with the following factors: selection of inappropriate comparators, selective reporting of more favourable per protocol analyses, pro-industry conclusions, as well as restrictions on publication and data sharing [63–65]. A comparison of Cochrane reviews versus industry-supported meta-analyses showed that the latter came to more favourable conclusions on the same drugs [66]. The available data on NPWT are insufficient to make definite conclusions about the impact of industrial sponsorship in this field.

Differences in primary study selection

Five systematic reviews on NPWT (including the IQWiG review) published in December 2004 or later showed considerable differences in the selection of primary studies published between 1994 and 2004 [6, 9, 10, 67, 68]. There was agreement between reviews regarding the selection of 4 RCTs [19, 20, 24, 30]. One further study [17] classified by IQWiG as an eligible RCT was not classified accordingly (or identified) by 2 other reviews [10, 67]. A study [69] classified as an eligible RCT by 2 reviews [9, 68] was classified as non-randomised by IQWiG, and one study [70] classified as an eligible RCT by one other review [68] was also classified as non-randomised by IQWiG. These findings indicate that in systematic reviews on NPWT, search strategies, inclusion criteria, and classification of primary studies are not applied in a standardised manner.

Conclusion

Multi-source comprehensive searches identify unpublished clinical trial data. However, lack of access to unpublished study results data raises doubts about the completeness of the evidence base on NPWT. The implementation of regulations such as prospective mandatory trial registration and the obligation to publish all results is needed to ensure that independent researchers have access to all outcomes of completed or discontinued clinical trials.