Study inclusion and PIL collection

PubMed was searched from January 2011 to November 2013 to identify RCTs of manual acupuncture, electroacupuncture, and laser acupuncture where sham or placebo device or procedure was used for control group. No language restriction was imposed. For the RCTs predating 2011, we adopted a list of included studies in another systematic review of randomized, placebo-controlled trials of acupuncture with blinding evaluation data, published in English between 1985 and 2011 [10].

When the list of eligible studies was completed, first or corresponding authors were invited via e-mail to send in PILs or informed consent forms used in their trials. If the authors refused to provide their PILs, or there was no response after the second reminder, studies were consequently excluded from our analysis. PILs in languages other than English were translated into English by native language speaker.

Content analysis of PILs

Through iterative reading of PILs and following steps of systematic methods developed for deductive content analysis [11–13], we reached a consensus of final three categories by description/explanation of placebo acupuncture: “full disclosure of placebo acupuncture (FD)”, “deceptive disclosure of placebo acupuncture (DD)”, and “missing information on placebo acupuncture (MI)”. The PILs had to use specific words such as ‘sham’, ‘placebo’, or ‘fake’ acupuncture, to be coded in FD category. PILs that used neutral or deceitful words like ‘group two’, or simply the ‘control group’ were put into DD category even though we trusted that the study authors never intended to be ‘deceptive’ on purpose in their PILs. However, if they were followed by additional explanation suggesting that neutrally/deceitfully named group would involve sham procedures (e.g. to receive non-acupoint acupuncture), it was considered to be in the FD category since it was judged that sufficient information was given to the study participants to perceive the existence of placebo acupuncture. For trials with more than one control groups applying different types of placebo acupuncture, each placebo group was counted as a separate unit of data. PILs that missed out on explaining anything about a placebo control were coded into MI category.

Information on placebo and blinding

From studies providing results on success of blinding, Bang’s blinding index (BI) [14] was calculated with 95% confidence intervals (CI) to further investigate the association between information on placebo and effectiveness of blinding in trials. Bang’s BI yields a value between − 1 and 1 for each arm in a study, providing the proportion of the potentially unblinded beyond chance [14]. BI of 1 would mean that everyone guessed correctly, and 0 means guesses were random, whereas − 1 would mean that all guesses were incorrect (e.g. all participants in the placebo arm answered that they believe they received real treatment). The number of participants in each response of each arm was extracted from the included studies and more information was sought from the original authors if necessary. If a study measured outcome at more than one time point, the data measured at the end of the last intervention or at the closest were extracted. Based on BIs, each study was classified into one out of nine possible blinding scenarios: in brief, random guess, unblinded or opposite guess is given to each arm in a study, and then, nine blinding scenarios are possible [10, 14, 15]. The threshold for blinding classification was random guess for − 0.2 < BI < 0.2; unblinded for BI ≥ 0.2; and opposite guess for BI ≤ − 0.2 [10]. It should be noted that throughout this paper, the term “unblinded” generally means “more correct guess,” not broken blinding literally.

Assigned scenarios and calculated BIs were compared between “disclosure (FD) and no disclosure (DD/MI)” groups using Mann-Whitney U test or independent t-test depending on the data distribution by Shapiro-Wilk test using SPSS (IBM Corp., Version 21.0, Armonk, NY), and the statistical significance was set at p < 0.05.

Information on placebo and trial outcomes

The aim of this meta-analysis was not to estimate the effects of acupuncture, but to synthesize the primary outcomes so that the summarised effects of studies from disclosure and no disclosure groups can be compared. For this analysis, following strategies were used to choose the outcome: (1) whether it is continuous outcome or dichotomous outcome, it should have data that are needed to calculate standardized mean difference (SMD) and standard deviation (SD); (2) the timing should be either indicated primary end-point or as close as possible to the last treatment session; (3) in the absence of predefined single main outcome measure, commonly used outcome in similar studies (e.g. visual analogue scale (VAS) for pain studies) was chosen or; (4) an outcome that reports clear data (e.g. if one outcome is reported in a table with specific values whereas another outcome is reported in a figure without any accurate values, the former) was chosen; (5) if a trial had more than one placebo group, the data were pooled separately as they were in content analysis, and number of participants in the shared intervention group was divided evenly [16]. For calculating SMD and SD from dichotomous outcomes, we used a formula, \( \mathrm{SMD}=\frac{\sqrt{3}}{\pi}\ln OR \)(OR, odds ratio) [17]. A random-effects model with the generic inverse variance method was adopted to estimate SMD and 95% CI using Review Manager (The Nordic Cochrane Centre, Version 5.3.4, Copenhagen, Denmark). The I² test was used to examine heterogeneity across the studies, and different PIL categories were subject to the subgroup analysis to investigate whether the estimates in the primary outcome varied between subgroups (FD or DD). In this meta-analysis using a Review Manager software, this test was undertaken using Cochran’s Q test and Higgins’ I² by comparing the subtotal estimates between the subgroups. In addition, a post hoc analysis grouped studies by its originating region (Asia and Non-Asia) in each PIL category and tested in the same manner above whether the estimates in the outcome of interest varied in different regions.

Included PILs

The initial PubMed search yielded 216 records. Among them, 107 articles were excluded for not meeting our inclusion criteria: studies with no acupuncture group per inclusion criteria (n = 39), non-RCTs (n = 26), protocols only (n = 20), studies with no placebo acupuncture control group (n = 19), animal studies (n = 2), and a duplicate (n = 1) [18, 19]. Additional 53 studies included in a previous systematic review [10] of placebo-controlled trials of acupuncture were then added to the remaining 109 studies. One study [20] from the adopted list was replaced by its original trial as it was identified as an additional report [21]. Two studies overlapping with our own search result were excluded, leaving a total of final 160 studies subject to invitation to share PILs.

The invitation e-mail was sent to the trialists in 24 countries in total, with China and USA on the top with 29 trials followed by Germany, Korea, and UK. Among 100 studies that replied to our invitation letter (62.5% of response rate), 63 studies provided PILs. In addition, one author generously provided two extra PILs on top of the requested documents and final 65 PILs from 15 different countries were collected for analysis (Fig. 1; Additional file 1: Appendix 1).

Placebo control groups and information disclosure in PILs

Types of placebo acupuncture implemented in the included studies were diverse and could be simply divided into three groups; non-penetrating placebo acupuncture, penetrating placebo acupuncture, and others (Additional file 1: Appendices 2 and 3).

Through the content analysis of information on placebo given to study participants, extracted data of 70 placebo control groups from 65 PILs were categorized into three groups. More than half of the PILs (57.1%) were straightforward to the participants when they were introducing placebo acupuncture control, and these were classified as FD group. In the 2nd group named as DD group, no words or phrases that clearly indicated placebo acupuncture would be used (35.7%), i.e., rather deceptive words such as ‘different (7),’ ‘control (5),’ ‘group 1 or group 2 (2),’ ‘non-traditional (2),’ ‘not typical (2),’ and ‘test (2),’ were given. Remaining five PILs (7.1%) fell into MI group and they missed out on explaining anything about a placebo control group. Detailed results of the content analysis are provided in Additional file 1: Appendix 4.

Our analysis of PILs by country of its origin revealed that PILs from non-Asian countries were more likely to fully inform the study participants of placebo acupuncture compared with those from Asian countries (χ² (1, n = 70) = 7.099, p = 0.008; Additional file 1: Appendix 5).

Information on placebo and blinding

BI values for each study were calculated with their 95% CI (Fig. 2). Average BI values for real acupuncture groups in FD and DD/MI categories were 0.42 and 0.41, respectively. In other words, 42% (95% CI 0.23 to 0.62) of participants in the real acupuncture groups who were given full disclosure of placebo information and 41% (95% CI 0.21 to 0.61) in the real acupuncture groups from the studies with no disclosure (DD/MI category) correctly guessed that the given treatment was real acupuncture beyond chance (Fig. 2a and b). On the other hand, 16% of the participants in the placebo group in the studies with full information disclosure believed that they received real acupuncture beyond chance (− 0.16, 95% CI −0.39 to 0.06, Fig. 2c). In the groups without adequate information on placebo acupuncture, 21% of the participants in the placebo group guessed they were given real acupuncture (− 0.21, 95% CI −0.47 to 0.05, Fig. 2d). There was no significant difference in BI values of both real and placebo acupuncture groups between the two categories, i.e. disclosure vs. no disclosure (real acupuncture, p = 0.92, independent t-test; placebo acupuncture, p = 0.38, Mann-Whitney U test).

Information on placebo and trial outcomes

Among 22 studies of which SMD and SD were available, two studies had two placebo control groups, leaving 24 units of data for a meta-analysis. No study from MI category provided adequate data for pooling. The meta-analysis of studies in DD category showed acupuncture produced better clinical outcomes compared to placebo acupuncture (SMD −0.43, 95% CI −0.68 to − 0.18, p = 0.0006; Fig. 3), whereas the studies of FD category found no significant difference between real and placebo groups (SMD −0.12, 95% CI −0.26 to 0.01, p = 0.08; Fig. 3). When two categories were assigned as subgroups, a test for subgroup differences also showed that pooled effect for real acupuncture from DD category is greater than that of FD category (χ² = 4.54, p = 0.03, I² = 78.0%; Fig. 3).

A post hoc subgroup analysis was performed within each category to see if there is any regional difference in trial results, but this analysis could not confirm the results of Vickers et al. [23] which found Asian countries were more likely to produce positive findings in acupuncture studies. In both categories, there was no difference between studies from Asian and non-Asian countries (test for subgroup differences; FD, p = 0.88; DD, p = 0.23; Additional file 1: Appendix 6).

Principal findings of the study

We analyzed PILs of placebo-controlled acupuncture trials to see how placebo acupuncture was described to the study participants. Approximately 57% of the studies fully informed the participants of placebo acupuncture while the rest of the studies either gave deceitful or no information on placebo. A half of the studies were evaluated as ideally blinded. When BI values of the included studies were compared between FD and no disclosure (DD/MI) categories, placebo groups in the latter were more likely than those in FD category to make opposite guesses on the intervention, i.e., guessed that they have received real acupuncture. Although this difference did not reach statistical significance, it may reflect more wishful thinking of placebo groups in no disclosure category than those in FD category. Studies with no disclosure reported greater treatment effects for real acupuncture than those with FD, possibly due to enhanced expectations.

Possible explanations and implications of the study for clinicians and researchers

In medicine, whether in clinical or research settings, there are high levels of ethical requirements that are expected to be met. Respecting trial participants’ autonomy by providing them adequate amount of information is one of those requisites for ethical practice in medical research, and often, the PIL is presented to the participants as a tool to meet the requirements. The main purpose of the PIL, in this sense, would be to help participants voluntarily make a well-informed decision on the introduced matter. When the control intervention of the trial involves a placebo, there has to be even more cautious approach and full disclosure of such information [1, 24, 25]. On the other hand, there are concerns on too much information may disadvantage the participants, as in increasing a nocebo effect [26, 27], or resulting in a lengthy document that rather compromises participants’ understanding [28]. Some argue that PILs are not in a reader friendly language [29], nor come with decision making aids [30]. Therefore, for a PIL to fully serve its purpose, it seems necessary to inform the participants in a way that is not only being clear about what is going to happen, but also being considerate of how it actually helps them to make an informed decision.

There has been a concerning voice that placebo-controlled trials of acupuncture are reluctant to fully disclose the existence of placebo acupuncture control [3, 4]. However, findings of the current study show change of practice ever since, showing number of studies that accurately inform the use of placebo control, being more than a half: out of 70 placebo groups from 65 PILs, 40 were classified into FD category, and 12 of those 40 were from the older studies that were included in the previous review. These findings confirm that acupuncture trials are also trying to conform to the ethical requirements [31, 32]. However, at the same time, given that there is still a room for improvement in providing complete and accurate information about placebos even in trials assessing conventional medicine [9], and device-based interventions are facing challenges in maintaining blinding with a full disclosure of placebos [33], information disclosure in PIL may be taken as an issue in clinical research across the board.

In East Asian countries, even after western medicine’s addition to the system, traditional medicine is still routinely used [34, 35]. As a result, acupuncture is easily accessible and embedded in cultural background. Therefore, participants as well as the researchers of acupuncture trials are assumed to be more familiar with acupuncture compared to those in non-Asia region [36]. This brings concern to Asian researchers that the study participants will easily distinguish real from placebo acupuncture. The rationale behind such fear is not simple, as studies examining the influence of the participant’s previous experience on detecting placebo acupuncture presented conflicting results [37, 38]. The notion that blinding may be hampered if trial participants are fully informed on details of placebo acupuncture, is partly explained in our BI analysis. While 21% of the participants in the placebo acupuncture group from DD/MI category made incorrect guesses beyond chance, i.e. they believed they received real acupuncture, 16% of those in the placebo acupuncture group from FD category made such guesses. It could be interpreted that though not statistically significant, the participants on the placebo acupuncture arm who were not very well informed about placebo acupuncture tended to guess that they received real acupuncture more than participants in FD category did. It reflects participants’ wishful thinking, which may lead to an augmented placebo effect [10]. This finding is in line with the result from the Cochrane review of all placebo interventions where a higher placebo effect was reported in trials without correct information of placebo than those with correct information [39]. An altered placebo labeling study also has shown that placebo effect goes higher as the label on placebo changes from placebo, uncertain to valid medicine [40]. Given our BI analysis and the results from other relevant studies, the hypothesis can be generated that full disclosure of placebo in PILs may undermine successful blinding of participants and this should be systematically evaluated in the future studies.

A meta-epidemiological study has reported that well maintained blinding resulted in a smaller effect of intervention compared to non-blinded trials [41]. In the present study, however, pooled effect size of trials in DD category, which showed better blinding results than FD category, was greater than that of FD category. There could be two possible explanations. First, we may suspect that informing the presence of placebo acupuncture in detail lowered the participants’ expectation and as a result, possibly undermined the effect of real acupuncture in studies in FD category, whereas the participants’ expectation was maintained or enhanced in DD category and the trials favored acupuncture to a larger extent. This is supported by a pooled analysis from four, large population based German RCTs which demonstrated higher expectation was associated with a better outcome regardless of given treatment [42]. Considering that our study was not designed to directly measure participants’ expectation of treatment, however, we cannot estimate to what extent expectation contributed to greater effect size from the studies in DD category than those in FD category. Another explanation is that the treatment effect in DD category is rather overestimated due to investigator’s bias. Not only the patients but also trialists have expectations, wittingly or unwittingly [43]. Considering the fact that more studies in DD/MI category originated from Asia, which is an acupuncture friendly region where researchers are eager to prove its efficacy, than FD category, it may be trialists’ expectation on top of the participants’ expectation that resulted in such higher estimation. Besides, when we evaluated whether outcome assessor was blinded to the intervention in the included studies, the proportion of studies with appropriate blinding for outcome assessor was slightly lower in DD category (20 out of 25, 80%) compared to FD category (33 out of 40, 82.5%). This too supports investigator bias may have possibly played a part in overestimation of treatment effect in the studies of DD category. Nonetheless, the fact that only small number of studies were included here and that this is merely observational in nature allows for an interpretation in a narrow context. To elucidate complicated relationships between PILs, blinding, and outcomes of a trial, further studies designed to compare the links between those are warranted for more discussion.

Strengths and weaknesses of the study

One of the strengths of this study is that we are addressing a fundamental question over so-called routine clinical research procedures. We tried to provide a more comprehensive understanding of any association of placebo disclosure in PILs with methodological issues in clinical research, i.e. blinding, and the effect size. Previous studies have usually focused on ethics [44], readability [45], or comprehensiveness [46], but few studies explored association of pre-trial information on placebo with blinding and trial outcomes, which definitely requires further investigations from a different point of view. Another strength would be that it encompasses qualitative and quantitative research designs. Collected PILs were subject to systematic qualitative content analysis method [12]. Then, numerical analyses such as calculating BI and effect size were conducted in conjunction with the results from content analysis. Such integrated method can help science achieve better understanding of its discoveries [47].

Some limitations also should be noted. Firstly, not all included documents were written in English and simple analysis such as word count, page count, or readability calculation was not feasible. Although, all documents were carefully translated into proper English by experts then analyzed in the main analysis in a standardized way, we cannot rule out the possibility that some subtle nuance of original language was not delivered in translation. Secondly, the number of studies included in blinding and outcome analysis was too small compared to that of all included studies for content analysis, which might have made it difficult to detect statistically significant differences, if any, thus to draw any firm conclusion. Even so, the number of studies included in the main content analysis was larger than 16 of a previous study [3] and was similar to the suggested ‘medium’ sample size for integrated qualitative-quantitative research [47]. More importantly, this study accomplished to collect the documents from many different countries of different environment and culture while the previous study had documents mostly from only few countries such as Germany, UK and USA. This diversity of documents in the present study made it possible to explore rich source of data. Last but not the least, the actual perception of the information on PIL by study participants may differ from degree of information disclosure judged in our study. Participants’ understanding and memory of the contents of PILs vary [33] and their behavior may be influenced by other multiple factors such as prior knowledge, health literacy, preference and expectation.

Unanswered questions and future research

Our findings raise questions on routinely accepted informed consent procedures: ethical standard requires every detailed information including placebo should be given to study participants. Placebo information in other than acupuncture trials, however, has been reported far from fully informing [9] and what information is provided to the patient about side effects is known to modulate treatment responses often to enhance nocebo effect [8, 48]. Moreover, as we have few empirical findings on how patients are told about placebo may affect blinding and study outcomes, future studies are needed to examine this unanswered question.

Other than methodological qualities such as allocation concealment or blinding that are known to affect the direction or magnitude of treatment effect [41, 49], we found information on placebo in PIL may be another potential factor for influencing the study outcome by possibly modulating patient expectation in the informed consent process. Therefore, further in-depth studies utilizing both qualitative and quantitative methods are warranted to better understand complicatedly intertwined factors in PILs, blinding, and outcomes of placebo-controlled trials.