Skip to main content

The difference between concealment and blinding in clinical trials and why both are important. A reply to Garg and Mickenautsch, BMC Medical Research Methodology (2022) 22:17

The Original Article was published on 15 June 2022

Peer Review reports

To the Editor,

We have been made aware of a paper in your journal [1] that purports to identify a risk of ‘selection bias’ in the two studies that comprised the NINDS rt-PA for Acute Ischemic Stroke Study [2], referred to from here on as the Studies. Upon careful reading, we find that the authors re-state some factual errors and misconceptions about the Studies. They also raise a new accusation that somehow patients with a more favorable prognosis were selectively directed into the rt-PA treated arm, thus biasing the trial toward a better outcome in the treated arm. In making this claim, the authors conflate two very different aspects of rigorous clinical trial design: treatment concealment versus blinding. We wish to untangle this confusion.

The authors first state there were 22 out of 624 (3.5%) cases that received incorrect treatment, of whom 21 received rt-PA instead of placebo. The Studies were conducted prior to the advent of the Internet or digital, web-based randomization systems: we used a manual randomization system of sequentially numbered envelopes. Vial numbers in the envelopes were assigned based on a stratified randomized block design and did not identify treatment group. To randomize, the investigator opened the next envelope in sequence to obtain a code number corresponding to a drug vial number. Although state of the art at the time, the system was complex and human errors occurred. There were 13 patients (2.1%) for whom the investigator chose the wrong vial ID number and of these, 11 (1.8%) received placebo but should have received rt-PA; two such patients received placebo and should have, so using the incorrect vial did not change treatment. These issues were all disclosed to the FDA in the final trial report. During review of the application for approval, an independent FDA reviewer commented that since the patients receiving placebo rather than rt-PA had better outcomes than the other placebo patients, “…this error in the randomization process appears not to have contributed any bias to overestimating the treatment effect.” (Full report is available at Clinical Review II for PLA 96–0350 (fda.gov)) In one patient, although the correct vial ID number was used, the coordinating center had packaged the wrong drug, and the patient received placebo but should have received rt-PA. In 18 (2.9%) patients, drug vials were chosen from the wrong treatment stratum, in part due to delays in drug re-supply. Of these 18 mis-randomizations, 11 ended up getting the wrong drug assignment: 10 (1.6%) received placebo who should have received active drug and one received rt-PA who should have been given placebo. Again, after considering the outcomes in these patients, the independent FDA reviewer concluded “…the errors do not seem to have altered the overall outcome of the studies.”

Next, the authors describe the well-known imbalance in mild stroke patients that favored the rt-PA treated group. This imbalance was adjusted for in the original analysis, and upon extensive, independent re-analysis, no impact on the results was found [3]. Then the authors go one step further and attempt to link the mis-randomized issue to the mild-NIHSS imbalance issue by noting that we opened the unblinding envelopes in 16 (2.6%) patients, 8 (1.3%) for patient bleeding, and 8 (1.3%) for unstated reasons. The authors mischaracterized these opened envelopes as “failed concealment” rather than unblinding.

The distinction between concealment and blinding is critically important. Concealment is used to reduce selection bias by assuring that the investigator who selects the subject and assigns treatment has no knowledge of the treatment about to be given. In contrast, blinding is used to reduce performance bias by assuring that all outcome assessments are done by an investigator without knowledge of the treatment group to avoid preferentially rating one group or the other as better. A failure of concealment occurs before treatment, while a failure of blinding occurs after treatment.

In no case was a treatment revealed (unblinded) prior to treatment. Unblinding occurred only after an adverse event or some other intercurrent clinical event. Bleeding is one such clinical event, and as a result, more unblinding occurred in the patients after rt-PA treatment, compared to placebo. It is crucial to note that there was not one single incident of failed concealment in the Studies, and no possibility of selection bias. Stated another way, the alleged ‘randomization subversion’ the authors assert was physically impossible.

The authors allude to the characteristic foaming of the drug and gum bleeding as further threats to concealment. First, the placebo drug clearly foamed the same as the active drug. In addition, gum bleeding could only occur AFTER treatment, and thus was not in any way a threat to concealment. Nothing enabled the investigator randomizing the patient to knowingly allocate patients with lower NIHSS scores to one group over another.

After a re-analysis of the data that the authors label ‘sensitivity analysis’ purporting to correct for selection bias, they present their ‘revised effect size’ in their Table 4. We note that all of the odds ratios still favor treatment, while some of the revised confidence intervals contain 1.0. The authors ignored, however, recalculating the primary outcome analysis of the trial, which used a global odds ratio. We wonder why? They offer their opinion that the global odds ratio has no clinical meaning, yet it was accepted by the FDA as representing the ability to statistically determine a “consistent and persuasive treatment effect” using multiple outcome measures.

The Studies’ results have been confirmed in further randomized, controlled trials [4, 5], and in the daily practice of vascular neurology where all of us witness the benefit of thrombolysis every day [6,7,8]. We write now to flatly deny that somehow treatment assignment was manipulated. Concealment failure was not only antithetical to us, it was physically impossible.

Most importantly, we write to firmly refute any suggestion that acute stroke patients should not be treated with thrombolysis. To withhold proven therapy from appropriate patients is unjustified. There is no equipoise in the vascular neurology community nor is there enthusiasm for a placebo-controlled rt-PA trial within 3 h of stroke onset. The benefit of thrombolytic therapy for acute ischemic stroke is supported by a considerable published literature and is considered standard of care around the world. We urge all readers to review the literature personally and carefully judge the results of the many large, rigorous, well-designed trials—including ours—that establish the considerable benefit of intravenous thrombolysis for acute ischemic stroke.

Availability of data and materials

All data from the original trial is available from the NINDS.

Abbreviations

NINDS:

National Institute of Neurological Disorders and Stroke

rt-PA:

Recombinant tissue type plasminogen activator

FDA:

Food and Drug Administration

References

  1. Garg R, Mickenautsch S. Risk of selection bias assessment in the NINDS rt-PA stroke study. BMC Med Res Methodol. 2022;22(1):172.

    Article  PubMed  PubMed Central  Google Scholar 

  2. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke. N Engl J Med. 1995;333(24):1581–7.

    Article  Google Scholar 

  3. Ingall TJ, O’Fallon WM, Asplund K, Goldfrank LR, Hertzberg VS, Louis TA, Christianson TJH. Findings from the reanalysis of the NINDS tissue plasminogen activator for acute ischemic stroke treatment trial. Stroke. 2004;35(10):2418.

    Article  CAS  PubMed  Google Scholar 

  4. Lees KR, Emberson J, Blackwell L, Bluhmki E, Davis SM, Donnan GA, Grotta JC, Kaste M, von Kummer R, Lansberg MG, et al. Effects of alteplase for acute stroke on the distribution of functional outcomes: a pooled analysis of 9 trials. Stroke. 2016;47(9):2373–9.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Wardlaw JM, Murray V, Berge E, del Zoppo GJ. Thrombolysis for acute ischaemic stroke. Cochrane Database Syst Rev. 2014;7:CD000213.

    Google Scholar 

  6. Hacke W, Lyden P, Emberson J, Baigent C, Blackwell L, Albers G, Bluhmki E, Brott T, Cohen G, Davis SM, et al. Effects of alteplase for acute stroke according to criteria defining the European Union and United States marketing authorizations: Individual-patient-data meta-analysis of randomized trials. Int J Stroke. 2018;13(2):175–89.

    Article  PubMed  Google Scholar 

  7. Frank B, Grotta JC, Alexandrov AV, Bluhmki E, Lyden P, Meretoja A, Mishra NK, Shuaib A, Wahlgren NG, Weimar C, et al. Thrombolysis in stroke despite contraindications or warnings? Stroke. 2013;44(3):727–33.

    Article  CAS  PubMed  Google Scholar 

  8. Schwamm LH, Ali SF, Reeves MJ, Smith EE, Saver JL, Messe S, Bhatt DL, Grau-Sepulveda MV, Peterson ED, Fonarow GC. Temporal trends in patient characteristics and treatment with intravenous thrombolysis among acute ischemic stroke patients at Get With The Guidelines-Stroke hospitals. Circ Cardiovasc Qual Outcomes. 2013;6(5):543–9.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Susan Fagan, PhD for careful review of the manuscript.

Funding

Supported by the National Institute of Neurological Disorders and Stroke (N01-NS-02382, N01-NS-02374 (JB), N01-NS-02377 (PL), N01-NS-02381, N01-NS-02379, N01-NS-02373 (SL), N01-NS-02378 (MF), N01-NS-02376 (ECH), and N01-NS-02380). Although the NINDS supported the original trial, the opinions herein are solely those of the authors and were not reviewed or approved by the NINDS who played no role in the design of this opinion or in collection, analysis, and interpretation of data and in writing this manuscript.

Author information

Authors and Affiliations

Authors

Contributions

All authors, PL, JB, JG, TK, SL, MF, ECH, BT, share equally in responsibility for the genesis and preparation of this letter.

Corresponding author

Correspondence to Patrick Lyden.

Ethics declarations

Ethics approval and consent to participate

Not applicable to this letter. Informed consent was obtained from all patients (or next-of-kin) enrolled in the NINDS rt-PA for Acute Ischemic Stroke.

Consent for publication

Not applicable.

Competing interests

The authors confirm that they do not have, and have never had, any financial interest in the drug tissue plasminogen activator.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lyden, P., Broderick, J., Grotta, J. et al. The difference between concealment and blinding in clinical trials and why both are important. A reply to Garg and Mickenautsch, BMC Medical Research Methodology (2022) 22:17. BMC Med Res Methodol 23, 275 (2023). https://doi.org/10.1186/s12874-023-02074-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-023-02074-5