- Open Access
Pure intervention effect or effect in routine health care - blinded or non-blinded randomized controlled trial
BMC Medical Research Methodology volume 18, Article number: 91 (2018)
Randomized trials provide the most valid evidence of effectiveness of interventions. The study aims to determine the primary study question for randomized controlled trials; to evaluate the study questions in trials on effectiveness of arthroscopic meniscectomy for meniscal rupture of the knee; and to explore the clinical and research implications.
Previous studies on benchmarking controlled trials were utilized. A literature search was undertaken to find the trials on arthroscopic surgery for meniscal rupture of the knee, data was extracted, and checked for accuracy twice.
The first question in RCTs is whether to assess the pure intervention effect, or intervention effect in routine health care circumstances. The former necessitates a double blinded design and the latter a non-blind design. The trials on arthroscopic meniscectomy of the knee showed considerable differences in study characteristics.
The study question in RCTs on pure intervention effect dictates use of blinded design, while question of intervention effect in routine health care dictates use of non-blinded design. Blinding should not be considered a validity criterion when study question is on effectiveness in routine health care. When informing patients, the potential for other effects besides the pure intervention effect should be considered.
The experimental studies, particularly randomized controlled trials (RCTs) provide the least biased information on effectiveness of medical interventions and create the basis for systematic reviews on effectiveness of interventions . Blinding of patients and care givers safeguards that knowledge of treatment allocation will not confound the effectiveness estimates . However, the effect estimates may vary between blinded and non-blinded RCTs, and this variation may be dependent on the outcome. Perceived outcomes, for example pain, may be reduced by the placebo effect in non-blinded real-world circumstances, while some objective outcomes, e.g. mortality may show less difference between blinded vs. non-blinded comparisons [2, 3]. The effectiveness of arthroscopic partial meniscectomy (APM) for a ruptured meniscus of the knee has been under debate in scientific journals [4,5,6,7,8,9,10], and was thus chosen for obtaining empirical data for the current study.
The aims of this study were to find and operationalize the study question to be considered first in randomized controlled trials (RCTs); to assess consequences of this operationalization both conceptually and by using empirical data on effectiveness of arthroscopic meniscectomy for meniscal rupture of the knee; and to explore the consequent clinical and research implications.
The present study utilized the methods for observational effectiveness studies, the benchmarking controlled trials (BCTs), where there is a need for a very detailed description of the study questions, selection of patients, characteristics of patients, interventions, and outcomes . In BCTs the study question is always on effectiveness in routine health care. The BCT framework was used to assess which is the first study question to be asked in experimental studies, the randomized controlled trials (RCTs).
Literature search was undertaken to find all randomized controlled trials published in peer-reviewed journals assessing effectiveness of arthroscopic partial meniscectomy of the knee in comparison to any other non-pharmacological treatment, including sham surgery, among patients having knee pain, with at least one-year follow-up. Trials focusing on knee osteoarthrosis were excluded. The following key words were used: arthroscopic partial meniscectomy, randomized controlled trial, systematic review. Cochrane CENTRAL, Ovid MEDLINE, and Web of Science databases till October 2017 were used to find the eligible articles by the author, who checked the search findings to exclude misclassifications. The search strategy is described in Additional file 1.
The descriptive information in each trial concerning blinded vs non-blinded study design, selection of patients, and characteristics of patients, interventions and outcomes were extracted by the author, who rechecked twice the accuracy of the data. The study question characteristics were also depicted in a flow chart to show the similarities and differences between the studies.
Characteristics of RCTs assessing pure intervention effect or effectiveness in real-world health care are shown in Table 1. The validity issues in RCTs utilizing a double blinded design and in RCTs using a non-blinded design are shown in Fig. 1. The double blinded design aims to assess the pure (specific) effects of an intervention, often biologic effects; while in routine health care the specific effect is complemented with a placebo effect and non-specific effects caused by the interaction between the patient and those providing care (Fig. 2).
Study question dictates double blinded study design
When the study question is to assess the pure intervention effect the answer can be obtained only by a double blinded design (Table 1). The comparison can be between active intervention and placebo, or between two or more active interventions. When the comparison is to placebo, the study assesses the incremental effectiveness of the intervention beyond the effect caused by placebo. The study question in a double blinded randomized controlled trial is on pure (usually biological) effect of the treatment, while the clinical question in routine health care, where the patient and the care provider are always aware of the treatment options, is on what is the biological effect of the intervention plus the placebo effect of this intervention, Furthermore the overall effect is increased with non-specific treatment effects (information, advice, support) related to the intervention under study (Fig. 2).Therefore, double blind placebo controlled trials do not answer the question of effectiveness in routine health care. When two or more active interventions are compared with another in a double blind RCT the study assesses difference in effectiveness between the interventions, when placebo effect has been controlled by the double blinded design.
Study question dictates non-blinded study design
In routine health care patients and their health care providers are always aware of the treatment options and to concord with this a non-blinded RCT is necessary (Table 1). When the comparison is between active intervention and no intervention (a choice to be made in clinical praxis), the study assesses the pure intervention effect plus its placebo effect plus non-specific treatment effects in comparison to none of these three effectiveness components (Fig. 2). Similarly, when the comparison is between two active interventions the study question again accords with the clinical question in routine health care, where in both groups the specific effect, placebo effect and non-specific effect add to the effectiveness estimates.
The study questions in RCTs on arthroscopic partial meniscectomy
The primary literature search of the three databases found altogether 2375 abstracts of articles. Based on data in abstracts altogether 6 randomized controlled trials (RCTs) assessing effectiveness of arthroscopic partial meniscectomy and fulfilling the inclusion criteria where found (Table 2). [12,13,14,15,16,17]
All the 6 trials had reported appropriately the items needed for assessment of the study question. Information of patients’ selection process to the study was limited in five studies (Table 2). In one study, Gauffin et al., it was reported that more than 95% of patients were referred by the general practitioners to the study hospital from its catchment area19. The number of patients recruited per year per hospital varied from 6 to 82 between the studies. The proportion of eligible patients declining participation varied from 3 to 55%.
Very little or nothing was reported on general health status, comorbid conditions, behavioural factors like degree of physical activity, environmental factors like work conditions, or on degree of education or other socioeconomic factors of the patients (Table 2).
The trial by Gauffin et al. was the only study, which had a prerequisite of 12 weeks of exercises before eligibility of the patients was considered. In three studies previous exercise was not required before randomizing patients (Table 2).
In four studies crossover from conservative treatment to surgery varied from 19 to 36% during one year follow-up. In the studies by Sihvonen et al. and Yim et al. the cross-over was negligible.
Four studies used KOOS (Knee injury and Osteoarthritis Outcome Score) as their primary outcome; one study used WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index), one study used visual analogue scale for pain and the Lysholm Knee Scoring Scale. Only one study used the meniscal lesion specific WOMET (Western Ontario Meniscal Evaluation Tool) as the primary outcome (Table 2). In one study no patient dropped out from the follow-up, in four trials at least 90% of patients did attend the primary follow-up; in one study (Gauffin et al.) 20% and 7% of patients were lost to follow-up in the exercise and APM groups, respectively.
Figure 3 shows in a flow chart the characteristics of the study questions: the pure intervention effect or effectiveness of intervention in routine praxis; representativeness of the study populations, the treatments before patients were considered eligible for the trial, the contents of the treatments in the actual experiment, and the primary outcome measures. Five of the RCTs had a non-blinded study design, and one had a double blinded design. Only one study, Gauffin et al., had recruited a comprehensive (and thus representative) patient population from its catchment area . There were between study differences in the degree of concomitant osteoarthrosis of the knee. Gauffin et al. was the only study, which had 12 weeks of exercises tried before surgery. The content of the index and control interventions varied as well as the primary outcomes and their decisive time-points. In the trial by Gauffin et al., arthroscopic partial meniscectomy was more effective than exercise therapy. In the four other RCTs no treatment effectiveness was found. The only trial with a double blinded design, Sihvonen et al., found no effectiveness of arthroscopic partial meniscectomy in comparison with sham surgery.
The main finding of this paper is that the study question of assessing pure intervention effect dictates use of a double blinded design, and the study question of assessing effectiveness of intervention in routine health care settings dictates use of non-blinded RCT (Table 1, Fig. 1).
Table 2 and Fig. 3 show that all the RCTs on effectiveness of arthroscopic meniscectomy have studied a different study question, and therefore it was not possible to assess empirically the differences in effectiveness between those studies with non-blinded vs those with a blinded study design.
Double-blinded placebo controlled RCTs are needed to ensure that the intervention has favorable biological effects, and that the effects exceed the potential harms. If there is enough evidence that the intervention does not have any biological effect in a particular patient group, it may not be justified to study effectiveness of this intervention among similar patients in routine health care. Non-blinded RCTs are needed to gain evidence of effectiveness in routine health care circumstances, and double blinded RCTs, which have shown effectiveness of an intervention per se, should be followed by non-blinded RCTs to evaluate the effect in routine health care circumstances.
The effectiveness shown by a double blinded RCT is different from the effectiveness in ordinary health care circumstances, because in the latter placebo effects and non-specific treatment effects add to the pure intervention effect. Consequently, evidence of effectiveness from blinded placebo controlled RCTs, may not be as such valid for decisions in routine health care, and neither for informing patients. The blinded RCTs may underestimate effectiveness in routine health care, where placebo and non-specific effects add to the pure intervention effect; making also the number needed to treat figures biased. Therefore, when informing patients the potential for the placebo effect and non-specific effects to increase further the pure intervention effect should be taken into account.
These findings have implications also for health economics. The most valid way to obtain evidence on cost-effectiveness of a particular intervention is by an economic analysis alongside a randomized controlled trial . However, if a double blinded design has been utilized, the effectiveness consists only of the effect of the pure intervention effect, and does not reflect the routine health care context, where there is also the placebo effect plus the effects provided by the interaction between the patient and the therapists. Thus, cost-effectiveness information from double blind RCTs with economic analysis should be followed by non-blinded trials to answer the routine health care study question. This applies also to modelling studies assessing the incremental cost-effectiveness ratios (ICERs): double blinded RCTs may not provide valid effectiveness estimates, as the placebo effect and other effects by routine health care are not considered. Thresholds for acceptable cost per one health related quality of life (HRQoL) are used in some countries like UK . Again, the ICER estimates based on double blinded placebo-controlled trials may be biased and lead to larger costs per health-related quality of life (HRQoL), than actually occur in routine health care circumstances for which the estimates are intended.
Blinded RCTs are not the best way to assess efficacy (i.e. effectiveness in ideal circumstances) of an intervention, but ideal circumstances (meticulously selected patient population, most competent staff) can be designed equally well also for the non-blinded, routine health care experiments. Double blinded randomized trials are often conducted in optimal circumstances, and in these cases may reveal the best attainable effectiveness estimates, but they may also be conducted - as pragmatic trials - in routine health care contexts. Consequently, the idea of categorically denoting efficacy to double blind RCTs and effectiveness to non-blinded RCTs may not be justifiable.
The extent of the placebo effects and non-specific effects in routine health care may be dependent on the individual patient, and on the health care provider, and how well they are able to communicate between each other. Additional potential modifying factors may be e.g. competence of staff, structures of the health care system and cultural features . There is a need to study these issues as modifiers of effectiveness in different patient groups and health care systems.
The assessment of risk of bias (internal validity) of the two study designs differs. When the study question is on intervention effectiveness per se, the success of double blinding and concealment of treatment allocation are of outmost importance. But, when the study aims to quantify the effectiveness of an intervention in routine health care circumstances, where both patients and health care staff are aware of the interventions chosen, blinding is not justified. Hitherto, the instructions for assessment of risk of bias in RCTs consider success of blinding in all RCT designs an important validity criterion . The present paper argues that this interpretation is not tenable, and a distinction should be made between the two main study questions, which dictate whether to use blinded RCTs or non-blinded RCTs.
The study question analysis on arthroscopic meniscectomy of the knee shows major differences in the characteristics of the six trials, and all the trials are clinically heterogeneous. Therefore comparison of the double blinded vs non-blinded trials is not appropriate. The trial by Gauffin et al., which found effectiveness in routine health care, might be reproduced in a double blinded design with a sham surgery comparison to quantify the pure intervention effect in this representative patient population. In the four other non-blinded RCTs no treatment effectiveness was found, and considering the invasive nature of the intervention, with potential harms there may not be an indication to repeat these experiments in a double blinded design. In the only double blinded trial by Sihvonen et al. no benefits of surgery were found. As the surgery involves potential risks, there is no justification to proceed into non-blinded study design using same patient, intervention and outcome characteristics.
The study question analysis can be applied for planning and assessing of future RCTs, and for assessment of clinical homogeneity in systematic reviews.
When the aim is to assess pure intervention effect, a double blinded RCT is indicated, and when the intention is to assess effectiveness of an intervention in routine health care circumstances, a non-blinded RCT is required. Appropriate blinding of patients and therapists is an essential validity criterion when assessing pure intervention effect, but blinding is contraindicated, when assessing effectiveness of interventions in routine health care. There is a need for non-blinded trials assessing effectiveness and cost-effectiveness in different patient groups and health care settings. When informing patients, the potential for additional effects besides the pure intervention effect should be considered. The study question analysis of the RCTs on arthroscopic meniscectomy of the knee showed that all the trials are clinically heterogeneous, and do not allow a meta-analysis or a comparison of the double blinded vs non-blinded trials.
Benchmarking Controlled Trial
Health Related Quality of Life
Incremental cost-effectiveness ratio
Randomized Controlled Trial
Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.
Wechsler ME, Kelley JM, Boyd IO, Dutile S, Marigowda G, Kirsch I, Israel E, Kaptchuk TJ. Active albuterol or placebo, sham acupuncture, or no intervention in asthma. N Engl J Med. 2011;365(2):119–26.
Veerus P, Fischer K, Hakama M, Hemminki E, EPHT Trial. Results from a blind and a non-blind randomised trial run in parallel: experience from the Estonian postmenopausal hormone therapy (EPHT) trial. BMC Med Res Methodol. 2012;12:44. 2288–12-44
Monk P, Garfjeld Roberts P, Palmer AJ, Bayliss L, Mafi R, Beard D, Hopewell S, Price A. The urgent need for evidence in arthroscopic meniscal surgery: a systematic review of the evidence for operative Management of Meniscal Tears. Am J Sports Med. 2017;45(4):965–73.
Swart NM, van Oudenaarde K, Reijnierse M, Nelissen RG, Verhaar JA, Bierma-Zeinstra SM, Luijsterburg PA. Effectiveness of exercise therapy for meniscal lesions in adults: a systematic review and meta-analysis. J Sci Med Sport. 2016;19(12):990–8.
van de Graaf VA, Wolterbeek N, Mutsaerts EL, Scholtes VA, Saris DB, de Gast A, Poolman RW. Arthroscopic partial meniscectomy or conservative treatment for nonobstructive meniscal tears: a systematic review and meta-analysis of randomized controlled trials. Arthroscopy. 2016;32(9):1855–65. e4
Lamplot JD, Brophy RH. The role for arthroscopic partial meniscectomy in knees with degenerative changes: a systematic review. Bone Joint J. 2016;98-B(7):934–8.
Thorlund JB, Juhl CB, Roos EM, Lohmander LS. Arthroscopic surgery for degenerative knee: systematic review and meta-analysis of benefits and harms. Br J Sports Med. 2015;49(19):1229–35.
Beaufils P, Becker R, Kopf S, Englund M, Verdonk R, Ollivier M, Seil R. Surgical management of degenerative meniscus lesions: the 2016 ESSKA meniscus consensus. Knee Surg Sports Traumatol Arthrosc. 2017;25(2):335–46.
Jarvinen TL, Guyatt GH. Arthroscopic surgery for knee pain. BMJ. 2016;354:i3934.
Malmivaara A. Benchmarking controlled trial-a novel concept covering all observational effectiveness studies. Ann Med. 2015;47(4):332–40.
Herrlin S, Hallander M, Wange P, Weidenhielm L, Werner S. Arthroscopic or conservative treatment of degenerative medial meniscal tears: a prospective randomised trial. Knee Surg Sports Traumatol Arthrosc. 2007;15(4):393–401.
Katz JN, Losina E. Surgery versus physical therapy for meniscal tear and osteoarthritis. N Engl J Med. 2013;369(7):677–8.
Sihvonen R, Paavola M, Malmivaara A, Itala A, Joukainen A, Nurmi H, Kalske J, Jarvinen TL, Finnish degenerative meniscal lesion study (FIDELITY) group. Arthroscopic partial meniscectomy versus sham surgery for a degenerative meniscal tear. N Engl J Med. 2013;369(26):2515–24.
Yim JH, Seon JK, Song EK, Choi JI, Kim MC, Lee KB, Seo HY. A comparative study of meniscectomy and nonoperative treatment for degenerative horizontal tears of the medial meniscus. Am J Sports Med. 2013;41(7):1565–70.
Gauffin H, Tagesson S, Meunier A, Magnusson H, Kvist J. Knee arthroscopic surgery is beneficial to middle-aged patients with meniscal symptoms: a prospective, randomised, single-blinded study. Osteoarthr Cartil. 2014;22(11):1808–16.
Kise NJ, Risberg MA, Stensrud S, Ranstam J, Engebretsen L, Roos EM. Exercise therapy versus arthroscopic partial meniscectomy for degenerative meniscal tear in middle aged patients: randomised controlled trial with two year follow-up. BMJ. 2016;354:i3740.
Petrou S, Gray A. Economic evaluation alongside randomised controlled trials: design, conduct, analysis, and reporting. BMJ. 2011;342:d1548.
McCabe C, Claxton K, Culyer AJ. The NICE cost-effectiveness threshold: what it is and what that means. Pharmacoeconomics. 2008;26(9):733–44.
Crum AJ, Leibowitz KA, Verghese A. Making mindset matter. BMJ. 2017;356:j674.
Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, Cochrane Bias methods group, Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
Asko Lukinmaa is acknowledged for valuable comments of the text; Seija Puro for the graphics; and Ritva Miikki for the literature search strategy. Riitta Malmivaara, BA, MSc is acknowledged for reviewing the English language.
The project was funded by the National Institute of Health and Welfare, Helsinki, Finland.
Availability of data and materials
All data generated or analysed during this study are included in this published article.
Ethics approval and consent to participate
Consent for publication
The author declares that he has no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Malmivaara, A. Pure intervention effect or effect in routine health care - blinded or non-blinded randomized controlled trial. BMC Med Res Methodol 18, 91 (2018). https://doi.org/10.1186/s12874-018-0549-z
- Randomized controlled trial
- Arthroscopic partial meniscectomy
- Clinical comparability
- Study question analysis