Skip to main content
Download PDF

The gap between statistical and clinical significance: time to pay attention to clinical relevance in patient-reported outcome measures of insomnia

BMC Medical Research Methodology volume 24, Article number: 177 (2024) Cite this article

Abstract

Background

Appropriately defining and using the minimal important change (MIC) and the minimal clinically important difference (MCID) are crucial for determining whether the results are clinically significant. The aim of this study is to survey the status of randomized controlled trials (RCTs) for insomnia interventions to assess the inclusion and interpretation of MIC/MCID values.

Methods

We conducted a cross-sectional study to survey the status of RCTs for insomnia interventions to assess the inclusion and appropriate interpretation of MIC/MCID values. A literature search was conducted by searching the main sleep medicine journals indexed in PubMed, the Excerpta Medica Database (EMBASE), and the Cochrane Central Register of Controlled Trials (CENTRAL) to identify a broad range of search terms. We included RCTs with no restriction on the intervention. The included studies used the Insomnia Severity Index (ISI) or the Pittsburgh Sleep Quality Index (PSQI) questionnaire as the outcome measures.

Results

81 eligible studies were identified, and more than one-third of the included studies used MIC/MCID (n = 31, 38.3%). Among them, 21 studies with ISI as the outcome used MIC defined as a relative decrease ranging from 3 to 8 points. The most frequently used MIC value was a 6-point decrease (n = 7), followed by 8-point (n = 6) and 7-point decrease (n = 4), a 4 to 5-points decrease (n = 3), and a 30% reduction from baseline; 6 studies used MCID values, ranging from 2.8 to 4 points. The most frequently used MCID value was a 4-point decrease in the ISI (n = 4). 4 studies with PSQI as the outcome used a 3-point change as the MIC (n = 2) and a 2.5 to 2.7-point difference as MCID (n = 2). 4 non-inferiority design studies considered interval estimation when drawing clinically significant conclusions in their MCID usage.

Conclusions

The lack of consistent MIC/MCID interpretation and usage in outcome measures for insomnia highlights the urgent need for further efforts to address this issue and improve reporting practices.

Peer Review reports

Introduction

Insomnia, characterized by difficulties initiating or maintaining sleep, accompanied by symptoms such as irritability or fatigue during wakefulness, is a prevalent and costly health complaint [1, 2]. The prevalence of insomnia disorders ranges from 5.8 to 19% in European countries [3], while the prevalence of insomnia symptoms in the general adult population ranges from 35 to 50% [4]. Given the subjective nature of insomnia symptoms, patient-reported outcome measures (PROMs) play an important role in for evaluating insomnia symptom severity and treatment-related changes. PROMs provide information directly from the patient, without interpretation by a clinician or any other party, and remain important for assessing insomnia despite the common use of objective assessments, such as actigraphy, in recent clinical studies.

The two most frequently utilized PROMs for insomnia are the Insomnia Severity Index (ISI) and the Pittsburgh Sleep Quality Index (PSQI), which are used to assess insomnia severity and evaluate treatment effects [5, 6]. To properly interpret changes in PROMs, it is essential to differentiate between statistically significant and clinically meaningful differences [7]. Various terms are used to describe clinically meaningful effects, often inconsistently, making interpretation difficult. The concept of the Minimal Important Change (MIC) represents the smallest change in health status perceivable by patients compared to a baseline. On the other hand, the Minimal Clinically Important Difference (MCID) refers to the smallest difference between two groups that is deemed beneficial enough by patients to be clinically important [8, 9].

Despite many studies demonstrating statistically significant effects of interventions for both pharmacological and non-pharmacological treatments of insomnia [10,11,12], the focus of current research often lies solely on statistical significance, neglecting clinical significance. Evaluating the efficacy of insomnia treatments requires considering both statistical and clinical significance. Furthermore, appropriately defining and reporting the MIC/MCID is crucial to concluding whether the results are clinically significant. Currently, the concepts of MIC and MCID are frequently confused in clinical study reporting. The question is then, what is the status of MIC/MCID reporting in clinical studies related to insomnia? The aim of this study is to survey the status of RCTs for insomnia interventions to assess the inclusion and appropriate interpretation of MIC/MCID values.

Methods

The reporting of the current study was based on the PRISMA 2020 statement whenever possible [13].

Literature search

The literature search was conducted in 30-Sep-2023. Considering the terminology of MIC/MCID various a lot, MIC/MCID were not defined as an item in the search strategies. First, literature search was conducted by searching the main sleep medicine journals indexed in PubMed, Excerpta Medica Database (EMBASE), and the Cochrane Central Register of Controlled Trials (CENTRAL) to identify a broad range of search terms. Next, Endnote X20 was used to screen the records. Two investigators independently reviewed titles and abstracts of search results to identify potentially eligible references. The full texts were screened by researchers independently based on inclusion and exclusion criteria. Discrepancies were resolved through discussion with corresponding authors. By a prior search in Scimago Journal & Country Rank (https://www.scimagojr.com/), we identified 22 journals in sleep medicine and general medicine, includes “Sleep”, “Sleep medicine”, “Journal of Sleep Research”, “Journal of clinical sleep medicine”. Four of them were excluded as they were listed as predatory journals. A full list of the remaining 18 journals detailed and search strategy was presented in the Supplementary file 1 and has been recorded in the previous study [14].

Study screening and selection criteria

We included randomized controlled trials with no restriction on the intervention. Included randomized controlled trials used ISI or/and PSQI questionnaire as the outcomes and published in the aforementioned journals of sleep medicine without restriction to study type (i.e., superiority, non-inferiority, or equivalence trails). Interim analyses, pooled analyses, and post-hoc analyses (i.e., subgroup analysis, secondary analysis) of RCTs were excluded. Non-human studies and studies not published as full papers (such as conference abstracts, study protocols) were excluded. Pre-print articles or grey literature without peer-review process were also excluded. Regarding the aspects of insomnia, the MIC values from Morin and Yang were the most frequently referenced sources, primarily estimated through an anchor-based approach [6, 9]. Table 1 provides a summary of the sources of MIC/MCID values for ISI and PSQI. Studies that defined the post-treatment endpoint to achieve certain value (remission) were excluded.

Data collection and statistics

An information extraction table using Microsoft Excel 365 was made. Before formal data collection, two researchers were trained. Three included studies were assessed using the extraction table as a pretest for consistency validation. Data was collected and coded by two researchers. Discrepancies were resolved through discussion and confirmed by corresponding authors. The category data was presented as number and percent. The details items were as follows: (1) Basic characteristics: title, authors, publication years and the journal, the country and region where the study was carried out, target population, sample size, type of intervention, and types of PROM. (2) MIC/MCID related characteristics: The complete article was reviewed for mention and definition of MIC/MCID. Credit was given if MIC/MCID was defined for the primary outcomes and was mentioned in any section of the article. As the definition of MIC/MCID various a lot, we mainly focus on the clinically instead of minimal, that is, the moderate clinical meaning was also included (e.g., more than 30% reduction compared with baseline). The studies were placed into 1 of 3 categories according to (a) only defined and used MIC, (b) only defined and used MCID, and (c) defined and used both MIC/MCID. The classification for MIC/MCID was based on a two-stage procedure. First, we classified the value based on the terminology. Specifically, if the authors used terminology such as response, change, decrease, or improvement, we expected it as MIC. Conversely, if the authors used terminology difference, we expected it as MCID. Second, we determine whether the MIC or MCID that was actually used by examining if the value was within-group (classified as MIC) or between-group (classified as MCID). For articles that did not mention MCID, the published literature was searched to determine if the MCID for the reported outcome existed prior to publication of the article.

Results

Search results

After screening titles and abstracts, a total of 164 studies were assessed for edibility. Among them, 81 RCTs met the inclusion criteria and were included in the final dataset. Figure 1 illustrates the study flowchart. The majority of included studies focused on non-pharmacological interventions, followed by complementary medicine and therapies. Cognitive Behavioral Therapy for Insomnia (CBT-I) studies compromised the largest proportion. Only 5 studies assessed the effectiveness of hypnotics. Out of the eligible studies, 31 used MIC/MCID values. Among the included RCTs, MIC/MCID were used under 11 different definitions including “treatment response”, “minimal significant difference”, “clinically meaningful improvement”, “clinically significant improvement”, “clinically significant treatment effect”, “clinically significant change”, “clinically relevant difference”, “minimally important difference”, “minimally clinically important differences”, “slight clinical improvement”, and “clinically important difference”. All included studies evaluated at least one intervention group and one comparison group. Among them, 3 studies utilized a cross-over design, 2 studies employed a cluster design, and 4 studies followed a non-inferiority design.

The MIC/MCID used condition

A total of 31 studies used MIC/MCID, 23 studies used MIC and 8 studies used MCID. Table 2 provides a summary of the characteristics, as well as the defined and used MIC/MCID values of the included studies. Most MIC were used as a relative decrease from 3 to 8 points. 6-points of ISI was the most frequently used value for the MIC (n = 7), followed by 8- and 7-points of ISI (n = 6 and 4, respectively), and a range from 4- to 4.7-points (n = 3). Two source to justify the choice of value or definition of MIC (ranges from 6 to 8) accompanied most of the definitions [6, 9]. A decrease of 30% reduction of ISI was also defined as a MIC in 1 study (Table 2). The eFigure 1 of supplemental file 1 present the chronological illustration of MIC/MCID source and citations by included studies. All MCIDs were used as a relative decrease from 2.8 to 4 points. 4-points of ISI was the most frequently used value for the MCID (n = 4). 4 studies with PSQI as the outcome used a 3-point change as the MIC (n = 2) and a 2.5 to 2.7-point difference as MCID (n = 2). Although all studies discount the MCID, the source to justify the choice of value or definition of MCID accompanied most [9]. For MCID usage, 4 non-inferiority designed studies considered the interval estimation when drawing the conclusion of clinically significant. Figure 2 illustrates the distribution pattern of MIC/MCID values in the included RCTs.

The mismatches from studies and sources

Based on the criteria, treatment responses reported by 6 studies were classified as MIC. Additionally, 5 out of 8 studies used MCID were judged as MIC. Among the 23 studies used MIC terminology, 4 were judged as MCID (Fig. 1). Among 19 studies referring MIC as a source, the values in 17 studies aligned with the source. Among 5 studies referring MCID as a source, the values in 3 studies matched the sources (Fig. 1). Table 3 represents the frequency of comparison of expected vs. used (judgement) MIC/MCID values.

Included studies reported responder analysis

We further aggregated the outcomes of responder analyses, which were documented in 11 studies utilizing 5 distinct definitions of responder. The selection of the cut-off value to define a responder appeared arbitrary. Among the various definitions employed, the most prevalent approach was based on an ISI reduction of less than 8 points (n = 5), followed by a reduction of over 50% in ISI (n = 3). These references are provided in Supplemental file 2.

Fig. 1
figure 1

Study flowchart

Full size image
Table 1 Source of MIC/MCID values for ISI and PSQI
Full size table
Table 2 Characteristics of reported MIC/MCID values of included studies
Full size table
Table 3 Comparison of expected vs. used (judgement) MIC/MCID values
Full size table
Fig. 2
figure 2

Distribution pattern of MIC/MCID values in the included RCTs

Full size image

Discussion

To date, there has been considerable variation in the definitions of MIC and MCID, leading to confusion in terminologies found within clinical study reports. However, it is important to note that this study does not aim to emphasize spelling of the distinction between “MIC” and “MCID” or similar words. Rather, its focus is to encourage clinical researchers to provide clarity on what constitutes clinically significant changes, whether in terms of change from baseline to end-of-treatment (MIC) or differences between intervention and control groups (MCID). The primary function of an RCT is to compare groups and identify differences between them. Consequently, in the reporting of RCT results, the focus should be on the difference between groups although the change within a group is important for individual. This emphasis on MCID is crucial, as relying on MIC could potentially diminish the distinctive role of RCTs. This is because a single-group study has the capacity to ascertain MIC, but not MCID.

In this study, we identified an analysis of 81 studies focusing on the MIC/MCID for two PROMs used to assess interventions for insomnia. Our findings revealed that out of these 81 studies, 31 reported MIC/MCID values. Most of the included studies focused on describing the MIC rather than the MCID. According to the criteria, out of 18 studies that reported and defined (minimal) clinically significant change, 12 were categorized as MIC and 5 were categorized as MCID. Out of 7 studies that reported and defined (minimal) clinically important difference, 3 were classified as MCID and 4 were categorized as MIC. This lack of standardized MIC/MCID values and inconsistent methodologies present a significant challenge. Different MIC/MCID values can lead to varying interpretations of clinical significance. Considering the crucial role that MIC/MCID plays in assessing the meaningfulness of changes or differences in PROMs, it is surprising that less than half of the RCTs published in prominent sleep medicine journals provide MIC/MCID values. Although many of these studies report statistically significant results, the absence of defined and reported MIC/MCID values is concerning. The lack of MIC/MCID reporting hampers the interpretation of clinical context of the study findings [50]. The current reporting practices surrounding MIC/MCID in RCTs investigating interventions for insomnia offer limited assistance in facilitating meaningful interpretation or providing additional insights. Given these challenges, urgent efforts are required to standardize the methodology for reporting MIC/MCID values in research on insomnia.

Several methods are available to estimate the MIC, including distribution-based method, anchor-based method, and consensus-based method. However, the appropriate values of MCID for insomnia related PROMs are still lacking. The most frequently reported MIC and MCID values for the ISI were 6 points and 4 points, respectively. It is crucial to note that MIC and MCID are fundamentally different concepts, although they are often used interchangeably, leading to confusion among clinicians and researchers. MIC represents the change in the score on an outcome measure compared to the score at an earlier time point within a person or the mean change within a group over time. However, the challenge arises when within-group mean change is erroneously referred to as the treatment effect or response to treatment. Change from baseline to follow-up incorporates changes resulting from the natural course of the condition, regression to the mean, nonspecific effects, and treatment effects. It is important to consider these factors, regardless of the specific condition and treatment being studied. Within-group change over time should not be equated with the treatment effect, although change scores provide valuable information. They estimate what is likely to happen when a patient receives the study treatment, but they are distinct from the treatment effect and do not represent the treatment response. On the other hand, the calculation of difference requires data from two groups of people. The between-group difference is determined by subtracting the mean score on an outcome measure in the comparison group from the mean score in the treatment group. This difference, typically assessed at follow-up or as the difference in change between the two groups, can reasonably be referred to as the treatment effect or treatment response because it does not include natural history, regression to the mean, or nonspecific effects. It is important to emphasize that the treatment effect is a comparative effect, reflecting what can be expected if a patient receives treatment compared to what can be expected if the same patient receives the comparison. Confusion may arise when authors draw conclusions solely based on within-group changes in RCTs, particularly when there is no difference between the treatment and comparison groups. Stating that both treatments are effective is rarely a valid conclusion from an RCT, unless specific circumstances warrant it. Within-group change in an RCT does not represent the treatment effect any more than the results from a single-group study, as it still incorporates natural history, regression to the mean, and nonspecific effects. Interpreting RCT findings in this manner undermines the purpose of randomization and the inclusion of a comparison group.

In addition to the mean change, reductions of more than 30% have been suggested to indicate the MIC one study [40], and consensus on this threshold has also been informed by researchers considering the effects of measurement error from distribution studies [51, 52]. Although some studies proposed values do not strictly define a MIC, they provide a responder analysis in terms of PROMs. A reduction of more than 50% has been reported in some studies to reflect a more substantial improvements compared to MIC. In a recent consensus statement to improve the interpretation of clinical trials, 50% was suggested to reflect an acceptable responsive ratio [53]. Some studies have also reported the effect size (Cohen’s d) as a parameter to suggest clinically significance. However, it is important to note that the responder analysis and effect size cannot replace MIC/MCID [54, 55]. Responder analyses, which often require continuous outcomes to be categorized, may result in the loss of information and reduced statistical power, especially when high rates missingness dilute the dataset, multiply imputing the continuous variable was less biased and had well-controlled coverage probabilities of the 95% confidence interval (CI) compared to imputing the dichotomous response [56]. Therefore, they are recommended as secondary analyses to enhance the interpretability of the main analysis. It is worth noting that responder analyses may introduce relative reporting methods (e.g. as percentage of participants who improved) which, in the case of small absolute differences, can make interventions appear more effective [57].

One major advantage in adding MIC/MCID into the interpretation is the ability to determine when the available evidence is insufficient to draw a conclusion. The recent CONSORT PRO Extension (Consolidated Standards of Reporting Trials patient reported outcomes) encourages authors to include discussion of an MIC/MCID in reports of clinical trials if the PROM used in the study is validated [58, 59]. This concept is also particularly valuable in systematic reviews that synthesize data from different studies, as it allows for an assessment of the adequacy of the evidence [60]. However, given the multiplicity of MCID estimates often available for a given patient reported outcome measure and non-standardized methodology, researchers and decision makers in search of MCIDs need to critically evaluate the quality of the estimates [61, 62], it is crucial to accurately interpret the MCID to arrive at an appropriate conclusion. CIs that fall within the range defined by the MCID indicate evidence of no difference, while CIs outside this range are considered establishing clinically significant differences. Most included studies (86.2%) reporting MIC/MCID did not take 95% CI into consideration when drawing a clinically significant conclusion. Only four non-inferiority designed RCTs reporting MCID studies take the CIs into consideration when drawing conclusions. Study results are considered statistically significant when the lower limit of the 95% CI exceeds the null effect. Similarly, study results could be considered clinically significant if the MCID lies outside the 95% CI of the PROM scores. A previous review summarized 4 conditions of clinical importance, depending on the relationship of the MCID of the intervention to the point estimate and the 95% CI surrounding it: (1) when the MCID is smaller than the lower limit of the 95% CI (definite clinical importance); (2) when the MCID is greater than the lower limit of the 95% CI, but smaller than the point estimate of the efficacy of the intervention (probable clinical importance); (3) when the MCID is less than the upper limit of the 95% CI, but greater than the point estimate of the efficacy of the intervention (possible clinical importance); and (4) when the MCID is greater than the upper limit of the 95% CI (definitely not clinical importance) [63]. Moreover, previous research has highlighted considerable variability in MIC/MCID as a function of estimation method, population and context, suggesting the importance of considering such factors when appraising the appropriateness of published MIC/MCID for use in clinical research and practice. However, we found that the included studies mostly only refer to the study by Yang et al. in 2009 [9]. Based on results of this study, a 6-point reduction is recommended to represent a clinically meaningful improvement in individuals with primary insomnia. It also noted that research on generalizability of the recommended MIC in this study to other patient populations and other type of treatment interventions is needed.

Several limitations should be acknowledged in this study. First, the data were exclusively obtained from mainstream sleep medicine journals, without considering other general medicine or psychiatric journals. Besides, the analysis focused solely on two frequently used PROMs for insomnia, while other relevant PROMs such as Sleep Assessment Questionnaire (SAQ), St. Mary’s Hospital Questionnaire, Leeds Sleep Evaluation Questionnaire, Post-Sleep Evaluation Questionnaire were not included [64]. Consequently, the generalizability of the findings may be constrained. Moreover, this study did not provide a recommendation in terms of choosing an appropriate MIC or MCID for further study. Last, the study did not explore the reporting of MIC/MCID values in meta-analysis and clinical practice guidelines, which is crucial for interpretation of results that encompass synthesized data with clinically significance [65, 66].

In conclusion, future researchers of insomnia related RCT should be more vigilant in using PROMs that have a defined MIC/MCID in the stage of the study design. If the PROM does not have a defined MCID, researchers should be cautious about interpreting results as clinically significant based solely on a P-value or MIC. Researchers should consider MIC/MCID and 95% CI in decision making. The width of the CIs and its location with respect to the MIC/MCID are important considerations in making clinical conclusion. Implementation guidelines on MIC/MCID reporting and estimation for future studies are urgently needed.

Data availability

The data that support the findings of this study are available in Supplementary files.

Abbreviations

MCID:

Minimal clinically important difference

MIC:

Minimal important change

PROM:

Patient-reported outcomes measures

RCT:

Randomized controlled trial

ISI:

Insomnia Severity Index

PSQI:

Pittsburgh Sleep Quality Index

CBT-I:

Cognitive Behavioral Therapy for Insomnia

CI:

Confidence interval

PRISMA:

Preferred Reporting Items for Systematic reviews and Meta-Analyses

References

  1. Sutton EL. Insomnia. Ann Intern Med. 2021;174(3):ITC33–48.

    Article  PubMed  Google Scholar 

  2. Albrecht JS, Wickwire EM, Vadlamani A, Scharf SM, Tom SE. Trends in Insomnia diagnosis and treatment among Medicare beneficiaries, 2006–2013. Am J Geriatr Psychiatry. 2019;27(3):301–9.

    Article  PubMed  Google Scholar 

  3. Riemann D, Baglioni C, Bassetti C, Bjorvatn B, Dolenc Groselj L, Ellis JG, Espie CA, Garcia-Borreguero D, Gjerstad M, Goncalves M, et al. European guideline for the diagnosis and treatment of insomnia. J Sleep Res. 2017;26(6):675–700.

    Article  PubMed  Google Scholar 

  4. Walsh JK, Coulouvrat C, Hajak G, Lakoma MD, Petukhova M, Roth T, Sampson NA, Shahly V, Shillington A, Stephenson JJ, et al. Nighttime Insomnia symptoms and perceived health in the America Insomnia Survey (AIS). Sleep. 2011;34(8):997–1011.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Buysse DJ, Reynolds CF 3rd, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28(2):193–213.

    Article  CAS  PubMed  Google Scholar 

  6. Morin CM, Belleville G, Belanger L, Ivers H. The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep. 2011;34(5):601–8.

    Article  PubMed  PubMed Central  Google Scholar 

  7. The B. Significance testing - are we ready yet to abandon its use? Curr Med Res Opin. 2011;27(11):2087–90.

    Article  PubMed  Google Scholar 

  8. King MT. A point of minimal important difference (MID): a critique of terminology and methods. Expert Rev Pharmacoecon Outcomes Res. 2011;11(2):171–84.

    Article  PubMed  Google Scholar 

  9. Yang M, Morin CM, Schaefer K, Wallenstein GV. Interpreting score differences in the Insomnia Severity Index: using health-related outcomes to define the minimally important difference. Curr Med Res Opin. 2009;25(10):2487–94.

    Article  PubMed  Google Scholar 

  10. Brasure M, Fuchs E, MacDonald R, Nelson VA, Koffel E, Olson CM, Khawaja IS, Diem S, Carlyle M, Wilt TJ, et al. Psychological and behavioral interventions for managing Insomnia Disorder: an evidence report for a clinical Practice Guideline by the American College of Physicians. Ann Intern Med. 2016;165(2):113–24.

    Article  PubMed  Google Scholar 

  11. Wilt TJ, MacDonald R, Brasure M, Olson CM, Carlyle M, Fuchs E, Khawaja IS, Diem S, Koffel E, Ouellette J, et al. Pharmacologic treatment of Insomnia Disorder: an evidence report for a clinical Practice Guideline by the American College of Physicians. Ann Intern Med. 2016;165(2):103–12.

    Article  PubMed  Google Scholar 

  12. Chan NY, Chan JWY, Li SX, Wing YK. Non-pharmacological approaches for management of Insomnia. Neurotherapeutics. 2021;18(1):32–43.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. 2021;10(1):89.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Xu C, Doi SAR, Zhou X, Lin L, Furuya-Kanamori L, Tao F. Data reproducibility issues and their potential impact on conclusions from evidence syntheses of randomized controlled trials in sleep medicine. Sleep Med Rev. 2022;66:101708.

    Article  PubMed  Google Scholar 

  15. Dworkin RH, Turk DC, McDermott MP, Peirce-Sandner S, Burke LB, Cowan P, Farrar JT, Hertz S, Raja SN, Rappaport BA, et al. Interpreting the clinical importance of group differences in chronic pain clinical trials: IMMPACT recommendations. Pain. 2009;146(3):238–44.

    Article  PubMed  Google Scholar 

  16. Krieger T, Urech A, Duss SB, Blattler L, Schmitt W, Gast H, Bassetti C, Berger T. A randomized controlled trial comparing guided internet-based multi-component treatment and internet-based guided sleep restriction treatment to care as usual in insomnia. Sleep Med. 2019;62:43–52.

    Article  PubMed  Google Scholar 

  17. Patterson PD, Martin SE, Brassil BN, Hsiao WH, Weaver MD, Okerman TS, Seitz SN, Patterson CG, Robinson K. The Emergency Medical Services Sleep Health Study: a cluster-randomized trial. Sleep Health. 2023;9(1):64–76.

    Article  PubMed  Google Scholar 

  18. Bergdahl L, Broman JE, Berman AH, Haglund K, von Knorring L, Markstrom A. Auricular Acupuncture and Cognitive Behavioural Therapy for Insomnia: A Randomised Controlled Study. Sleep Disord 2016, 2016:7057282.

  19. Yeung WF, Chung KF, Tso KC, Zhang SP, Zhang ZJ, Ho LM. Electroacupuncture for residual insomnia associated with major depressive disorder: a randomized controlled trial. Sleep. 2011;34(6):807–15.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Schuffelen J, Maurer LF, Lorenz N, Rotger A, Pietrowsky R, Gieselmann A. The clinical effects of digital cognitive behavioral therapy for insomnia in a heterogenous study sample: results from a randomized controlled trial. Sleep 2023.

  21. McCurry SM, Shortreed SM, Von Korff M, Balderson BH, Baker LD, Rybarczyk BD, Vitiello MV. Who benefits from CBT for Insomnia in primary care? Important patient selection and trial design lessons from longitudinal results of the lifestyles trial. Sleep. 2014;37(2):299–308.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Lancee J, van Straten A, Morina N, Kaldo V, Kamphuis JH. Guided online or face-to-face cognitive behavioral treatment for Insomnia: a Randomized wait-list controlled trial. Sleep. 2016;39(1):183–91.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Fu C, Zhao N, Liu Z, Yuan LH, Xie C, Yang WJ, Yu XT, Yu H, Chen YF. Acupuncture improves peri-menopausal insomnia: a Randomized Controlled Trial. Sleep 2017, 40(11).

  24. Bean HR, Diggens J, Ftanou M, Alexander M, Stafford L, Bei B, Francis PA, Wiley JF. Light enhanced cognitive behavioral therapy for insomnia and fatigue during chemotherapy for breast cancer: a randomized controlled trial. Sleep 2022, 45(3).

  25. Arnedt JT, Conroy DA, Mooney A, Furgal A, Sen A, Eisenberg D. Telemedicine versus face-to-face delivery of cognitive behavioral therapy for insomnia: a randomized controlled noninferiority trial. Sleep 2021, 44(1).

  26. Lee B, Kim BK, Kim HJ, Jung IC, Kim AR, Park HJ, Kwon OJ, Lee JH, Kim JH. Efficacy and safety of electroacupuncture for Insomnia Disorder: a Multicenter, Randomized, Assessor-Blinded, Controlled Trial. Nat Sci Sleep. 2020;12:1145–59.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Ousmen A, Touraine C, Deliu N, Cottone F, Bonnetain F, Efficace F, Bredart A, Mollevi C, Anota A. Distribution- and anchor-based methods to determine the minimally important difference on patient-reported outcome questionnaires in oncology: a structured review. Health Qual Life Outcomes. 2018;16(1):228.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Yeung WF, Chung KF, Zhang SP, Yap TG, Law AC. Electroacupuncture for primary insomnia: a randomized controlled trial. Sleep. 2009;32(8):1039–47.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Blom K, Jernelov S, Ruck C, Lindefors N, Kaldo V. Three-year Follow-Up of Insomnia and Hypnotics after Controlled Internet treatment for Insomnia. Sleep. 2016;39(6):1267–74.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Birling Y, Zhu X, Avard N, Tannous C, Fahey PP, Sarris J, Bensoussan A. Zao Ren An Shen capsule for insomnia: a double-blind, randomized, placebo-controlled trial. Sleep 2022, 45(2).

  31. Alessi CA, Fung CH, Dzierzewski JM, Fiorentino L, Stepnowsky C, Rodriguez Tapia JC, Song Y, Zeidler MR, Josephson K, Mitchell MN et al. Randomized controlled trial of an integrated approach to treating insomnia and improving the use of positive airway pressure therapy in veterans with comorbid insomnia disorder and obstructive sleep apnea. Sleep 2021, 44(4).

  32. Store SJ, Tillfors M, Wastlund E, Angelhoff C, Andersson G, Norell-Clarke A. The effects of a sleep robot intervention on sleep, depression and anxiety in adults with insomnia-A randomized waitlist-controlled trial. J Sleep Res. 2023;32(3):e13758.

    Article  PubMed  Google Scholar 

  33. Vedaa O, Hagatun S, Kallestad H, Pallesen S, Smith ORF, Thorndike FP, Ritterband LM, Sivertsen B. Long-Term effects of an unguided online cognitive behavioral therapy for chronic insomnia. J Clin Sleep Med. 2019;15(1):101–10.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Savard J, Ivers H, Savard MH, Morin CM, Caplette-Gingras A, Bouchard S, Lacroix G. Efficacy of a stepped care approach to deliver cognitive-behavioral therapy for insomnia in cancer patients: a noninferiority randomized controlled trial. Sleep 2021, 44(11).

  35. Buysse DJ, Germain A, Moul DE, Franzen PL, Brar LK, Fletcher ME, Begley A, Houck PR, Mazumdar S, Reynolds CF. Efficacy of brief behavioral treatment for chronic insomnia in older adults. Arch Intern Med. 2011;171(10):887–95. 3rd et al.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Perrault AA, Pomares FB, Smith D, Cross NE, Gong K, Maltezos A, McCarthy M, Madigan E, Tarelli L, McGrath JJ, et al. Effects of cognitive behavioral therapy for insomnia on subjective and objective measures of sleep and cognition. Sleep Med. 2022;97:13–26.

    Article  PubMed  Google Scholar 

  37. Ford ME, Geurtsen GJ, Groet E, Rambaran Mishre RD, Van Bennekom CAM, Van Someren EJW. A blended eHealth intervention for insomnia following acquired brain injury: a randomised controlled trial. J Sleep Res. 2023;32(1):e13629.

    Article  PubMed  Google Scholar 

  38. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care. 2003;41(5):582–92.

    Article  PubMed  Google Scholar 

  39. Rahman SA, Nathan MD, Wiley A, Crawford S, Cohn AY, Harder JA, Grant LK, Erickson A, Srivastava A, McCormick K et al. A double-blind, randomized, placebo-controlled trial of suvorexant for the treatment of vasomotor symptom-associated insomnia disorder in midlife women. Sleep 2022, 45(3).

  40. Kallestad H, Scott J, Vedaa O, Lydersen S, Vethe D, Morken G, Stiles TC, Sivertsen B, Langsrud K. Mode of delivery of cognitive behavioral therapy for Insomnia: a randomized controlled non-inferiority trial of digital and face-to-face therapy. Sleep 2021, 44(12).

  41. Kalmbach DA, Cheng P, O’Brien LM, Swanson LM, Sangha R, Sen S, Guille C, Cuamatzi-Castelan A, Henry AL, Roth T, et al. A randomized controlled trial of digital cognitive behavioral therapy for insomnia in pregnant women. Sleep Med. 2020;72:82–92.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Ayabe N, Okajima I, Nakajima S, Inoue Y, Watanabe N, Yamadera W, Uchimura N, Tachimori H, Kamei Y, Mishima K. Effectiveness of cognitive behavioral therapy for pharmacotherapy-resistant chronic insomnia: a multi-center randomized controlled trial in Japan. Sleep Med. 2018;50:105–12.

    Article  PubMed  Google Scholar 

  43. Redeker NS, Yaggi HK, Jacoby D, Hollenbeak CS, Breazeale S, Conley S, Hwang Y, Iennaco J, Linsky S, Nwanaji-Enwerem U et al. Cognitive behavioral therapy for insomnia has sustained effects on insomnia, fatigue, and function among people with chronic heart failure and insomnia: the HeartSleep Study. Sleep 2022, 45(1).

  44. Mercier J, Ivers H, Savard J. A non-inferiority randomized controlled trial comparing a home-based aerobic exercise program to a self-administered cognitive-behavioral therapy for insomnia in cancer patients. Sleep 2018, 41(10).

  45. Hartescu I, Morgan K, Stevinson CD. Increased physical activity improves sleep and mood outcomes in inactive people with insomnia: a randomized controlled trial. J Sleep Res. 2015;24(5):526–34.

    Article  PubMed  Google Scholar 

  46. Redeker NS, Jeon S, Andrews L, Cline J, Jacoby D, Mohsenin V. Feasibility and efficacy of a self-management intervention for Insomnia in stable heart failure. J Clin Sleep Med. 2015;11(10):1109–19.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Feuerstein S, Hodges SE, Keenaghan B, Bessette A, Forselius E, Morgan PT. Computerized cognitive behavioral therapy for Insomnia in a Community Health setting. J Clin Sleep Med. 2017;13(2):267–74.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Yeh CH, Suen LK, Shen J, Chien LC, Liang Z, Glick RM, Morone NE, Chasens ER. Changes in Sleep with Auricular Point Acupressure for Chronic Low Back Pain. Behav Sleep Med. 2016;14(3):279–94.

    Article  PubMed  Google Scholar 

  49. Wong KY, Chung KF, Au CH. Low-intensity cognitive behavioral therapy for Insomnia as the entry of the stepped-care model in the community: a Randomized Controlled Trial. Behav Sleep Med. 2021;19(3):378–94.

    Article  PubMed  Google Scholar 

  50. Carrasco-Labra A, Devji T, Qasim A, Phillips M, Johnston BC, Devasenapathy N, Zeraatkar D, Bhatt M, Jin X, Brignardello-Petersen R, et al. Serious reporting deficiencies exist in minimal important difference studies: current state and suggestions for improvement. J Clin Epidemiol. 2022;150:25–32.

    Article  PubMed  Google Scholar 

  51. Snapinn SM, Jiang Q. Responder analyses and the assessment of a clinically relevant treatment effect. Trials. 2007;8:31.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Health USDo, Human Services FDACfDE, Research, Health USDo, Human Services FDACfBE, Research, Health USDo, Human Services FDACfD, Radiological H. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4:79.

    Article  Google Scholar 

  53. Kroenke K, Miksch TA, Spaulding AC, Mazza GL, DeStephano CC, Niazi SK, Illies AJC, Bydon M, Novotny PJ, Goyal A, et al. Choosing and using patient-reported outcome measures in clinical practice. Arch Phys Med Rehabil. 2022;103(5S):S108–17.

    Article  PubMed  Google Scholar 

  54. Johnston BC, Alonso-Coello P, Friedrich JO, Mustafa RA, Tikkinen KAO, Neumann I, Vandvik PO, Akl EA, da Costa BR, Adhikari NK, et al. Do clinicians understand the size of treatment effects? A randomized survey across 8 countries. CMAJ. 2016;188(1):25–32.

    Article  PubMed  Google Scholar 

  55. Collister D, Bangdiwala S, Walsh M, Mian R, Lee SF, Furukawa TA, Guyatt G. Patient reported outcome measures in clinical trials should be initially analyzed as continuous outcomes for statistical significance and responder analyses should be reserved as secondary analyses. J Clin Epidemiol. 2021;134:95–102.

    Article  PubMed  Google Scholar 

  56. Floden L, Bell ML. Imputation strategies when a continuous outcome is to be dichotomized for responder analysis: a simulation study. BMC Med Res Methodol. 2019;19(1):161.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Cates C, Karner C. Clinical importance cannot be ruled out using mean difference alone. BMJ. 2015;351:h5496.

    Article  PubMed  Google Scholar 

  58. Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD, Group CP. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA. 2013;309(8):814–22.

    Article  CAS  PubMed  Google Scholar 

  59. Calvert M, Kyte D, Mercieca-Bebber R, Slade A, Chan AW, King MT, the, Hunn S-PROG, Bottomley A, Regnault A et al. A : Guidelines for Inclusion of Patient-Reported Outcomes in Clinical Trial Protocols: The SPIRIT-PRO Extension. JAMA 2018, 319(5):483–494.

  60. Yu CW, Nanji K, Hatamnejad A, Gemae M, Joarder I, Achunair A, Devji T, Phillips M, Zeraatkar D, Steel DH, et al. Patient-reported outcome measure use in Guidelines published by the American Academy of Ophthalmology: a review. Ophthalmology. 2023;130(11):1201–11.

    Article  PubMed  Google Scholar 

  61. Devji T, Carrasco-Labra A, Qasim A, Phillips M, Johnston BC, Devasenapathy N, Zeraatkar D, Bhatt M, Jin X, Brignardello-Petersen R, et al. Evaluating the credibility of anchor based estimates of minimal important differences for patient reported outcomes: instrument development and reliability study. BMJ. 2020;369:m1714.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Carrasco-Labra A, Devji T, Qasim A, Phillips MR, Wang Y, Johnston BC, Devasenapathy N, Zeraatkar D, Bhatt M, Jin X, et al. Minimal important difference estimates for patient-reported outcomes: a systematic survey. J Clin Epidemiol. 2021;133:61–71.

    Article  PubMed  Google Scholar 

  63. Man-Son-Hing M, Laupacis A, O’Rourke K, Molnar FJ, Mahon J, Chan KB, Wells G. Determination of the clinical importance of study results. J Gen Intern Med. 2002;17(6):469–76.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Vernon MK, Dugar A, Revicki D, Treglia M, Buysse D. Measurement of non-restorative sleep in insomnia: a review of the literature. Sleep Med Rev. 2010;14(3):205–12.

    Article  PubMed  Google Scholar 

  65. Chan LS. Minimal clinically important difference (MCID)--adding meaning to statistical inference. Am J Public Health. 2013;103(11):e24–25.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Takano Y, Ibata R, Machida N, Ubara A, Okajima I. Effect of cognitive behavioral therapy for insomnia in workers: a systematic review and meta-analysis of randomized controlled trials. Sleep Med Rev. 2023;71:101839.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors thank Prof. Chang Xu for his professional suggestion and helps regarding study concepts and data extraction process.

Funding

This study is funded by the International Postdoctoral Exchange Fellowship Program (Talent-Introduction Program, Grant No. YJ20220238) and China Postdoctoral Science Foundation (Grant No. 2023TQ0018).

Author information

Author notes

  1. Zongshi Qin and Yidan Zhu contributed equally to this study.

Authors and Affiliations

  1. Clinical Research Institute, Institute of Advanced Clinical Medicine, Peking University, Beijing, China

    Zongshi Qin & Yidan Zhu

  2. Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    Dong-Dong Shi

  3. School of Life Sciences, Beijing University of Chinese Medicine, Beijing, China

    Rumeng Chen & Sen Li

  4. Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China

    Jiani Wu

Authors
  1. Zongshi Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yidan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dong-Dong Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rumeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiani Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conception and design: ZQ, SL, and JW; Manuscript drafting: ZQ and YZ; Data collection: ZQ, YZ, and RC; Data analysis and result interpretation: ZQ, YZ, DS, SL, and JW; Methodology guidance: ZQ and JW; Manuscript editing: ZQ, YZ, SL, and JW. All authors have read and approved the manuscript.

Corresponding authors

Correspondence to Zongshi Qin, Sen Li or Jiani Wu.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, Z., Zhu, Y., Shi, DD. et al. The gap between statistical and clinical significance: time to pay attention to clinical relevance in patient-reported outcome measures of insomnia. BMC Med Res Methodol 24, 177 (2024). https://doi.org/10.1186/s12874-024-02297-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-024-02297-0

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us