Table 2 An example of the application of quality criteria

Selected study: Shirani A, Zhao Y, Karim ME, et al. Association between use of interferon beta and progression of disability in patients with relapsing-remitting multiple sclerosis. JAMA 2012;308(3):247–256.

Summary

This observational study investigated the association between interferon beta exposure and disability progression in patients with relapsing-remitting multiple sclerosis (MS). It was a retrospective cohort study based on prospectively collected data (1985–2008) within the British Columbia Multiple Sclerosis (BCMS) database, Canada which compared 868 patients receiving interferon with 829 untreated contemporary and 959 historical patients. The main outcome measure was time from interferon treatment eligibility (baseline) to a confirmed and sustained score of 6 (requiring a cane to walk 100 m; confirmed at > 150 days with no measurable improvement) on the Expanded Disability Status Scale (EDSS) (range, 0–10, with higher scores indicating higher disability). A multivariable Cox regression model with interferon treatment included as a time-varying covariate was used to assess the hazard of disease progression associated with interferon treatment. Analyses also included propensity score adjustment to address confounding by indication. Median active follow-up times (first to last EDSS measurement) were as follows: for the interferon–treated cohort, 5.1 years (interquartile range [IQR], 3.0–7.0 years); for the contemporary control cohort, 4.0 years (IQR, 2.1–6.4 years); and for the historical control cohort, 10.8 years (IQR, 6.3–14.7 years). The observed outcome rates for reaching a sustained EDSS score of 6 were 10.8, 5.3, and 23.1% in the 3 cohorts, respectively. After adjustment for potential baseline confounders (sex, age, disease duration, and EDSS score), exposure to interferon was not associated with a statistically significant difference in the hazard of reaching an EDSS score of 6 when either the contemporary control cohort (hazard ratio, 1.30; 95% CI, 0.92–1.83) or the historical control cohort (hazard ratio, 0.77; 95% CI, 0.58–1.02) were considered. Further adjustment for comorbidities and socioeconomic status, and separate propensity score adjustment did not substantially change the results.

Justification for observational design

1. RCTs non-existent or inadequate - Not satisfied: A multicenter, randomized, double-blind, placebo-controlled trial of interferon beta-1b (IFNB) in 372 ambulatory patients with relapsing-remitting multiple sclerosis, entry EDSS score of 0 to 5.5 and at least two exacerbations in the previous 2 years. Patients were randomised 1:1:1 to placebo, 1.6 million international units (MIU) of IFNB, and 8 MIU of IFNB three times a week for 2 years. Annual exacerbation rates were significantly lower in both treatment groups compared with the placebo, and more patients in the 8 MIU group were exacerbation-free at 2 years compared with placebo group (p = 0.007). EDSS scores changed little from baseline in both the placebo and treatment arms. The IFNB Multiple Sclerosis Study Group. Interferon beta-1b is effective in relapsing-remitting multiple sclerosis. I. Clinical results of a multicenter, randomised, double-blind, placebo-controlled trial. Neurology 1993;43:655–61.

A multicenter, randomized, double-blind, placebo-controlled trial of interferon beta-1b (IFNB) in 560 patients with entry EDSS scores of 0–5·0 were randomised 1:1:1 to interferon 22 μg, 44 μg, or placebo three times a week for 2 years. While the mean change in EDSS (− 0.25, 95% CI − 0.50 – 0.0) just reached statistical significance (p = 0·05), the effect was not clinically significant as the minimal clinically important difference in the 10-point EDSS scores is − 1.0. PRISMS (Prevention of Relapses and Disability by Interferon β-1a Subcutaneously in Multiple Sclerosis) Study Group. Randomised double-blind placebo-controlled study of interferon β-1a in relapsing/remitting multiple sclerosis. Lancet 1998; 352: 1498–1504.

2. RCTs not feasible – Partially satisfied: The authors justified their study for the following reasons: 1) typically, drug efficacy (as established through randomized clinical trials conducted under optimal conditions) is greater than drug effectiveness (as measured in “real-world” settings); 2) patients participating in clinical trials tend to be highly selected in terms of comorbidities, motivation, cognition, and ability to adhere to medication schedules; 3) follow-up protocols are highly structured, supportive, and specialized, and the duration of therapy in clinical trials is typically shorter than under usual care conditions. For all these reasons, the relationship between interferon exposure and disease progression is difficult to delineate based on clinical trials. We argue that large scale pragmatic trials with long term follow-up are possible and indeed the two RCTs cited above constituted multi-centre trials with broad inclusion criteria and follow-up of at least 2 years.

Minimisation of bias in study design and data collection

1. Pre-specified study protocol: Fully satisfied - detailed protocol with hypothesis, participant selection criteria, data sources and collection methods, description of interventions, primary and secondary outcomes, methods of analysis and adjustment.

2. Clearly stated patient selection criteria: Fully satisfied – patients with confirmed diagnosis of remitting and relapsing multiple sclerosis registered with British Columbia multiple sclerosis (BCMS) clinics. Patients were either receiving interferon or not.

3. Representative study population: Fully satisfied – The BCMS database was established in 1980 and is estimated to capture 80% of the BC multiple sclerosis population.

4. Prospective and verifiable data collection: Partially satisfied – Neurologists collect clinical data relating to disease severity using standardised scoring instruments for level of disability. No mention of inter-rater reliability checks.

5. Validation checks for coded administrative data: Not satisfied – Data relating to exposure to interferon was collected from province-wide health administrative databases, data related to comorbid conditions captured from hospital discharge abstracts and services provided by practitioners derived from Medical Service Plan Payment Information database. No comment on validation or audit processes for ensuring accuracy and completeness of these data sources.

6. Validation checks for longitudinal data linkage processes: Fully satisfied – Linkage was performed through Population Data BC, a pan-provincial population health data resource, with the linkage algorithms posted on its website. Patients were identified through BCMS database and linked via their personal health number, a unique lifelong identifier.

7. Minimisation of recording bias in administrative data: Fully satisfied – No evidence of recording bias with no system of care incentives to upcode data to maximise revenue.

8. Minimisation of surveillance bias in clinical registry data: Fully satisfied – rate of clinic visits and recording of measures of disease severity disease were not different between treatment and no treatment cohorts.

9. Minimisation of recall bias: Not applicable

10. Minimisation of social desirability bias: Not applicable

11. Independent assessment of outcomes: Not satisfied – Clinicians caring for patients were the same individuals scoring measures of disease severity and who were aware of whether patient was receiving interferon or not. No comment on measures being performed, even in a random subset, by independent assessors blind to treatment status.

Appropriate methods used to create comparable groups

12. Appropriate statistical regression models for balancing populations: Fully satisfied – A multivariable Cox regression model with interferon beta treatment included as a time-varying covariate was used to assess the hazard of disease progression with treatment. Analyses also included propensity score adjustment to address confounding by indication.

13. Model includes all important predictor variables: Partially satisfied – Model was adjusted for sex, age, disease duration, EDSS score at baseline, socioeconomic status and Charlson co-morbidity index. However other potentially relevant predictors were absent: previous relapse rate, prior use of other treatments such as high dose steroids and copaxone, clinic volumes (as proxy measures of clinician experience)

14. Appropriate selection and measurement of all important predictor variables: Fully satisfied – Selection of variables is self-explanatory and detailed discussion of how they were measured.

15. Majority of population sample included in analysis: Fully satisfied – more than 80% of the sample was included.

16. Imputation methods for missing data: Not satisfied - patients with fewer than 2 prospective EDSS measurements from baseline to study end were excluded; however, patients with more than 2 measurements but then lost to follow-up were censored at the last recorded EDSS measurement, and this measurement included in the Cox proportional hazards model. No comment as to what proportion of the sample comprised patients lost to follow-up.

17. Comparison groups well balanced: Not applicable – used Cox multivariable regression models for primary analysis

Appropriate adjustment of observed effects

18. Subgroup analyses and Interaction testing for identifying independent prognostic variables: Not satisfied – no comments made as to interaction testing or subgroup analyses

19. Avoidance of unplanned post-hoc analysis: Fully satisfied – no post-hoc analyses reported

20. Correction for multiple outcome analyses: Not applicable – there was only one outcome measure: change in EDSS over time.

21. Adjustment for clustering effects in multicentre studies: Not satisfied – there were 4 separate clinics so clustering effects could operate, especially as the clinicians in each who were reporting EDSS were not blind to treatment and there may be inter-clinic differences in other co-interventions

22. Adjustment for time-dependent bias: Fully satisfied – disease duration and timing of interferon treatment were treated as time-varying co-variates in Cox proportional hazards model.

Validation of observed effects

23. Large effect size: Not applicable – there was no difference between groups

24. Exclusion of possible benefit in presence of negative results: Not satisfied – the point estimate of the hazard ratio comparing interferon treated with contemporary untreated patients was 1.30 with the upper 95% CI limit of 1.83, suggesting potential for benefit despite non-significant result with lower 95%CI limit being 0.92.

25. Sensitivity analysis for unmeasured confounders: Not satisfied – no analyses done

26. Plausibility of intervention mechanism of action: Fully satisfied – biologically plausible

27. Temporal relation between intervention and outcomes: Fully satisfied – extended period of follow-up for all three cohorts which gave adequate time for benefit to show if treatment was effective

28. Dose-response relationship: Not applicable – as no effect was seen

29. Consistency with other studies of same intervention: Fully satisfied – previously performed randomised trials and systematic review have not confirmed that interferon has clinically meaningful effects on the disease course

30. Coherence with other studies of similar interventions: Not applicable – no other similar interventions at the time – in later years a pegylated form of interferon was entered into trials

31. Falsification test for intervention effect specificity: Not satisfied – no mention of falsification tests in the study protocol in the event of benefit or harm being observed.

Authors interpretations

32. Study limitations acknowledged – Fully satisfied – extensive discussion stating limitations such as differential patient recruitment according to disease severity, limited choice of disease outcome measures and limitations of EDSS to assess all relevant functional domains, and inadequate study power.

33. Impartial statement of study implications: Partially satisfied – authors state their findings question the routine use of interferon in preventing or delaying long-term disability, despite previously stating the limitations of the EDSS. They do concede that it is possible a subgroup of patients may benefit, and may be identified using pharmacogenomic or biomarker studies.