Table 3 List of the criteria for determining which interactions are taken into account that were evaluated in the study

1

Always include all interactions

Sometimes referred to as “never pool” [3, 22]. Avoids bias, but has lower power unless interactions are very large [3,4,5,6]. May be particularly appropriate for economic evaluation [1] since this focuses on maximising expected net benefit subject to current information [8]. This approach is statistically consistent, in that we would always adopt the treatment that truly had the highest NMB if the sample size were infinite, although results may be more sensitive to chance than approaches excluding some interactions.

Interactions were included in analyses on all trial samples.

2

Never include any interactions

Sometimes referred to as “always pool” [3, 22]. Maximises statistical power unless interactions are very large, but is biased unless the true interaction is zero [3,4,5,6]. This approach is not statistically consistent and would cause us to adopt a suboptimal treatment whenever there is a qualitative interaction in NMB that changes which treatment had highest NMB, even with an infinite sample size.

No interactions were included analyses on any trial samples.

3

Include interactions where p < 0.05

Reflects standard practice for clinical endpoints, where only interactions that are statistically significant in an initial test are included in the main analysis [3, 4, 7]. Significance levels > 0.05 are sometimes used for the test on interactions [23]. However, most studies are underpowered for main effects in costs and QALYs [24,25,26,27,28,29,30], which are likely to have variances a quarter of the size of those found interaction terms [31,32,33]. Statistical inference may be irrelevant for decision-making, as health gains from the budget are maximised by adopting the treatment with the highest expected net benefit [8].

Interaction for cost [or benefits] was included if it was statistically significantly different from zero in the model that included interactions for both costs and benefits.

4

Include interactions where p < 0.10

5

Include interactions where p < 0.25

6

Include interactions decreasing AIC

Information criteria trade efficiency against bias, taking account of sample size [34], although this trade-off is based on information theory, rather than decision analysis. These measures have been used outside of healthcare to decide whether to include interactions in factorial experiments [35].

Results are based on the mixed model with lowest AIC/BIC.

7

Include interactions decreasing BIC

8

Include qualitative interactions in cost or benefits

Interactions that change the ranking of treatments for cost or benefits may also have a high chance of changing the ranking of treatments for net benefits and therefore could also change the conclusions. This approach is simpler to implement than the criteria based on interactions for net benefit as it does not depend on the ceiling ratio. However, at ceiling ratios other than zero and infinity, the conclusions of economic evaluation could be sensitive to interactions even if this criterion does not pick up qualitative interactions for either costs or benefits.

Includes interactions for cost [benefits] that change rankings of treatments for cost [benefits]: i.e. those that are larger than and have the opposite sign from one or both of the simple effects (which will have interaction:effect ratios <− 1).

9

Include interactions for cost or benefits if >simple effect

This criterion includes super-additive interactions for cost or benefits that are larger than as the smaller of the two simple effects, as well as the qualitative interactions included in criterion 8. However, like 8, it may not identify all qualitative interactions for net benefit.

All interactions with an absolute magnitude larger than the smaller of the two simple effects (i.e. all those with interaction:effect ratios <− 1 or > 1) are included.

10

Include interactions for cost or benefits if p < 0.05 or > simple effect

This approach takes account of statistical significance and interactions that are larger than main effects.

As for 9, but also including smaller interactions that are statistically significantly different from zero.

11

Include qualitative interactions for cost, benefits or NMB

Allowing for interactions will have no effect on the conclusions about which treatment is adopted unless the interactions are qualitative on a NMB scale (i.e. change the ranking of treatments) at the ceiling ratio(s) of interest. However, since the true shadow price of a QALY is unknown, this approach requires arbitrary judgements about the ceiling ratio(s) at which the interactions are assessed. Including all interactions that are qualitative at any ceiling ratio would generally result in inclusion of all interactions, since any quantitative interaction in either costs or QALYs will produce a qualitative interaction in NMB at some ceiling ratio whenever the treatment lies in the north-east or south-west quadrants [1].

Interactions were calculated at a series of 8 evenly-spaced ceiling ratios between £5000 and £40,000 per unit benefit using the coefficients for all 4 mixed models. Interactions for costs [or benefits] were included if the interaction for cost [or benefits] was qualitative, or if the coefficients from the mixed model that included an interaction for cost [or benefits] but not benefits [or cost] produced a qualitative interaction for NMB at any of the ceiling ratios.

12

Include interactions for cost, benefit or NMB if >simple effect

Includes all qualitative interactions in cost, benefits or NMB and any super-additive interactions that are larger than the smaller of the two simple effects. Calculated as for 11, but also including large super-additive interactions.

13

Include |interactions| ≥0.25 or ≥ £250

An absolute limit for the size of interaction that can safely be ignored could be pre-specified. However, there is no general rule for how large this limit should be and it may vary between applications. The size thresholds used were chosen arbitrarily.

Only interactions above the designated size threshold were taken into account. For example, criterion 13 includes interactions in benefits that are ≥0.25 (or ≤ −0.25) units in size and interactions in cost that are ≥£250 (or ≤ −£250) in size.

14

Include |interactions| ≥0.5 or ≥ £500

15

Include |interactions| ≥1 or ≥ £1000