Table 4 The measures used to assess performance of the criteria for deciding which interactions are considered

Sensitivity for including non-zero interactions

Sensitivity and specificity evaluate the extent to which criteria identify non-zero interactions, but do not reflect the consequences of ignoring them.

The proportion of samples in which interactions in cost [or benefit] were taken into account in the analysis when the true interaction was not zero

Specificity for excluding interactions equal to zero

The proportion of samples in which interactions in cost [or benefit] were excluded from the analysis when the true interaction equalled zero

Probability of adopting treatment with highest NMB

This focuses on the purpose of economic evaluation: namely to inform a treatment adoption decision regarding which treatment has highest expected NMB and to thereby maximise health gain from the budget. It assumes that inference is irrelevant to decision-making [8], but nonetheless acknowledges that inefficient analysis and small sample sizes may cause us to adopt the wrong treatment by chance. The probability of making the wrong decision may be relevant risk-averse decision-makers. However, it does not take account of the consequences of making the wrong decision.

The treatment arm with highest expected NMB was identified at the ceiling ratio of interest for (a) the “true” parameters used to generate the data and (b) based on the mixed model coefficients estimated on each sample. The proportion of samples in which the treatment predicted to have highest NMB (b) was the same as the “true” best treatment (a) was calculated for each scenario.

Opportunity cost associated with adopting a suboptimal treatment

This measure takes account of the opportunity cost of adopting the wrong treatment, as well as the probability of adopting the wrong treatment [1]. It is similar to the opportunity cost of ignoring interactions [1] but is based on a contrast between the genuine best treatment and the treatment predicted to be best, rather than a comparison between two imperfect analyses on finite samples. As such, the opportunity cost estimated here takes account of situations where allowing for spurious interactions causes us to adopt the wrong treatment by chance, as well as situations where ignoring interactions biases the analysis.

For each sample, the opportunity cost was defined as the NMB for the “true” best treatment (a) minus the NMB for the treatment predicted to have highest NMB in that analysis of that sample (b). In both cases, NMB for each treatment was calculated using the “true” parameters used for data generation. Opportunity cost was therefore zero for all samples in which the “true” best treatment was adopted and positive in all other cases. Opportunity cost was then averaged across samples and scenarios.