Table 2 Inter-rater agreement (percentage agreement) and reliability (kappa coefficients) of the items from the COSMIN checklist (COSMIN step 3)

Box A Internal consistency (n = 195)^b

A1

Does the scale consist of effect indicators, i.e. is it based on a reflective model?

Design requirements

A2^c

Was the percentage of missing items given?

A3^c

Was there a description of how missing items were handled?

A4

Was the sample size included in the internal consistency analysis adequate?

A5^c

Was the unidimensionality of the scale checked? i.e. was factor analysis or IRT model applied?

A6

Was the sample size included in the unidimensionality analysis adequate?

A7

Was an internal consistency statistic calculated for each (unidimensional) (sub)scale separately?

A8^c

Were there any important flaws in the design or methods of the study?

Statistical methods

A9

for Classical Test Theory (CTT): Was Cronbach's alpha calculated?

A10

for dichotomous scores: Was Cronbach's alpha or KR-20 calculated?

A11

for IRT: Was a goodness of fit statistic at a global level calculated? e.g. χ², reliability coefficient of estimated latent trait value (index of (subject or item) separation)

Box B. Reliability (n = 141) ^b

Design requirements

B1^c

Was the percentage of missing items given?

B2^c

Was there a description of how missing items were handled?

B3

Was the sample size included in the analysis adequate?

B4^c

Were at least two measurements available?

B5

Were the administrations independent?

B6^c

Was the time interval stated?

B7

Were patients stable in the interim period on the construct to be measured?

B8

Was the time interval appropriate?

B9

Were the test conditions similar for both measurements? e.g. type of administration, environment, instructions

B10^c

Were there any important flaws in the design or methods of the study?

Statistical methods

B11

for continuous scores: Was an intraclass correlation coefficient (ICC) calculated?

B12

for dichotomous/nominal/ordinal scores: Was kappa calculated?

B13

for ordinal scores: Was a weighted kappa calculated?

B14

for ordinal scores: Was the weighting scheme described? e.g. linear, quadratic

Box D. Content validity (n = 83) ^b

Design requirements

D1

Was there an assessment of whether all items refer to relevant aspects of the construct to be measured?

D2

Was there an assessment of whether all items are relevant for the study population? (e.g. age, gender, disease characteristics, country, setting)

D3

Was there an assessment of whether all items are relevant for the purpose of the measurement instrument? (discriminative, evaluative, and/or predictive)

D4

Was there an assessment of whether all items together comprehensively reflect the construct to be measured?

D5^c

Were there any important flaws in the design or methods of the study?

Box E. Structural validity (n = 118) ^b

E1

Does the scale consist of effect indicators, i.e. is it based on a reflective model?

Design requirements

E2^c

Was the percentage of missing items given?

E3^c

Was there a description of how missing items were handled?

E4

Was the sample size included in the analysis adequate?

E5^c

Were there any important flaws in the design or methods of the study?

Statistical methods

E6

for CTT: Was exploratory or confirmatory factor analysis performed?

E7

for IRT: Were IRT tests for determining the (uni-) dimensionality of the items performed?

Box F. Hypotheses testing (n = 170) ^b

Design requirements

F1^c

Was the percentage of missing items given?

F2^c

Was there a description of how missing items were handled?

F3

Was the sample size included in the analysis adequate?

F4

Were hypotheses regarding correlations or mean differences formulated a priori (i.e. before data collection)?

F5

Was the expected direction of correlations or mean differences included in the hypotheses?

F6

Was the expected absolute or relative magnitude of correlations or mean differences included in the hypotheses?

F7^c

for convergent validity: Was an adequate description provided of the comparator instrument(s)?

F8^c

for convergent validity: Were the measurement properties of the comparator instrument(s) adequately described?

F9^c

Were there any important flaws in the design or methods of the study?

Statistical methods

F10

Were design and statistical methods adequate for the hypotheses to be tested?

Box G. Cross-cultural validity (n = 33) ^b

Design requirements

G1^c

Was the percentage of missing items given?

G2^c

Was there a description of how missing items were handled?

G3

Was the sample size included in the analysis adequate?

G4^c

Were both the original language in which the HR-PRO instrument was developed, and the language in which the HR-PRO instrument was translated described?

G5^c

Was the expertise of the people involved in the translation process adequately described? e.g. expertise in the disease(s) involved, expertise in the construct to be measured, expertise in both languages

G6

Did the translators work independently from each other?

G7

Were items translated forward and backward?

G8^c

Was there an adequate description of how differences between the original and translated versions were resolved?

G9^c

Was the translation reviewed by a committee (e.g. original developers)?

G10^c

Was the HR-PRO instrument pre-tested (e.g. cognitive interviews) to check interpretation, cultural relevance of the translation, and ease of comprehension?

G11^c

Was the sample used in the pre-test adequately described?

G12

Were the samples similar for all characteristics except language and/or cultural background?

G13^c

Were there any important flaws in the design or methods of the study?

Statistical methods

G14

for CTT: Was confirmatory factor analysis performed?

G15

for IRT: Was differential item function (DIF) between language groups assessed?

Box H. Criterion validity (n = 57) ^b

Design requirements

H1^c

Was the percentage of missing items given?

H2^c

Was there a description of how missing items were handled?

H3

Was the sample size included in the analysis adequate?

H4

Can the criterion used or employed be considered as a reasonable 'gold standard'?

H5^c

Were there any important flaws in the design or methods of the study?

Statistical methods

H6

for continuous scores: Were correlations, or the area under the receiver operating curve calculated?

H7

for dichotomous scores: Were sensitivity and specificity determined?

Box I. Responsiviness (n = 79) ^b

Design requirements

I1^c

Was the percentage of missing items given?

I2^c

Was there a description of how missing items were handled?

I3

Was the sample size included in the analysis adequate?

I4^c

Was a longitudinal design with at least two measurement used?

I5^c

Was the time interval stated?

I6^c

If anything occurred in the interim period (e.g. intervention, other relevant events), was it adequately described?

I7^c

Was a proportion of the patients changed (i.e. improvement or deterioration)?

Design requirements for hypotheses testing

For constructs for which a gold standard was not available

I8

Were hypotheses about changes in scores formulated a priori (i.e. before data collection)?

I9

Was the expected direction of correlations or mean differences of the change scores of HR-PRO instruments included in these hypotheses?

I10

Were the expected absolute or relative magnitude of correlations or mean differences of the change scores of HR-PRO instruments included in these hypotheses?

I11^c

Was an adequate description provided of the comparator instrument(s)?

I12^c

Were the measurement properties of the comparator instrument(s) adequately described?

I13^c

Were there any important flaws in the design or methods of the study?

Statistical methods

I14

Were design and statistical methods adequate for the hypotheses to be tested?

Design requirements for comparison to a gold standard

For constructs for which a gold standards was available:

I15

Can the criterion for change be considered as a reasonable 'gold standard'?

I16^c

Were there any important flaws in the design or methods of the study?

Statistical methods

I17

for continuous scores: Were correlations between change scores, or the area under the Receiver Operator Curve (ROC) curve calculated?

I18

for dichotomous scales: Were sensitivity and specificity (changed versus not changed) determined?

Box J. Interpretability (n = 42) ^b

J1^c

Was the percentage of missing items given?

J2^c

Was there a description of how missing items were handled?

J3

Was the sample size included in the analysis adequate?

J4^c

Was the distribution of the (total) scores in the study sample described?

J5^c

Was the percentage of the respondents who had the lowest possible (total) score described?

J6^c

Was the percentage of the respondents who had the highest possible (total) score described?

J7^c

Were scores and change scores (i.e. means and SD) presented for relevant (sub) groups? e.g. for normative groups, subgroups of patients, or the general population

J8^c

Was the minimal important change (MIC) or the minimal important difference (MID) determined?

J9^c

Were there any important flaws in the design or methods of the study?