Skip to main content

Table 4 Elements of a comprehensive research question of a study on reliability or measurement error

From: COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study

Element of the research question

Consensusa (%)

1

the name of the outcome measurement instrument

42/45 (93%) (R1b)

2

the version of the outcome measurement instrument or way of operationalization of the measurement protocolc

42/45 (93%) (R1) (version)

33/45 (73%) (R1) (operationalization)

3

the construct measured by the measurement instrument

40/45 (89%) (R1)

4

a specification whether one is interested in a reliability parameter (i.e. a relative parameter such as an ICC, Generalizability coefficient φ, or Kappa κ) or a parameter of measurement error (i.e. an absolute parameter expressed in the unit of measurement e.g. SEM, LoA or SDC; or expressed as agreement or misclassification, e.g. the percentage specific agreement).

36/42 (86%) (R2d)

5

a specification of the components of the measurement instrument that will be repeated (especially when only part of the measurement instrument is repeated, e.g. only assignment of the score based on the same images)

38/45 (84%) (R1)

6

a specification of the source(s) of variation that will be variede

41/45 (91%) (R1)

7

a specification of the patientf populationg studied

42/45 (93%) (R1)

  1. a Consensus was set at 67% of the panelists (strongly) agreed to a proposal, the denominator can be decreased because panelists considered themselves to have ‘no expertise’ on a specific proposal or dropped-out; b R1: consensus reached in Round 1; c In Generalizability theory these are the facets of stratification (FoS), when patients are nested in a facet [16]; d R2: consensus reached in Round 2; e In Generalizability theory these are the random or fixed facets of generalizability (FoG), e.g. time or occasion, the (level of expertise of) professionals, the machines, or other components of the measurement [16]; f In the COSMIN methodology we use the word patient. However, sometimes the target population doesn’t consist of patients, but e.g. healthy individuals, caregivers, clinicians, or the body structures (e.g. joints, or lesions). In these cases, the word patient should be read as e.g. healthy volunteer, or clinician; g In Generalizability theory these are the Object of Measurement (OoM) or the facet of differentiation [16]