Table 1 Methodological framework for assessment of experimental vignette studies

1. Credibility

• The degree to which vignettes credibly represent critical aspects of a clinical scenario or patient to potential participants is crucial to the success of an experimental vignette study [37].

• Lens model approaches (studies which compare optimal versus actual decisions in a given situation, originally developed by Brunswick in 1950) have demonstrated empirically that the decision-making performance of participants is improved when situations are realistic [47].

• Basing vignettes on real-life data, clinical expertise, and existing guidelines are recommended ways of enhancing credibility [26, 45, 46].

2. Number

• Presenting participants with more than one vignette enables examination of variations in judgement within individuals as well as between them – that is, the extent to which each participant is differentially influenced by each experimental factor in making their decisions. Where this is required to address study aims, for example, in vignette approaches based on the lens model [48], it is typically recommended that there are at least five different representations for each experimental factor.

• Additional considerations are needed when several vignettes are used,, such as controlling for the order in which vignettes are presented and taking account of clustering within individuals in the analysis (see: wider study design).

3. Variability

• Developing or using a number of different representations of each experimental factor may increase study generalisability, by reducing the possibility that idiosyncrasies in one particular representation are responsible for findings. For example, using one female and one male actor in video vignettes may lead not to participants responding to the constructs of gender, but to that particular female or that particular male.

• Where participants do view more than one vignette, analysis must account for clustering of vignettes by respondent, to avoid over-estimating the statistical significance of any effect [49].

4. Mode

• The mode through which vignettes are delivered has an important influence on the research question an experimental vignette study can answer.

• Vignette mode has historically been textual only, with participants presented with a written scenario. Text-based vignettes may constrain not just the information the respondent is given, but how this information is framed.

• More recently the use of pictures, videos, actors, and interactive environments have been developed [22, 46].

• Pictorial modes are particularly suited to examination of characteristics, such as ethnicity, where visual representation removes the need for explicit statement (and prior framing) of the characteristic.

• Studies using video vignettes extend this still further by enabling participants to form judgements on body language and speech patterns in addition to visual cues.

• Interactive formats, such as unannounced standardised patients or virtual reality set-ups, have the potential to mimic real delivery which enables exploration of how inequalities may unfold during a clinical encounter, through enabling explorations of variations in the information that clinical participants elicit from patients or in both parties’ non-verbal communication. Such approaches are more complex to construct and more costly to develop than static vignette formats, which may limit their feasibility.

5. Evaluation

• Evaluation of vignettes’ face validity – during vignette construction and once data are collected – is key to understanding the validity of findings in studies using vignettes.

• Thinking through in advance what is needed to make particular vignettes ‘successful’ for their target audience will guide the nature of and approach to evaluation.

• Options include assessment by an expert panel, feedback from participants, or comparing responses to the vignettes to an additional data source such as clinical data [26, 46].

6. Description

• Readers of vignette study papers need to be able to form their own judgments of vignette credibility. An entire vignette should be provided to enable them to do so.

1. Concealment

• When investigating unwarranted variations in care, it is important to conceal the purpose of such studies, given that few people will volunteer behaviours or attitudes that they recognise as poor or biased.

• If the study’s purpose is not adequately masked it can bias results, even with carefully constructed vignettes [31]. Participants may learn of the study purpose directly (from study information shared at recruitment) but also may infer it indirectly, through other cues in study materials (e.g. funder’s name), or pre-specified responses that prime participants to consider certain answers.

2. Realism

• External validity of vignette studies is enhanced when studies are conducted in a setting as close as possible to the natural ecology of decision-making [47, 50].

• The generalisability of studies to investigate unwarranted variation in healthcare may be improved by collecting data in a setting that mimics key aspects of clinical settings, whether that be the actual environment, other inclusion of features such as the imposition of time constraints.

3. Sampling & response

• The representativeness of any survey rests on sampling, coverage, and nonresponse.

• This is particularly important for studies of healthcare variations, where a biased sample or responses – for physician or patient participants – may lead to over- or under-estimation of variations.

• Studies need to justify their sample design, sample size, approach to recruitment, response and completion rates, and reasons for excluding data [51].

• The implications of low or biased responses should be considered.

4. Analysis

• Experimental vignette studies are often complex in how data are structured. Analysis must appropriately account for hierarchies within the data [22].