Table 1 System of bias factors, which may affect internal and external validity

Selection bias

Problem

Treatment and control group are different, e.g. differences in age, severity of disease

Study group and "target group" are different, study group is not representative, e.g. differences in age, severity of disease

Solution

Randomisation, matched pairs

Identification (and adjustment as far as possible) of relevant epidemiological factors, e.g. by comparison with patients who have not consented in the study

False negative/positive results may occur:

–: relevant distinctions/subgroups unknown → levelled outcome

+: e.g. treatment group with more "responders"

+/–: e.g. study group with more advanced disease (university hospital)

+: e.g. study group without concomitant diseases (better prognosis than "usual" patients)

Key questions

Is randomisation adequate?

Are (known) relevant factors distributed equally?

Are relevant epidemiological factors taken into account?

Performance bias

Problem

Apart from the intervention tested, groups are treated differently

Study treatment does not reflect the actual variability in managing disease and patients' problems

Solution

Blinding, documentation of possible differences, change to open label design (COLA design)

Treatment as realistic as possible with individualised modification if necessary (pragmatic controlled trials)

False negative/positive results my occur:

–: concomitant therapy in control group; non compliance in verum group;

+: concomitant therapy in verum group

+: high compliance (e.g. in hospitals); highly specialised therapists; high dosages of medication

–: relevant context factors are missing (patient-therapists relationship, accessibility to therapy); inexperienced therapists; low dosages of medication

Key questions

Is blinding adequate and checked? Are concomitant interventions documented?

Are realistic interventions applied which are carried out by physicians in everyday practice?

Attrition bias

Problem

Drop out rates between groups are different or that large that analysis is not reliable any more

Drop out rates between study group an target group are different, e.g. different compliance and/or motivation

Solution

Intention to treat analysis (note: drop out rates > 10 % have a high risk of bias)

Compliance control and assessment

False negative/positive results may occur:

– : intention to treat analyses

+: drop out rates are higher in treatment group (with per protocol analyses)

–: drop outs due to adverse effects (and intention to treat analysis)

+: drop outs due to ineffectiveness of therapy (and per protocol analysis)

Key questions

Is the drop out rate documented? Are adequate analyses performed?

Are the reasons for dropping out documented? Do the reasons for dropping out have an impact on the assessment of compliance, effectiveness or safety?

Detection bias

Problem

Differences in the perception of outcome parameters between groups and within the the course of the study

Outcome parameters and/or length of follow up have no practical relevance to patients' problems

Solution

Blinding of assessors; if blinding is not possible: assessment of two independent persons; objective parameters

Selection of clinically relevant and generally available outcome parameters; adequate length of follow up

False negative/positive results may occur:

–/+: inadequate blinding and respective expectations by assessors

–: outcome parameters do not reflect actual improvement; inadequate follow up

+: significant but irrelevant outcomes

Key questions

Blinding procedures of assessors adequate? Independent assessors?

Are outcome parameters, length of follow up and detected differences relevant?