Skip to main content

Table 1 System of bias factors, which may affect internal and external validity

From: Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity

Bias factors   Internal validity External validity/generalisability
Selection bias Problem Treatment and control group are different, e.g. differences in age, severity of disease Study group and "target group" are different, study group is not representative, e.g. differences in age, severity of disease
  Solution Randomisation, matched pairs Identification (and adjustment as far as possible) of relevant epidemiological factors, e.g. by comparison with patients who have not consented in the study
  False negative/positive results may occur: –: relevant distinctions/subgroups unknown → levelled outcome
+: e.g. treatment group with more "responders"
+/–: e.g. study group with more advanced disease (university hospital)
+: e.g. study group without concomitant diseases (better prognosis than "usual" patients)
  Key questions Is randomisation adequate?
Are (known) relevant factors distributed equally?
Are relevant epidemiological factors taken into account?
Performance bias Problem Apart from the intervention tested, groups are treated differently Study treatment does not reflect the actual variability in managing disease and patients' problems
  Solution Blinding, documentation of possible differences, change to open label design (COLA design) Treatment as realistic as possible with individualised modification if necessary (pragmatic controlled trials)
  False negative/positive results my occur: –: concomitant therapy in control group; non compliance in verum group;
+: concomitant therapy in verum group
+: high compliance (e.g. in hospitals); highly specialised therapists; high dosages of medication
–: relevant context factors are missing (patient-therapists relationship, accessibility to therapy); inexperienced therapists; low dosages of medication
  Key questions Is blinding adequate and checked? Are concomitant interventions documented? Are realistic interventions applied which are carried out by physicians in everyday practice?
Attrition bias Problem Drop out rates between groups are different or that large that analysis is not reliable any more Drop out rates between study group an target group are different, e.g. different compliance and/or motivation
  Solution Intention to treat analysis (note: drop out rates > 10 % have a high risk of bias) Compliance control and assessment
  False negative/positive results may occur: – : intention to treat analyses
+: drop out rates are higher in treatment group (with per protocol analyses)
–: drop outs due to adverse effects (and intention to treat analysis)
+: drop outs due to ineffectiveness of therapy (and per protocol analysis)
  Key questions Is the drop out rate documented? Are adequate analyses performed? Are the reasons for dropping out documented? Do the reasons for dropping out have an impact on the assessment of compliance, effectiveness or safety?
Detection bias Problem Differences in the perception of outcome parameters between groups and within the the course of the study Outcome parameters and/or length of follow up have no practical relevance to patients' problems
  Solution Blinding of assessors; if blinding is not possible: assessment of two independent persons; objective parameters Selection of clinically relevant and generally available outcome parameters; adequate length of follow up
  False negative/positive results may occur: –/+: inadequate blinding and respective expectations by assessors –: outcome parameters do not reflect actual improvement; inadequate follow up
+: significant but irrelevant outcomes
  Key questions Blinding procedures of assessors adequate? Independent assessors? Are outcome parameters, length of follow up and detected differences relevant?