We use Mathematica software, version 8, to calculate the expectation and bias of I2 analytically. This Methods section introduces notation, assumptions, and statistical properties, and describes the calculations that we submitted to Mathematica. The Results section will give the results of those calculations.
Meta-analysis
Meta-analysis summarizes the results of K studies, each of which has sample size n
k
, k = 1,…,K. In each study, there is a true effect β
k
estimated by \( {\widehat{\beta}}_k \), with a true standard error σ
k
estimated by \( {\widehat{\sigma}}_k \), or, equivalently, a true variance \( {\sigma}_k^2 \) estimated by \( {\widehat{\sigma}}_k^2 \). With large n
k
, the quantity \( \left({\widehat{\beta}}_k-{\beta}_k\right)/{\widehat{\sigma}}_k \) approaches a standard normal distribution according to the central limit theorem.
Two models can be used in meta-analysis: a fixed-effects model and a random-effects model. Some confusion is possible because the term fixed effects is used in two different senses [15]. In some literature, the term fixed effects means that the K study effects β
k
are assumed to be homogeneous. We use the term fixed effects in its other sense, where it means that we seek only to generalize about the K studies in the meta-analysis. The true effects β
k
can be either homogeneous or heterogeneous, but they are regarded as fixed quantities. Because of sampling error, the K studies would produce different estimates \( {\widehat{\beta}}_k \) and \( {\widehat{\sigma}}_k \) if they were repeated, but the true effects β
k
and true standard errors σ
k
would not change.
Under a random-effects model, by contrast, we assume that the true effects β
k
in the meta-analysis were drawn at random from a larger population of effects, and we seek to make inferences about that larger population [16]. So the β
k
are not fixed quantities but random variables that would be different if a different sample were drawn from the population of effects.
The estimand ι2
In order to understand the properties of the estimator I2, we must first define the quantity that is being estimated. We call the estimand ι2. It represents the fraction of variance in the estimated effects \( {\widehat{\beta}}_k \) that is due to heterogeneity rather than measurement error.
More formally, the \( {\widehat{\beta}}_k \) vary from one study to another. The variance in \( {\widehat{\beta}}_k \) is partly due to the heterogeneity of the true effects β
k
and partly due to estimation error summarized by the standard errors σ
k
. By the law of total variance we have
$$ \begin{array}{c}V\left({\widehat{\beta}}_k\right)=V\left({\beta}_k\right)+E\left({\sigma}_k^2\right)\\ {}={\tau}^2+{\sigma}^2\end{array} $$
(1)
where τ2 = V(β
k
) is the heterogeneity variance or between-study variance, and \( {\sigma}^2=E\left({\sigma}_k^2\right) \) is the average within-study variance. Under a fixed-effects model these variances and expectations refer only to the K effects β
k
and standard errors σ
k
in the meta-analysis. Under a random effects model τ2 refers to the larger population of effects, but σ2 still refers only to the K standard errors σ
k
in the meta-analysis, unless we are willing to regard the σ
k
as well as the β
k
as samples from a larger population.
The fraction of variance that is due to heterogeneity is
$$ {\iota}^2=\frac{V\left({\beta}_k\right)}{V\left({\widehat{\beta}}_k\right)}=\frac{\tau^2}{\tau^2+{\sigma}^2} $$
(2)
If ι2 = 0 then the effects β
k
are homogeneous; if ι2 > 0 then they are heterogeneous.
Note that, unlike some past definitions [6], our definition of ι2 does not assume equal standard errors σ1 = σ2 = … = σ
K
. Note also that ι2 is not an absolute measure of heterogeneity. Instead, τ2 is an absolute measure of heterogeneity, while ι2 compares τ2 to σ2. When the estimation error is small, as it is if n
k
is large, then ι2 can be large even if τ2 is small [17].
The naïve estimator \( {\widehat{\boldsymbol{\iota}}}^{\mathbf{2}} \)
To estimate the fraction ι2, Higgins and Thompson [6] first derived the naïve estimator
$$ {\widehat{\iota}}^2=1-\frac{df}{Q} $$
(3)
where df = K–1, Q is Cochran’s Q statistic [4]
$$ Q={\displaystyle \sum_{k=1}^K}\frac{{\left({\widehat{\beta}}_k-\widehat{\overline{\beta}}\right)}^2}{{\widehat{\sigma}}_k^2} $$
(4)
and
$$ \widehat{\overline{\beta}}=\frac{{\displaystyle {\sum}_{k=1}^K}{\widehat{\sigma}}_k^{-2}{\widehat{\beta}}_k}{{\displaystyle {\sum}_{k=1}^K}{\widehat{\sigma}}_k^{-2}} $$
(5)
is the precision-weighted average of the estimated effects.
The distribution of \( {\widehat{\iota}}^2 \) depends on the distribution of Q. Under homogeneity, with large n
k
, Q has a central chi-square distribution with df degrees of freedom.
Under heterogeneity, the large-n
k
distribution of Q depends on whether we regard the effects as fixed or random. Under a random-effects model, Q is distributed like a weighted sum of K–1 central \( {\chi}_1^2 \) variables, where the weights are given by a matrix function of τ2 and \( {\sigma}_k^2 \) [18]. If we make the simplifying assumption that all the standard errors are equal (σ
k
= σ) then the weights are all equal to 1 + τ2/σ2 [18] or, in our notation (1 − ι2)− 1, so that
$$ X=\left(1-{\iota}^2\right)Q $$
(6)
has a central chi-square distribution with df degrees of freedom [18]. As ι2 gets small, we converge toward the homogeneous situation where Q itself has a central chi-square distribution with df degrees of freedom.
Under a fixed-effects model, by contrast, Q has a non-central chi-square distribution with df degrees of freedom and a non-centrality parameter of [19]
$$ \lambda ={\displaystyle \sum_{k=1}^K}\frac{{\left({\beta}_k-\overline{\beta}\right)}^2}{\sigma_k^2} $$
(7)
where \( \overline{\beta} \) is the precision-weighted mean of the true effects β
k
. If we make the simplifying assumption that all the standard errors are equal (σ
k
= σ) then the non-centrality parameter reduces to
$$ \begin{array}{c}\lambda =\frac{1}{\sigma^2}{\displaystyle \sum_{k=1}^K}{\left({\beta}_k-\overline{\beta}\right)}^2\\ {}=K\frac{\tau^2}{\sigma^2}\\ {}=K\frac{\iota^2}{1-{\iota}^2}\end{array} $$
(8)
The last line shows that λ is an increasing function of ι2 and that that λ = 0 if ι2 = 0. So again, as ι2 gets small, Q converges toward the central chi-square distribution that it has under homogeneity.
The truncated estimator I2
A shortcoming of the naïve estimator \( {\widehat{\iota}}^2 \) is that it can be negative even though the estimand ι2 cannot. Negative values of \( {\widehat{\iota}}^2 \) occur whenever Q < df, which is not a rare event. Figure 1 shows the probability that \( {\widehat{\iota}}^2 \) is negative when the effects are homogeneous. The probability decreases as df increases, but the probability is always greater than 50%.
To avoid negative estimates, Higgins and Thompson [6] suggested rounding them up to zero. The rounded or truncated estimator
$$ {I}^2= \max \left(0,{\widehat{\iota}}^2\right) $$
(9)
is the estimator that is widely used today. I2 cannot be negative but can be zero. Values of I2 = 0 occur in about one-quarter of published meta-analyses [20].
Expectation and bias of the estimators
The expectation of the naïve estimator \( {\widehat{\iota}}^2 \) is
$$ E\left({\widehat{\iota}}^2\right)=1-df\ E\left(\frac{1}{Q}\right) $$
(10)
This is easily calculated in the homogeneous case, where 1/Q is an inverse chi-square variable whose expectation is 1/(df – 2). It is just as easily calculated in the heterogeneous case with fixed effects; in that case, 1/Q is a scaled inverse chi-square variable with an expectation of (1 − ι2)/(df − 2). The calculation is harder in the heterogeneous case with random effects; in that case, 1/Q is the scaled inverse of a noncentral chi-square variable. Although the expectation of this inverse has a closed-form solution [21], it is not transparent or easy to calculate by hand. However, we can calculate it using Mathematica.
The expectation of the truncated estimator I2 is a little harder to calculate. It is the weighted average of two conditional expectations: the expectation of I2 when I2 = 0 and the expectation of I2 when I2 > 0. The probability that I2 = 0 is P(Q < df), and the probability that I2 > 0 is P(Q > df). Therefore the expectation of I2 is
$$ \begin{array}{c}E\left({I}^2\right)=P\left(Q<df\right)\times 0+P\left(Q>df\right)\times E\left({I}^2\left|Q>df\right.\right)\\ {} = P\left(Q>df\right)\times E\left(1-\frac{df}{Q}\left|Q>df\right.\right)\end{array} $$
(11)
Under homogeneity, Q has a central chi-square distribution and the expectation E(I2) has a closed-form solution which Mathematica can calculate.
Under heterogeneity, the expectation E(I2) depends on whether we regard the effects as fixed or random. If effects are random, then X = (1 − ι2)Q has a central chi-square distribution. The probability that I2 = 0 is P(X < (1 − ι2)df), and the probability that I2 > 0 is P(X > (1 − ι2)df). Therefore the expectation of I2 is
$$ E\left({I}^2\right) = P\left(X>\left(1-{\iota}^2\right)df\right)\times E\left(1-\left(1-{\iota}^2\right)\frac{df}{X}\left|X>\left(1-{\iota}^2\right)df\right.\right) $$
(12)
which again has a closed-form solution which Mathematica can calculate.
If instead effects are fixed, then the expectation E(I2) in (11) has no closed-form solution. But the expectation for specific values of ι2 and df can be calculated using numerical integration in Mathematica.