- Research Article
- Open Access
- Open Peer Review
Meta-analysis of incidence rate data in the presence of zero events
- Matthew J Spittal^{1}Email author,
- Jane Pirkis^{1} and
- Lyle C Gurrin^{1}
https://doi.org/10.1186/s12874-015-0031-0
© Spittal et al.; licensee BioMed Central. 2015
- Received: 6 March 2015
- Accepted: 9 April 2015
- Published: 30 April 2015
Abstract
Background
When summary results from studies of counts of events in time contain zeros, the study-specific incidence rate ratio (IRR) and its standard error cannot be calculated because the log of zero is undefined. This poses problems for the widely used inverse-variance method that weights the study-specific IRRs to generate a pooled estimate.
Methods
We conducted a simulation study to compare the inverse-variance method of conducting a meta-analysis (with and without the continuity correction) with alternative methods based on either Poisson regression with fixed interventions effects or Poisson regression with random intervention effects. We manipulated the percentage of zeros in the intervention group (from no zeros to approximately 80 percent zeros), the levels of baseline variability and heterogeneity in the intervention effect, and the number of studies that comprise each meta-analysis. We applied these methods to an example from our own work in suicide prevention and to a recent meta-analysis of the effectiveness of condoms in preventing HIV transmission.
Results
As the percentage of zeros in the data increased, the inverse-variance method of pooling data shows increased bias and reduced coverage. Estimates from Poisson regression with fixed interventions effects also display evidence of bias and poor coverage, due to their inability to account for heterogeneity. Pooled IRRs from Poisson regression with random intervention effects were unaffected by the percentage of zeros in the data or the amount of heterogeneity.
Conclusion
Inverse-variance methods perform poorly when the data contains zeros in either the control or intervention arms. Methods based on Poisson regression with random effect terms for the variance components are very flexible offer substantial improvement.
Keywords
- Meta-analysis
- Person-time
- Incidence rate ratio
- Sparse data
Background
Meta-analysis is widely used in medical research to combine information from independent studies to evaluate the effectiveness of an intervention. When the outcome in each independent study is a binary variable, the data can be viewed as a two-by-two contingency table, with each cell corresponding to counts of events (e.g. the number of people with and without disease) in separate groups, for example, participants assigned to treatment and control arms of an intervention study. The pooled effect size typically of interest – a risk difference, relative risk or odds ratio – is then based on the summary information collected from these studies. A related effect size, the pooled incidence rate ratio (IRR), is instead based on counts of events over time, for example per person-year, recorded separately for each study arm. We refer to this type of data as incidence rate data.
Our interest in this problem was initiated by an analysis we recently undertook to evaluate the effectiveness of installing barriers for reducing jumping deaths at known suicide hotspots [1]. This is based on the premise that restricting access to means is one of the few successful suicide prevention strategies [2]. A total of eight studies had previously counted the number of suicide deaths at hotspots in the periods before and after the installation of barriers and safety nets. In six of these studies there were, however, no deaths following the installation of barriers.
Two approaches have been proposed for conducting meta-analyses of incidence rate data. These are the inverse-variance method [3,4] and using a Poisson regression model with fixed intervention effects [5]. The inverse-variance method is problematic when there are “structural zeros” (data like ours where multiple studies have counts of zero in one or both arms of a study) because when a study contains a zero count, the study-specific log IRR and the variance of the study-specific log IRR are both undefined. Thus all studies with zero counts are omitted from the analysis. One proposed remedy is to apply a continuity correction [6]; although this has generally only been considered when a single study has a zero count, not multiple studies. Using a Poisson regression model with fixed intervention effects has been proposed as a means of dealing with varying exposure time [5], but may also be useful for addressing problems where there is a number of zero counts in the data. Another option, which has not been applied to meta-analysis problems before, is to extend the previous model by estimating a Poisson regression models with random intervention effects instead. The advantage of this approach is that, in addition to potentially dealing with zero counts and varying exposure time, it may resolve problems that occur when there is heterogeneity in the intervention effect. This method, although widely used, has not previously been applied to meta-analysis problems.
Study aims
The purpose of this study is to evaluate the usefulness of a Poisson regression model with random intervention effects for meta-analysis when the data contains structural zeros. We explore this using frequentist and Bayesian implementations. We make comparisons with the inverse-variance method (with and without the continuity correction) and the Poisson regression model with fixed intervention effects. This extends previous work which has focused on rare events and varying exposure time [3-5,7], but not situations that result in structural zeros or heterogeneity in the intervention effect. We evaluate these methods through Monte Carlo simulation, manipulating the number of zero counts, the amount of heterogeneity in the control and intervention groups and the number of studies within each meta-analysis. We then apply each of these methods to data from two published studies: the suicide prevention work outlined above [1], and a Cochrane review which evaluated the effectiveness of condom use in reducing heterosexual HIV transmission [8].
Methods
Inverse-variance methods
The inverse-variance method of meta-analysis synthesises information from multiple sources by calculating a pooled estimate of the effect size of an intervention by taking a weighted average of point estimates from independent studies. This method is widely used and recommended in the Cochrane Handbook [6] as well as other sources [3,4] for pooling incidence rate data.
When the original effect sizes are odds ratios, hazard ratios or risk ratios, then the estimates are first transformed onto the log scale, since the sampling distribution of the pooled estimate will be more approximately normally distributed than on the untransformed scale (thus improving the accuracy of inferences based on asymptotic theory) and because there is no closed form formula for the variance of these effect sizes on the untransformed scale. Risk-differences, or other absolute measures, are left on their original scale. We briefly review the formalities of the inverse-variance approach as it applies to estimating pooled IRRs and the assumptions this requires.
and the summations are over the study index i. In this formulation, the parameter τ^{2} represents the between-study variance of θ_{ i } to account for any heterogeneity in the effect size. The parameter τ^{2} can be estimated using the method of moments formula [12]. Other estimators of heterogeneity have been proposed, for instance I^{2} [13,14], as well as methods for constructing a confidence interval of the heterogeneity estimate [15].
Before generalising the above by progressing to a regression setting, we mention a simpler method that aggregates study-specific counts and the corresponding exposure time in each arm across studies, and then calculates the pooled incidence rate ratio using these totals. This approach, sometimes referred to as the “naive method” [16,17], estimates a pooled effect size that is the ratio of exposure time-weighted averages of study-specific rates in the intervention and control arms. The study by Weller and David-Beaty [8], to which we return in the application section, provides an example of this approach. The deficiency with this method is that in order for it to produce an unbiased estimate of an intervention effect it requires the strong assumption that the population events rates in both the control arms and intervention arms do not vary between studies. Although this assumption can be tested empirically using the data it seems unrealistic and is likely to hold only very infrequently in practice. In addition, this approach has been criticised because it fails to account for between-study heterogeneity, and in the context of network meta-analysis, breaks randomisation [16,17]. As such, we do not consider this approach further.
Poisson regression with fixed intervention effects
where i=1,2,…,K represents the study index across K studies contributing data to the analysis, j represents the intervention index (coded 0 in the control group and 1 in the intervention group), μ_{ ij } is the expectation of y_{ ij }, y_{ ij } and time_{ ij } are the event count and exposure time respectively for the j^{th} group in the i^{th} study, β_{ i } is the logarithm of the event rate in the control group of the i^{th} study, and β_{int} is the logarithm of the pooled IRR.
However, this would produce a model that simply reproduces the data (i.e. observed and fitted values would be the same) and likely over-fitted (i.e. have poor predictive performance in out-of-sample testing). In the next section we consider extending the Poisson regression model to relax the assumption that β_{int} is constant across studies (i.e. the effect size is homogeneous across studies) and to allow for (and quantify) any heterogeneity in a parsimonious way.
Poisson regression with random intervention effects
where γ_{i0}∼N(0,σ^{2}) and γ_{i1}∼N(0,τ^{2}). The parameters β_{1} and β_{int} are fixed effect regression coefficients, whereas γ_{i0} represents the study-specific deviation from the average event rate and γ_{i1} represents the study-specific deviations from the average intervention effect. Their variances, σ^{2} and τ^{2}, estimate the baseline variability and between-study heterogeneity in the control group and intervention effect respectively. These are referred to as random effects variance parameters.
As with the fixed effect model, exp(β_{int}) represents the pooled IRR. It appears in this model, however, with the explicit acknowledgement that it is a population-averaged parameter and that specific instances of the IRR may vary between the sub-populations that are the targets of each study. Note that while we focus on a two-group comparison here, this approach is very flexible and can accommodate designs where there are multiple treatment arms (i.e. a three-group comparison) through the use of indicator variables and random-effects parameters for each treatment arm in the model. We refer to this approach as “Poisson regression with random intervention effects”.
Mixed-effect models can also be fitted by taking a Bayesian approach, that is, specifying a full probability model with distributional assumptions for both the observed data and the model parameters. Such a specification is particularly suited to hierarchical models like those used in meta-analysis, where the distribution of the data (in the case of meta-analyses, the event counts) are governed by parameters (the study-specific event rates and intervention effect) that themselves have a population distribution defined by a set of hyper-parameters (the random effects variance parameters) and so on as we progress through the levels of the hierarchy. The advantage of a full probability model specification is that it produces a joint posterior probability distribution for the parameters, which allows for more flexible approach to inference and incorporates explicitly the uncertainty of estimation in all parameters. We refer to this approach as “Bayesian Poisson regression with random intervention effects”.
The Bayesian approach comes at a cost, however, and that is the need to specify prior probability distributions for the unknown parameters (a sampling distribution for the data is also required, but this is usually implicit in the proposed regression model). It is well known that inferences about variance parameters when the data are sparse can be especially sensitive to the choice of prior distributions [18-20]. In this situation, non-informative prior distributions (prior distributions that are intended to allow Bayesian inference when not much is known beyond what is available in the data) are often employed. Several non-informative prior distributions have been proposed for estimating the variance parameters for continuous (normal) outcomes, i.e. estimating \(\sigma _{\alpha }^{2}\) when \(\theta _{\textit {ij}} \sim \text {Normal}(\mu + \alpha _{j}, {\sigma _{y}^{2}})\) where \(\alpha _{j} \sim \text {Normal}(0, \sigma _{\alpha }^{2})\). These are the inverse-Gamma distribution, the log-normal distribution and the half-Cauchy distribution [19,21,22].
When the inverse-gamma distribution is used as a prior distribution for the variance parameters, σ^{2}∼1/z and τ^{2}∼1/z where z=Gamma(ε,ε). When ε=0.001 this is a proper prior distribution (i.e. it does not depend on the data and integrates to 1 [23]) and close to uniform on log(σ) and log(τ). The inverse-gamma(0.001, 0.001) distribution has a peak close to zero and a long tail, meaning that low values for the variance components are supported (although when σ^{2} or τ^{2} are not close to zero this may unreasonably influence the posterior distribution).
If a log-normal prior distribution is employed as a prior distribution for the variance parameters, then the log standard deviations are normally distributed (e.g. σ∼Normal[0,100^{2}] and τ∼Normal[0,100^{2}]). A related prior, which allows estimation of the standard deviations on their natural scale is the half-Cauchy prior distribution, i.e. σ∼half-Cauchy(C) and τ∼half-Cauchy(C). The parameter C is the population median standard deviation. In a pure Bayesian analysis, the value of C would be based on prior information.
A final strategy is to take an empirical Bayes approach; for example, allowing the specification of the prior distribution for the variance parameters to depend on estimates of their magnitude and precision based on results from a non-Bayesian (frequentist) analysis. A possible prior distribution has the form \(\sigma \sim \text {Uniform}[0, \hat {\sigma } + \text {SE}(\hat {\sigma })]\) and \(\tau \sim \text {Uniform}[0, \hat {\tau } + \text {SE}(\hat {\tau })]\).
Simulation study
Overview
We use simulation methods to evaluate the performance of six different methods of pooling IRRs. These are (1) the inverse-variance method; (2) the inverse-variance method with the continuity correction; (3) Poisson regression with fixed intervention effects; (4) Poisson regression with random intervention effects; (5) Bayesian Poisson regression with random intervention effects with a inverse-gamma prior for the variance parameters; and (6) Bayesian Poisson regression with random intervention effects with a half-Cauchy prior for the variance parameters. The two key manipulations were the number of zero counts and the level of baseline variability and heterogeneity in each meta-analysis sample. We also varied the number of studies within each meta-analysis.
Data generation
Values used in the simulation study
Assigned values | |
---|---|
Methods evaluated | |
Inverse-variance method | |
Inverse-variance method with continuity correction | |
Poisson regression model with fixed intervention effects | |
Poisson regression model with random intervention effects | |
Bayesian Poisson regression model with random intervention effects using inverse-gamma priors for τ^{2} | |
Bayesian Poisson regression model with random intervention effects using half-Cauchy priors for τ | |
Fixed parameters | |
Incidence rate ratio, exp(β_{int}) | 0.2 |
Time, intervention group, t_{1} | t_{1}∼Uniform(2,5) |
Time, control group, t_{0} | t_{0}∼Uniform(2,10) |
Number of simulated datasets per scenario, B_{ s } | 500 |
Varied parameters | |
Percent of zero counts in the intervention group, Poisson(β_{1}∗0.2) | 0.09%, 5%, 14%, 37%, 55%, 82% |
Heterogeneity: control and intervention groups, σ,τ | |
Scenario A | (0.1,0.5) |
Scenario B | (0.1,2.5) |
Scenario C | (1.0,0.5) |
Scenario D | (1.0,2.5) |
Scenario E | (0.1×β_{1},2.5) |
Number of studies, k | 5, 10, 20 |
We varied the percentage of zeros in the intervention group from approximately 0.09 percent zeros to 82 percent zeros. We did this through setting the values of β_{1} to log(35), log(15), log(10), log(5), log(3) and log(1) so that when we drew random observations a Poisson distribution with mean μ= exp(β_{1})×0.2, the probability y=0 would be 0.1, 5, 14, 37, 55 and 82 percent, respectively (ignoring random-effects). These values provided a range in which to explore the effect of increasing the percentage of zeros in the data.
We also varied the amount of baseline variability and between-study heterogeneity. We considered scenarios where baseline variability was either σ=0.1 or 1.0 or where the baseline variability was proportionate to the baseline event rate (σ=0.1× exp(β_{1})). Similarly, we examined two values of between-study heterogeneity in the intervention effect, τ=0.5 or 2.5. In all, we examined five combinations of σ and τ representing the broad spectrum in which baseline variability and heterogeneity may influence real-world data. We refer to these as “Scenario A”, “Scenario B”, and so on.
Finally, we varied the number of studies that comprised each meta-analysis, setting k= 5, 10 and 20. We primarily focus on reporting the results of k=5 and k=10 studies, given that this represents the typical size of a systematic review in medicine [9,24].
Implementation
Varying six values of β_{1}, the five heterogeneity conditions and the three study sizes, produced s=90 scenarios for comparison. We simulated B_{ s }=500 datasets for each of the 90 scenarios, giving a total of 45,000 simulations. (We chose this number of simulations to keep the computation time manageable.) We used Stata 13.1 [25] to generate the data and estimate parameters for the first four methods of interest. We estimated parameters from the inverse-variance method using the metan package [26] in Stata. We used the poisson command in Stata to estimate a Poisson regression model with fixed intervention effects and the meqrpoisson to estimate the Poisson regression model with random intervention effects. The Bayesian Poisson regression models with random intervention effects were estimated using JAGS 3.10 [27] and rjags in R 3.1.1 [28]. For the Bayesian Poisson regression model with an inverse-gamma prior distribution for the variance parameters we used ε=0.001. For the equivalent model with a half-Cauchy prior distribution we used C=1. The Bayesian models were fit using two chains with an initial burn-in of 1000 iterations, followed by sampling of 5000 iterations. We checked a random sample of simulations from each scenario to determine if the chains had mixed together and encountered no problems.
Data extraction and analysis
From each simulation we extracted the estimated log IRR, \(\hat {\beta }_{\text {int}i}\), its standard error, \(\text {SE}(\hat {\beta }_{\text {int}i})\), and the estimates of the variance parameters, \(\hat {\sigma }_{i}\) and \(\hat {\tau }_{i}\). For \(\hat {\beta }_{\text {int}i}\), we evaluate bias, accuracy and coverage for each of the 90 scenarios of interest; for \(\hat {\sigma }_{i}\) and \(\hat {\tau }_{i}\) we evaluate bias only [29].
Bias in the log IRR was estimated by the percentage bias \((\bar {\hat {\beta }}_{\text {int}} - \beta _{\text {int}}/\beta _{\text {int}}) \times 100\), where \(\bar {\hat {\beta }}_{\text {int}} = \sum _{i = 1}^{B_{s}} \hat {\beta }_{\text {int}i} / B_{s}\) and β_{int}= log(0.2)=−1.609. Accuracy was measured by the mean square error, \((\bar {\hat {\beta }}_{\text {int}} - \beta _{\text {int}})^{2} + (\text {SE}(\hat {\beta }_{\text {int}}))^{2}\), where \(\text {SE}(\hat {\beta }_{\text {int}})\) is the empircal standard error over the B_{ s } simulations. We calculated coverage, the proportion of simulation in which the 95% confidence interval \(\hat {\beta }_{\text {int}i} \pm 1.96 \times \text {SE}(\hat {\beta }_{\text {int}i})\) includes β_{int}.
The percentage bias in \(\hat {\tau }_{i}\) was estimated for all models except the Poisson regression model with fixed intervention effects by \((\bar {\hat {\tau }} - \tau / \tau) \times 100\), where \(\bar {\hat {\tau }} = \sum _{i = 1}^{B_{s}} \hat {\tau }_{i} / B_{s}\). Only the Poisson regression methods with random intervention effects estimate \(\hat {\sigma }_{i}\). We therefore only extracted this in these cases, estimating percent bias in the same way as for \(\hat {\tau }_{i}\).
Empirical studies
Suicides from known jumping hotspots
In the introduction we briefly described the motivating example for this simulation study – a meta-analysis of the effectiveness of installing barriers on reducing suicide by jumping at known hotspots [1]. Jumps from these sites (bridges, viaducts and cliffs) generally have high fatality rates, can cause significant distress or injury to bystanders and often receive prominent media coverage, increasing the risk of copycat acts [30]. A number of studies have investigated the effectiveness of structural interventions – such as barriers, fences or safety nets – on reducing suicide by jumping at these sites [31-39]. Individual studies are typically before-and-after designs, with the pre-intervention period considered the “control” group and the post-intervention the “intervention” group. (Although we do not show the data here, these studies also compare suicide rates at nearby sites before and after the introduction of barriers at the hotspot, thereby providing additional information on the effectiveness of barriers.)
Suicide counts and exposure time by study
Pre-intervention | Post-intervention | |||
---|---|---|---|---|
Study no. | No. events | Time (years) | No. events | Time (years) |
1 | 19 | 6 | 0 | 4 |
2 | 41 | 5 | 20 | 5 |
3 | 221 | 14 | 0 | 0.4 |
4 | 25 | 7 | 1 | 5 |
5 | 14 | 22 | 0 | 22 |
6 | 7 | 3 | 0 | 3 |
7 | 96 | 9 | 0 | 4 |
8 | 13 | 10 | 0 | 2 |
Condom effectiveness in reducing heterosexual HIV transmission
Weller and Davis-Beaty [8] used meta-analysis to evaluate the effectiveness of condoms in reducing the incidence of HIV infection between heterosexual couples. They included studies that examined the direction of transmission from male to female partners, female to male partners, and studies where the direction of transmission was unknown. This was based on observational data, so the “control” group was couples who never used condoms and the “intervention” group was those who always used condoms. The outcome of interest was the incidence of HIV transmission.
Heterosexual HIV transmission counts and exposure time by study
Always use condoms | Never use condoms | |||
---|---|---|---|---|
Study no. | No. events | Person-years | No. events | Person-years |
1 | 0 | 11.5 † | 2 | 6.9 † |
2 | 0 | 8.4 † | 4 | 21.1 |
3 | 1 | 101 † | 13 | 185.3 |
4 | 1 | 8.54 † | - | - |
5 | - | - | 1 | .006 |
6 | 0 | 45.2 † | - | - |
7 | 4 | 136.1 † | - | - |
8 | 0 | 28 † | - | - |
9 | 5 | 362.5 † | - | - |
10 | - | - | 0 | 5 † |
11 | - | - | 2 | 60.4 † |
12 | - | - | 10 | 147 † |
13 | - | - | 8 | 139.3 |
14 | 0 | 7.5 † | 0 | 9.6 |
15 | 0 | 249.6 † | - | - |
16 | 0 | 6 † | 0 | 24 † |
Results
Simulation study
As the percentage of sparse data within the intervention group increases, the point estimate of the pooled log IRR derived from the inverse-variance method display increased bias (column 1). For instance, in scenario A (σ=0.1,τ=0.5) with k=5 studies, the percentage bias is approximately 0% when there is effectively no zeros in the data. With 5% zeros, the percentage bias is 5%, increasing to 50% when there is around 82% zeros in the data. This pattern is replicated for all other scenarios and when k=10 and k=20 (see the Table A1 in the Additional file 1: Appendix). This pattern occurs because the log IRR for an individual study is undefined when there are zero events, and as a result, studies with zeros are excluded from the estimate of the pooled effect size, biasing the results. This pattern is exacerbated by larger heterogeneity values (τ=2.5 in scenarios B, D and E). For instance, when there is 82% zeros in the data, the percent bias is 133%, 116% and 130% in these scenarios respectively for k=5, and by a similar amount for k=10 and 20. The continuity correction does not remedy any of these problems (column 2).
The Poisson regression model with a fixed intervention effect (column 3) displays only a small amount of bias in the pooled log IRR when there is low baseline variability (σ=0.1 in scenarios A and C). In these scenarios, the percentage bias ranged from 0.6% to 8% for all values of k. But when heterogeneity was larger (τ=2.5 in scenarios B, D and E) then this method produced point estimates of the pooled log IRR that diverge substantially from their true values. For example, the percentage bias in scenario B is approximately 100% for k=5 regardless of the amount of zeros in the data, and range from 120% to 134% when k=10. In contrast to this, the Poisson regression model with random intervention effects (column 4) produced estimates of the pooled log IRR that were close to their true value in all scenarios. The percentage bias ranged from approximately 0% to 11% for k=5 and between 0% and 7% when k=10. The size of any bias was unrelated to the percentage of zeros in the data or by baseline variability and between-study heterogeneity.
A Bayesian approach to the Poisson regression model with random intervention effects produced biased pooled log IRRs. When the variance parameters were estimated using an inverse-gamma prior distribution, the pooled log IRR was close to the true value in scenarios when there was only a small amount of zeros in the data (column 5). For instance, when k=5 and there was no zeros in the data, the percentage bias was just 2.2% in scenario A, 16% in scenario B, 6% in scenario C, 13% in scenario D and 29% in scenario E. As the percentage of zeros in the data increased, the amount of bias increased, such that, for instance, when there were 82% zeros in the data, the percentage bias was 61%, 121%, 73%, 95% and 112% in scenarios A through E respectively. While these effects were attenuated when k=10 and k=20 (Additional file 1: Appendix, Table A1), the general pattern remained. A similar picture emerged when the variance parameters were estimated using a half-Cauchy prior distribution (column 6).
Empirical studies
Table 2 shows the counts of jumping suicides and the exposure time for each study reported by Pirkis et al. [1]. The zero values in six of the intervention groups means the study-specific log IRR and its standard error can only be calculated for the two remaining studies (studies 2 and 4). Therefore, analysis using the inverse-variance method estimated a pooled IRR of 0.207 with 95% confidence interval (CI) 0.026 to 1.646. This finding can be interpreted as providing insufficient evidence to conclude whether or not the barriers reduce the number of suicide jumping deaths per year. Repeating the analysis using the continuity correction meant that all eight studies could be included in the analysis and this approach yielded a pooled IRR of 0.085 with 95% CI 0.026 to 0.284, suggesting strong evidence of a protective effect. Analysis using a Poisson regression model with fixed intervention effects estimated a pooled IRR of 0.151 with 95% CI 0.089 to 0.229 and a Poisson regression model with random intervention effects estimated a pooled IRR of 0.008 with 95% CI 0.0002 to 0.300. The estimates of the random effects parameters varied between methods. Using inverse-variance methods, \(\hat {\tau } = 1.34\) and with the continuity correction \(\hat {\tau } = 1.25\). In a a Poisson regression model with random intervention effects, \(\hat {\tau } = 2.48\). The results using the two Bayesian approaches gave a similar effect size for the pooled IRR and the estimate of heterogeneity.
Table 3 shows the counts of HIV infections and exposure time in 11 studies that followed heterosexual couples who “always” used condoms (the intervention group) and 10 studies of couples who “never” used condoms (the control group). The data pose a number challenges for traditional meta-analysis. Only 5 studies have data in both treatment arms, and unusually, there is less data available for the control group than the intervention group. The data were also sparse, with seven studies in the intervention and three studies in the control group having zero counts. The combination of these two elements means that the study-specific log IRR and its standard error were undefined for all studies. Therefore, it is not possible to calculate a pooled IRR using the inverse-variance method. Weller and Davis-Beaty [8] overcame this problem by collapsing all the data into a single table and calculating a pooled IRR from this aggregated information. This approach estimated a pooled IRR of 0.198. Weller and Davis-Beaty derive their confidence limits using a best case/worst case scenario. But using an aggregated analysis gives a 95% CI of 0.081 to 0.470.
Estimates from a Poisson regression model with fixed intervention effects could not be estimated reliably (due to the imbalance in the number of studies in the control and intervention groups), but they could be derived when the indicator variables were omitted. This gave a pooled IRR of 0.198 (95% CI 0.090 to 0.437) which is very similar to the estimates from the aggregated analysis. Analysis of this data using a Poisson regression model with random intervention effects estimated a pooled IRR of 0.171 with 95% CI 0.057 to 0.515. Weller and Davis-Beaty [8] excluded a number of studies from their primary analysis because of concerns about heterogeneity. Our simulation results suggested that the intervention effect parameters are unbiased in the presence of heterogeneity when using a Poisson regression model with random intervention effects. Therefore, we re-analysed their data using all the available information (this information also contained in Table 3). The revised analysis estimated a pooled IRR of 0.147 (95% CI 0.053 to 0.407). Using a Bayesian Poisson regression approach with an inverse-gamma distribution for the variance components gave an IRR of 0.122 with 95% credible interval 0.014 to 0.396. Using a half-Cauchy prior distribution for the variance components yielded a pooled IRR of 0.102 with 95% credible interval 0.010 to 0.500. Turning to the random effects, the Poisson regression model with random intervention effects gave an estimate of \(\hat {\tau } = 0.616\). Using the inverse-gamma prior gave an estimate of \(\hat {\tau } = 0.666\) while the half-Cauchy prior gave \(\hat {\tau } = 0.841\).
Discussion
Methods for the meta-analysis of incidence rate data (counts of events in time) have received relatively little attention [40], and no work has addressed how to undertake a meta-analysis when there are structural zeros in the data (multiple studies within a meta-analysis which have counts of zero events). Nonetheless, there is a need to undertake meta-analyses on this type of data. We have shown that the inverse-variance method of meta-analysis (one of the most commonly used and recommended methods) is biased in the presence of structural zeros. We show that this finding holds even after adjustment using the continuity correction as recommended by the Cochrane Handbook [6].
We explored several alternatives to the inverse method. In the context of pooling rates when exposure time varies between groups, Guevara et al. [5] proposed using Poisson regression with indicator variables for each study. Since the Poisson distribution includes zeros [41,42], this suggests a potentially useful means of pooling incidence rate data with structural zeros. The issue that then arises is how the pooled IRR and the other fixed-effects estimates are effected by baseline variability and heterogeneity. Theoretically, this approach accounts for baseline variability (i.e. variability in the incidence rates in the control group) via the use of indicator variables for each study. Our simulations show that this method produces relatively unbiased estimates when there is low baseline variability. But as the level of baseline variability increases the method displays bias in the pooled IRR. Interestingly, this bias is constant across simulations regardless of the amount of zeros in the data. Our simulations also show that Poisson regression with fixed intervention effects has high mean square error (relative to the competing method) under high heterogeneity conditions and has poor coverage. As such, we do not recommend using Poisson regression with fixed intervention effects for meta-analysis unless the baseline variability and between-study heterogeneity are both close to zero.
Our primary interest was in comparing the two aforementioned methods to Poisson regression with random intervention effects. This method extends Poisson regression with fixed intervention effects by allowing study-specific random intercepts and slopes. These parameters therefore estimate the baseline variability and between-study heterogeneity in the intervention effect. This latter estimate is often of interest when conducting meta-analysis. We explored several implementations of this method: one based on adaptive Gaussian quadrature estimation (referred to as a frequentist model), and two based on Bayesian techniques (a full probability model with prior distributions for all parameters including the study-specific intercepts and slopes). We tested the usefulness of an inverse-gamma prior distribution and the half-Cauchy distribution for the random-effect parameters while using the traditional non-informative normal distribution for the fixed-effect parameters.
Our simulations show that the (frequentist) Poisson regression model with random intervention effects estimated the pooled IRR without bias, generally had the lowest mean square error and had good coverage. These results held in a variety of situations, for instance when there was only a small number of studies in each meta-analysis, when there was high baseline variability or high heterogeneity, and when there was a large number of zeros in the data. The estimates of baseline variability and between-study heterogeneity were close to their true values, but did exhibit bias in some circumstances – most notability when there was a small number of studies in each meta-analysis and when there was a large number of zeros in the data. It is worth pointing out, however, that all methods did poorly in these situations, and that Poisson regression with random intervention effects had the lowest bias of those tested. Neither Bayesian implementation of this method were able to estimate the pooled IRR or the variance components as accurately. Based on these findings, we see Poisson regression with random intervention effects as a useful method for conducting a meta-analysis of incidence rate data, especially when the data contains structural zeros. In line with this, we give code in the Additional file 1: Appendix for setting up the data and undertaking analysis using Stata [25]. There are two important caveats to this recommendation. First, our simulations show that the accuracy of the pooled IRR improves as the number of studies increases. Thus, while it is possible to conduct a meta-analysis using, for example five studies, a meta-analysis with more studies than this will provide more stable estimates for all parameters. Second, our results show that the estimates of baseline variability and between-study heterogeneity remain biased regardless of the number of studies in the meta-analysis. As such, while the pooled IRR is likely to be accurate, the variance parameters will be estimated with error.
Although not reported here, our simulations included several other methods. We evaluated two other fixed-effects methods – complete pooling of the data to calculate the pooled incidence rate ratio and stratified pooling (by study) to calculate the pooled effect size [43]. In simulations, these results were effectively the same as those for the fixed-effects Poisson regression. We also explored several other prior distributions for the variance components in a Bayesian analysis – a log normal prior distribution for the variance components and an Empirical Bayes approach. Results for both were similar to that reported for the Bayesian methods reported here. In general, we found that the Bayesian approach was able to reproduce the results from adaptive Gaussian quadrature but we believe its performance could be improved by taking an iterative approach to determine the parameters defining the prior distribution of the variance components – an approach that has been demonstrated previously [44]. Finally, it is worth noting, that the BUGs language, the tool used to implement Bayesian analysis, is very flexible and able to draw from a complex structure for the random-effects parameters. Thus, there is likely to be situations where a Bayesian approach will out-perform a frequentist analysis.
Our study has several limitations. First, we evaluated only six methods of analysing incidence rate data, but a number of different methods have been proposed, for example, the Mantel Haenszel method [45], Peto’s method [46], the Binomial-Normal method [7]. It may be fruitful for future research to compare and contrast these methods with our preferred approach. Second, our simulations did not allow for a correlation between baseline variability and heterogeneity. This was mainly because with k=5 or 10 studies, there is likely to be insufficient information in the data to estimate this parameter reliability. Nonetheless, in real-world data, such an association could plausibly occur. Finally, we did not directly manipulate the sample sizes in each study. Yet, in typical meta-analyses, for example where the effect size of interest is a pooled odds ratio or rate ratio, then studies with large numbers will tend to have a strong influence on the overall result. This effect of this on estimating a pooled IRR using Poisson regression with random intervention effects is unknown.
Conclusion
Our approach is a simple yet flexible method of undertaking meta-analyses on incidence rate data when there are zero counts in the data. Our proposed method of using Poisson regression with random intervention effects has several merits. First, many popular statistical programs (e.g. Stata, R) can perform the analysis using routinely available command. In Stata, the command is meqrpoisson and in R, the glmer command in the lme4 package. We give example Stata code in the Additional file 1: Appendix. The commands also enable the basic model to be extended – for instance it is trivial to estimate a correlation between σ and τ with modern statistical software. This is also true of Bayesian methods as implemented by JAGS, WinBUGS and OpenBUGS. Second, because the method is based on regression techniques, in principle it is possible that the models themselves can be extended to include additional covariates. For example, it is common to report separate meta-analyses for subgroups such as males and females, or for observational studies and randomised control trials. When data is available at the subgroup level, parameters representing these groups could be entered into the model either as additive terms or multiplicative terms (for instance, with the variable representing the treatment arm). Further research could investigate this more fully.
Availability of supporting data
Declarations
Authors’ Affiliations
References
- Pirkis J, Spittal MJ, Cox G, Robinson J, Cheung YTD, Studdert D. The effectiveness of structural interventions at suicide hotspots: a meta-analysis. Int J Epidemiol. 2013; 42(2):541–8.PubMedGoogle Scholar
- Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, et al. Suicide prevention strategies: a systematic review. JAMA. 2005; 294(16):2064–074.PubMedGoogle Scholar
- Hasselblad VV, McCrory DCD. Meta-analytic tools for medical decision making: a practical guide. Med Decis Making. 1994; 15(1):81–96.Google Scholar
- Bagos PG, Nikolopoulos GK. Mixed-effects Poisson regression models for meta-analysis of follow-up studies with constant or varying durations. Int J Biostat. 2009; 5(1):1–33.Google Scholar
- Guevara JP, Berlin JA, Wolf FM. Meta-analytic methods for pooling rates when follow-up duration varies: a case study. BMC Med Res Methodol. 2004; 4:17.PubMedPubMed CentralGoogle Scholar
- Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. West Sussex: The Cochrane Collabortion and John Wiley & Sons, Ltd; 2008.Google Scholar
- Stijnen T, Hamza TH, Ozdemir P. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat Med. 2010; 29:3046–67.PubMedGoogle Scholar
- Weller SC, Davis-Beaty K. Condom effectiveness in reducing heterosexual HIV transmission. Cochrane Database Syst Rev. 2002; 1:CD003255.PubMedGoogle Scholar
- Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med. 2004; 23(9):1351–75.PubMedGoogle Scholar
- Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med. 2007; 26:53–77.PubMedGoogle Scholar
- Bhaumik DK, Amatya A, Normand SL, Greenhouse J, Kaizar E, Neelon B, Gibbons RD. Meta-Analysis of Rare Binary Adverse Event Data. J Am Stat Assoc. 2012; 107(498):555–67.PubMedPubMed CentralGoogle Scholar
- DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177–88.PubMedGoogle Scholar
- Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539–1558.PubMedGoogle Scholar
- Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003; 327(7414):557–60.PubMedPubMed CentralGoogle Scholar
- Viechtbauer W. Confidence intervals for the amount of heterogeneity in meta-analysis. Stat Med. 2007; 26(1):37–52.PubMedGoogle Scholar
- Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, et al. Indirect comparisons of competing interventions. Health Technol Assess (Winchester, England). 2005; 9(26):1.Google Scholar
- Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR task force on indirect treatment comparisons good research practices: part 1. Value Health. 2011; 14(4):417–28.PubMedGoogle Scholar
- Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR. How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med. 2005; 24(15):2401–428.PubMedGoogle Scholar
- Gelman A. Prior distributions for variance parameters in hierarchical models. Bayesian Anal. 2006; 1(3):515–34.Google Scholar
- Jamsen KM, Zaloumis SG, Scurrah KJ, Gurrin LC. Specification of generalized linear mixed models for family data using Markov Chain Monte Carlo Methods. J Biomet Biostat. 2012; S1:003.Google Scholar
- Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-care Evaluation, 1st. West Sussex: Wiley; 2004.Google Scholar
- Lunn D, Jackson C, Spiegelhalter DJ, Best N, Thomas A. The Bugs Book. A practical introduction to Bayesian analysis. Boca Raton: Chapman & Hall; 2012.Google Scholar
- Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian data analysis, 2nd. Boca Raton: Chapman and Hall/CRC; 2004.Google Scholar
- Kontopantelis E, Springate DA, Reeves D. A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. PLoS One. 2012; 8(7):69930.Google Scholar
- StataCorp. Stata: Release 13. College Station TX: StataCorp LP; 2013.Google Scholar
- Harris R, Bradburn M, Deeks J, Harbord R, Altman D, Sterne J. metan: fixed- and random-effects meta-analysis. Stata J. 2008; 8(1):3–28.Google Scholar
- Plummer M. JAGS Version 3.1.0 User Manual; 2011. http://sourceforge.net/projects/mcmcjags/files/Manuals/3.x/jags_user_manual.pdf/download.
- R Core Team. R: a language and environment for statistical computing. Vienna; 2014. R Foundation for Statistical Computing, http://www.R-project.org/.
- Burton A, Altman DG, Royston P, Holder RL. The design of simulation studies in medical statistics. Stat Med. 2006; 25(24):4279–292.PubMedGoogle Scholar
- Chen Y-Y, Yu K, Yip P. International handbook for suicide prevention: research, policy and practice In: O’Conner R, Platt S, Gordon J, editors. International handbook for suicide prevention: research, policy andpractice. Chichester: Wiley-Blackwell: 2011.Google Scholar
- O’Carroll PW, Silverman MM, Berman AL. Community suicide prevention: the effectiveness of bridge barriers. Suicide Life Threat Behav. 1994; 24(1):89–99.PubMedGoogle Scholar
- Beautrais AL, Gibb SJ, Fergusson DM, Horwood LJ, Larkin GL. Removing bridge barriers stimulates suicides: an unfortunate natural experiment. Aust N Z J Psychiatry. 2009; 43(6):495–7.PubMedGoogle Scholar
- Beautrais AL. Effectiveness of barriers at suicide jumping sites: a case study. Aust N Z J Psychiatry. 2001; 35(5):557–62.PubMedGoogle Scholar
- Reisch TT, Michel KK. Securing a suicide hot spot: effects of a safety net at the Bern Muenster Terrace. Suicide Life Threat Behav. 2005; 35(4):460–7.PubMedGoogle Scholar
- Bennewith O, Nowers M, Gunnell D. Effect of barriers on the Clifton suspension bridge, England, on local patterns of suicide: implications for prevention. Br J Psychiatry. 2007; 190:266–7.PubMedGoogle Scholar
- Pelletier AR. Preventing suicide by jumping: the effect of a bridge safety fence. Inj Prev. 2007; 13(1):57–9.PubMedPubMed CentralGoogle Scholar
- Skegg K, Herbison P. Effect of Restricting Access to a Suicide Jumping Site. Aust N Z J Psychiatry. 2009; 43(6):498–502.PubMedGoogle Scholar
- Sinyor M, Levitt AJ. Effect of a barrier at Bloor Street Viaduct on suicide rates in Toronto: natural experiment. BMJ. 2009; 341:2884.Google Scholar
- Issac M, Bennett J. Prevention of suicide by jumping: The impact of restriction of access at Beachy Head, Sussex during the foot and mouth crisis 2001. J Public Health Med. 2005; 6(1):19–22.Google Scholar
- Sutton AJ, Higgins JPT. Recent developments in meta-analysis. Stat Med. 2008; 27(5):625–50.PubMedGoogle Scholar
- Hilbe J. Negative binomial regression. Cambridge: Cambridge University Press; 2011.Google Scholar
- Cameron AC, Trivedi P. Regression analysis of count data. Cambridge: Cambridge University Press; 1998.Google Scholar
- Rothman KJ, Greenland S, Lash TL. Modern epidemiology, 3rd. Philadephia: Lippincott Williams & Wilkins; 2008.Google Scholar
- Burton PR, Tiller KJ, Gurrin LC, Cookson WOCM, Musk AW, Palmer LJ. Genetic variance components analysis for binary phenotypes using generalized linear mixed models (GLMMs) and Gibbs sampling. Genet Epidemiol. 1999; 17:118–40.PubMedGoogle Scholar
- Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst. 1959; 22(4):719–48.PubMedGoogle Scholar
- Peto R, Pike MC, Armitage P, Breslow NE, Cox DR, Howard SV, et al. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. II. analysis and examples. Br J Cancer. 1977; 35(1):1–39.PubMedPubMed CentralGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.