Evaluating quasi-experimental approaches for estimating epidemiological efficacy of non-randomised field trials: applications in Wolbachia interventions for dengue

Abstract

Background

Wolbachia symbiosis in Aedes aegypti is an emerging biocontrol measure against dengue. However, assessing its real-world efficacy is challenging due to the non-randomised, field-based nature of most intervention studies. This research re-evaluates the spatial–temporal impact of Wolbachia interventions on dengue incidence using a large battery of quasi-experimental methods and assesses each method’s validity.

Methods

A systematic search for Wolbachia intervention data was conducted via PUBMED. Efficacy was reassessed using commonly-used quasi-experimental approaches with extensive robustness checks, including geospatial placebo tests and a simulation study. Intervention efficacies across multiple study sites were computed using high-resolution aggregations to examine heterogeneities across sites and study periods. We further designed a stochastic simulation framework to assess the methods’ ability to estimate intervention efficacies (IE).

Results

Wolbachia interventions in Singapore, Malaysia, and Brazil significantly decreased dengue incidence, with reductions ranging from 48.17% to 69.19%. IEs varied with location and duration. Malaysia showed increasing efficacy over time, while Brazil exhibited initial success with subsequent decline, hinting at operational challenges. Singapore's strategy was highly effective despite partial saturation. Simulations identified Synthetic Control Methods (SCM) and its variant, count Synthetic Control Method (cSCM), as superior in precision, with the smallest percentage errors in efficacy estimation. These methods also demonstrated robustness in placebo tests.

Conclusions

Wolbachia interventions exhibit consistent protective effects against dengue. SCM and cSCM provided the most precise and robust estimates of IEs, validated across simulated and real-world settings.

Introduction

The burden of dengue increased globally by 30-fold over the past five decades [1], placing a heavy burden on healthcare systems. With limited effective therapeutics [2, 3] and vaccines [4, 5], vector-control remains the primary means to reduce dengue transmission [6, 7].

Emerging vector-control strategies, such as the Sterile Insect Technique (SIT) and Wolbachia-based methods, show promise in reducing mosquito populations and the transmission of dengue. SIT utilises radiation to sterilise male mosquitoes, limiting population growth. Wolbachia-based techniques employ two methods: the Incompatible Insect Technique (IIT), where Wolbachia-infected males render females infertile, and introgression (also referred to as “replacement”), which spreads Wolbachia within mosquito populations through infected females, reducing their vectorial capacity.

Despite the initial success of Wolbachia-infected Aedes aegypti mosquitoes in field trials across several countries like Malaysia [8, 9], Indonesia [10], Singapore [11, 12], Brazil [13, 14], Colombia [15], and Australia [16, 17], accurately gauging the intervention’s epidemiological impact presents substantial challenges. These challenges stem from factors that can obscure causal inferences regarding intervention efficacy (IE), such as (1) intervention spillovers to control areas, (2) heterogeneities in baseline disease trajectories in spatial units prior to intervention, (3) staggered adoption of intervention due to resource limitations, (4) environmental heterogeneity, leading to different doses of interventions across target units despite uniform implementation of interventions, and (5) biological intricacies of the intervention, such as imperfect suppression of wildtype mosquitoes or incomplete introgression.

While gold-standard, cluster randomised controlled trials (cRCTs) have successfully evaluated the epidemiological efficacy of Wolbachia introgression [10], they face practical and ethical constraints in population-level infectious disease control [18,19,20]. Additionally, although cRCTs balance baseline characteristics and historical dengue risk in intervention and control arms, differences in disease pre-trends cannot be explicitly balanced even if randomisation equalises historical dengue risk. Spillover effects due to the close proximity of intervention sites to controls also cannot be easily mitigated even with geographical features, underestimating the epidemiological efficacy of interventions.

Given these constraints, cRCTs can be augmented with alternative, quasi-experimental methods to triangulate IEs, including Pre-post analysis [16], Bayesian time series analysis [8], Difference-in-Differences (DiD) [13, 14, 21] and Regression Discontinuity Design (RDD) [22]. Currently, there is no consensus on a universally superior method, and choices often depend on study context. Other methodologies, such as compartmental models [17] can account for complicated disease transmission processes, and the Synthetic Control Method (SCM) [12] has been employed to simultaneously accommodate differences in spatiotemporal characteristics and historical dengue risk. These approaches aim to counteract selection bias and confounding factors, helping in robust causal inference in the absence of traditional randomisation.

While existing literature acknowledges the challenges of evaluating Wolbachia interventions, there is yet to be an examination of the robustness of quasi-experimental methods in this application. To address this gap, our study conducted a thorough analysis of various quasi-experimental designs to determine the causal impact of Wolbachia on dengue incidence across diverse settings in Singapore, Malaysia, and Brazil. Our study provides a threefold contribution to the literature on quasi-experimental methodology. Firstly, we systematically searched for and re-evaluated all available open datasets pertaining to Wolbachia field trials using a large battery of quasi-experimental tools to re-estimate IEs by spatial units and event time. Secondly, we implemented a series of spatial and temporal placebo tests to verify the methodological robustness of each method. Finally, we stress-tested each method against a diverse range of hypothetical dengue epidemic scenarios through a simulation study to further underscore the practical relevance of these tools in public health planning.

By comparing these methods, we aim to identify the quasi-experimental techniques that have the necessary rigor for reliable evaluations of public health measures, a critical step to aid in the implementation of evidence-based strategies against dengue and similar diseases when randomisation is impractical.

Methods

Study cohorts and data collection

We sought to re-evaluate the efficacies of Wolbachia interventions to stem dengue transmission where reports were publicly available, using quasi-experimental tools. First, eligible study cohorts were identified through a systematic review conducted in accordance with the PRISMA guidelines (Supplementary Fig. 1). A search was performed on the PUBMED platform from December 2023 to January 2024 using specific terms: (1) "dengue" OR "breakbone fever," (2) "population" OR "community," and (3) "Wolbachia". The final search was conducted on January 4, 2024.

The inclusion criteria for article selection were studies that investigated the epidemiological impact of Wolbachia on dengue. On the other hand, the following exclusion criteria were applied: (1) articles without open-sourced data, (2) articles that only reported aggregated dengue incidence across all control and intervention sites, (3) articles with no control arm or only a single control arm, (4) articles where the final reported metric was not dengue incidence, (5) review articles, and (6) articles not published in English. These criteria ensured that the included studies were accessible for re-evaluating IEs using selected quasi-experimental methods as described later.

The initial search yielded a total of 212 articles. Following screening and application of the specified inclusion and exclusion criteria, 2 articles met the requirements and were selected for data extraction. The third study setting, Singapore, was included in the analysis despite not having open-sourced data due to the team’s pre-existing access to the relevant data from previous studies [12, 23]. For more detailed data descriptions, see Supplementary Methods.

Quasi-experimental methods for understanding intervention effects

We evaluate IEs of Wolbachia interventions to reduce dengue incidence rates using 6 commonly employed quasi-experimental methods, which include the (1) Pre-post design – where dengue incidence rates are compared based on mean differences by utilising a simple regression model; (2) regression discontinuity design (RDD) – where dengue incidence rates are compared in pre- and post-intervention time for intervention units while controlling for temporal trends; (3) 2 × 2 difference-in-differences (2 × 2 DiD), which computes the intervention efficacies by comparing first, the Pre-post intervention differences in dengue incidence for intervention units, and the differences in dengue incidence between intervention and control units over time; (4) difference-in-differences with multiple time periods (Panel DiD), which is an extension of 2 × 2 DiD that explicitly accounts for the intervention adoption time in each treated unit, allowing for estimation of treatment effects that vary spatially and temporally (Supplementary Table S2, S3); (5) the canonical synthetic control methods (SCM), which generates counterfactuals for the intervention site by minimizing the differences in the dengue incidence between a weighted set of donors (controls) versus the intervention unit in the pre-intervention period; and (6) count SCM (cSCM), which relaxes the bounding of synthetic control weights but still maintains the zero-lower bound for dengue incidence for the synthetic controls. (1) – (2) can be considered a single-group design as it includes only treated units in the model for evaluation, while (3) – (6) can be considered multiple-group designs which have both treated and control groups for evaluation of IEs.

For regression-based analyses (1) – (3) with the Malaysian and Singapore data, we utilised zero-inflated negative binomial models from the 'pscl' R package, as the data exhibited non-negative, continuous characteristics with a significant number of zero values. Meanwhile, for the Brazil data, we employed truncated normal regressions using the R package 'truncreg'. This decision was based on the absence of a population to use as an offset and the continuous, non-negative nature of the data. Supplementary Methods provides specific details on implementation of all quasi-experimental methods, computation of IEs and the counterfactuals.

Aggregates of intervention efficacy (IE)

As outcomes were both temporally and spatially explicit, we could re-aggregate IEs in meaningful ways, to examine potential spatial heterogeneity in IEs and considering time needed for Wolbachia interventions to take effect – either the time required for Wolbachia to introgress into local populations for the introgression approach, or suppression of wild-type mosquitoes to take place using the incompatible insect technique (IIT). In the preceding sections, $I{E}_{RDD,i}, I{E}_{DiD,i}$ and $I{E}_{SCM,i}$ can be considered as aggregations of IEs for a specific treated unit $i$. However, we can also compute.

• (1) the overall effect of the intervention across all study sites ${(IE}_{I}),$ as the difference between the cumulative sum of actual cases ${y}_{i,t}$ and cumulative sum of counterfactual cases ${\widehat{y}}_{i,t}$ across all sites. This is calculated from each site $i$’s specific start time of intervention ${t}_{start\left(i\right)}$ to the end of the study ${t}_{end}$.

$${IE}_{I}= \frac{\left(\sum_{i=1}^{I}\sum_{{t= t}_{start\left(i\right)}}^{{t}_{end}}{\widehat{y}}_{i,t} \right)- (\sum_{i=1}^{I}\sum_{{t= t}_{start\left(i\right)}}^{{t}_{end}}{y}_{i,t})}{\left(\sum_{i=1}^{I}\sum_{{t= t}_{start\left(i\right)}}^{{t}_{end}}{\widehat{y}}_{i,t} \right)}\times 100$$

(1)

• (2) the IE by event-study time (${IE}_{{t}_{1}:{t}_{2}})$, where ${t}_{1}:{t}_{2}$ are the start and end periods for some pre-specified time in the post-intervention period. Intervention and counterfactuals can similarly be obtained in preceding sections.

$${IE}_{{t}_{1}:{t}_{2}}=\left({\sum }_{t={t}_{1}:{t}_{2}}\frac{{\text{counterfactual}}_{t}-{\text{intervention}}_{\text{t}}}{{\text{counterfactual}}_{t}} \right)\times 100$$

(2)

Robustness checks for quasi-experimental methods

Each method was stress-tested to validate efficacy estimates. Placebo interventions, set two years prior to actual interventions, discerned if observed effects were due to pre-intervention dengue incidence trends or the Wolbachia, ensuring that each methods fits well to the pre-intervention data and that the measured effects were attributable to the intervention rather than to poorly predictive counterfactuals. For each dataset and method, we truncated all observations to be the pre-intervention period data, and recalculated placebo IEs after adjusting the placebo-treatment to two years prior to the actual intervention date. Post-adjustment placebo efficacies were then compared to actual intervention effects in the complete dataset.

To test for spatial validity, in-space placebo tests were conducted by reassigning the control group to be the treated units and measuring whether there were any large intervention effects for control sites, while excluding the actual treated units from the donor pool. Supplementary Methods provides further details on the execution of in-space placebos for each method.

Simulation study

We conducted a simulation study to examine whether quasi-experimental methods can recover IEs of Wolbachia interventions. We defined data generating processes using a custom-built ASEI-SEIR transmission dynamic model which incorporates host-vector components with waning immunity with stochasticity in the observation process for the Wolbachia introgression intervention (Supplementary Methods, Supplementary Table S1). In summary, the simulation was conducted 1000 times for various control pool sizes, ranging from 5 to 40 sites with increments of 5 per 1000 simulations.

Each simulation run created two intervention sites first with the absence, then the presence of Wolbachia releases using the same initial parameters (i.e. initial populations of susceptible humans ${S}_{h}$, dengue-infected humans ${I}_{h}$, and wild-type juvenile mosquitoes ${A}_{\left(n,w\right)}$). After the simulations were run, we assumed stochasticity using a Poisson distribution for the number of infected individuals in both intervention and control sites. The ‘true’ IE was then computed by:

$${IE}_{{t}_{1}:{t}_{2}}=\left({\sum }_{t={t}_{1}:{t}_{2}}\frac{{\text{no intervention}}_{t}-{\text{intervention}}_{\text{t}}}{{\text{no intervention}}_{t}} \right)\times 100$$

(3)

Each quasi-experimental tool was then fitted according to the observable number of infected individuals to generate counterfactuals for the case where the intervened group had no interventions. The $IE$ of Wolbachia introgression was hence calculated using each method and its respective formulae. The percentage error between the ‘true’ $IE$ and the calculated values was subsequently computed to determine the ability of each method to recover the ‘true’ $IE$.

Data analysis was performed using R software (V. 4.3.1) and RStudio (V.2023.06.1 + 524).

Results

Characteristics and outcomes of Wolbachia trials across study settings

Table 1 provides a summary of the essential characteristics pertaining to the Wolbachia trials conducted in Malaysia, Brazil, and Singapore. The Malaysia trial involved wAlbB Wolbachia introgression across six sites, with a pre-intervention period of 221 to 255 weeks and a post-intervention period of 78 to 88 weeks, achieving the highest intervention saturation among the studies at 74.62%. In Malaysia, saturation is calculated as the average percentage of wAlbB detected by qPCR in Ae. aegypti mosquitoes, starting 4 or 8 weeks post-release depending on the study site. In Brazil, a similar intervention with wMel Wolbachia spanned 32 sites, with a pre-intervention duration of 42 to 51 months and a shorter post-intervention timeframe of 3 to 12 months, resulting in a 50.65% saturation rate. In Brazil, saturation is the average percentage of wMel detected by qPCR, starting 1 month post-release. Conversely, Singapore utilised a combination of IIT and SIT with high-fidelity sex sorting across four sites with saturation gradually expanding over the study period with the largest control group of 30 sites. Saturation in Singapore is calculated as the sum of (intervention area * intervention timepoints) / (total area * max timepoints). The study in Singapore featured the most extensive pre-intervention data with 247 to 335 weeks of observation time, and post-intervention data of 109 to 209 weeks.

Table 1 Summary of Wolbachia intervention approaches over the 3 study settings (Malaysia, Brazil, Singapore)

Aggregated intervention efficacies by site and event time

Reductions in dengue incidence were attributed to Wolbachia interventions in all study settings, indicated by positive IEs for methods which passed all robustness checks. For Malaysia, the IE estimates from quasi-experimental methods ranged from 46.73% to 69.19% (Table 2), surpassing the 40.00% (95%CI 5.06–64.59) estimate from the original study [8]. Brazil estimates showed a high degree of efficacy (55.35% to 89.40%), broadly consistent with the original study [14]. However, only SCM and cSCM passed spatial robustness checks in this location. In Singapore, SCM and cSCM passed both placebo checks and these methods respectively produced positive IEs despite incomplete saturation of interventions in intervention sites over the study period, similar to the original study [12]. Estimated IEs across study settings revealed distinct patterns when both in-time and in-space checks were employed.

Table 2 Aggregate intervention efficacy (IE) estimates (%) of Wolbachia releases on total dengue incidence rates across all intervention sites. The estimates are reported for each method and are accompanied by the corresponding confidence intervals (in parentheses). The IE estimate reported in the respective original papers, highlighted in blue, is provided for comparison. Numbers in parenthesis represent lower and upper bounds for 95% confidence intervals, estimated using the bootstrapping procedure [8, 12, 14]

The efficacy of Wolbachia interventions can be mediated by their duration. As quasi-experimental tools can generate counterfactuals for each location and time post-intervention, we re-aggregated IEs by event time relative to intervention start (Table 3). Results indicated that Wolbachia releases were associated with reduced dengue incidence in all locations, but geographic variation in magnitude of efficacy was evident, likely due to local factors such as saturation levels, release strategies and Wolbachia strain employed (Table 3).

Table 3 Intervention efficacy (IE) estimates of Wolbachia releases on total dengue incidence rates across all intervention sites aggregated by event time, with corresponding intervention saturation. For Malaysia and Brazil, saturation is determined by the average percentage of the wAlbB/wMel Wolbachia detected by qPCR in Ae. aegypti mosquitoes across all sites during monitoring phase relative to the start of intervention (introgression). In Singapore, saturation is defined as the area covered by the intervention (IIT-SIT)

Malaysia’s IE estimates were consistently high across methods (Table 3, Supplementary Figure S2a), demonstrating an upward trend over time, suggesting that the intervention is stable and increasingly effective as Wolbachia introgresses into the local Aedes population. Conversely, Brazil showed significant variability and a notable decline in efficacy in later months (Table 3, Supplementary Figure S2b). However, Brazil’s IE estimates lack robustness, failing all temporal checks and passing few spatial checks. In Singapore, IIT-SIT showed reductions in dengue incidence ranging from 35.77–53.11%, using SCM and cSCM (Table 3, Supplementary Figure S2c), stabilising as intervention saturation increases.

This variation in IE reflects the complexity of factors affecting Wolbachia intervention outcomes, highlighting the need to account for local conditions in efficacy assessments.

Empirical validation by simulation study

Table 4 presents the results of the simulation study, displaying the average percentage errors across 1000 simulation runs along with their corresponding standard deviations (s.d.) across methods. Both the Pre-post and RDD methods exhibited underestimation of the true IE. However, the RDD method, with a percentage error of -34.98% (s.d. 23.848), demonstrated the greatest extent of underestimation and spread when compared to other methods.

Table 4 Percentage errors of intervention efficacies for each method compared to the true intervention efficacy (IE). The values in parentheses represent the standard deviations, reflecting the variability observed across 1,000 bootstrapping iterations. These errors were computed using the formula (true IE–- IE from method) / (true IE). Here, ‘true IE’ represents the IE determined by simulating the dengue epidemic cycle with the same initial parameters and comparing scenarios with and without the intervention. For the intervention scenario, the initial Wolbachia-infected mosquito population is set to a random multiplier (1.2 to 2 times) of the wild-type mosquito population. In the non-intervention scenario, the initial Wolbachia mosquito population is set to zero

2 × 2 DiD consistently underestimated the true IE across all control pool sizes, with a percentage error of approximately -2.5% (s.d. 3.39), suggesting a minor bias that remained consistent irrespective of the control pool size.

Meanwhile, SCM and cSCM showed notable improvements in performance as the control pool size increased. SCM’s error decreased from 2.52% (s.d. 3.327) to 0.23% (s.d. 1.370), and its s.d. decreased as the number of controls increased, indicating greater accuracy and consistency with larger control pools. On the other hand, cSCM exhibited a transition from a small positive error of 1.62 (s.d. 2.465) to a small negative error of -0.27 (s.d. 0.572) as the control pool expanded, along with diminishing s.d., suggesting enhanced precision, but a tendency to slightly underestimate the IE with increasing control pool size. cSCM also exhibited the least variability in its estimates across varying control pool sizes, reflected by consistently narrow standard deviations, indicating a high level of precision in estimates. The results also suggest a requisite minimum number of control groups for accurate estimation in SCM and cSCM. With a baseline of 10 control groups, SCM's accuracy improved by a factor of four (2.52 to 0.60). Likewise, cSCM's accuracy saw a tenfold increase (1.62 to -0.15) with at least 5 control groups, but declined when more than 20 groups were used.

Discussion

Dengue necessitates robust vector control strategies. Yet, assessing the effectiveness of these interventions in field conditions is difficult, due to the challenge of undertaking randomised control studies and confounding in non-randomised studies. This study revisits the IEs of three Wolbachia field trials targeting dengue from a systematic search through a variety of commonly-used quasi-experimental methods, all underpinned by stringent validity checks. Despite incomplete intervention saturation over the study period, methods passing these checks confirm Wolbachia’s protective effect against dengue (Tables 2 and 3), aligning with existing literature [8,9,10, 12,13,14,15,16,17, 21].

Comparative analysis reveals how different statistical methods which complied with spatio-temporal placebo checks can yield diverse IE estimates. The Bayesian model from the Malaysia study yielded a conservative IE estimate of 40.00% (95%CI 5.06–64.59) [8] while other methods in this study reported higher efficacies 46.73 – 69.19%. However, a revised Bayesian analysis based on a longer timespan and additional study sites were concordant with our higher estimates of intervention efficacy (IE: 62.4% (95%CI: 50.00–71.00)) [9]. In Singapore, SCM and cSCM derived IEs (IE: 48.71–53.33%) were in close agreement with the original study's 56.88% (95%CI 51.88–58.46) [12], with the application of similar methodologies and data, confirming the intervention’s effectiveness even with partial Wolbachia saturation. These consistent results, validated by robust placebo checks, highlight how the choice of analytical method can significantly impact intervention assessments.

The Brazil study's application of Interrupted Time Series Analysis (ITSA), operationally mirroring 2 × 2 DiD, resulted in a higher reported IE of 69.40 (95%CI: 54.40–79.40) [14] compared to 66.59% (95%CI: 60.18–70.83) and 55.35% (95%CI 52.16–65.95) from our use of SCM and cSCM respectively. Discrepancies in IE estimates may arise from different methodological approaches. Firstly, SCM and cSCM have theoretically enhanced accuracy over DiD through better balancing of pre-intervention historical outcomes compared to DiD, thus accounting for unobserved confounders. Additionally, due to absence of population data at the neighbourhood level, the granularity of data differs; our study’s neighbourhood-level analysis with min–max normalisation to adjust for population sizes and reporting disparities contrasts with the original study’s zone-level approach, in which the authors could directly account for population. Therefore, interpreting IE differences between studies requires concomitant consideration of each analytical approach's unique assumptions and constraints.

Aggregating IEs by the period post-intervention (Table 3, Supplementary Figure S2), our analysis facilitates equitable comparison of intervention impacts across sites, currently unexamined in existing literature, illuminating the dose–response relationship of Wolbachia interventions. The IEs in Malaysia displayed an upward trend with longer intervention periods, indicating that sustained Wolbachia presence may lead to more significant reductions in dengue incidence. This observation indicates the potential long-term benefits of sustained Wolbachia releases until Wolbachia frequencies reach a high and stable level without further releases, and is also consistent with the recent Bayesian analysis completed on additional Malaysian data [9]. In Singapore, IIT-SIT first showed a moderate efficacy, which may be due to wildtype vector declines as Wolbachia-infected males began reducing viable mosquito offspring. The dip in mid-term efficacy likely reflected interepidemic dengue transmission or ecological adjustments to the new intervention–emphasising the need for persistent releases to maintain pressure on the mosquito population. The recovery and eventual stabilisation of IE in the later months signifies that the consistent release of Wolbachia-infected males leads to a progressive and sustained reduction in the mosquito population despite incomplete saturation across the study period. In Brazil, a decline in IE over time pointed to challenges in maintaining the intervention's effectiveness. Varied factors affect the wMel strain’s homogenous spread within mosquito populations, including persistent wild-type eggs, urban infrastructure barriers, and disruptions like the COVID-19 pandemic impeding mosquito dispersal [13, 14]. The Brazil dataset, inclusive of both suspected and confirmed dengue cases, confronts limitations in the specificity and sensitivity of case reporting, wherein potential misclassified or unreported cases could skew the assessment of Wolbachia’s efficacy. Declines in maternal transmission of Wolbachia at higher temperatures may also account for decreased efficacy, especially in Brazil's dense urban environment with its variable microclimates [24, 25]. The need for adaptive management of Wolbachia interventions, as emphasised in public health discussions [26], is critical to respond to these environmental and biological challenges to maintain the success of the program.

Our study introduces a novel empirical validation of quasi-experimental methods using the ASEI-SEIR model to simulate dengue dynamics with Wolbachia interventions, addressing the previously unexplored efficacy of such methods in recovering IEs in vector-borne disease transmission and interventions. Consistently minimal errors affirm SCM and cSCM’s exceptional empirical precision and accuracy in recovering IEs, outperforming other methods and demonstrating their reliability across different control configurations (Table 4). Conversely, RDD showed a notable negative bias, and the Pre-post and 2 × 2 DiD methods, despite relatively modest errors, did not match SCM and cSCM’s performance. Although parallel trends were validated in the Brazil study's DiD analysis, our simulations point out that 2 × 2 DiD's sensitivity to unaccounted factors can lead to erroneous estimations of IE, even with historically consistent parallel trends. This illustrates the advantage of SCM and cSCM in adjusting for time-varying confounding factors, offering a reduction in bias within quasi-experimental settings, given an adequate pre-intervention duration [27].

In public health research, the choice between cRCTs and quasi-experimental designs hinges on the study’s resources, objectives, and constraints. cRCTs may offer high validity but face logistical and ethical issues in outbreak situations. Our study illustrates that quasi-experimental methods can provide comparable validity with thorough checks, yet require cautious interpretation due to potential biases from non-randomisation. In the simple one-group Pre-post design, researchers must adjust for confounding factors, including disease incidence fluctuations (regression towards the mean) and behavioural changes (maturation), to attribute effects to the intervention, given that this method cannot infer causality on its own due to the lack of a control group [28]. For SCM and cSCM to produce reliable counterfactuals, certain conditions should be met. While pre-intervention data directly before the intervention reflects the most recent trends, a longer pre-intervention period better captures historical changes and strengthens counterfactuals [27, 29]. While a larger donor pool offers more control options for weight selection, it is essential to select well-matched controls to minimise over-fitting and interpolation biases [30]. A well-matched control unit is characterised by its ability to closely mimics the outcome trajectory of the treated unit over an extended pre-intervention period [30]. Valid causal inference with 2 × 2 DiD requires confirmed parallel trends [31]. RDD's effectiveness is bandwidth-dependent, requiring the balance between breadth of data against the risk of bias [32, 33]. Lastly, traditional quasi-experimental methods approximate treatment effects as binary step functions, lacking the ability to account for the increasing magnitude of treatment effects over time. This approach fails to capture the gradual and cumulative impacts of interventions like Wolbachia, where small levels of saturation can significantly impact over time. Our time-aggregated analysis (Table 3) demonstrates how variations in treatment saturation correlate with changes in intervention efficacy providing a more accurate understanding of the intervention's incremental effects. These findings highlight that when applied rigorously, quasi-experimental methods extend the toolkit for reliable causal inferences in public health interventions where cRCTs are impractical.

Our study has several strengths. (1) We systematically reviewed publicly available data on Wolbachia field trials to minimise selection bias of included studies for re-evaluation of IEs. (2) We employed a comprehensive suite of quasi-experimental methods, accompanied by extensive robustness checks, to confidently demonstrate Wolbachia's consistent protective effect against dengue. (3) We used a novel data generating process using a custom-built transmission dynamic model to validate the efficacy of these quasi-experimental methods in estimating true IEs. However, there are limitations. The study did not account for spillover effects, possibly underestimating Wolbachia's efficacy by diluting the observed differential in disease incidence between intervention and control sites. Subsequent research could use spatial analysis to account for indirect protection effects due to the migration of Wolbachia-infected mosquitoes into control areas. Moreover, data constraints precluded the inclusion of relevant covariates and demographic subgroup analyses. Future studies with richer datasets could perform more granular analyses using spatial interpolation to better understand Wolbachia’s differential impact across population strata and control for confounders. Lastly, our study focused on a single-serotype, single-season simulation to clarify IE recovery in simplified dengue dynamics. Multi-season, multi-serotype expansions are recommended for future work to triangulate long-term efficacy while encompassing the interplay of cross-immunity to antibody-dependent enhancement.

Conclusion

In conclusion, our study rigorously re-evaluated the impact of Wolbachia-based vector control interventions on dengue incidence across three locations using quasi-experimental methods. The findings revealed a consistent protective effect against dengue, particularly in analyses supported by robust internal validity checks. SCM and cSCM stand out for their precision and validity in estimating IEs in both real-world and simulated settings, displaying promise for guiding public health strategies where randomised controlled trials are not always feasible.