Skip to main content

Advertisement

Download PDF

Structuring and validating a cost-effectiveness model of primary asthma prevention amongst children

BMC Medical Research Methodology volume 11, Article number: 150 (2011) Cite this article

Abstract

Background

Given the rising number of asthma cases and the increasing costs of health care, prevention may be the best cure. Decisions regarding the implementation of prevention programmes in general and choosing between unifaceted and multifaceted strategies in particular are urgently needed. Existing trials on the primary prevention of asthma are, however, insufficient on their own to inform the decision of stakeholders regarding the cost-effectiveness of such prevention strategies. Decision analytic modelling synthesises available data for the cost-effectiveness evaluation of strategies in an explicit manner. Published reports on model development should provide the detail and transparency required to increase the acceptability of cost-effectiveness modelling. But, detail on the explicit steps and the involvement of experts in structuring a model is often unevenly reported. In this paper, we describe a procedure to structure and validate a model for the primary prevention of asthma in children.

Methods

An expert panel was convened for round-table discussions to frame the cost-effectiveness research question and to select and structure a model. The model's structural validity, which indicates how well a model reflects the reality, was determined through descriptive and parallel validation. Descriptive validation was performed with the experts. Parallel validation qualitatively compared similarity between other published models with different decision problems.

Results

The multidisciplinary input of experts helped to develop a decision-tree structure which compares the current situation with screening and prevention. The prevention was further divided between multifaceted and unifaceted approaches to analyse the differences. The clinical outcome was diagnosis of asthma. No similar model was found in the literature discussing the same decision problem. Structural validity in terms of descriptive validity was achieved with the experts and was supported by parallel validation.

Conclusions

A decision-tree model developed with experts in round-table discussions benefits from a systematic and transparent approach and the multidisciplinary contributions of the experts. Parallel validation provides a feasible alternative to validating novel models. The process of structuring and validating a model presented in this paper could be a useful guide to increase transparency, credibility, and acceptability of (future, novel) models when experts are involved.

Peer Review reports

Background

Asthma affects more people [1, 2] and costs more to manage than ever before [3–6]. Given limited resources for competing needs, prevention of asthma could pay off. Currently, primary prevention programmes on asthma generally differ in the approach to reduce or avoid exposure to one (unifaceted, e.g. an anti-house dust mite [HDM] measure) or more environmental factors (multifaceted, e.g. exclusive breastfeeding until six months and an anti-HDM measure) [7].

The decision to ultimately implement asthma prevention programmes, let alone choose between a unifaceted and a multifaceted approach, is not made easier with favourable [8, 9] and unfavourable [10, 11] reports on the effectiveness of prevention. Differences in the conclusions could be due to factors such as the essential difference between the approaches (unifaceted versus multifaceted), the population (open versus captive), "room for improvement" (pre-existing high-levels of allergen avoidance or adapted smoking behaviour), and even the selected outcome measures or (asthma) definitions (e.g. asthma-related symptoms, physician-diagnosis). However, the meta-analyses by van Schayck and colleagues [12, 13] show that multifaceted programmes significantly reduce the risk for asthma compared to unifaceted programmes.

Although several studies have been performed on primary prevention of asthma amongst children, none could singly determine the cost-effectiveness of primary prevention [8–11, 14–18]. Specifically, there is no known study that directly compares the various alternatives in primary asthma prevention [12, 13]. Ideally, it could be studied in a clinical trial or, alternatively, using a model which synthesises the evidence from all available sources and levels of evidentiary quality [19, 20]. Such a model could estimate the uncertainty of decisions and cost associated with asthma given imperfect, current data in an explicit and transparent manner [19].

A permutation of models is possible depending on various considerations but, despite the pervasive use of models in economic evaluation, there is still wariness about their appropriateness and validity [19, 21]. However, movement towards uniformity through good practice guidelines [21–24] have helped to allay such wariness. Still, there is a need for more transparency and explicitness in modelling practice [25]. Already, this is being met in clinical effectiveness research by the increasing practice of reporting study protocols separate from the trial results [26, 27], which reduces publication bias and increases transparency and credibility. In this paper, we describe the design and validation of the structure of a decision analytic model which compares multifaceted and unifaceted programmes of primary prevention of asthma in children versus the current situation (i.e. unstructured advice on avoiding allergen sensitization and symptomatic treatment).

Methods

Expert panel sessions and model structuring

Consultation with experts, just as personal experience or a review of the literature, may provide the background information needed when developing a model [28]. For this paper, experts were consulted to provide biomedical insight into the natural course of asthma and its care in the Netherlands. We developed a protocol (Additional File 1) on conducting the expert panel held as round-table sessions [29]. The protocol identified relevant inputs and expected outcomes of each session, including the guiding questions to structure the discussions. Following a review of relevant health technology assessment (HTA) literature [28–30], the protocol consists of the following elements:

  1. 1.

    Description of the study's background, motivation, and objectives including the role of experts in the modelling process;

  2. 2.

    Identification of criteria for selecting experts;

  3. 3.

    Identification and inclusion of experts;

  4. 4.

    Identification of the number and purpose of the expert sessions;

  5. 5.

    Description of the structure of the sessions in terms of knowledge, process events, and expected session outcomes;

  6. 6.

    Description of the techniques to referee between experts, including the feedback communication with the experts; and

  7. 7.

    Documentation of the results.

The foremost criterion for expert selection was the extent of their expertise and experience with asthma and its prevention in children and/or research experience in the field of efficacy and cost-effectiveness [29]. This was discussed within the research group which resulted in nominations for candidate experts. Once identified and contacted, the experts' willingness to participate and ability to attend the sessions [29] were also considered. The experts received copies of the protocol for their guidance to the process. Initially, only two sessions were planned but because of logistical issues as well as significant changes in the model structure after consulting the research group, another session was added. Consensus was not formalised as strict unanimity on discussion points because of the explorative nature of the round-table discussions [29].

The expert panel sessions were conducted in 2010 at Maastricht University, the Netherlands. Eight expert nominees were contacted but the final combination of experts were two from general practice and (paediatric) pulmonology and one each from epidemiology and HTA. During the first session, the HTA expert joined in by telephone and thereafter kept contact through e-mail. Because of unavailability, one pulmonologist and one general practitioner (GP) dropped out after the first and second sessions, respectively.

The first session was meant to frame the decision problem and generate assumptions needed to determine the structure of the model. Input to this session was an overview of the research line as background for the decision problem and purpose of the model. The experts were asked about other aspects of the decision framework covering the natural history of the disease, gold standards of measure, possible interaction and recursion of disease states, as well as appropriate comparators. Since time horizon, population, and perspective were already naturally defined by the existing clinical studies from the PREVention of ASthma amongst Children (PREVASC) research line [14–18], these elements were not solicited from the experts. The first session resulted in a set of assumptions and a decision framework.

After the first expert panel session, the literature was systematically searched for existing models that could be adapted for the current decision problem [31]. When a new model was required, the type was chosen using guidelines and recommendations in the literature [21–24]. Consideration was given to the purpose (single-use vs. programmatic), level of simulation (individual vs. cohort), relationship between individuals (independent vs. interacting), and frequency of outcomes/health states (recursive vs. non-recursive). The model structure was drawn based on the type of model, the decision framework and assumptions, and published recommendations [32].

The last sessions were meant to refine the model structure and determine its descriptive validity. For the second session, the input was an overview of the modelling process and the general types of models, to provide some information to the experts with limited exposure to economic modelling. It was only given at that time so that the generation of assumptions and the framing of the decision problem during the first session would not be restricted by bias for one model or another. During this second session, the model structure was scrutinised on how well it reflected the set of assumptions and decision framework from the first expert panel session. The outcome at this stage was a list of comments for improving the correspondence of the model structure with the set of assumptions and decision framework. On the third and last expert panel session, the input was the latest model structure which was again evaluated. After satisfying the descriptive validity, the model was finalised.

Validating the model

Descriptive validity (i.e. whether the essential characteristics of reality is captured in the model [33–35]) was determined during the second and third expert panel sessions. This validation was carried out with the experts' review of how the model structure was able to incorporate the assumptions about the development and current care of the disease into a simplified structure. Specifically, it was determined during the sessions whether the model was parsimonious and sufficient to answer the decision problem [36]. This was an iterative process with the experts after the initial session.

In the absence of comparable models that evaluate allergen-avoidance measures in the primary prevention of asthma in children, we focused instead on the parallelisms (i.e. similarities) between characteristics of the model structure, namely: model type, time horizon, perspective, population, intervention, comparator, and health states/effect measures [37]. The other characteristics are related to the natural history of a disease and its treatment which are currently important determinants in the chain of choices that lead to the selection of the model structure [37]. Aside from including studies on the secondary and tertiary prevention of asthma (i.e. preventing allergic sensitisation from progressing to asthma or preventing the exacerbations of established asthma [1]) we proceeded with this "parallel validation" by increasing the scope to studies on lower respiratory tract infections. These infections in infants, particularly with the respiratory syncytial virus, have been associated with the development of asthma later on [38]. To test for the parallelisms, we first performed a systematic search of the literature to identify the relevant studies. The search was conducted in the PubMed databases based on an extensive search strategy for economic evaluations [39, 40]. The database was searched until 08 September 2010 and was restricted to English-language records of titles and abstracts of evaluations that used decision-analytic modelling. Only evaluations of at least two alternatives on asthma prevention which relate costs to consequences [41] were included in the review. Following the screening process (Figure 1), the results were reviewed by one of the co-authors and any uncertainty was resolved by direct discussion. Technical or internal validity of the calculations and consistency of formulae and parameter inputs with the assumptions were checked.

Figure 1
figure1

Selection of economic evaluations for the modified convergent validation. This flow chart shows the process of selecting the studies which would be included in the analysis for parallel validation.

Full size image

Model operation

Once the decision problem was framed and the model's structure visualised, its operational feasibility or technical validity [35] was tested by populating it with data from the PREVASC studies. Detailed description of the PREVASC programme (Figure 2) can be found elsewhere [14–18]. Children were included in the studies based on risk for first-degree familial asthma history (i.e. positive or negative, PFH or NFH respectively) [14, 15]. Phase 1 and 2 studies investigated the clinical effects of multifaceted primary asthma prevention at age two and six years, respectively. It was found that over the short or long term, there were no significant reductions of asthma diagnosis between intervention and control groups [16]. However, a subgroup of the PREVASC children seemed to benefit from the prevention programme [14]. Phase 3 was a natural history study which found that there were more respiratory-tract morbidities and atopy in PFH than NFH children in the first two years of life [19, 20].

Figure 2
figure2

The PREVASC Research Line. Schematic flow of the participants in the different studies of the PREVASC research line beginning with the risk-screening of the sample population indicates that data synthesis through modelling is necessary to determine the value of multifaceted primary asthma prevention amongst children. Measurement around the age of six for subjects with negative history is part of the current (Phase 4) study.

Full size image

Branch probabilities were calculated as the ratio of one situation over its sum with the complement possibilities. A diagnostic algorithm (Table 1) was applied to determine asthma status based on inhaled corticosteroid (ICS) use, wheezing three months prior to the lung function measurements, and the measurements on reversibility and hyperreactivity. Because the original PREVASC clinical studies did not explicitly study the effects of unifaceted prevention, this had to be derived from the clinical data. A per-protocol analysis of compliance was used to reclassify PFH participants into multifaceted, unifaceted, or control. The current situation was a combination of the PFH control and NFH participants weighted by the respective ratios of 20% and 80% found in the general population [42].

Table 1 Algorithm for objective asthma diagnosis of children
Full size table

Direct medical as well as intervention costs were assessed and valued using standardised unit prices for the Netherlands indexed using 2009 figures [43]. These direct medical costs include GP and specialist consultations for asthma-related complaints, any related hospital admissions, the diagnostic tests (i.e. chest x-ray and radioimmunoassay test for allergens), and medication use. For those who received intervention, costs on research nurse visits, the use of house dust mite-impermeable materials for the beds of participants and parents, hypoallergenic formula feeding, and education materials in the form of brochures were included. The model outcome was the incremental cost-effectiveness ratio which was assessed for various threshold values (€0 - €70,000 per asthma case avoided).

Results

Expert panel and sessions

Clinical insight into the natural course of asthma, diagnosis, as well as current care and alternatives were provided by experts from the first three fields. The HTA expert provided insight into important economic considerations such as how costs and consequences can be incorporated in the comparison of alternatives. The involvement of the experts in the development of the model structure is illustrated in Figure 3.

Figure 3
figure3

Expert input in the model development. Flowchart showing the influence of experts regarding the design and validation of the structure of the model to determine the economic value of primary prevention in asthma.

Full size image

Decision framework and assumptions for the model structure

Discussion amongst the experts resulted in a decision framework (Table 2) that determined the most appropriate model. This framework was incumbent on a number of expert-derived assumptions largely drawn from the biomedical model of asthma. The experts considered asthma to be more than the sum of genetic expression and environmental influences that might be best addressed by multifaceted programmes. They estimated clinicians would see a larger number of allergic children with transient wheezing than the classic asthma presentation of persistent wheezing combined with shortness of breath and (nocturnal) coughing. Most asthma cases that would be seen, they thought, would be the allergic variant typically seen in children exposed to common allergens for the first time. The experts argued that the processes that lead to asthma may have begun with allergic sensitisation in utero. They cited that the current paradigm of asthma prevention in the Netherlands consists of unstructured pre- and post-natal advice to mothers and their families regarding exposing the child to environmental tobacco smoke and air-borne particulates for children in general and pharmacotherapy for those with symptoms. The experts assumed that any alternative intervention must also be instituted during gestation and after birth to prevent asthma. For any prevention programme, a reasonable screening strategy is necessary to identify the recipients who are most at risk and stand to gain the most from the programme. It was thought that this would be best done through first-degree familial asthma history.

Table 2 Decision problem framework for the primary prevention of asthma in children
Full size table

Although differentiating between phenotypes was recognised as invaluable in refining asthma diagnosis, it was contended that the diagnosis of asthma alone was sufficient and necessary to determine the value of primary prevention at the health care level. Even if diagnosis were to be established at age six years, it was assumed that the pathological processes may have begun long before diagnosis. Furthermore, these processes may continue even in the absence of clinical signs and symptoms at later years. However, they noted that diagnosis of asthma prior to age six years is difficult or changes over time. Patients would be seen when the flare of symptoms as well as the pathologic process are subsiding which may not give a very accurate picture of the disease. At a later age, clinical remission may occur during pubescence or it may recur during the adult years. The difficulty, they said, is compounded by a lack of gold standard for its diagnosis. Around age six years, large groups of children who may have asthma are symptom-free but physiologic changes such as exaggerated inflammatory reactions and airway remodelling may already be occurring which could be detected using objective measurements. According to the experts, bronchioalveolar lavage and biopsies are the most reliable tests for airway inflammation. But since the procedures are also invasive and difficult to perform in practice, those were considered primarily research more than routine clinical tools. In contrast, the experts said that spirometry is common in the general practice setting and suitable in documenting abnormal lung function described by reversibility and hyperresponsiveness of the bronchial tubes. Combining objective lung function tests with symptomatology seemed to them to be the best approach to asthma diagnosis to date. However, they said that since symptomatic children in the Netherlands receive asthma medication, particularly ICS, an objective diagnosis at age six years may be contaminated. It was then assumed that children who received ICS therapy three months prior to the tests are asthmatic when either test of reversibility or hyperreactivity is positive. When both tests are negative, diagnosis could go either way. Amongst children who did not receive ICS therapy, positive hyperreactivity was assumed to signify presence of asthma and negative hyperreactivity meant absence of asthma. It was also assumed that evaluating the costs and consequences over a time horizon of six years after birth would be sufficient since this is the age when lung function tests would be first reliable in children.

Model structure

A search of the literature for economic models of primary asthma prevention in children based on the expert-developed decision framework came up empty-handed (Table 3). This meant that a new model (Figure 4) had to be developed which began with the problem or situation of administering to children a primary prevention programme on asthma. Based on the expert sessions, it was determined that the decision tree structure suited the decision framework and assumptions. Identifying the target population for the intervention on the basis of risk became the first node for the intervention arm of the decision tree. It was assumed that NFH children would receive current care [44] whereas PFH children would be given the intervention either as a unifaceted or multifaceted prevention depicted by the second decision node coming from the PFH branch. A child would receive unifaceted prevention when only one measure against air- or food-borne allergens is employed and multifaceted when at least two measures against both triggers are used. It was also assumed that children could not shift between these two prevention categories. The model resulted in three situations of care: prevention with unifaceted or multifaceted approach and current situation care. For these situations, children were assumed to either remain healthy or manifest some asthma signs and symptoms. Consequent upon the clinical picture, the children may or may not receive pharmacotherapy as required. The primary outcome is either a positive or a negative objective diagnosis of asthma, which relies on reversibility and hyperreactivity tests and symptomatology.

Table 3 Results for the systematic review for decision analytic models on asthma prevention
Full size table
Figure 4
figure4

The PREVASC Decision Tree. Structure of the decision analytic model to assess the cost-effectiveness of primary prevention of asthma. The tree compares current situation without primary prevention with the alternative of primary prevention of asthma amongst children (first decision node) as well as whether multifaceted or unifaceted primary prevention approaches is more cost-effective (second decision node).

Full size image

Validation of the model structure

The model structure was revised and refined in the last two sessions with the experts who were asked to consider the intuitiveness of the model structure in reflecting the decision framework and the assumptions. The review was at the same time a process of descriptively validating the model structure. Concerning the parallel validity with models in the literature, the strategy and resulting hits are shown in Table 3. In reviewing the titles and abstracts, none of the 406 search results referred to a trial-based economic evaluation of primary or secondary asthma prevention in children which used a non-pharmacological approach. When the survey focused on models, there were 62 hits, of which only seven were studies on the pharmacological prevention of asthma or lower respiratory tract infection in children. After a critical appraisal, two studies were further excluded because they did not use decision analytic models in the economic evaluation. Of the remaining five studies (Table 4), three [45–47] used decision-tree models and two [48, 49] used Markov models. Two studies focused on asthma prevention [46, 49] and others on RSV infection [45, 47, 48]. Omnes and colleagues [46] reported a model which has the most characteristics similar to ours in terms of model structure, time horizon, perspective, population, comparator, and effects. The intervention was, however, not a perfect correspondence which is not surprising given the lack of comparable models with our decision framework. In short, their study supports the structural validity of our model and decision problem's most salient concerns and so validates in a way the structure of the model presented here through parallelisms.

Table 4 Important characteristics of studies for parallel validation
Full size table

Model operation

For this demonstration of the model's operation and internal validity, we have used solely the data from the PREVASC studies. Only cases with relevant diagnostic data were included in the modelling which whittled down the possible 443 PFH cases in the intervention and control group to 324 cases. Their compliance with at least two, one, or none of the measures against exposure to airborne and food allergens was the basis for respectively regrouping them into multifaceted (N = 259), unifaceted (N = 53), and control (N = 12). Branch probabilities were calculated and folded back for the expected outcomes and costs. Between the prevention approaches, the multifaceted had a higher costs but also a higher expected probability of asthma cases avoided than the unifaceted approach (€539.49 vs. €445.19 for costs and 0.93191 vs. 0.92610 for probability of asthma avoidance, respectively). This means that the multifaceted approach costs around €16,000 more to avoid one asthma case than in the unifaceted approach. However, when compared with the current situation, which has the lowest cost and highest probability (€302.30 and 0.96494), the two prevention approaches were strongly dominated. Even for various cost-effectiveness thresholds, the current situation dominated the two approaches.

Discussion

The model structure

Use of models in health economic evaluation has increased over the years and most often for one of two things [50]. First, when the relevant clinical trial has not (yet) been conducted or economic data is missing or incomplete. Second, when the clinical trial used intermediate outcomes or when the follow-up period is short whilst the decision problem is concerned with endpoints long into the future. Though modelling allows decision makers to be critical and realistic about alternatives, to be conscious and clear of valorisation, and to reveal the relationships between inputs and outputs [51], models can seem like "black boxes" [33, 50] to consumers unless steps are made explicit and transparent. This was the rationale for explicitly reporting the steps we took in structuring our model.

In the primary prevention of asthma in children, several studies [8–11, 14–18] have provided information on effectiveness but none could singly determine whether primary prevention of asthma is cost-effective and whether a unifaceted or a multifaceted approach is better. Like others [52, 53], we have found that the literature is surprisingly silent regarding health economic studies on the value of health care strategies other than those concerned with asthma control. Consequently, one may cast a wary eye on the policy and care decisions relating to the prevention of asthma in children and questions whether the decisions were adequately informed. By synthesising the best available evidence through this model, we hope to take the first step toward properly evaluating and substantiating the choice amongst primary asthma prevention alternatives.

A number of structures could have been used in developing an economic model from regression equations to system dynamic models [21–24]. Some of the more common structures include decision trees, Markov-types, and discrete event simulations (DES) [21–24]. It has always been advocated that the choice of a model structure should be on the consideration of the simplest, and thus best, fit [34] between three elements: the decision problem, natural history of the disease, and current paradigms of care [33]. Since Markov-type models are used in decision problems with time-dependent or recurrent situations [21–24], these were considered inappropriate for this study primarily because asthma was assumed to be a stable diagnosis. Although asthma is understood to be a chronic condition, we considered that its diagnosis would not change in the six-year time frame of our study. This consideration is supported by findings that physiologic changes occur even for so-called mild cases who are more likely to be asymptomatic or only rarely symptomatic [54]. The various wheezing phenotypes could have been modelled instead as the recurring disease states of asthma because of their strong relationship with each other. Despite this, the experts did not think that wheezing was strictly causative of asthma. DES structures, which are more detailed and complicated extensions of Markov-type models [21–24], are similarly not appropriate for this study because asthma does not have a universally accepted measure of severity particularly in population-based studies [55]. Also, the choice for a more complex model should be balanced by any change to the results compared to use of a simpler model [33]. For these reasons, in combination with the decision framework, a decision tree structure was chosen and developed for our specific research question that aims to assess the cost-effectiveness of a programme to prevent the presence of asthma in children around age six years.

The current cost-effectiveness analysis limited itself to the use of the clinical data from the PREVASC studies to demonstrate the model's operation and evaluate its internal validity. The results of this demonstration showed that based on the probability of developing asthma, the current situation dominated either approach to primary prevention of asthma in children. It is notable though that a multifaceted approach was better than a unifaceted one. Use of the PREVASC data enabled technical validation by debugging and refining the model to handle problems like empty cells. For this feasibility assessment of the model, it should be noted that originally, there was no distinction made between the unifaceted and multifaceted groups in the PREVASC studies. The distinction was made only between control and intervention.

Value of expert panels

Experts are frequently involved in modelling studies to provide the initial model structure, validate the final structure, estimate values of transition probabilities, usage, and cost units [29]. Their reported involvement is, however, uneven [25]. Many are unclear in their reports even of the extent of expert participation in the process. In this paper, we have presented a detailed account of the explicit inputs from experts and their role in structuring a model. We have taken a naïve approach to reduce the bias of the modeller in the selection of one model structure over another but more importantly to benefit from the multi-disciplinary interaction between experts. The round-table discussions provided the opportunity for a dynamic engagement of the experts critical to framing the decision problem. Through the multidisciplinary experts, we hope to have gained better insight into daily practice which is not always possible through systematic reviews alone. All of these should contribute to the improvement of the descriptive and eventual face validity of the model's structure through the iterative reflection on the elicited assumptions. An important consideration is the selection of experts since the breadth and depth of their (practical) knowledge would determine the richness of the endeavour. This could be aided by a brief review of the literature to identify potential experts who are often cited.

Eliciting information and feedback from a panel of experts may range from informal consultations, to round-table discussions, and then Delphi approaches [25, 29]. The first approach simply and usually provides background information which may or may not be used. It is typically unilateral and high-handed. Round-table discussions are usually semi-structured sessions where participants are less restricted to explore issues and arguments without necessarily concluding in a consensus which is in contrast to the Delphi approaches. Although Delphi approaches may be seen as the most "democratic" approach to elicit expert opinion because of the intermediary, it is also a handicap since some element of exploration is lost as statements or opinions are "edited" by the intermediary. In round-table discussions, model developers may benefit from the dynamic and personal interaction with and amongst the experts so that no significantly relevant opinion, alternative, or scenario is left out of the model. There is, however, a caveat to round-table discussions: more dominant experts may "drown" the opinions of other experts so the modeller must be able to deftly control the panel. We have tried to address this partly through the imposition of a time limit for the discussions. Also, contributions of each expert were actively sought.

Parallel validation

Given that novel models do not have comparable models on the same decision problem, the standard validation techniques on the face, convergent, and predictive attributes of this model could not be ascertained at this stage of our study. The comparison of the outputs between the developed model and other models on the same decision problem [33–35] usually covered in the discussion section of a scientific report [56] hints at the prevalent approach to validate novel models. Indeed, validation of novel models "is superseded by the concept of model credibility" particularly because complete validation of a model is not possible [57]. Face validity through expert opinion has been suggested as an approach to validate the structure of novel models. Another solution that we propose and have described here may be through drawing parallelisms with other models, which is to perform what we refer as "parallel validation". Similar to convergent validation in testing between model corroboration [33–35], the distinguishing feature of parallel validation stems from the absence of a model built around the same decision problem. The proposed validation compares other models with the same structure but concern different diseases, populations, or interventions. This is therefore a qualitative consideration of the parallelisms from one model to another. Structural comparisons are admittedly more difficult than comparing input parameters [58]. Nevertheless, decision makers still need to make judicious choices with the highest possible level of confidence. Existing standards on modelling and reporting have been used to serve as markers of similarity and dissimilarity [59]. Figuratively, one can speak of models on the same decision problems as coming from one nuclear family. Parallel validation may be a viable way to compare the structure of one model to a model from a different but closely related family, even if only qualitatively. How modest the literature on (cost-) effectiveness of pharmacotherapies may be, there is even less on prevention [52–54]. Parallel validation reveals that more needs to be done to increase the literature on economic evaluation of primary prevention of asthma amongst children.

Conclusions

In this paper we showed that a decision-tree structure developed with experts in round-table discussions is descriptively valid in reflecting reality. In addition it is able to incorporate the probable scenarios that would help determine the value of primary asthma prevention. By explicitly detailing the steps in the development of a model from initial problem to final structure, the transparency and use of models for economic evaluation may increase and enrich the toolbox of decision-makers who need to take informed choices. Future studies could determine the number and method of panel sessions that would increase a model's structural validity.

References

  1. 1.

    Johansson SGO, Haahtela T: Introduction. Prevention of Allergy and Allergic Asthma: World Allergy Organization Project Report and Guidelines. Edited by: Johansson SGO, Haahtela T. 2004, Basel, CH: Karger, XIII-XIX. [Ring J, Adorini L, Berek C, Blaser K, Capron M, Denburg JA, Holgate ST, Marone G, Saito H (Series Editors): Chemical Immunology and Allergy, vol. 84.]

  2. 2.

    Peters SP, Ferguson G, Deniz Y, Reisner C: Uncontrolled asthma: a review of the prevalence, disease burden and options for treatment. Respir Med. 2006, 100: 1139-1151. 10.1016/j.rmed.2006.03.031.

  3. 3.

    Lane S, Molina J, Plusa T: An international observational prospective study to determine the Cost of Asthma eXacerbations (COAX). Respir Med. 2006, 100: 434-450. 10.1016/j.rmed.2005.06.012.

  4. 4.

    Bahadori K, Doyle-Waters MM, Marra C, Lynd L, Alasaly K, Swiston J, FitzGerald JM: Economic burden of asthma: a systematic review. BMC Pulm Med. 2009, 9: 24-10.1186/1471-2466-9-24.

  5. 5.

    Williams AE, Lloyd AC, Watson L, Rabe KF: Cost of scheduled and unscheduled asthma management in seven European Union countries. Eur Respir Rev. 2006, 15: 4-9. 10.1183/09059180.06.00009801.

  6. 6.

    Sculpher MJ, Price M: Measuring costs and consequences in economic evaluation in asthma. Respir Med. 2003, 97: 508-520. 10.1053/rmed.2002.1474.

  7. 7.

    Arshad SH: Primary prevention of asthma and allergy. J Allergy Clin Immunol. 2005, 116: 3-14. 10.1016/j.jaci.2005.03.043.

  8. 8.

    Arshad SH, Bateman B, Matthews SM: Primary prevention of asthma and atopy during childhood by allergen avoidance in infancy: a randomised controlled study. Thorax. 2003, 58: 489-493. 10.1136/thorax.58.6.489.

  9. 9.

    Chan-Yeung M, Watson W, Ferguson A, Dimich-Ward H, Rousseau R, Lilley M, DyBuncio A, Becker A: The Canadian Childhood Asthma Primary Prevention Study: outcomes at 7 years of age. J Allergy Clin Immunol. 2005, 116: 49-55. 10.1016/j.jaci.2005.03.029.

  10. 10.

    Brunekreef B, Smit J, de Jongste J, Neijens H, Gerritsen J, Postma D, Aalberse R, Koopman L, Kerkhof M, Wijga A, van Strien R: The prevention and incidence of asthma and mite allergy (PIAMA) birth cohort study: design and first results. Pediatr Allergy Immunol. 2002, 13 (Suppl 15): 55-60.

  11. 11.

    Marks GB, Mihrshahi S, Kemp AS, Tovey ER, Webb K, Almqvist C, Ampon RD, Crisafulli C, Belousova EG, Mellis CM, Peat JK, Leeder SR: Prevention of asthma during first 5 years of life: a randomized controlled trial. J Allergy Clin Immunol. 2006, 118: 53-61. 10.1016/j.jaci.2006.04.004.

  12. 12.

    van Schayck OCP, Maas T, Kaper J, Knottnerus AJA, Sheikh A: Is there any role for allergen avoidance in the primary prevention of childhood asthma?. J Allergy Clin Immunol. 2007, 119: 1323-1328. 10.1016/j.jaci.2007.02.024.

  13. 13.

    Maas T, Kaper J, Sheikh A, Knottnerus JA, Wesseling G, Dompeling E, Muris JW, van Schayck CP: Mono and multifaceted allergen reduction interventions for preventing asthma in children at high risk of developing asthma. Cochrane Database Syst Rev. 2009, 3: CD006480-

  14. 14.

    Schönberger HJAM, Dompeling E, Knottnerus JA, Maas T, Muris JWM, van Weel C, van Schayck CP: The PREVASC study: the clinical effect of a multifaceted educational intervention to prevent childhood asthma. Eur Respir J. 2005, 25: 660-670. 10.1183/09031936.05.00067704.

  15. 15.

    Kuiper S, Maas T, van Schayck CP, Muris JW, Schönberger HJ, Dompeling E, Gijsbers B, van Weel C, Knottnerus JA, PREVASC Group: The primary prevention of asthma in children study: design of a multifaceted prevention program. Pediatr Allergy Immunol. 2005, 16: 321-331. 10.1111/j.1399-3038.2005.00278.x.

  16. 16.

    Maas T, Dompeling E, Muris J, Wesseling , Knottnerus A, van Schayck O: Randomized controlled trial on environmental exposure reduction aimed at prevention of asthma in susceptible children: outcome of an intervention suitable for implementing in primary care [abstract]. Eur RespirJ. 2010, 36: 367s-

  17. 17.

    Kuiper S, Muris JW, Dompeling E, Kester AD, Wesseling G, Knottnerus JA, van Schayck CP: Interactive effect of family history and environmental factors on respiratory tract-related morbidity in infancy. J Allergy Clin Immunol. 2007, 120: 388-395. 10.1016/j.jaci.2007.03.038.

  18. 18.

    Kuiper S, Muris JW, Dompeling E, van Schayck CP, Schönberger HJ, Wesseling G, Knottnerus JA: Association between first-degree familial predisposition of asthma and atopy (total IgE) in newborns. Clin Exp Allergy. 2006, 36: 594-601. 10.1111/j.1365-2222.2006.02467.x.

  19. 19.

    Brennan A, Akehurst R: Modelling in health economic evaluation: what is its place? what is its value?. Pharmacoeconomics. 2000, 17: 445-459. 10.2165/00019053-200017050-00004.

  20. 20.

    Sculpher MJ, Claxton K, Drummond M, McCabe C: Whither trial-based economic evaluation for health care decision making?. Health Econ. 2006, 15: 677-687. 10.1002/hec.1093.

  21. 21.

    Barton P, Bryan S, Robinson S: Modelling in the economic evaluation of health care: Selecting the appropriate approach. J Health Serv Res Policy. 2004, 9: 110-118. 10.1258/135581904322987535.

  22. 22.

    Karnon J, Brown J: Selecting a decision model for economic evaluation: A case study and review. Health Care Manag Sci. 1998, 1: 133-140. 10.1023/A:1019090401655.

  23. 23.

    Brennan A, Chick SE, Davis R: A taxonomy of model structures for economic evaluation of health technologies. Health Econ. 2006, 15: 1295-1310. 10.1002/hec.1148.

  24. 24.

    Stahl JE: Modelling methods for pharmacoeconomics and health technology assessment: An overview and guide. Pharmacoeconomics. 2008, 26: 131-148. 10.2165/00019053-200826020-00004.

  25. 25.

    Evans C, Crawford B: Expert use in pharmacoeconomic studies: Guidance and future use. Pharmacoeconomics. 2000, 17: 545-553. 10.2165/00019053-200017060-00002.

  26. 26.

    van Keulen HM, Mesters I, Brug J, Ausems M, Campbell M, Resnicow K, Zwietering PJ, van Breukelen G, van Mechelen W, Severens JL, de Vries H: Vitalum study design: RCT evaluating the efficacy of tailored print communication and telephone motivational interviewing on multiple health behaviors. BMC Public Health. 2008, 8: 216-10.1186/1471-2458-8-216.

  27. 27.

    Albers-Heitner P, Berghmans B, Joore M, Lagro-Janssen T, Severens J, Nieman F, Winkens R: The effects of involving a nurse practitioner in primary care for adult patients with urinary incontinence: The PromoCon study (Promoting Continence). BMC Health Serv Res. 2008, 8: 84-10.1186/1472-6963-8-84.

  28. 28.

    Varian HR: How to build an economic model in your spare time [monograph on the internet]. 2009, Berkeley: University of California, Downloaded 07 April 2010, [http://people.ischool.berkeley.edu/~hal/Papers/how.pdf]

  29. 29.

    Evans C: The use of consensus methods and expert panels in pharmacoeconomic studies. Pharmacoeconomics. 1997, 12: 121-129. 10.2165/00019053-199712020-00003.

  30. 30.

    van der Fels-Klerx IHJ, Goosens LHJ, Saatkamp HW, Horst SHS: Elicitation of quantitative data from a heterogeneous expert panel: Formal process and application in animal health. Risk Anal. 2002, 22: 67-81. 10.1111/0272-4332.t01-1-00007.

  31. 31.

    Steuten L, Palmer S, Vrijhoef B, van Merode F, Spreeuwenberg C, Severens H: Cost-utility of a disease management program for patients with asthma. Int J Technol Assess Health Care. 2007, 23: 184-191.

  32. 32.

    Detsky AS, Naglie G, Krahn MD, Naimark D, Redelmeier DA: Primer on medical decision analysis: Part 2--Building a tree. Med Decis Making. 1997, 17: 126-135. 10.1177/0272989X9701700202.

  33. 33.

    Weinstein MC, O'Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, Luce BR: Principles of good practice for decision analytic modelling in health-care evaluation: Report of the ISPOR Task Force on Good Research Practices--Modeling Studies. Value Health. 2003, 6: 9-17. 10.1046/j.1524-4733.2003.00234.x.

  34. 34.

    McCabe C, Dixon S: Testing the validity of cost-effectiveness models. Pharmacoeconomics. 2000, 17: 501-513. 10.2165/00019053-200017050-00007.

  35. 35.

    Hammerschmidt T, Goertz A, Wagenpfeil S, Neiss A, Wutzler P, Banz K: Validation of health economic models: The example of EVITA. Value Health. 2003, 6: 551-559. 10.1046/j.1524-4733.2003.65241.x.

  36. 36.

    Akehurst R, Anderson P, Brazier J, Brennan A, Briggs A, Buxton M, Cairns J, Calvert N, Claxton K, Dixon S, Fryback D, Gallivan S, Green C, Lloyd A, McCabe C, Mitchell A, Nicholl J, O'Brien B, Roberts J, Sculpher M, Severens H, Sullivan S, Veenstra D: Decision analytic modelling in the economic evaluation of health technologies: A consensus statement. Pharmacoeconomics. 2000, 17: 443-444. 10.2165/00019053-200017050-00003.

  37. 37.

    Russell LB: Comparing model structures in cost-effectiveness analysis. Decis Making. 2005, 25: 485-486. 10.1177/0272989X05281155.

  38. 38.

    Warner JO: Origins of asthma. Childhood Asthma. Edited by: Szefler SJ, Pedersen S. 2006, New York: Taylor & Francis, [Lenfant C (Series Editor): Lung Biology in Health and Disease, vol. 209].

  39. 39.

    Alton V, Eckerlund I, Norlund A: Health economic evaluations: How to find them. Int J Tech Assess Health Care. 2006, 22: 512-517.

  40. 40.

    Sassi F, Archard L, McDaid D: Searching literature databases for health care economic evaluations: How systematic can we afford to be?. Med Care. 2002, 40: 387-394. 10.1097/00005650-200205000-00004.

  41. 41.

    Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL: Methods for the Economic Evaluation of Health Care Programmes. 2005, Oxford: Oxford University Press, 3

  42. 42.

    Schönberger HJAM, van Schayck CPM: Prevention of asthma in genetically predisposed children in primary care--from clinical efficacy to a feasible intervention programme. Clin Exp Allergy. 1998, 28: 1325-1331. 10.1046/j.1365-2222.1998.00416.x.

  43. 43.

    Hakkaart-van Roijen L, Tan SS, Bouwmans CAM: Manual for cost investigation. 2010, Amstelveen, Netherlands: Dutch College of Health Insurers

  44. 44.

    Dirksen WJ, Geijer RMM, de Haan M, de Koning G, Flikweert S, Kolnaar BGM: NHG-Standaard astma bij kinderen. Huisarts Wet. 1998, 41: 130-143.

  45. 45.

    Nuijten MJ, Wittenberg W, Lebmeier M: Cost-effectiveness of palivizumab for respiratory syncytial virus prophylaxis in high-risk children: A UK analysis. Pharmacoeconomics. 2007, 25: 55-71. 10.2165/00019053-200725010-00006.

  46. 46.

    Omnes LF, Bousquet J, Scheinmann P, Neukirch F, Jasso-Mosqueda G, Chicoye A, Champion L, Fadel R: Pharmacoeconomic assessment of specific immunotherapy versus current symptomatic treatment of allergic rhinitis and asthma in France. Eur Ann Allergy Clin Immunol. 2007, 39: 148-156.

  47. 47.

    Resch B, Gusenleitner W, Nuijten MJ, Lebmeier M, Wittenberg W: Cost-effectiveness of palivizumab against respiratory syncytial viral infection in high-risk children in Austria. Clin Ther. 2008, 30: 749-760. 10.1016/j.clinthera.2008.03.014.

  48. 48.

    El Hassan NO, Sorbero ME, Hall CB, Stevens TP, Dick AW: Cost-effectiveness analysis of palivizumab in premature infants without chronic disease. Arch Pediatr Adolesc Med. 2006, 160: 1070-1076. 10.1001/archpedi.160.10.1070.

  49. 49.

    Brüggenjürgen B, Reinhold T, Brehler R, Laake E, Wiese G, Machate U, Willich SN: Cost-effectiveness of specific subcutaneous immunotherapy in patients with allergic rhinitis and allergic asthma. Ann Allergy Asthma Immunol. 2008, 101: 316-324. 10.1016/S1081-1206(10)60498-X.

  50. 50.

    Buxton MJ, Drummond MF, van Hout BA, Prince RL, Sheldon TA, Szucs T, Vray M: Modelling in economic evaluation: An unavoidable fact of life. Health Econ. 1997, 6: 217-227. 10.1002/(SICI)1099-1050(199705)6:3<217::AID-HEC267>3.0.CO;2-W.

  51. 51.

    Akl EA, Grant BJB, Guyatt GH, Montori VM, Schünemann : A decision aid for COPD patients considering inhaled steroid therapy: development and before and after pilot testing. BMC Med Inform Decis Mak. 2007, 7: 12-10.1186/1472-6947-7-12.

  52. 52.

    Weiss K, Haus M, Iikura Y: The costs of allergy and asthma and the potential benefit of prevention strategies. Prevention of Allergy and Allergic Asthma: World Allergy Organization Project Report and Guidelines. Edited by: Johansson SGO, Haahtela T. 2004, Basel CH: Karger, XIII-XIX. [Ring J, Adorini L, Berek C, Blaser K, Capron M, Denburg JA, Holgate ST, Marone G, Saito H (Series Editors): Chemical Immunology and Allergy, vol. 84.]

  53. 53.

    Schwappach DLB: The economic evaluation of prevention--Let's talk about values and the case of discounting. Int J Public Health. 2007, 52: 335-336. 10.1007/s00038-007-0231-4.

  54. 54.

    Shiba K, Kashara K, Nakajima H, Adachi M: Structural changes of the airway wall impair respiratory function, even in mild asthma. Chest. 2002, 122: 1622-1666. 10.1378/chest.122.5.1622.

  55. 55.

    Lukrafka JL, Fuchs SC, Moreira LB, Picon RV, Fischer GB, Fuchs FD: Performance of the ISAAC Questionnaire to establish the prevalence of asthma in adolescents: A population-based study. J Asthma. 2010, 47: 166-169. 10.3109/02770900903483766.

  56. 56.

    Annemans L: Methodological issues in evaluating cost effectiveness of adjuvant aromatase inhibitors in early breast cancer: A need for improved modelling to aid decision making. Pharmacoeconomics. 2008, 26: 409-423. 10.2165/00019053-200826050-00005.

  57. 57.

    Gass SI: Decision-aiding models: validation, assessment, and related issues for policy analysis. Oper Res. 1983, 31: 603-631. 10.1287/opre.31.4.603.

  58. 58.

    Drummond MF, Barbieri M, Wong JB: Analytic choices in economic models of treatments for rheumatoid arthritis: what makes a difference?. Med Decis Making. 2005, 25: 520-533. 10.1177/0272989X05280561.

  59. 59.

    Goldhaber-Fiebert JD, Stout NK, Goldie SJ: Empirically evaluating decision-analytic models. Value Health. 2010, 13: 667-674. 10.1111/j.1524-4733.2010.00698.x.

Pre-publication history

  1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/11/150/prepub

Download references

Acknowledgements

We thank Dr. Jean Muris MD, Dr. Geertjan Wesseling MD, Annemiek Nijholt MD, and Dr. Lotte Steuten, for graciously contributing as experts for this study. The authors appreciate the time and effort put in the study by the general practitioners from the Nijmegen Monitoring Project (NMP), the Regional Network of General practitioners (RNH), CEL and EXTRA, and their practice staff members to recruit participants and record and collect the required data. This study is funded by a grant (120610019) from the Netherlands Organisation for Health Research and Development (ZonMw).

Author information

Affiliations

  1. Department of General Practice, CAPHRI School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands
    • G Feljandro P Ramos
    • , Sandra Kuiper
    • , J André Knottnerus
    •  & Onno CP van Schayck
  2. Department of Paediatric Pulmonology, Maastricht University Medical Centre, Maastricht, The Netherlands
    • Edward Dompeling
  3. Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre, Maastricht, The Netherlands
    • Antoinette DI van Asselt
  4. Department of Primary and Community Care, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
    • Wim JC de Grauw
    •  & Tjard RJ Schermer
  5. Department of Health, Organisation, and Policy Economics, CAPHRI School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands
    • Johan L Severens
  6. Institute of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands
    • Johan L Severens
Authors
  1. G Feljandro P Ramos
    View author publications
    You can also search for this author in
  2. Sandra Kuiper
    View author publications
    You can also search for this author in
  3. Edward Dompeling
    View author publications
    You can also search for this author in
  4. Antoinette DI van Asselt
    View author publications
    You can also search for this author in
  5. Wim JC de Grauw
    View author publications
    You can also search for this author in
  6. J André Knottnerus
    View author publications
    You can also search for this author in
  7. Onno CP van Schayck
    View author publications
    You can also search for this author in
  8. Tjard RJ Schermer
    View author publications
    You can also search for this author in
  9. Johan L Severens
    View author publications
    You can also search for this author in

Corresponding author

Correspondence to G Feljandro P Ramos.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

GFPR wrote the draft of the manuscript, collated and analysed the data as well as reported the results. JLS was involved in all aspects of the study and advised the project group with regard to the economic evaluation. ED was also involved in all aspects of the study, gave advice on effects and process evaluation, and was one of the experts consulted. ADIvA was involved in supervising the technical operation of the model. TRJS and WJCdG were involved in coordinating the data collection as well as writing the manuscript. SK was also involved in writing the manuscript and together with JAK and chief grant applicant OCPvS were responsible for conceptualizing, planning, and supervising the study. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Ramos, G.F.P., Kuiper, S., Dompeling, E. et al. Structuring and validating a cost-effectiveness model of primary asthma prevention amongst children. BMC Med Res Methodol 11, 150 (2011). https://doi.org/10.1186/1471-2288-11-150

Download citation

Keywords

  • Asthma
  • Primary Prevention
  • Decision Problem
  • Decision Framework
  • Asthma Prevention

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us