Realist evaluation

Realist evaluation seeks to understand how and why, for whom, and under what circumstances a programme works (or not) [19, 20]. The philosophical basis of realist evaluation is scientific realism [19]. The realist evaluator aims to identify the context-mechanism-outcome (CMO) causal relationship in order to explain “how, why, for whom and under what conditions a programme works” [21]. Following the understanding that people are not passive recipients of innovations [22] and that programmes can only work when the relevant actors adopt either all or parts of the intervention modalities, Van Belle [23] and Mukumbang et al. [24] elaborated on the CMO configurational logic proposed by Pawson and Tilley [19] to include components of the ‘Intervention’ and the ‘Actors’. These authors suggested that an intervention-context-actor-mechanism-outcome (ICAMO) configuration would provide a better analytical tool because aspects of the intervention, which provide the mechanisms and the actors through whom the intervention works are accounted for. Thus, in this article, we adopted the ICAMO heuristic tool.

Because realist evaluation is focused on providing explanatory models, it has the potential to open the “black box” of programmes by making explicit the generative mechanisms to explain how the programme modalities lead to the intended outcome(s) [25]. For this reason, realist evaluation is recommended as an appropriate approach for capturing the complexities of health care interventions during their evaluations [26, 27].

The realist methodology is a ‘theory driven, interpretative approach to uncovering underlying middle-range theories (or logics) driving interventions and their multiple components, as well as illuminating the contextual factors that influence mechanisms of change to produce outcomes’ [28]. The interpretive approach is driven by various forms of reasoning; deductive methods (based on testing specific hypothesis) and inductive reasoning (formulating general inferences), but central to the realist method of inquiry is abductive or retroductive reasoning [29]. Retroduction is a mode of inference in which events are explained by postulating (and identifying) mechanisms which are capable of producing outcomes [30]. According to Wynn and Williams [31], retroduction is characterised by the use of causal mechanisms as the basis for explanation, the possibility for multiple potential explanations, and the understanding that these causal mechanisms may or may not be observable empirically.

Typically, realist evaluations start with an initial programme theory (hypothesis) and end with a more refined theory. Therefore, the evaluator hypothesises in advance the intervention (I) (or its components), the relevant actors (A), mechanisms that are likely to operate (M), the contexts in which they might operate (C) and the outcomes that will be observed if they operate as expected (O). This hypothesis is formulated by conceptualising the components of a programme implementation process (programme modalities, context, actors, mechanisms and outcomes) to form theories about the underlying causes to arrive at explanations of what we observe. Based on this, realist evaluators seek to understand how and why programmes work by formulating programme theories.

According to Westhorp [32], “Realist evaluation is most appropriate for evaluating new initiatives or programmes that seem to work but where ‘how and for whom’ is not yet understood; programmes that have previously demonstrated mixed patterns of outcomes; and those that will be scaled up, to understand how to adapt the intervention to new contexts.” Since the adherence club intervention fulfils all the above conditions, we adopted the realist evaluation approach for the evaluation of the adherence club intervention.

The Programme theory

One of the central elements of realist evaluation is the programme theory. Realist evaluation starts and ends with a theory or with theories. Thus, eliciting an initial programme theory is a “pre-requisite of realist evaluation” methodology [32]. Developing a programme theory follows from the notion that programmes are theory-incarnate [19]. A programme theory is described as “a set of explicit or implicit assumptions of how the programme should be organised and why the programme is expected to work” [33].

Programme theories link activities and outcomes to explain how and why the desired change is expected to take place and represent how “the mechanisms introduced by the programme into pre-existing contexts can generate outcomes” [34]. This process is guided by a ‘generative’ model of causality, in which causal links are demonstrated through a fine-grained explanation of what happens between cause and effect.

Developing programme theories serves two main purposes: as a planning tool and/or as an evaluation tool. If used for planning, Sharpe [35] suggests that it is beneficial to develop a programme theory prior to the start of the programme. In this instance, the programme theory indicates how different elements of the programme are intended to work together and to identify the intermediate outcomes of a programme or an intervention [36]. This gives a clear indication of the goals and objectives of the programme and of the pathways through which they could be attained.

Programme theories are also used to guide monitoring and evaluation [36]. They are especially important for the evaluation of complicated and complex aspects of programmes [37]. To this end, the goal of the evaluator(s) is to understand not only patterns related to the outcome of the intervention but also to reveal how and why the intervention attains the outcome of interest [38]. It is noteworthy that the evaluation of the programme’s theory is an evaluation of the programme rather than the evaluation of the programme theory [35]. Whatever their use, programme theories should be concrete enough to be tested and refined through empirical research, and abstract enough to generalise from the case-specific theories [39].

What differentiates the programme theory in realist evaluation from programme theories as conceived in other theory-driven approaches, such as Theory of Change [40] or Theory-driven evaluation [41, 42], is that realist evaluation specifies what mechanisms will generate the outcomes in what context. Thus, a realist programme theory provides a conceptual framework for putting the underlying CMO components centre stage [43].

Collating findings into an initial programme theory

After identifying the various aspects of the relevant actor, context, mechanism and outcome components (steps 1–3), we used the ICAMO configuration (Fig. 4) as a heuristic to formulate the initial programme theory [23, 24].

Based on the ICAMO heuristic tool, we applied the process of configuration mapping [48] – an approach to causality, whereby, outcomes are considered to follow from the alignment of a specific combination of attributes [20]. The configuration process was achieved through the logic of retroduction. By applying retroduction, the realist researcher moves from the description and analysis of concrete phenomena (usually obtained from actors by interview) to reconstructing the basic conditions (theory) for these phenomena [49].

Following the logic of retroduction, first, we identified mechanisms that were related to the different actors (patients, health professionals) and the different outcomes that these mechanisms are likely to perpetuate. Second, we examined the context conditions that were associated with the mechanisms identified as informed by the data. Finally, to elicit the initial programme theory based on our data, we found it useful to distinguish between different sub-elements of the adherence club programme, including the individual-level, interpersonal-level, group-level, family-level and community-level. Figure 5 represents the configuration map developed through this process.

Formulating the testable hypotheses

The testable hypotheses were formulated by translating the ICAMO configurations using the “if…, then…, because …” statement [32, 48]. Through this exercise, we identified two alternative programme theories.

Objectives of the adherence club programme

The adherence club programme was designed to shift the majority of consultations and ART collections for stable patients to ‘clubs’ organised and facilitated by facility counsellors or peer educators [50]. In so doing, it strives (a) to retain patients – reduce the number of patients lost to follow-up – in ART care by providing a more efficient way to manage stable patients; (b) to achieve and maintain good long-term adherence in PLWHA on ART treatment by maintaining a good quality of care – creating an environment for more convenient clinical visits that accommodate their lifestyle needs; and (c) to decongest the health facility – fewer patients using the mainstream ART scheme – through group sessions that are facilitated by trained non-clinical staff [51].

Eligibility to the adherence club

An adherence club is made of 25–30 ‘stable’ patients on ART. Evidence from the document review shows that although patients are encouraged to request their admission into the clubs, it is clinicians who determine whether a patient qualifies for admission. Eligibility is centred on the patient being ‘stable’ – clinically well and adhering to treatment [12]. The patients can be recruited if they are 18 years or more, weigh more than 40 kg, and have been on the same ART regimen (1 or 2) for at least 12 months. Clinically, the two most recent consecutive viral loads of the patient should be undetectable (< 400 copies/mL) with a CD4 count of more than 200 cells/mm³ in the last 6 months [50, 52]. In addition, the patients must have a reliable clinical attendance record as evidenced by attendance card, and no medical condition requiring regular clinical consultations including tuberculosis (TB).

Club membership can be terminated if the patient has a viral load above 400 copies/mL or significantly abnormally low CD4 count < 200 cells/mm³, and when he/she develops an active TB infection, switches ART regimen for clinical reasons, becomes pregnant (for female patients), or develops any condition that requires frequent clinical follow-up [8]. In addition, when a patient fails to regularly attend mandatory club sessions or when he/she fails to send a ‘treatment buddy’ to collect their medication from the club facilitator or club nurse within 5 days (grace period), he/she is returned to the main clinic for care [53].

In practice, more considerations are taken into account when it comes to allowing patients in an adherence club. For instance, if a patient in the club develops a chronic co-morbidity, such as hypertension and diabetes, that is well-controlled and if the patient is ‘stable’ as per the clinician’s assessment, he/she then will still be eligible for a club on condition he/she fulfils the “HIV-related” criteria. In a similar manner, some facilities choose to keep pregnant women in the club as long as their viral load is undetectable, in which case their antenatal visits are managed by the staff responsible for antenatal care separately. However, in some settings, pregnant women are placed out of clubs and are only allowed to re-enter after pregnancy, particularly where mother and baby will be managed as a pair in the post-natal period or if antenatal care is delivered at a facility other than the facility rendering club services.

Management structure and running of the club activities

Activities of the adherence club

The observations showed that facility-based adherence clubs take place in available spaces or rooms, such as the clinic boardroom, makeshift or purpose-built buildings. During the introductory visit to the club, patients are provided with the rules governing the adherence club and its activities. Patients are considered enrolled in the club after they have attended their first club meeting and have picked up their pre-packed medication. According to the document review and as confirmed by our observations, if a patient is unable to attend a club appointment for any reason, they are entitled to send someone known as a ‘treatment buddy’ – usually a family member, friend or companion who supports an ART patient on treatment – to collect their treatment from their adherence club [17].

The activities of the adherence club are organised bi-monthly. A regular club session lasts approximately one to one and a half hours [50]. On days when blood tests are carried out for routine adherence monitoring through CD4 count and viral load, the sessions take much longer. Once a year, club members attend a regular clinic. Table 4 and Fig. 6 below show a standard attendance schedule of the adherence club programme.

Type of club visit	Activities	Script and CDU visits
	Recruitment + clinician scripting for 3 months	1 month supply by pharmacy
Enrolment visit	Scripting for 6 months	1 × 2 months pre-packed
Routine visit		1 × 2 months pre-packed
Blood visit	Bloods taken	2 × 2 months pre-packed
Clinical visit	Clinical consultation + re-scripting for 6 months	3 × 2 months pre-packed
Routine visit		1 × 2 months pre-packed
Routine visit		2 × 2 months pre-packed
Re-scripting visit	Re-scripting	4 × 2 months pre-packed

Note: The cycle repeats from month 12

Based on our observations and as reported in some of the programme documents, during a standard club session (medication pick-up), the patients arrive in the morning and hand their clinic cards to the clinic facilitator. At each visit, club members are clinically assessed (weight and symptom screen) and participate in a group adherence support and/or education activity. They are then issued with 2 months’ pre-dispensed medication. Next, the club facilitator gives health education talks to the assembled group on relevant topics. These topics include challenges that patients face in taking their medication. Sometimes a patient is asked to give a talk. The club facilitators do not have the licence to dispense medication; therefore, they collect the pre-packaged medication (labelled for each patient and supplied by the Chronic Dispensing Unit) and distribute them to the patient (or treatment ‘buddy’). The drug distribution task can also be done by an ‘expert’ patient.

According to the adherence club guidelines [51] and adherence club toolkit [50], any patient reporting or presenting with symptoms suggesting illness, adverse drug effects or weight loss are referred by the club facilitator to the club nurse for further consultations. Based on the outcome of the consultation, they are either sent to collect their medication from the club facilitator (minor ailments and medication side effects) or removed from the club (uncontrolled comorbidity like diabetes or hypertension). Figure 7 below outlines the activities of a standard adherence club session.

Based on the adherence club toolkit [50] and as confirmed by our observations, on the appointment days for blood checks and annual clinic visits, the patient attends in person. If a patient sends a buddy on a blood-taking or a clinical visit day or for the second time in succession, the buddy is told to inform the client that they should visit the club’s manager at the facility within five working days [51].

Based on the adherence club guidelines [51], when a club patient has not sent a ‘buddy’ to collect his/her medication and does not come to the facility within a ‘grace’ period of five working days, he/she will be referred to the club’s manager. The guideline recommends that at the end of the ‘grace’ period, the club facilitator should record the appropriate outcome of the patients who did not attend the group session after efforts to contact the patient has been made. Our observation showed that the grace period is sometimes longer than 5 days as the operational staffs tend to be more lenient. The club manager informs the patient that he/she can no longer attend the club and must return to mainstream care.

Where a patient has been returned to mainstream care for clinical reasons, the club clinician can set the conditions for the patient’s re-enrolment in the club system. In such case, the patient is considered a new club patient from a monitoring and evaluation perspective [53]. Relevant information in the club register is transferred to the facility ARV register (paper or electronic) by the clinic data clerk. Information, such as the CD4 count and annual viral load tests results, is recorded in the club register by the club facilitator and transferred to the electronic register by the data clerk.

Comparisons and effectiveness of the adherence club to standard clinic care

Both the facility-based and community-based adherence club models have shown better results and effectiveness in retaining patients in care and improving adherence to medication compared to the regular clinic services under pilot conditions [11, 12]. An evaluation of the adherence club intervention under implementation conditions confirms the findings of the pilot studies [54]. Recent evidence shows that patients receiving care in the adherence care model show more satisfaction with ART care compared to those who receive care from the standard ART scheme [55]. Evidence also shows that the adherence club model of ART care is cost-effective compared to standard care [56].

Programme theory 1

Programme theory 2