Building capacity for evidence generation, synthesis and implementation to improve the care of mothers and babies in South East Asia: methods and design of the SEA-ORCHID Project using a logical framework approach
© McDonald et al; licensee BioMed Central Ltd. 2010
Received: 1 December 2009
Accepted: 1 July 2010
Published: 1 July 2010
Rates of maternal and perinatal mortality remain high in developing countries despite the existence of effective interventions. Efforts to strengthen evidence-based approaches to improve health in these settings are partly hindered by restricted access to the best available evidence, limited training in evidence-based practice and concerns about the relevance of existing evidence. South East Asia - Optimising Reproductive and Child Health in Developing Countries (SEA-ORCHID) was a five-year project that aimed to determine whether a multifaceted intervention designed to strengthen the capacity for research synthesis, evidence-based care and knowledge implementation improved clinical practice and led to better health outcomes for mothers and babies. This paper describes the development and design of the SEA-ORCHID intervention plan using a logical framework approach.
SEA-ORCHID used a before-and-after design to evaluate the impact of a multifaceted tailored intervention at nine sites across Thailand, Malaysia, Philippines and Indonesia, supported by three centres in Australia. We used a logical framework approach to systematically prepare and summarise the project plan in a clear and logical way. The development and design of the SEA-ORCHID project was based around the three components of a logical framework (problem analysis, project plan and evaluation strategy).
The SEA-ORCHID logical framework defined the project's goal and purpose (To improve the health of mothers and babies in South East Asia and To improve clinical practice in reproductive health in South East Asia), and outlined a series of project objectives and activities designed to achieve these. The logical framework also established outcome and process measures appropriate to each level of the project plan, and guided project work in each of the participating countries and hospitals.
Development of a logical framework in the SEA-ORCHID project enabled a reasoned, logical approach to the project design that ensured the project activities would achieve the desired outcomes and that the evaluation plan would assess both the process and outcome of the project. The logical framework was also valuable over the course of the project to facilitate communication, assess progress and build a shared understanding of the project activities, purpose and goal.
Poor health and pregnancy care leads to over half a million maternal deaths and almost eight million perinatal deaths each year. For every woman who dies, about 20 more suffer injuries, infection and disabilities . The burden of death and injury falls disproportionately on low and middle income countries, as is indicated by the lifetime risk of maternal death in South East Asia which is one in 130 compared to one in 7,300 in developed regions . Importantly, much of this burden could be prevented if interventions which have been demonstrated by research to be effective and feasible were implemented at scale .
Implementation of the results of research into practice has been identified by the WHO as the most significant challenge to health care . An overview of approaches to bridging the gap between research and practice has highlighted the lack of primary research about what interventions are effective in increasing use of research in health care in low-income settings . In part this reflects the low priority afforded to health services research in these settings . This is a concern given that increasing the generation and use of research in health care are essential steps in improving health systems and achieving better health, particularly in low-income settings [7–9].
Ensuring access to scientifically valid and up-to-date information is a prerequisite to use of research in guiding health care, or 'evidence-based practice'. In spite of the profusion of health information available in print and electronic media (and several international initiatives to promote access), healthcare workers in low-income settings are still disadvantaged when it comes to accessing reliable information on effective care [10, 11]. Organisational, financial and technical barriers to accessing research (including lack of computers and internet access, limited scientific literacy, lack of availability of research, inappropriate format of research publications, etc.) and concerns about relevance to local settings are compounded by a lack of skills among practitioners and other stakeholders to appraise and interpret research findings .
The South East Asia - Optimising Reproductive and Child Health in Developing Countries (SEA-ORCHID) project aimed to address some of these issues. This five-year project (2004-08) was funded through the International Collaborative Research Grants Scheme, a partnership between the UK-based Wellcome Trust and Australia's National Health and Medical Research Council. The scheme was designed to foster collaborative research between countries in South East Asia and Australia by funding research into major health issues affecting developing countries and developing research capacity.
The objective of SEA-ORCHID was to evaluate whether a intervention designed to strengthen the capacity for research generation, synthesis, and use, improved perinatal practice and led to better health outcomes for mothers and babies.
The SEA-ORCHID study protocol reporting the overall research design and project plan has been published previously . The protocol outlined the core groups targeted by the intervention and key intervention activities. Planning and designing the intervention was undertaken in two stages: development of a project-wide logical framework before the intervention phase; and a site-specific action research-based operationalisation of this framework undertaken during the intervention phase. In this paper, we describe the methods and results of the first stage.
The project was approved by the ethics committee relevant to each participating hospital and by the ethics committee of the administering institution in Australia (University of Sydney).
Logical Framework Approach
In complex projects that have broad goals, a logical framework approach can help summarise and describe the multiple strands of the project clearly and systematically, and clarify the core goals, broad objectives, activities and expected outcomes . There are three main components of a logical framework: project rationale (analysis of main problems and their causes); project plan (goal, objectives, activities, outputs) and evaluation and measurement strategy. The logical framework tool helps develop and demonstrate the logic of the project in two dimensions. The first, vertical dimension, is whether the activities of the project are likely to achieve the objectives and whether these objectives are likely to achieve the broader goals. The second, horizontal dimension, is whether the evaluation plan will measure the project's progress (process measures) and whether the progress of the project will in turn address the identified problems (outcome measures).
Development of the logical framework
The SEA-ORCHID project coordinator (SM) and one of the Australian-based project staff (CC) managed the development of the logical framework, drawing on information from several sources, along with contributions from members of the SEA-ORCHID team.
The information sources used to develop the logical framework included the study protocol , a brief survey of perceived current practice, and in-depth discussions at investigator and project meetings (to capture the experiences and perspectives of the local decision-makers). The study protocol set the parameters of the project in terms of its overall aims and design.
At the start of the project planning phase, the SE Asian investigators completed an informal survey of current practice which provided a snap shot of the extent to which care was in line with research (as perceived by the investigators who were senior clinicians working in the participating sites) at different levels within the hospital system in each country (tertiary, regional, provincial and district). The survey also collected data on any complementary research activities that might inform or affect the intervention design, such as participation in the WHO Reproductive Health Library cluster randomised trial  (Additional file 1: SEA-ORCHID survey).
Discussions at the early investigator meetings were pivotal in articulating the key issues and beginning the process of translating these into objectives linked to activities. At the first full meeting of all the investigators and Australian-based educators in Adelaide in March 2005, several general principles were agreed with respect to intervention planning:
sustainability and capacity-building should underpin all intervention strategies;
interdisciplinary teamwork, including both clinical and non-clinical staff, such as librarians and biostatisticians, should be encouraged;
SEA-ORCHID project baseline data on clinical practices should be used wherever possible to determine local priorities.
During the same meeting, each SE Asian investigator summarised the current situation at the project hospitals in their country in terms of awareness, acceptance and use of research in practice; outlined particular barriers and enablers that could affect the intervention; and suggested activities they considered priorities.
The information from these three sources served as the raw material for the logical framework.
The logical framework was developed with input from the seven chief investigators (one from each of the SE Asian countries, and three from Australia), the two project coordinators (from Australia and Thailand) and the Australian project staff.
The SE Asian investigators were senior obstetricians or neonatologists responsible for co-ordinating the project and determining both local and national priorities within the overall framework of the project.
The three Australian investigators were from centres that have strong links with The Cochrane Collaboration and international research networks for maternal and perinatal health. The Australian Research Centre for Health of Women and Babies (ARCH) at the University of Adelaide http://www.health.adelaide.edu.au/og/research/arch.html comprises several research divisions. The research synthesis division, where the regional co-ordinating unit for the Cochrane Pregnancy and Childbirth Group is based, also includes the ARCTURUS Clinical Trials division and the national coordinating office for the WOMBAT Collaboration http://www.wombatcollaboration.net and promotes and supports high quality randomised trials in maternal and perinatal health.
The Centre for Perinatal Health Services Research at the University of Sydney aims to improve the process and outcome of care for mothers and infants in New South Wales, nationally and internationally, through research and education promoting evidence-based practice http://www.psn.org.au/. The Centre is also the regional co-ordinating base for the Cochrane Neonatal Group. The third Australian site was at Monash University and houses the Australasian Cochrane Centre http://www.cochrane.org.au/, which is responsible for training and supporting review authors and fostering the development of Cochrane activity in Australasia and South East Asia.
Hospitals in the SEA-ORCHID Project
Type of hospital
Average births per month (in 2005)
Dr Sardjito Hospital, Yogyakarta
Sleman District Hospital, Yogyakarta
Ipoh Hospital, Ipoh
Kelantan Hospital, Kota Bharu
Philippine General Hospital, Manila
Jose Fabella Hospital, Manila
Srinagarind Hospital, Khon Kaen
Khon Kaen Hospital, Khon Kaen
Kalasin Hospital, Kalasin
Development of the logical framework included establishing a project rationale by defining and unpacking the central problem - the problem analysis - and developing a project goal that captured the overarching aim of the project in addressing the central problem.
The project planning component of the process sought to generate a range of possible activities and outputs designed to meet the objectives underpinning the project's overall aim. The third component of the logical framework - evaluation and measurement - outlined the process measures designed to monitor the project's progress and the outcome measures designed to assess whether the project was meeting its goal and purpose.
Refining the logical framework was a four-month, iterative process with drafts circulated among the project teams for feedback and comments. The final version of the logical framework was approved in June 2005.
Inappropriate clinical practices
Inadequate access to clinical resources
Lack of access to maternity care services
Since SEA-ORCHID was not a multi-million dollar international aid project, our focus was on the first of these. The purpose of the SEA-ORCHID project was thus defined as 'To improve clinical practice in reproductive health in South East Asia'.
The problem analysis outlined several reasons why clinical practice was not always in line with the results of research. These included a lack of skills in generating and using evidence; inadequate access to reliable healthcare information; lack of locally derived and relevant research evidence; and lack of evidence-based policies and clinical practice guidelines (Figure 3).
These reasons were determined to have several underlying causes:
low participation in research synthesis
lack of evidence-based practice training in medical and nursing curricula
lack of financial resources to pay for subscriptions, etc
lack of computer access and literacy
low priority given to research
lack of research skills
large centralized systems of establishing and changing clinical practice
role of hospital policy-makers in determining clinical practice.
to build capacity in research synthesis
to increase capacity and skills for evidence-based practice
to demonstrate effective implementation strategies for evidence-based practice change
to improve access to quality healthcare information
to increase locally derived and relevant research activity
to increase evidence-based policy making
to influence the broader socio-economic and health policy environment.
In relation to the first objective 'To build capacity in research synthesis', proposed activities included training in systematic reviews and guideline development, developing structures to sustain research synthesis activity in the longer term, and facilitating recognition of systematic reviews as a valid form of academic research.
Activities linked to the second objective 'To increase capacity and skills for evidence-based practise' included developing curriculum materials, training educators and users of evidence, and working towards the integration of EBP within existing curricula.
In relation to the third objective 'To demonstrate effective implementation strategies for evidence-based practice change', activities focused on assessing barriers and educational needs, promulgating adult learning principles, monitoring clinical practice and disseminating findings, and developing collaborative implementation strategies.
The fourth objective 'To improve access to quality healthcare information' gave rise to activities that included pursuing consortia with medical schools or colleges, promoting access to various professional groups, and developing a longer term strategy to secure national provision to The Cochrane Library.
The fifth objective 'To increase locally derived and relevant research activity' included activities aimed at identifying local research needs and appropriate questions for systematic reviews, facilitating and mentoring those involved in research synthesis, and incorporating research skills into training programs.
Activities in relation to objective six 'To increase evidence-based policy making' included identifying evidence gaps and organising workshops for policy makers.
The final objective 'To influence the broader socio-economic and health policy environment' was concerned with activities around advocacy and awareness raising, and collaborating with other agencies.
These broad objectives and activity descriptions were used by the project teams in each country and site to develop and implement detailed project plans reflecting local priorities, needs and resources. Descriptions of the specific activities undertaken at each participating hospital are available at http://www.seaorchid.org.
Evaluation and measurement
Objectives and related outcome measures of the SEA-ORCHID Project
To build capacity in research synthesis
SR and CPG workshops conducted
SRs and CPGs produced
SR trainers on database
Participants in Cochrane Collaboration
Branches of Australasian Cochrane Centre in SE Asia
Academic sites recognising SRs in SE Asia
To increase capacity and skills for evidence-based practice
Training materials developed
EBP sessions undertaken and number of participants
EBP in curricula pre- and post-intervention
Nodes with documented recognition of CME points
Persons receiving CME points for EBP training
To demonstrate effective implementation strategies for evidence-based practice change
Pre- and post-intervention measurement of primary and secondary outcomes
Qualitative assessment of barriers and enablers to EBP
Knowledge, beliefs and attitudes to EBP
Program logic framework with input and consensus from all project partners
Process evaluation of project implementation
Description of collaboration and communication strategies
Publication describing current practice
To improve access to quality healthcare information
Cochrane Library usage statistics
National subscriptions to Cochrane Library
To increase locally derived and relevant research activity
Reports of trials identified and submitted to the Cochrane Central Register of Controlled Trials
Research trials registered
Training sessions conducted
Participants attending training sessions
To increase evidence-based policy making
Description of networking arrangements and communication strategy and members
Participants attending workshops
EB policies and CPGs
To influence the broader socio-economic and health policy environment
Documented advocacy strategies
Advocacy and lobbying networks and activities
Responses and outcomes to advocacy activities
Assessment of the progress of the project was done through face-to-face project meetings, teleconferences and regular reporting. Project meetings involving team members from all SEA-ORCHID sites were held every nine months to provide an opportunity for sharing successes, brainstorming approaches to overcoming barriers, encouraging networking across nodes and refining the planned activities for the next period of the intervention phase. Between project meetings, the Australian educators initiated regular two-monthly teleconferences with each of the project nodes to ensure progress was on track, provide support and resources, and help with any problems or issues that arose. The project investigators also had regular teleconferences. SEA-ORCHID team members reported every two months on the activities undertaken at their hospital by completing activity statements which were then uploaded to the SEA-ORCHID website http://www.seaorchid.org to give a cumulative summary of all activities undertaken during the project.
There are many different program logic models and frameworks used for planning and evaluating programs. Despite differences in approach, they share a common purpose which is to describe the rationale behind a program and to clarify the relationships between monitoring and evaluation and the project activities. In development projects, logical frameworks are the predominant program logic method used and are particularly important as communication tools in projects that cover diverse countries, cultures and languages . The purpose of using a logical framework in the SEA-ORCHID project was to:
assess whether the project activities were addressing perceived needs;
clarify the expected cause-and-effect relationships between the project activities and expected outcomes;
predict the extent to which the planned outcome measures were likely to evaluate the impact of the project activities; and
assist project workers to reach a consensus about realistic goals and outcomes of the project.
The logical framework became the core tool that guided the overall intervention strategy and was taken by each site and used as the basis for devising a tailored suite of activities that was implemented within an action research framework during the two-year intervention phase (2006-07). Over the course of the project the logical framework was a powerful tool for facilitating communication within and between sites, and for maintaining a shared understanding of the purpose and progress of a complicated, multilayered intervention.
There are several limitations of the logical framework approach for projects of this nature. Firstly, the linear model does not always do justice to the complex multifaceted influences, dynamic interactions, cultural differences and unforeseen consequences that are part of everyday reality. Therefore they need to be used flexibly and adapted as projects evolve. Secondly, the logical framework only represents the consensus of the individuals involved in developing it. However, they can be used to clarify the assumptions on which project plans are made and allow incoming members of the project to check whether these assumptions are relevant to their local context. Finally, the logical framework approach in this instance had to be adapted to meet the realities of planning a project intervention within the constraints of a research grant. Under ideal conditions, projects are planned systematically by looking first at the problem, then developing goals and objectives, and finally devising an accompanying evaluation plan . In SEA-ORCHID, the broad issue of building capacity for evidence-based practice and the pre- and post-intervention evaluation strategy was defined first and the specific goals and objectives defined subsequently within the requirements of a research protocol.
Experts in healthcare improvement emphasise the crucial importance of acquiring a good understanding of the problem, the target group, its setting and obstacles to change . Logic models, such as the logical framework, value participatory processes in project planning as a means of achieving this understanding. SEA-ORCHID is an international collaborative project that aims to improve health outcomes for mothers and babies in South East Asia by strengthening capacity for generating, synthesising and implementing evidence. The logical framework approach is an established project design method that is based on a systematic analysis of the problem or situation and an exploration of possible solutions. For SEA-ORCHID, the use of a project-wide logical framework has provided a mechanism for further developing site-specific intervention plans. Overall, this approach is both robust and flexible and appropriate for real-world research into change processes.
The process of developing a logical framework in the SEA-ORCHID project was valuable because it enabled a reasoned, logical approach to the project design that ensured the project activities would achieve the desired outcomes and that the evaluation plan would assess both the process and outcome of the project. The logical framework was also valuable over the course of the project to facilitate communication, provide a reference for assessment of progress and build a shared understanding of the project activities, purpose and goal.
- SE Asia:
South East Asia
South East Asia - Optimising Reproductive and Child Health in, Developing Countries
World Health Organization
All authors were funded individually by their respective university/institution for the development and design of the project. This study is jointly funded by an International Collaborative Research Grant from the National Health and Medical Research Council of Australia (No. 307703) and Wellcome Trust, United Kingdom (071672/Z/03/Z).
- Hill K, Thomas K, AbouZahr C, Walker N, Say L, Inoue M, Suzuki E: Estimates of maternal mortality worldwide between 1990 and 2005: an assessment of available data. Lancet. 2007, 370: 1311-19. 10.1016/S0140-6736(07)61572-4.View ArticlePubMedGoogle Scholar
- WHO, UNICEF, UNFPA and The World Bank: Maternal mortality in 2005: estimates developed by WHO, UNICEF, UNFPA and The World Bank. Geneva. 2007, [http://www.childinfo.org/maternal_mortality_countrydata.php]Google Scholar
- Jones G, Steketee RW, Black RE, Bhutta ZA, Morris SS, the Bellagio Child Survival Study Group: How many child deaths can we prevent this year?. Lancet. 2003, 362: 65-71. 10.1016/S0140-6736(03)13811-1.View ArticlePubMedGoogle Scholar
- World Health Organization: Bridging the "Know-Do" Gap: Meeting on Knowledge Translation in Global Health. 2005, Geneva, Switzerland. World Health OrganizationGoogle Scholar
- Haines A, Kuruvilla S, Borchert M: Bridging the implementation gap between knowledge and action for health. Bull World Health Organ. 2004, 82: 724-32.PubMedPubMed CentralGoogle Scholar
- Sanders D, Haines A: Implementation research is needed to achieve international health goals. PLoS Med. 2006, 3 (6): e186-10.1371/journal.pmed.0030186. DOI: 10.1371/journal.pubmed.0030186View ArticlePubMedPubMed CentralGoogle Scholar
- Pang T, Sadana R, Hanney S, Bhutta ZA, Hyder AA, Simon J: Knowledge for better health: a conceptual framework and foundation for health research systems. Bull World Health Organ. 2003, 81: 815-20.PubMedGoogle Scholar
- Lansang M, Dennis R: Building capacity in health research in the developing world. Bull World Health Organ. 2004, 82: 764-70.PubMedPubMed CentralGoogle Scholar
- Global Forum for Health Research: Statement by the Global Forum for Health Research Forum 8, Mexico City. 2004, [http://www.globalforumhealth.org/Forums/Annual-Forums/Previous-Forums/Forum-8/Statement-by-the-Global-Forum-at-the-conclusion-of-Forum-8]Google Scholar
- Volmink J, Siegfried N, Robertson K, Gülmezoglu AM: Research synthesis and dissemination as a bridge to knowledge management: the Cochrane Collaboration. Bull World Health Organ. 2004, 82: 778-83.PubMedPubMed CentralGoogle Scholar
- Martis R, Ho JJ, Crowther CA: Survey of evidence-based practice knowledge and clinical change among maternal and infant health practitioners in South East Asia.
- Godlee F, Pakenham-Walsh N, Ncayiyana D, Cohen B, Packer A: Can we achieve health information for all by 2015?. Lancet. 2004, 364: 295-300. 10.1016/S0140-6736(04)16681-6.View ArticlePubMedGoogle Scholar
- Henderson-Smart DJ, Lumbiganon P, Festin MR, Ho JJ, Mohammad H, McDonald SJ, Green S, Crowther CA: Optimising reproductive and child health outcomes by building evidence-based research and practice in South East Asia (SEA-ORCHID): study protocol. BMC Med Res Methodol. 2007, 7: 43-10.1186/1471-2288-7-43. DOI:10.1186/1471-2288-7-43View ArticlePubMedPubMed CentralGoogle Scholar
- AusAID: AusGuideline: the logical framework approach. 2005, Canberra, [http://www.ausaid.gov.au/ausguide/pdf/ausguideline3.3.pdf]Google Scholar
- Gülmezoglu AM, Langer A, Piaggio G, Lumbiganon P, Villar J, Grimshaw J: Cluster randomized trial of an active, multifaceted educational intervention based on the WHO Reproductive Health Library to improve obstetrics practices. BJOG. 2007, 114: 16-23. 10.1111/j.1471-0528.2006.01091.x.View ArticlePubMedGoogle Scholar
- Furnell S: Developing and using a program theory matrix for program evaluation and performance monitoring. Program theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation. Edited by: Rogers P, Hacsi T, Petrosino A, Heubner T. 2000, San Francisco: Jossey-Bass, 87: 91-101.Google Scholar
- Owen JM: Program Evaluation, Forms and Approaches. 1993, St Leonards, NSW: Allen and UnwinGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/10/61/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.