Skip to main content

Should we embed randomized controlled trials within action research: arguing from a case study of telemonitoring

Abstract

Background

Action research (AR) and randomized controlled trials (RCTs) are usually considered to be theoretically and practically incompatible. However, we argue that their respective strengths and weaknesses can be complementary. We illustrate our argument from a recent study assessing the effect of telemonitoring on health-related quality of life, self-care, hospital use, costs and the experiences of patients, informal carers and health care professionals in two urban hospital services and one remote rural primary care service in New Zealand.

Methods

Data came from authors’ observations and field notes of discussions with three groups: the healthcare providers and healthcare consumers who participated in the research, and a group of 17 researchers and collaborators. The consumers had heart failure (Site A, urban), airways disease (Site B, urban), and diabetes (Site C, rural). The research ran from 2008 (project inception) until 2012 (project close-off). Researchers came from a wide range of disciplines. Both RCT and AR methods were recognised from early in the process but often worked in parallel rather than together. In retrospect, we have mapped our observed research processes to the AR cycle characteristics (creation of communicative space, democracy and participation, iterative learning and improvement, emergence, and accommodation of different ways of knowing).

Results

We describe the context, conduct and outcomes of the telemonitoring trial, framing the overall process in the language of AR. Although not fully articulated at the time, AR processes made the RCT sensitive to important context, e.g. clinical processes. They resulted in substantive changes to the design and conduct of the RCT, and to interpretation and uptake of findings, e.g. a simpler technology procurement process emerged. Creating a communicative space enabled co-design between the researcher group and collaborators from the provider participant group, and a stronger RCT design.

Conclusions

It appears possible to enhance the utility of RCTs by explicitly embedding them in an AR framework to shape stronger RCT design. The AR process and characteristics may enable researchers to evaluate telehealth while enhancing rather than compromising the quality of an RCT, where research results are returned to practice as part of the research process.

Trial registration

Australian New Zealand Clinical Trials Registry, reference ACTRN12610000269033.

Peer Review reports

Background

Good clinical decisions require evidence on more than just effectiveness [1]. Clinicians, consumers, and health systems, need evidence about processes, social context, patient engagement, equity, and health literacy; such factors are typically and explicitly eliminated from Randomised Controlled Trial (RCT) designs as sources of bias and confounding. Interventions whose effectiveness may vary with context, such as information and communication technologies (ICT), and many other health services interventions, challenge the ability to reduce bias and even the wisdom of attempting to do so.

Multiparadigm inquiry offers different ways of knowing [2], which in turn, offer opportunities to learn about factors excluded from positivist RCT reports. Rich data aiming at a thick description as described by Geertz [3] that includes nuanced, multi-perspective, qualitative and quantitative data can be collected and used, e.g., to offer explanations for RCT findings and identify unanticipated consequences of interventions [4]. Action Research (AR), in contrast to RCTs, consists of cycles of planning, acting, doing, and adjusting the plan, as well as characteristics such as [1] participation and democracy, [2] communicative space, [3] iterative improvement, [4] emergence, and [5] different ways of knowing [5]. AR naturally falls within critical and interpretivist approaches [6] and aims at change. RCT and AR are therefore well placed for theoretical pluralism because what is weak in one is strong in the other for the purpose of evaluating complex healthcare interventions such as telemonitoring, as outlined in Table 1.

Table 1 Strengths and weaknesses of RCTs and Action Research

The purpose of this article is to examine the use of AR cycles and characteristics as a frame for designing and executing a telemonitoring RCT. The case we use in our argument aimed to assess the effect of telemonitoring on health-related quality of life, self-care, hospital use, costs and the experiences of patients, informal carers, and health care professionals. We present background literature, a telemonitoring RCT as the case study, and a critical analysis of our AR experience related to the case study.

Acquiring evidence for practice

RCTs are not enough for acquiring evidence and applying it in practice. It can take up to 17 years for research from RCTs to become everyday clinical practice, because of repetitive research projects, and difficulties in identifying relevant research for practice [7]. Research into telehealth is not compatible with that timetable, as technology is constantly changing as refinements and innovations emerge [7, 8]. Kaldoudi, Chatzopoulou, and Vargemezis [9] argue that RCTs alone may not be enough for telehealth because the intervention is more than the technology. Furthermore, the intervention can influence and be influenced by uncontrollable confounding factors, e.g., local context of the patient, the organisation, and healthcare processes. Bias cannot be controlled because telehealth RCTs cannot be blinded - it is obvious to participants that groups with and without the technology are being compared [10].

Telehealth consists of complex interventions. The UK Medical Research Council’s (MRC) evaluation framework defines complex interventions as having a number of intervention components, and associated behaviours, groups, organisational levels, intervention flexibility, and number and variability of outcomes [11]. Telehealth research is usually complex, involving patients doing self-care, people being cared for at a distance with complex processes that are not bounded by the tyranny of geography or time, and extend beyond the patient-clinician relationship and a single organisation, to include multiple members of the multi-disciplinary team and the technology vendor.

Complex interventions require complex evaluations. Different methods can and should be used to evaluate interventions, and RCTs have a place, but do not sufficiently evaluate complex interventions, especially those that involve ICT. Lewin, Glenton and Oxman [12] conducted a systematic literature review of RCTs that included qualitative research, revealing that there is usually no overt link between the qualitative report and the RCT. The qualitative data is usually reported in the form of feasibility studies prior to an RCT. Campbell, Fitzpatrick, Haines, et al. [13] present a design for complex RCTs that includes a staged approach and collecting qualitative data, but does not address the difficulties associated with lack of blinding, and the rapid development of technologies. Mohr, Cheung, Schueller, et al. [7] propose a rapid cyclic form of RCT, but their method is limited to rapidly developing mobile technologies with high levels of sign-up and attrition. Murray, Teweek, Pope, et al. [14] propose that normalisation (i.e. an intervention becomes a normal part of everyday work) should be the goal of intervention evaluations such as RCTs. Their Normalisation Process Theory provides a framework for intervention designers to focus on implementation, such as context and change impact before a trial begins. Although they advocate for co-design, implementation considerations, and reflexivity (which are also AR features), once the trial begins these activities are set aside until the trial’s results are available. The AR feature of returning the results to the participants as part of the research is lacking.

Greenhalgh, Russel, Ashcroft, et al. [4] argue that ehealth evaluations require more than experimental cause and effect measurement because of the richness of what can be learned from the particular in complex situations. The MRC evaluation framework [11] has potential for thick description that can arise from AR. The MRC framework is a cycle consisting of feasibility study, evaluation of effectiveness and change process, implementation (implementation, surveillance and follow up), and development (theory, modelling and evidence base development). There are opportunities for gathering the rich data that falls around RCTs, but without the properties and processes of Action Research, these opportunities to apply learning from each step of the cycle and effect change can be lost.

Since telehealth interventions are complex, and although RCTs are increasingly built to accommodate complexity, the evaluation of these interventions requires a fuller, richer and therefore thick description of the intervention and its effects. The acquisition and use of evidence should be part of the research process, built in from the first design step, and fed back to practitioners at every stage of the research. We propose Action Research as a framework to address the limitations of RCTs in evaluating telehealth interventions.

Action research and the RCT

Action Research (AR) is an approach that can be used to frame research methods, adding dimensions of learning, developing and testing theory, and applying that learning in subsequent cycles [15]. AR is defined as

‘a participatory, democratic process concerned with developing practical knowledge in the pursuit of worthwhile human purposes….It seems to bring together action and reflection, theory and practice, in participation with others, in the pursuit of practical solutions to issues of pressing concern to people, and more generally the flourishing of individual persons and their communities.’ [16]

AR is usually described as a cyclic research process in which participants plan, act, reflect on the results of their actions, and modify the plan for the next cycle, repeated until they achieve their goal [17]. Research and action are conducted together, the research informing and informed by each cycle of action. All this is done in a mutually agreed upon ethical framework [18]. AR involves participants in defining a problem and designing and testing potential solutions, e.g. interventions, service improvements, trial methods. This is more than co-design and could provide useful input to build a strong RCT design that results in successful intervention implementation and return of research results to the involved community.

Reason [5], describes five characteristics of AR:

  1. 1.

    Create a communicative space at the beginning of a project. Participants are identified and agree to contribute to the project’s direction and content.

  2. 2.

    Iterative improvement is core to an AR project. It sets the tone for the change desired by the participants.

  3. 3.

    Emergence occurs as a result of informed action – as research produces knowledge and shared understandings, new (expected and unexpected) knowledge can be incorporated into subsequent iterations.

  4. 4.

    There are different ways of knowing, many perspectives. Each way of knowing is valid.

  5. 5.

    Democracy and participation are the foundation of AR.

Methods

The case: telemonitoring for long term condition management

We designed an RCT aimed at assessing the effects of telemonitoring on health-related quality of life, self-care, hospital use, costs and the experiences of patients, informal carers and health care professionals [19]. The RCT intervention consisted of telemonitoring of vital signs measured by participating patients plus patient answers to daily questions about symptoms. Data were collected in an information system, which the research nurses assessed and responded to according to an established clinical protocol as described by Kenealy et al. [19].

Participating patients in the intervention arm were given the equipment and trained on how to use it. The control patients continued to receive ‘usual care’ by their clinicians. There were three sites. Site A was urban and the patients had heart failure. Site B was also urban and the patients had Chronic Obstructive Airways Disease (COPD). Site A and B patients were recruited from hospital specialists. Site C was rural, and the patients were recruited from general practitioners, regardless of diagnosis. These patients were mostly diabetic.

The RCT participants fell into two groups as depicted in Table 2.

Table 2 RCT participants grouped as healthcare providers and consumers

During the planning phase of the RCT a small sub-group of the researchers negotiated the use of AR to frame this complex RCT. The goal was to use stages and sub-stages of the RCT plus the AR characteristics to enhance the implementation of the intervention and return the results of the study as soon as possible into practice. While it was considered good practice by the research team to negotiate the intervention design, the AR advocates were not able to negotiate the inclusion of potential RCT participants (i.e. specifically the patients/consumers) to co-define the research problem and participate in the co-design of the intervention. At the time it appeared appropriate to exclude potential RCT participants (consumers) because of the requirement to eliminate confounding factors and bias – inclusion was considered to undermine the RCT purpose, design and results. We did not manage to negotiate some of the funding to be focussed on the AR aspects of the fully funded RCT process and activities.

The academic who initiated the RCT project recruited clinicians, managers and community leaders from his extensive network to participate as co-researchers in the initial stages. There was a collaborator from each site – two were specialist clinicians and the third site was represented by a manager. Once the design had been established for the purpose of funding applications, these people became participants in the different sites that they belonged to, and are represented as such in Table 2. These three people remained in the AR team as well.

The research team was large and multidisciplinary. There were 13 academics from two faculties (Medical and Health Sciences, and Business), and covered the fields of medicine, nursing, health informatics, general practice, population health, and accounting and economics. A separate project team was established to support the funding applications, procurement of the telemonitoring equipment and information management system, and provide budget and project management services. For the purpose of this article, the AR team consisted of the academics, project manager, and site collaborators, totalling 17. The vendor supplied and supported the technology but did not become a member of the AR team as they were contracted to provide a service. The research was described and funded as an RCT, not an AR project, although we negotiated AR as a frame.

The research group became the AR group, and persisted for the duration of the research project, i.e. October 2008 until June 2012, as per Table 3. The RCT itself ran from September 2010 until August 2011.

Table 3 Research timeline and activities

In summary, three groups formed. The researcher group consisted of academics, some of whom were initially also clinicians from the different sites. Once the design was clarified, clinicians from Sites A and B shifted to the healthcare provider participant group, and the manager from Site C joined the researchers. The healthcare provider participant group was considered to be participants in that that they provided clinical services associated with the telemonitoring. They recruited patients, who made up the third group of consumer participants.

Results

AR as a frame

Action researchers use a cyclic approach – plan, act, reflect and modify the plan for future cycles, and then repeat the cycle [17]. The repeating cycles are not limited to repeated content. Cycles change as new aspects emerge due to learning from previous cycles. Our AR team applied the principles of the cyclic learning process to inform each step of the planning and execution of the RCT, as depicted in Fig. 1. Reflection, learning and improvement were the goals, including the capacity to adopt what was learned into practice. The overall research project was treated as a single AR cycle, with sub-cycles. The problem definition and intervention design were done by the research team. Inclusion of the site clinicians and mangers in the research team in the early stages of the overall project was deemed appropriate for problem co-definition and intervention co-design.

Fig. 1
figure 1

RCT framed by AR as a cyclic research process consisting of sub-cycles

Execution of the RCT then became a set of AR cycles as per Fig. 2. The implementation of each arm of the RCT at the three sites was each considered a sub-cycle, with learnings from the first sites (A and C) being implemented when the second site (B) was activated. Sites A and C were started in the same month. Learnings were exchanged between the sites until enrolments were standardised and initial problems had been resolved. Learnings from Sites A and C were then applied to site B when it joined the trial. The adjustments aimed to improve the RCT and intervention implementation, and did not result in material changes to the RCT design itself.

Fig. 2
figure 2

Site A, B, and C implementations as an AR cycle

As the overall research project progressed, and sub-cycles of the intervention were implemented, the degrees of distance between the research team and the research nurses executing the intervention increased, resulting in lost opportunities for AR characteristics and cycles to be applied. Since the AR aspect of the project was not funded, the discipline of reflexivity, documenting observations and lessons learned, and the contributions of participants, were not built into the research design or followed up. As the project progressed the commitment to AR reduced as the commitment to the RCT increased.

Meetings commenced in the first cycle (starting in 2008) and continued until after the study finished in June 2012. Reflection diaries [20] were not included in our project, although they are standard practice in AR. During this time the emphasis progressively shifted from planning the RCT to interpreting data, planning for the dissemination of findings, and future implementation of telehealth in New Zealand.

AR characteristics to co-design an RCT

Retrospectively we were able to see how the five characteristics of AR could be used in the design and execution of a telehealth RCT and the dissemination of the results.

  1. 1.

    Creating a communicative space to achieve co-design

Creating a communicative space within the research team occurred during the funding and ethics activities, and was sustained throughout the project in different ways, as per the objectives in Table 3. Funding and ethics approval applications required discussions and collaborative decision making and writing. Once funding had been approved a workshop was held to further develop relationships with a broad range of healthcare service leaders, academics and others who wanted to participate. This workshop attracted attendees from several sites all over New Zealand; not all attendees were able to commit their service to the project but wanted to participate in its planning.

Once the communicative space had been created the research and healthcare provider groups finalised which services would participate and under what circumstances, e.g., one urban site (Site A) wanted to include patients with heart failure, while a clinician from another urban site offered his COPD service (Site B). In contrast, the participants from the rural site were from a remote community, Māori, and had diabetes (Site C). Terms of engagement for Site C followed a collaborative and participative protocol that involved community elders, and a review of the research protocol prior to formal engagement in the project. This process leveraged the democracy and participation characteristic of AR by means of Kaupapa Māori [21]. Smith argues that “Kaupapa Māori is a 'local' theoretical positioning which is the modality through which the emancipatory goal of critical theory, in a specific historical, political and social context, is practised” [22]. Kaupapa Māori research reflects a Māori world view privileging Māori values in the interests of developing a research framework that is culturally safe. Kaupapa Maori research can strengthen community relationships, particularly when the research is driven by those who live in the community of interest. As Maori seek to be self-determining, Smith argues that such research allows for solutions and interventions that can change and improve lived realities.

The first meeting with aspiring research leaders and participants (the workshop) established the communicative space, although it was not a named AR workshop. Once the funding and ethics approval had been obtained, weekly and then fortnightly meetings among the researchers occurred. These meetings were a time for updating one another (communicative space), reflection on progress and problems (step 3 of the AR cycle), problem solving, decisions about adapting the research design and/or providing feedback to participants from the reflections, and determining next steps for the research to progress. During this time clinical processes were documented and modified for the purpose of implementing the telemonitoring intervention. These regular meetings of the broader research team discontinued once the RCT participants began being enrolled, and focussed irregular meetings occurred with a smaller team that was primarily focussed on the trial itself.

  1. 2.

    Different ways of knowing, many perspectives

The research group was a large team with 17 members who came from different research perspectives (positivist, interpretivist, critical). At times conflict arose from perceptions of incommensurability of these different perspectives, as well as interpersonal conflict. There was heated debate about contextual data collection and the value (or lack thereof) of qualitative data collection, e.g. patient perceptions of the technology. Stressors, such as research assistant turnover, the complex and complicated nature of the research, and differences in opinion, resulted in periodic interpersonal conflict, which was mostly resolved when participants were redirected to the common cause of their endeavours, i.e. the purpose of the research project.

  1. 3.

    Democracy and participation

Democracy and participation refers only to the research/AR group as described above. The researchers had different interests in the RCT and were asking different questions, e.g. consumer perspectives, technological aspects, costs and economics, management, clinical outcomes. Each researcher’s contribution was valued.

The leaders and managers in the remote site (Site C) took AR a step further, insisting on participative decision-making throughout the study, extending it to participating clinicians and patients using the principles of Kaupapa Māori Research [21]. As reported in [19] this rural community was “…an hour’s drive away from a regional city, in an impoverished area with a high proportion of indigenous Māori. Patients nearly all had type 2 diabetes and most had other long-term conditions. [Site C] contributed only intervention patients [i.e. no control patients], as the area is underserved and they wanted to use the opportunity to increase services for their patients and considered it unethical to enrol their patients into a control group.” Contrary to predictions that patient participation would undermine the RCT results, the differences between the sites and the lack of a control arm in the remote site did not appear to influence the trial results [19], but the outcomes of the opportunity to leverage AR practices and characteristics were limited to the site itself.

  1. 4.

    Constant iterative improvement

Constant iterative improvement was achieved by using AR cycles (Figs. 1 and 2). The project manager used the typical project management PDSA cycle (plan, do, study, act) [23], to ensure that this complex RCT was appropriately executed.

We were obliged to use the New Zealand government’s procurement process to secure a vendor for the telemonitoring equipment. This process was drawn out and complex, and one of the incidental outcomes of the research process was a contribution to the development of a simpler process, ‘active procurement’ [24]. The development of ‘active procurement’ could be attributed to the AR reflexivity that was being practiced in the early stage of the project. Such incidental outcomes (emergence) are common with AR [5].

We learned from each cycle and sub-cycle of the entire project. Refinement of clinical guidelines, improvement in the functioning of the technology, improvements as we progressed from one site implementation to the next, all produced learning that was used to inform the next stage of the project. Refinement of the RCT protocol and associated processes in Fig. 2 occurred in sub-cycles as each of the two urban sites began and completed their data gathering stages. Once a group of participants was locked into the RCT intervention and data gathering there was limited opportunity to use the AR cycle because these had to be consistent among participants. We used the AR cycle of learning and application but were only really successful in the first half of the research project.

  1. 5.

    Emergence

Emergence is to be expected in AR. The cyclical constant improvement approach enabled us to look for consequences and address any adverse events early. One should expect unintended consequences to emerge in RCTs, and more so in health information technology research [25]. We learned that the technology, despite being among the best available internationally at the time of selection, failed in a number of ways, e.g. lost data. Data quality can be compromised when patients are required to enter data themselves, rather than allow the system to automatically collect and transfer vital measurements. New processes of care emerged for remote patients (Site C) who otherwise would not have had access to service, but problems associated with distance and the need for additional support were also identified. An RCT focussed solely on the intervention may not have given sufficient space to identify these issues and address them and learn from action taken. Sites A and B started their version of the RCT at a different time from Site C (Table 3), taking advantage of what was learned from unintended consequences and continuous improvement emerging in previous months.

Successes and challenges of using AR to frame an RCT

Change is the main reason for AR. The desire for change arises from a critique of the current situation and the view that there could be something better in its place. This concurs with the motivation for RCT-based research – to identify and examine an intervention that has the potential to improve the health of individuals and/or communities. In answering the case’s research question our aim was to influence national policy on the development of telehealth as a valid, funded, and cost-effective choice for delivering health services that have a likelihood of improving health outcomes for people with chronic conditions. As the patients and their clinicians and service managers learned from their participation, processes and attitudes changed, and improvements occurred, thereby translating research into action. Many of the patients who participated were reluctant to return the telemonitoring equipment as they claimed it had made a difference in their lives. This raised an ethical debate about the consequences of withdrawing a research intervention when a project (and its funding) comes to an end.

We were new to applying AR characteristics and cyclic iterative learning to quantitative studies. At times this pluralism was natural and at other times the interpretivist nature of AR and positivist underpinning of RCTs seemed incommensurate, e.g. potential patient participants were excluded from co-definition of the problem and co-design of the intervention and RCT. For the most part the different approaches worked well together and offered opportunities for more rigour, stronger analysis of the data, and more useful findings. We did experience some challenges. Some of the researchers were strongly positivist, making it difficult for them to see value in what they saw as subjective, biased, and unreliable practices, e.g. positioning researchers inside the research and applying results as they become available. The positivist requirement to generalise findings that are bias free appears to be incommensurable with the interpretivist requirement for transferability and trustworthiness of findings. The main challenge was the lack of funding to build AR deliberately into the overall research project, ensuring that the discipline of practicing and documenting reflexivity was regularly practiced in meetings. The effects of adjusting the intervention during implementation (after learning from the implementation at other sites), and the effects of the research on the organisational context, could have been more rigorously investigated and reported. This could have informed future decisions about including possible patient participants in the early stages of the research. A significant challenge was the increasing degree of distance (and decreasing degree of influence) between the researchers and implementers of the trial, limiting the opportunities to document, analyse, share and apply any influence of AR practices on the trial.

The project leadership changed, resources became scarce for continued participation of academics, and the research/AR team shrank as researchers and collaborators turned to the demands of their daily work lives because the discipline of AR was not documented in the research project’s plan. The opportunities for AR contributions from those who initially negotiated AR as a frame became limited and as funding became more focussed on the RCT cycle, the Action Researchers withdrew.

Discussion

An RCT was conducted to assess the effect of telemonitoring on health-related quality of life, self-care, hospital use, costs and the experiences of patients, informal carers and health care professionals. AR was negotiated as a frame for the RCT and was conducted by Action Researchers new to using AR to frame an RCT, with mixed levels of commitment from the research project’s team members.

We were able to apply an AR cycle to the process of developing and executing an RCT design, and implement sub-cycles to each step of the overarching cycle (Figs. 1 and 2). Initially we aimed to only use the AR cycle but found ourselves using AR characteristics as well. The characteristics of AR were applied, especially in the early stages of the project. AR offers ways to deliberately and systematically collect data that is not usually collected in RCTs. It is interesting to note in our RCT report that although the RCT itself did not reveal a strong argument to implement tele-monitoring, the qualitative data that was collected revealed that tele-monitoring is a valuable intervention, especially from the patient’s point of view [19].

Why bother with AR? AR and RCTs complement one another in many ways, as can be seen in Table 1. We know that RCT is a good way of measuring the effect of a telehealth intervention, and we know how important it is to design an RCT well [26]. In the critical approach the researchers are saying that the situation under study should be better in some way, i.e. they critique the status quo, identify a vision for improvement and, using AR cycles and characteristics, improve the situation [27]. This could include an RCT to examine the effectiveness of their actions as a sub-cycle of an overall AR cycle [28]. On the other hand, there is a need to extend the scope of an RCT to include other data about the phenomenon being studied, e.g. to deliberately include the data that falls around an RCT that is often discarded due to concerns about bias and confounding factors [29]. In the interests of quality, the process of defining the scope and purpose of an RCT should be documented for other researchers to use: AR offers a rigorous approach to this documentation and process.

AR appears to be at odds with positivist research [28, 29], and this was a challenge at times during our project. Critical, interpretivist and positivist ways of knowing could be seen as mutually exclusive [28, 30]. They could complement one another, as natural responses to different types of knowing [8, 31]. Positivist research is good for identifying efficacy of clinical interventions where causal links are useful in evidence based care. A broad range of theories about telehealth (including tele-monitoring) are therefore embraced, e.g. socio-technical (e.g. technology acceptance model by Davis), health behaviour, and economics theories [32]. AR thus embraces different ways of knowing to enable multiple aspects of an intervention to be researched simultaneously, while also informing the actual implementation of the intervention in the context of an organisation [33]. Including patient participants in the co-definition and co-design of an RCT was initially seen to compromise the execution of the RCT. Instead of conducting a feasibility study, one could include those participants in the early stage of an RCT design process to strengthen the focus and applicability of the intervention and the RCT itself.

The inclusion of evidence from RCTs into evidence based care is challenging. Clinicians are unable to absorb the rising number of RCT publications, of which the quality is variable [34]. Furthermore, there is a time lag between the performance of an RCT, its publication and the use of its results in everyday clinical practice, further complicated by the slow moving organisational and regulatory environment in which healthcare is practiced [29, 34]. AR offers opportunities to initiate change where change can be implemented as the result of incremental, cyclic improvements. Formative evaluation that is concurrent to an RCT offers opportunities for change in medical practice that are not available in the summative orientation of the RCT process itself [29].

Formative and summative evaluation of a tele-monitoring intervention, framed by AR, and involving Information and Communications Technology (ICT), implies a form of ‘action design research’ [35]. This approach aims to meet the needs of practitioners by designing an ICT artefact for organisational implementation, while simultaneously conducting theoretical research using formative and summative research methods. ICT is not an intervention in the same way as a medication or surgical procedure is; it is an ‘ensemble artefact’ incorporating clinical and organisational processes and structures into the technology [35]. Consequently, when measuring the tele-monitoring impact on clinical outcomes for people with long term conditions the research data generated represents the embedded processes and structures as well as cause and effect links.

An RCT is not enough to demonstrate whether the ‘intervention’ works or not [36]. It is not enough to design complex RCTs as proposed by Campbell et al. [13], or to create continuous RCTs to accommodate rapid technological changes as proposed by Mohr et al. [7]. It is not enough to conduct a feasibility study or to simply include qualitative analyses arising from an RCT [12]. We propose that the AR process and characteristics enable telehealth researchers to evaluate formatively and summatively, and enable research teams and collaborators to bring about change as and when opportunities arise without compromising the quality of an RCT.

AR as a frame for planning and executing RCTs offers a structure, processes and characteristics. Researchers with different takes on the research question are able to explore their different ways of knowing, e.g. organisational researchers using social sciences, technology researchers using socio-technical theory and clinicians using positivism to establish causal effects. Action research and positivism both aim to improve a situation. At times AR and RCTs are at odds with one another, but with AR as the overarching frame that allows for democratic planning (and permits different types of participants), responsiveness to emergence at appropriate times, and iterative learning and emergence, then the data that falls around an RCT can, and should, be used purposively.

Conclusion

Our research team agreed to use AR to frame an RCT to assess the effect of tele-monitoring on health-related quality of life, self-care, hospital use, costs and the experiences of patients, informal carers and health care professionals. We were new to the idea of deliberately framing an RCT as an AR project and we learnt how to work together within a communicative space to build an RCT that reflected different ways of knowing. It was more than a mixed methods study. Our study accommodated theoretical pluralism (although challenging at times, especially when the different approaches appeared to be philosophically incommensurable, and when funding limited our capacity to use the discipline associated with AR) to enable researchers representing different ways of knowing. At times doing an RCT framed by AR seemed clumsy, but the way the cycles in Figs. 1 and 2 worked meant that certain ways of knowing dominated depending on the purpose of that cycle.

We propose that the cyclic framework plus the characteristics of AR should be used in the planning, execution and close-off of RCTs to enable stronger design, richer data collection and earlier adoption of evidence into practice for patients, clinicians, the multi-disciplinary team, investors, and improvement of the healthcare environment [37]. AR and RCTs share a purpose – to make improvements. AR enables clinicians and other healthcare practitioners to contribute to research and return the results into everyday practice. Based on our experience, we recommend that all RCTs could be framed as an AR project to enhance the quality, value, and impact of the research on everyday health care.

Abbreviations

AR, action research; COPD, chronic obstructive pulmonary disease; ICT, information and communication technologies; PDSA, Plan, Do, Study, Act (cycle); RCT, randomised controlled trials

References

  1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. Br Med J. 1996;312(7023):71.

    Article  CAS  Google Scholar 

  2. Lewis MW, Kelemen ML. Multiparadigm inquiry: Exploring organizational pluralism and paradox. Hum Relat. 2002;55:251.

    Article  Google Scholar 

  3. Geertz C. Thick description: Toward an interpretive theory of culture. In M. Martin & L.C. McIntyre (eds.). Readings in the philosophy of social science. Cambridge, Mass: MIT Press; 1994. pp. 213–31.

  4. Greenhalgh T, Russel J, Ashcroft RE, Parsons W. Why National eHealth Programs Need Dead Philosophers:Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History. Milbank Q. 2011;89(4):533–63.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Reason P. Choice and quality in action research practice. J Manag Inq. 2006;15(2):187–203.

    Google Scholar 

  6. Myers MD. Qualitative research in information systems. MIS Quarterly. 1997;June:241–2.

  7. Mohr DC, Cheung K, Schueller SM, Brown CH, Duan N. Continuous evaluation of evolving behavioral intervention technologies. Am J Prev Med. 2013;45(4):517–23.

    Article  PubMed  Google Scholar 

  8. Liu JL, Wyatt JC. The case for randomized controlled trials to assess the impact of clinical information systems. JAMIA. 2011;18:173–80.

    PubMed  PubMed Central  Google Scholar 

  9. Kaldoudi E, Chatzopoulou A, Vargemezis V. Adapting the STARE-HI guidelines for the evaluation of home care telehealth applications: An interpretive approach. J Inf Technol Healthc. 2009;7(5):293–303.

    Google Scholar 

  10. Ekeland AG, Bowes A, Flottrop S. Methodologies for assessing telemedicine: A systematic review of reviews. Int J Med Inform. 2012;81:1–11.

    Article  PubMed  Google Scholar 

  11. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi:10.1136/bmj.a1655.

  12. Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694–6.

  14. Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8(1):1.

    Article  Google Scholar 

  15. Reason P. Integrating action and reflection through co-operative inquiry. Manag Learn. 1999;30(2):207–26.

    Article  Google Scholar 

  16. Reason P, Bradbury H. Handbook of action research: Participative inquiry and practice. London: Sage; 2001.

  17. Dick B. Action research: Action and research. In: Shankar S, Dick B, Passfield R, Swepson P, editors. Effective change management using action learning and action research. Lismore, NSW: Southern Cross Press; 2002.

    Google Scholar 

  18. Rapoport RN. Three dilemmas in action research with special reference to the Tavistock experience. Human Relations. 1970;23(6):499–513.

    Article  Google Scholar 

  19. Kenealy TW, Parsons MJ, Rouse APB, Doughty RN, Sheridan NF, Hindmarsh JKH, et al. Telecare for Diabetes, CHF or COPD: Effect on Quality of Life, Hospital Use and Costs. A Randomised Controlled Trial and Qualitative Evaluation. PloS one. 2015;10(3).

  20. Baskerville RL, Wood-Harper AT. A critical perspective on action research as a method for information systems research. J Inf Technol. 1996;11(3):235–46.

  21. Kerr S, Penney L, Moewaka Barnes H, McCreanor T. Kaupapa Maori action research to improve heart disease services in Aotearoa, New Zealand. Ethn Health. 2010;15(1):15–31.

    Article  PubMed  Google Scholar 

  22. Smith LT. Decolonizing methodologies: Research and indigenous peoples. New York: Zed Books & Dunedin: University of Otago Press; 1999.

    Google Scholar 

  23. Zhang W, Xu X. Six Sigma and information systems project management: A revised theoretical model. Proj Manag J. 2008;39(3):59–74.

    Article  Google Scholar 

  24. Pollock M, Warren J, Gu Y, editors. An approach to active procurement. HINZ 2011 Conference and Exibition 2011; Rotorua, New Zealand. Available from https://www.yumpu.com/en/document/view/37198334/an-approach-to-active-procurement-health-informatics-new-zealand. Accessed on 25 Feb 2016.

  25. Ash J, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Inform Assoc. 2004;11:104–12.

    Article  Google Scholar 

  26. Augestad KM, Bernsten G, Lassen K, Bellika JG, Wootton R, Lindsetmo RO, et al. Standards for reporting randomized controlled trials in medical informatics: A systematic review of CONSORT adherence in RCTs on clinical decision support. J Med Inform Assoc. 2012;19(1):13–21.

    Article  CAS  Google Scholar 

  27. Baskerville R, Wood-Harper AT. A critical perspective on action research as a method for information systems research. In: Myers MD, Avison D, editors. Qualitative Research in Information Systems. London: SAGE Publications; 2002.

    Google Scholar 

  28. Susman GI, Evered RD. An assessment of the scientific merits of action research. Adm Sci Q. 1978;582–603.

  29. Scriven M. A summative evaluation of RCT methodology: And an alternative approach to causal research. J Multidiscip Eval. 2008;5(9):11–24.

    Google Scholar 

  30. Klein HK, Myers MD. A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Q. 1999;23(1):67–94.

    Article  Google Scholar 

  31. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health. 2003;57(7):527–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Gammon D, Johannessen LK, Sørensen T, Wynn R, Whitten P. An overview and analysis of theories employed in telemedicine studies. Methods Inf Med. 2008;47:260–9.

    CAS  PubMed  Google Scholar 

  33. Leykum LK, Pugh JA, Lanham HJ, Harmon J, McDaniel Jr RR. Implementation research design: integrating participatory action research into randomized controlled trials. Implement Sci. 2009;4:69.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Heneghan C, Godlee F. Where next for evidence based healthcare? BMJ. 2013;346.

  35. Sein MK, Henfridsson O, Purao S, Rossi M, Lindgren R. Action design research. MIS Q. 2011;35(1):37–56.

    Google Scholar 

  36. Steventon A, Bardsley M, Billings J, Dixon J, Doll H, Hirani S, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ. 2012;344, e3874.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Orr M, Day K. Knowledge and learning in 'successful' IT projects: a case study. Health Care and Informatics Review Online [Internet]. 2004 (17/06/2004). Available from: http://c.ymcdn.com/sites/www.hinz.org.nz/resource/collection/0F09C2E4-7A05-49FB-8324-709F1AB2AA2F/Knowledge_And_Learning_In_Successful_IT_Projects_Martin_Orr_and_Karen_Day_June_2004.pdf. Accessed 25 Feb 2016.

Download references

Acknowledgements

We wish to thank those involved in the RCT from the different services involved, the university researchers, the project team, and Uniservices, upon which this article is based.

Funding

The RCT was funded by the New Zealand Ministry of Health and District Health Boards New Zealand. This report is independent research arising from reflection on the methodology by the three authors and was not funded.

Availability of data and materials

This article is a reflection on the use of Action Research as part of the RCT reported in Kenealy, T. W., Parsons, M. J., Rouse, A. P. B., Doughty, R. N., Sheridan, N. F., Hindmarsh, J. K. H.,…Rea, H. H. (2015). Telecare for Diabetes, CHF or COPD: Effect on Quality of Life, Hospital Use and Costs. A Randomised Controlled Trial and Qualitative Evaluation. PLOS One, 10 [3]. Associated data and materials are available as part of the PLOS One publication.

Authors’ contributions

The idea for the paper was originally developed by KD and TK. TK analysed and reported the RCT used in the case study central to this paper. KD, TK and NS drafted the manuscript, revised the paper critically and approved the final version.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Consent was granted as part of the consenting process as per the ethics approval.

Ethics approval and consent to participate

Ethics permission for the RCT was granted by the New Zealand Ministry of Health Multi-region Ethics 83 Committee, reference MEC/10/04/031.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Karen Day.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Day, K., Kenealy, T.W. & Sheridan, N.F. Should we embed randomized controlled trials within action research: arguing from a case study of telemonitoring. BMC Med Res Methodol 16, 70 (2016). https://doi.org/10.1186/s12874-016-0175-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-016-0175-6

Keywords