The generation of consensus guidelines for carrying out process evaluations in rehabilitation research

Background Although in recent years there has been a strong increase in published research on theories (e.g. realist evaluation, normalization process theory) driving and guiding process evaluations of complex interventions, there is limited guidance to help rehabilitation researchers design and carry out process evaluations. This can lead to the risk of process evaluations being unsystematic. This paper reports on the development of new consensus guidelines that address the specific challenges of conducting process evaluations alongside clinical trials of rehabilitation interventions. Methods A formal consensus process was carried out based on a modified nominal group technique, which comprised two phases. Phase I was informed by the findings of a systematic review, and included a nominal group meeting with an expert panel of participants to rate and discuss the proposed statements. Phase II was an in depth semi-structured telephone interviews with expert panel participants in order to further discuss the structure and contents of the revised guidelines. Frequency of rating responses to each statement was calculated and thematic analysis was carried out on all qualitative data. Results The guidelines for carrying out process evaluations within complex intervention rehabilitation research were produced by combining findings from Phase I and Phase II. The consensus guidelines include recommendations that are grouped in seven sections. These sections are theoretical work, design and methods, context, recruitment and retention, intervention staff, delivery of the intervention and results. These sections represent different aspects or stages of the evaluation process. Conclusion The consensus guidelines here presented can play a role at assisting rehabilitation researchers at the time of designing and conducting process evaluations alongside trials of complex interventions. The guidelines break new ground in terms of concepts and theory and works towards a consensus in regards to how rehabilitation researchers should go about carrying out process evaluations and how this evaluation should be linked into the proposed trials. These guidelines may be used, adapted and tested by rehabilitation researchers depending on the research stage or study design (e.g. feasibility trial, pilot trial, etc.). Electronic supplementary material The online version of this article (10.1186/s12874-018-0647-y) contains supplementary material, which is available to authorized users.

Complex interventions are defined as those made up of a number of components or active ingredients that interact with each other and with outside factors to bring about changes to outcomes. It is important to be clear regarding what is 'complicated' and what is 'complex': complicated problems are formed of a number of parts that can be solved and their functioning can be predicted by using identified formulae and instructions; complex problems however rarely benefit from these tools, since they are uncertain. Complex problems are solved allowing time for learning about each component and for making sense of events taking place. Thus, the evaluation of complex interventions represents a great challenge since their path to success is variable and cannot be accurately predicted. Crucially, the difficulty in defining in detail rehabilitation treatments in terms of what are their 'active ingredients' and what is their impact is very challenging. Most of rehabilitation interventions will have several active ingredients.
Complex rehabilitation interventions can often be: -Offered multiple times to multiple participants that can belong to a number of different groups. -Complex behavioural treatments to the contrary of passive or surgical treatments.
-Delivered in a personal way where interactions therapist/patient play a significant role.
-Tailored to patient's needs at the time of defining goals or treatment plans.
-Designed in a number of sessions to allow time for individuals to learn and comprehend its content. -Delivered in different locations and sites which can change.
-Delivered to individuals who are not 'passive recipients' of the intervention, individuals who will perceive and take on board the intervention in their own unique manner (for 4. There should be a clear explanation of how the methodological guidance is applied to the process evaluation (e.g. if a guidance is chosen it is necessary to clearly explain how was the guidance followed and how did the process evaluation remain in line with the guidance's proposed frameworks/steps) Disagree Agree 1 2 3 4 5 6 7 8 9 example individuals will decide how intensively they want to get involved in the intervention) -Furthermore rehabilitation research is often context specific and defined as the interaction between the individual and the environment. In other words, identifying contextual processes (physical, psychological, social, etc.) and acknowledging that researchers bring their values into situations is of great importance when thinking about the science of rehabilitation. Therefore, researchers working in this field need to design strategies and ways to explore and measure context.
It is feasible to describe an intervention in terms of its 'active ingredients'. However, throughout the research process the intervention should be seen as a whole which is greater than the sum of its parts. Reducing the complex intervention to a number of components and understanding how these work individually might make the intervention loose its essence. Understanding how parts of the intervention work should always be considered in close relation to how the intervention works as a whole.

Theories and rehabilitation
Rehabilitation professionals share assumption regarding firstly, the nature of their work: they need to be apolitical, relevant and useful. Secondly, the nature of their goals: to increase function, independence and quality of life and finally, the nature of the relationship with the client, which has to be holistic and client-centred. The problem is that these theoretical assumptions so far lack in evidence base support. Many areas of rehabilitation are underdeveloped from a theoretical perspective and energy should be invested, as it is spend in empirical research, in developing well-articulated theories and consequent theoretical models. The theory behind the structure and delivery of a proposed rehabilitation intervention will need to reflect its complexity and address it.
What is the 'theory of change' behind the proposed rehabilitation intervention? How many theories are needed to guide rehabilitation research, or should there be an overarching one? Many theories appear relevant to rehabilitation, for example learning theories, theories of goal setting, theories related to self-management and also theories looking at changes at the person-environment interface such as theories of diffusion of innovation.

5.
The organizational context prior to the intervention being implemented should be clearly described through the use of both qualitative and quantitative methods.

Regarding context
It is of vital importance for researchers to acknowledge the vital role that context plays in explaining how interventions work. Context can be described as all surrounding systems in which the intervention is embedded. In other words, context is involved not only with the surrounding environment (e.g. institutions, organizations) but also their culture in terms of social behaviours, interactions amongst members and individual perceptions and preconceptions.
Complex rehabilitation interventions will be determined and embedded in a context which will not remain passive but will change with time. For example: -It can be said that often rehabilitation interventions will be politically determined (e.g. a government accepting or rejecting national service frameworks).
-In rehabilitation interventions the quality and characteristics of the interactions between the patient and the health professionals can play a major role in shaping their success or failure (e.g. If an OT is not able to build rapport with a patient the level of engagement and motivation of both, patient and OT, most likely will be affected). For example: -At present rehabilitation practice lacks of a nationally standardized and accepted set of outcome measures, therefore researchers often have to use and individual and trial/specific screening tools in order to identify and assess the suitability of the participants. Having a recruitment criteria that is therapeutically based is a more complicated procedure and therefore is more expensive and time consuming that the recruitment process in medical trials who often utilise a simple chart review.
-Rehabilitation researchers often have to give special attention to retention due to the nature of the patients, for example, their recruitment budget will often need to include cost of participants transportation to and from the research base or 'reminding methods' such as postcards or phone calls.
-It is often hard to reach patients who are not registered as being part of rehabilitation services. The recruiting effort will be considerable and often needs to use other alternative sources and venues which can be time consuming and costly.
-Recruiting effort will need to account for the characteristics of this group of service user who will often have mobility and/or cognitive difficulties which might have led to limited social involvement and very little time spent out of his/her home.

11.
Strategies to recruit participants to the process evaluation should be clearly described. Neurological rehabilitation interventions often require a level of skill and understanding of different techniques and methods. They often involve treating complex patients with complex needs. Thus, it is vital to have a good understanding of the characteristics of all staff responsible for delivering the neurological rehabilitation intervention. Staff's previous experience and level of skill will potentially have an impact on the way the intervention is being delivered and also on the way this intervention will bring about changes to outcomes.

Description of intervention
22. The study intervention should be detailed in a protocol/manual.

Tailoring rehabilitation interventions
The 'science of replication' in rehabilitation research requires further development.
There is more to delivering a rehabilitation intervention than just measuring how many elements were delivered. Rehabilitation research should avoid a 'cookbook' approach if it intends to understand the vital role played by contextual factors. As a result there is an increased awareness of the need to tailor rehabilitation interventions to patients' needs and cultural background in order to increase their potential to be effective. To be able to replicate a rehabilitation intervention across different settings it will be necessary to adapt it (tailor it) to some extent and this is likely to create tension between the need to tailor and the need to maximise treatment integrity.
Tailoring should not mean intervention staff 'improvising as they go along', it should mean that what is standardized will be contrasted and clearly defined and monitored against what is customized. As a result, the assessment of how the rehabilitation intervention was administered according to the plan will have to be standardized and tailored to the actual level of standardization and tailoring of the trialled intervention. Succeeding at this can be extremely challenging for rehabilitation researchers. A first vital step could involve identifying and recording the delivery of unplanned components (for example using specific recording sheets). This information can help for example, to identify which aspects need to be included in the re-training of intervention staff on the requirements to follow the protocol. It can also help identify aspects of the intervention which need modifying.

Preparing and assessing intervention staff
27. The training provided to intervention staff involved in the research should be clearly described (e.g. details on when and where will the training take place, who needs to attend, who will deliver it, etc.)

Training staff to provide a rehabilitation intervention
It is widely accepted that training the staff responsible for the implementation of the trialled rehabilitation intervention is beneficial: -Through training and supervision you can refine the work of the providers who in most cases will already have experience in this trialled intervention.
-The training can help teach the provider to not use their usual approaches if they are not part of the intervention -staff should familiarize themselves with the trial's manual/protocol during the training.
-Training provides a chance to discuss the philosophy underlying the intervention.
-Training will give a chance to intervention providers to practice the necessary skill set.
In rehabilitation trials it should be feasible to assess professionals' skills prior to the start of the trial. However, training staff involved in rehabilitation trials is often ongoing in order to assure that skills are maintained over time. In such cases an initial skill assessment could not be used as a basis for participation but regular/periodical assessments could be the solution. Staff involved in delivering a rehabilitation intervention will learn overtime and they will become more familiarized with the techniques, patient characteristics, organizational contexts etc. Therefore, investigating staff learning curves throughout the trial and how these might explain trends in outcomes would be highly beneficial. It is equally important to have measures in place to assess how intervention staff have maintained skills over time. Whilst for example in drug trials the delivery of the intervention is relatively simplistic in the case of rehabilitation interventions it often is not. For example, in the case of rehabilitation the accurate delivery of the intervention can be highly dependent on for example: -The level of skill, previous experience and knowledge of the intervention staff -possible biases and previous experience can influence or clash with intervention -Individual characteristics of patients beyond the intervention (e.g. depression, cognitive impairment, acceptance or attitude towards the intervention, personal factors, geographical factors, etc.). Heterogeneity of trial participants will be likely even after detailed screening according to inclusion and exclusion criteria.
-The difficulty with blinding, participants will know, in most cases, the intervention they are receiving.
-Difficulty with assessing participants understanding of the purpose of the intervention (for example when participants have some level of cognitive impairment which is often the case in rehabilitation research).

38.
Process evaluations should clearly define quantitative and qualitative indicators that reflect acceptable quality in the delivery of the study intervention. Understanding if the intervention has been carried out as initially planned can therefore prove both, very challenging and highly dependent on the quality and level of detailed information included in the plan to execute procedures and assessment. Strategies in order to address this need to be clearly described and in place throughout the research trial.