Table 1 Critical appraisal of the synthesis paper as a meta-narrative review

1. In the title, identify the document as a meta-narrative review or synthesis.

0, 3, 0, 2

Called a ‘systematic review and synthesised framework’ rather than a meta-narrative review or synthesis.

2. While acknowledging publication requirements and house style, abstracts should ideally contain brief details of: the study’s background, review questions or objectives; search strategy; methods of selection, appraisal, analysis and synthesis of sources; main results; and implications for practice.

1, 2, 1, 3

Most required elements are present but no research question stated and no mention of any of the methodological approaches outlined for meta-narrative review in the RAMESES publication standards.

3. Rationale for review: explain why the review is needed and what it is likely to contribute to existing understanding of the topic area.

1, 2, 2, 3

Rationale for doing a systematic review given but no mention of meta-narrative review in the rationale.

4. Objectives and focus of review: state the objective(s) of the review and/or the review question(s). Define and provide a rationale for the focus of the review.

0, 2, 1, 1

The broad objective of the review is stated but the review question(s) are never explicitly stated.

5. Changes in the review process: any changes made to the review process that was initially planned should be briefly described and justified.

0, 0, 0, 0

No mention made of any changes in the review process.

6. Rationale for using meta-narrative review: explain why meta-narrative review was considered the most appropriate method to use.

0, 1, 1, 1

The first mention of the employment of the meta-narrative approach is very late – under the Analysis section, and the approach is broadly referenced rather than rationalized. Inadequate mention is made of the research tradition and epistemological synthesis and critique etc., and the attempt to build an overarching narrative which characterize meta-narratives.

7. Evidence of adherence to guiding principles of meta-narrative review: where appropriate, show how each of the six guiding principles (pragmatism, pluralism, historicity, contestation, reflexivity and peer review) has been followed.

0, 0, 0, 0

No evidence of adherence to guiding principles of meta-narrative review and no mention of any of the guiding principles.

8. Scoping the literature: describe and justify the initial process of exploratory scoping of literature.

1, 2, 0, 3

There is evidence of an initial systematic review of the literature which can be considered a scoping exercise. This is coherent and covers a good range of databases. However, this is not identified as a scoping of the literature preliminary to a meta-narrative review.

9. Searching processes: while considering specific requirements of the journal or other publication outlet, state and provide a rationale for how the iterative searching was done. Provide details on all the sources accessed for information in the review. Where searching in electronic databases has taken place, the details should include (for example) name of database, search terms, dates of coverage and date last searched. If individuals familiar with the relevant literature and/or topic area were contacted, indicate how they were identified and selected.

2, 1, 1, 1

Even as a traditional systematic review this was imperfect, for example, there is no clear statement of the review question or search terms. Other aspects of a traditional systematic review are adequately described.

10. Selection and appraisal of documents: explain how judgements were made about including and excluding data from documents, and justify these.

2, 1, 1, 0

Inclusion and exclusion criteria for the systematic review are not detailed sufficiently and there is insufficient clarity on sources used. Methods for data extraction are unclear, there is little information available regarding databases used, search terms, dates of coverage and so on.

11. Data extraction: described and explain which data or information were extracted from the included documents and justify this selection.

2, 0, 2, 1

Data extraction is described but not justified. Study selection section provides very little justification. Team disagreements leading to inclusion and further scrutiny is the only example of this.

12. Analysis and synthesis processes: describe the analysis and synthesis processes in detail. This section should include information on the constructs analysed and describe the analytic process.

0, 0, 0, 0

A meta-narrative approach is claimed but no details are given on data analysis and synthesis. The paper does not include an exploration of different traditions and constructs, but instead appears to synthesize different components and processes related to PSUE outlined in the literature reviewed, and their inter-relations.

13. Document flow diagram: provide details on the number of documents assessed for eligibility and included in the review with reasons for exclusion at each stage as well as an indication of their source of origin (for example, from searching databases, reference lists and so on).

2, 1, 2, 2

A document flow diagram is included which shows a reduction from 202 studies in the systematic review to 41 studies included in synthesised framework. The text explains the rationale for why the 41 were included and the others excluded. Little other detail is given at this stage, for example their source of origin. Most importantly, there is no list or table of the 41 included studies so the authors’ decision-making cannot be examined or tested.

14. Document characteristics: provide information on the characteristics of the documents included in the review.

0, 0, 0, 2

No details are given on the 41 studies included in the analysis.

15. Main findings: Present the key findings with a specific focus on theory building and testing.

2, 2, 1, 1

The four components are presented clearly but where the different phases are introduced the focus on theory building and testing becomes unclear.

16. Summary of findings: summarise the main findings, taking into account the review’s objective(s), research question(s), focus and intended audience(s).

2, 2, 0, 1

The findings are potentially interesting as it is the first (to our knowledge) attempt to provide a synthesis of conceptual frameworks for PPI based on a comprehensive review of other frameworks; however the lack of a clear statement of the review’s objectives and research question, make it difficult to judge how well they have achieved their objectives.

17. Strengths, limitations and future research directions: discuss both the strengths of the review and its limitations. These should include (but need not be restricted to) (a) consideration of all the steps in the review process and (b) comment on the overall strength of evidence supporting the explanatory insights which emerged. The limitations identified may point to areas where further work is needed.

1, 1, 2, 2

The main strength of the review is that they have produced a conceptual framework for PPI based on a systematic review and a synthesis of 41 other conceptual papers. Many other conceptual frameworks for PPI appear to have little or no prior review of previous conceptualisations.

The main weaknesses/limitations of the review are its many methodological inconsistencies and omissions, and in particular that the authors do not appear to have conducted anything that can be recognised as the meta-narrative approach they claim. Most of the stages of a meta-narrative review are either not present or only minimally present. There is not enough detail on the analysis to judge the validity of the framework of four components for PPI arrived at.

18. Comparison with existing literature: where applicable, compare and contrast the review’s findings with the existing literature (for example, other reviews) on the same topic.

2, 1, 1, 1

The ‘Comparison with other systematic reviews’ section (p.1160) carries out some of the necessary comparative critique but is tied to processes, not paradigms or research tradition analysis, and therefore does not meet the requirements of meta-narrative synthesis.

19. Conclusion and recommendations: list the main implications of the findings and place these in the context of other relevant literature. If appropriate, offer recommendations for policy and practice.

1, 1, 1, 1

The findings are compared with other systematic reviews but not placed in context of other conceptual frameworks. Authors claim their work provides “a framework with broad applicability and cohesive underpinnings necessary to integrate existing knowledge and guide future endeavours” but due to its significant methodological weaknesses we do not consider their framework to be robust.

20. Funding: provide details of funding sources (if any) for the review, the role played by the funder (if any) and any conflicts of interests of the reviewers.

3, 2, 3, 2

Partial funding source given (Patient-Centered Outcomes Research Institute) but no information on rest of funding.

Overall

1.15

This paper does not fulfil the criteria for a meta-narrative review as set out in the Rameses publication standards. Overall methodological quality is poor, and the conceptual model of PPI presented is therefore not well supported.