Skip to main content

Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study

Abstract

Background

Many clinical practice guidelines are based on randomised controlled trials conducted in secondary or tertiary care setting and general practitioners frequently question their relevance for primary care patients. Our aim was to compare the intervention effect estimates between primary care setting randomised controlled trials (PC-RCTs) and secondary or tertiary care setting randomised controlled trials (ST-RCTs).

Methods

Meta-epidemiological study of meta-analyses (MAs) of a binary outcome including at least one PC-RCT and one ST-RCT. PC-RCTs were defined as trials recruiting patients in general practices, primary care practices, family practices, community centers or community pharmacies. ST-RCTs were defined as trials recruiting in hospitals, including hospitalized patients, hospital outpatients and patients from emergency departments. For each MA, we estimated a ratio of odds ratio (ROR) by using random-effects meta-regression, with an ROR less than 1 indicating lower estimates of the intervention effect in PC-RCTs than ST-RCTs. Finally, we estimated a combined ROR across MAs by using a random-effects meta-analysis. We performed subgroup analyses considering the type of outcomes (objective vs subjective), type of experimental intervention (pharmacological vs non-pharmacological), and control group (active vs inactive) as well as analyses adjusted on items of the Cochrane Risk of Bias tool.

Results

Among 1765 screened reviews, 76 MAs with 230 PC-RCTs and 384 ST-RCTs were selected. The main medical fields were pneumology (13.2%) and psychiatry or addictology (38.2%). Intervention effect estimates did not significantly differ between PC-RCTs and ST-RCTs (ROR = 0.97, 95% confidence interval 0.88 to 1.08), with moderate heterogeneity across MAs (I2 = 45%). Subgroup and adjusted analyses led to consistent results.

Conclusion

We did not observe any significant difference in intervention effect estimates between PC-RCTs and ST-RCTs. Nevertheless, most of the medical fields in this meta-epidemiological study were not representative of the pathologies encountered in primary care. Further studies with pathologies more frequently encountered in primary care are needed.

Peer Review reports

Background

Many studies have highlighted the lack of adherence to clinical practice guidelines by general practitioners (GPs) [1, 2]. GPs generally argue a lack of relevance or difficulties in applying guidelines [3, 4], and they particularly express concerns about the applicability of clinical trial results used to establish guidelines to their own patients [5, 6]. The first reason is that many randomised controlled trials (RCTs) include highly selected patients, which restricts the generalizability of their results [7]. As an example, a study found that less than half of primary-care patients would have been included in hypertension trials [8]. The second, highly related to the first one reason is that in most RCTs, patients are recruited in secondary- or tertiary-care settings [9] rather than in primary-care. However, as compared with secondary- or tertiary-care setting patients, primary-care patients have less severe disease and more undifferentiated symptoms [10] but more multi-morbidity [11]. Such differences may lead to differences in benefits and harms of the intervention. Our hypothesis is that intervention effect estimates differ between primary care setting randomised controlled trials (PC-RCTs) and secondary or tertiary care setting RCTs (ST-RCTs).

We performed a meta-epidemiological study to assess whether intervention effect estimates differ between PC-RCTs and ST-RCTs.

Methods

This was a meta-epidemiological study, a study design used to compare intervention effect estimates between trials with and without a characteristic of interest [12]. In this study, we focused on the setting, with the aim of comparing PC-RCTs to ST-RCTs. A meta-epidemiological study is performed by generally using a two-step approach and, contrary to traditional epidemiological studies, units of analysis are studies rather than patients [13]. First, for each selected MA, we assessed the difference in intervention effects between studies with the characteristics of interest (i.e., primary care setting) and without the characteristics of interest. This step involved using a meta-regression for each selected MA (with the intervention effect considered the outcome and the characteristic of interest the independent variable). Second, results from the meta-regression were meta-analysed over the different MAs.

Search strategy

On July 9, 2020, we searched the Cochrane Database of Systematic Reviews to retrieve all systematic reviews with meta-analyses (MAs) that were published, with no restriction on time, by using the following text words in the full text: “primary care” OR “primary healthcare” OR “general practice” OR “family practice”. We only used primary care keywords to ensure that we had PC-RCTs in each MA.

Selection of relevant MAs and trials

We screened the full text of potentially eligible systematic reviews to select MAs of binary outcomes that had at least 3 trials (3 studies is the minimum to perform a meta-regression) and at least one PC-RCT and one ST-RCT. PC-RCTs were defined as trials recruiting patients in general practices, primary care practices, family practices, community centers or community pharmacies according to the definition by Afonso et al. [14]. Trials in nursing homes or at home were not considered PC-RCTs and were excluded. ST-RCTs were defined as trials recruiting in hospitals, including hospitalized patients, hospital outpatients and patients from emergency departments. Hospital-at-home trials were not considered ST-RCTs and were excluded. Trials with both primary and secondary or tertiary care settings were excluded, as were trials with an unclear setting or trials including patients in other than primary or secondary or tertiary care settings, such as schools. For this study, we selected only RCTs and excluded non-randomised or quasi-randomised trials.

If more than one MA was eligible within the same review, we selected the MA for the primary efficacy outcome, then the MA with the highest number of trials. MAs of adverse events were not included because of the uncertainty in the direction of bias. We also discarded MAs when it was impossible to determine which group was the experimental and control group.

When a trial was included several times within the same MA, we kept only the duplicate with the largest sample size. When a trial was included in several selected MAs, we kept the one in the most recent systematic review.

All this selection process was performed independently by two reviewers (A.D., A.H.), with disagreements resolved by discussion, referring to a third opinion (C.D.D.) when necessary.

Data collection

Two independent reviewers (A.D., A.H.) collected data from all selected MAs and trials by using a standardised data collection form. Disagreements were resolved by discussion, referring to a third opinion (C.D.D.) when necessary. The following characteristics were extracted:

For each meta-analysis:

  • General information: year of publication, first author name

  • Medical field

  • Type of intervention (pharmacological, non-pharmacological), type of comparator (active control or inactive control, with inactive control defined as placebo or no added intervention to usual care, sham or other)

  • Outcome and whether it was subjective or objective following the classification provided by Savović et al. [15]

    • All-cause mortality

    • Objectively assessed (e.g., laboratory results)

    • Objectively assessed but potentially influenced by a clinician or patient (e.g., smoking cessation)

    • Subjectively assessed (e.g., pain)

      • Objective outcomes were the two first categories and subjective outcomes were the two last categories.

  • Number of included studies

For each eligible RCT included in the MA:

  • Author

  • Year of publication

  • First author name

  • Trial type (PC-RCT or ST-RCT)

  • Number of centers (for primary care, a center was defined as a practice, and for secondary and tertiary care, it was a hospital)

  • Sample size

  • For each group, the number of events and the number of patients analysed

  • Domains of the Cochrane Risk of Bias tool [16] as was reported by the authors of the systematic reviews

For PC- RCTs, we extracted the following additional information if mentioned in the MA:

  • Type of primary care setting

  • Healthcare professionals who recruited patients (when mentioned)

  • Whether the settings where patients are seen during the study for recruitment, follow-up and primary outcome assessment were the same or not

  • Country(ies) where the study was performed because of existing differences in healthcare systems between countries in primary-care

All data were collected from the systematic review report except for the number of events, which was collected from the trial reports when missing in the MA report.

Statistical analysis

We estimated intervention effects as odds ratios (OR). Outcome events were re-coded so that an OR less than 1 indicated a beneficial effect of the experimental intervention. Randomised controlled trials with no event in both groups did not contribute to the analysis.

Meta-epidemiological analysis

To compare PC-RCTs and ST-RCTs, we used the two-step approach described by Sterne et al. [12]. First for each meta-analysis, we estimated a ratio of odds ratio (ROR) by using random-effects meta-regression. In our study, the ROR was the ratio of the OR for PC-RCTs to the OR for ST-RCTs, An ROR less than 1 indicates lower intervention effects for PC-RCTs. Second, we estimated a combined ROR across meta-analyses and the 95% confidence interval (CI) by using a random-effects meta-analysis model. The heterogeneity across MAs was assessed with the I2 statistic and its 95% CI and the between–meta-analysis variance τ2.

Subgroup and sensitivity analyses

Subgroup analyses were planned according to the objectivity of outcomes, type of intervention in the experimental group (pharmacological vs non-pharmacological), and control group (active vs inactive). We used an interaction test to assess whether the combined ROR varied between subgroups. We also performed sensitivity analyses by adjusting meta-regression models on each item of the Risk of Bias tool [16] (high or unclear risk vs low risk).

Results

Study selection and general characteristics

From 1765 identified systematic reviews, we selected 76 MAs with 1685 trials, from which 614 trials were selected (Fig. 1, full references of the 76 MAs in Additional file 1 and 2). Of these 76 MAs, 33 (43.4%) evaluated a pharmacological intervention, 15 (19.7%) had an active control and 32 (42.1%) concerned a subjective outcome. The median number of trials included per MA was 5 (interquartile range [IQR] 3.8–10). MA characteristics are detailed in Table 1.

Fig. 1
figure 1

Flowchart of the selection process

Table 1 Characteristics of the meta-analyses (MAs) included (n = 76)

No eligible MA refers to meta-analyses with several reasons for not selecting it: fewer than 3 trials, without both primary and secondary or tertiary care settings or without a primary efficacy outcome.

MA: meta-analysis.

MA: meta-analysis; IQR: interquartile range; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.

Characteristics of selected trials

Among 614 trials selected, 230 were PC-RCTs and 384 ST-RCTs (Table 2). The median sample size was 276 (IQR 146–561.5) for PC-RCTs and 139 (IQR 66–284.3) for ST-RCTs. The median number of centers was 13 (IQR 3–40.5) and 1 (IQR 1–3), respectively. A total of 61 (62.9%) PC-RCTs were at low risk of bias for blinding of outcome assessors as compared with 83 (43.4%) ST-RCTs. More characteristics of the PC-RCTs included in this study are available in Additional file 3.

Table 2 Characteristics of the included trials

IQR: interquartile range, PC-RCT: primary care setting randomised controlled trial, ST-RCT: secondary or tertiary care setting randomised controlled trial.

Differences in intervention effect estimates between PC-RCTs and ST-RCTs

We found no statistical difference in intervention effect estimates between PC-RCTs and ST-RCTs; the combined ROR was 0.97 (95% CI 0.88 to 1.08) (Fig. 2). The heterogeneity across MAs was moderate (I2 = 45, 95%CI 29 to 58%, between–meta-analysis variance τ2 = 0.07).

Fig. 2
figure 2

Difference in the intervention effect estimates between randomised controlled trials in primary care versus secondary or tertiary settings

PC-RCTs: primary care setting randomised controlled trial.

ST-RCTs: secondary or tertiary care setting randomised controlled trial.

Subgroup analyses

For the 33 MAs assessing pharmacological interventions, the combined ROR was estimated at 0.90 (95% CI 0.76 to 1.07), and for the 43 MAs assessing non-pharmacological interventions, the combined ROR was 1.01 (95% CI 0.90 to 1.14). The interaction test was not statistically significant (P = 0.97). We did not find a significant interaction in the other subgroup analyses (Fig. 3).

Fig. 3
figure 3

Difference in intervention effect estimates for the subgroup analyses

ROR: ratio of odds ratio; CI: confidence interval; MA: meta-analysis; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.

Sensitivity analyses

Results were consistent when adjusting on each item of the risk of bias (Fig. 4).

Fig. 4
figure 4

Difference in intervention effect estimates when adjusting on risk of bias

ROR: ratio of odds ratio; CI: confidence interval; MA: meta-analysis; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.

Discussion

In this meta-epidemiological study of 76 MAs, we did not observe a significant difference in intervention effect estimates between randomised controlled trials conducted in primary care versus secondary or tertiary care settings. Consistent results were found in subgroup and sensitivity analyses. Our result and the way we worked to obtain it satisfy the 10 criteria as treated by Moustgaard et al. [ 17].

To our knowledge, our meta-epidemiological study is the first to compare intervention effect estimates between PC-RCTs and ST-RCTs. The number of MAs analysed is the major strength of this study. In 2016, Dechartres et al. found that the median number of MAs per meta-epidemiological study was 29 versus 76 MAs in our study [18]. Nevertheless, we restricted our study to Cochrane reviews, which could limit the generalizability of our results. Cochrane reviews were shown to have better methodological quality [19] and to be better reported [20] than non-Cochrane reviews. We performed an analysis of MAs of binary outcomes, so our results cannot be extrapolated to MAs with continuous outcomes. MAs with continuous outcomes have more subjective outcomes, are less frequently blinded [21]and have higher heterogeneity [22] than those with binary outcomes. Finally, another limitation may be the definition of primary care settings. Primary care is often difficult and poorly defined in the literature and can include a large range of professions and professionals [14]. In Cochrane reviews, the description of the type of healthcare professionals in charge of including, following up and assessing patients was seldom reported. This omission could have led us to misclassify some trials or also to exclude some trials because of uncertainty. Nevertheless, the selection and the extraction process were performed in duplicate by health professionals (i.e., a pharmacy resident, a general medicine resident and a professor of general medicine in case of discrepancy), which helped limit the uncertainty of some terms and reduced the risk of errors.

Our result does not support GPs’ general feeling that results from PC-RCTs differ from those derived from ST-RCTs [3, 5]. Our study is sufficiently powered, and the ROR point estimate was very close to 1, which indeed supports the conclusion of an absence of differences in intervention effect estimates between PC-RCTs and ST-RCTs. However, among the 76 MAs, 29 (38.2%) were related to psychiatry (mainly depression) or addictology (mainly tobacco use) and 10 (13.2%) to pneumology. Very few MAs were dedicated to cardiovascular conditions (n = 4, 5.3%) or diabetes (n = 1, 1.3%). So the medical fields represented in our study do not well match those mostly frequently encountered in primary care, typically cardiovascular diseases (13.6%), muskuloskeletal troubles (12.6%) and pneumology (12.5%), with psychatric diseases representing only 8.5% of the consultations [23]. Therefore, how we selected the MAs for our study led to a sample of diseases that does not perfectly reflect the diseases mainly managed in primary care. Other studies should probably focus on specific medical fields or diseases rather than considering all MAs, whatever the medical field.

Another point is that we focused on binary outcomes, but in cardiovascular diseases or diabetes, most trials use surrogate or intermediate endpoints, which are mainly continuous ones, such as cholesterol level, blood pressure or HbA1c level [24, 25]. This situation may also have led to discarding some MAs focused on diseases that are highly prevalent in primary care.

Finally, although the present study focused on the settings, intervention effect estimates also depend on the target population, defined by selection criteria and/or the type/intensity of the intervention assessed. Due to the limited size of our study, no adjustment on these characteristics could be done while analyzing data. Therefore, one of the remaining questions, which is beyond the scope of the present work, is whether patients included in PC-RCTs are similar to patients included in ST-RCTs.

Conclusion

We found no significant difference in intervention effect estimates between PC-RCTs and ST-RCTs. Nevertheless, the main medical fields selected are not fully representative of primary care consultations. Further studies focusing on medical fields highly prevalent in primary care are needed.

Availability of data and materials

The datasets used and/or analysed during the current study are available in the Additional file.

References

  1. Ekesbo R, Midlöv P, Gerward S, et al. Lack of adherence to hypertension treatment guidelines among GPs in southern Sweden-a case report-based survey. BMC Fam Pract. 2012;13:34. https://doi.org/10.1186/1471-2296-13-34.

    Article  Google Scholar 

  2. Grol R, Dalhuijsen J, Thomas S, et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317:858–61. https://doi.org/10.1136/bmj.317.7162.858.

    Article  CAS  Google Scholar 

  3. Carlsen B, Norheim OF. « what lies beneath it all? » – an interview study of GPs’ attitudes to the use of guidelines. BMC Health Serv Res. 2008;8:218. https://doi.org/10.1186/1472-6963-8-218.

    Article  Google Scholar 

  4. Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS. Why don’t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implement Sci. 2009;4:54. https://doi.org/10.1186/1748-5908-4-54.

    Article  Google Scholar 

  5. Cranney M, Warren E, Barton S, et al. Why do GPs not implement evidence-based guidelines? A descriptive study Fam Pract. 2001;18(4):359–63. https://doi.org/10.1093/fampra/18.4.359.

    Article  CAS  Google Scholar 

  6. Steel N, Abdelhamid A, Stokes T, et al. A review of clinical practice guidelines found that they were often based on evidence of uncertain relevance to primary care patients. J Clin Epidemiol. 2014;67:1251–7. https://doi.org/10.1016/j.jclinepi.2014.05.020.

    Article  Google Scholar 

  7. Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?”. The Lancet. 2005;365(9453):82–93. https://doi.org/10.1016/S0140-6736(04)17670-8.

    Article  Google Scholar 

  8. Uijen AA, Bakx JC, Mokkink HGA, et al. Hypertension patients participating in trials differ in many aspects from patients treated in general practices. J Clin Epidemiol. 2007:6. https://doi.org/10.1016/j.jclinepi.2006.05.015.

  9. Fahey T. Applying the results of clinical trials to patients to general practice: perceived problems, strengths, assumptions, and challenges for the future. Br J Gen Pract J R Coll Gen Pract. 1998;48:1173–8.

    CAS  Google Scholar 

  10. Sonis J. Applicability of clinical trial results to primary care. JAMA. 1998;280(20):1746–a. https://doi.org/10.1001/jama.280.20.1746-a.

    Article  CAS  Google Scholar 

  11. Barnett K, Mercer SW, Norbury M, et al. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet. 2012;380:37–43. https://doi.org/10.1016/S0140-6736(12)60240-2.

    Article  Google Scholar 

  12. Sterne JAC, Jüni P, Schulz KF, et al. Statistical methods for assessing the influence of study characteristics on treatment effects in « meta-epidemiological » research. Stat Med. 2002;21:1513–24.

    Article  Google Scholar 

  13. Zhang W. Meta-epidemiology: building the bridge from research evidence to clinical practice. Osteoarthr Cartil. 2010;18(Suppl 2):S1.

    Article  CAS  Google Scholar 

  14. Afonso MM, Bénard A, Chapron A, et al. Les soins primaires : une définition du champ pour développer la recherche. Epidemiol Public Health Rev Epidémiologie. Revue d'Épidémiologie et de Santé Publique. 2018;66:157–62 doi : https://doi.org/10.1016/j.respe.2017.09.004.

    Article  Google Scholar 

  15. Savović J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157:429–38. https://doi.org/10.1002/sim.1184.

    Article  Google Scholar 

  16. Cochrane Bias Methods Group Cochrane Statistical Methods Group, Higgins Julian PT, Altman Douglas G, Gøtzsche Peter C, Peter J, David M, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:889–93. https://doi.org/10.1136/bmj.d5928.

    Article  Google Scholar 

  17. Moustgaard H, Jones HE, Savović J, et al. Ten questions to consider when interpreting results of a meta-epidemiological study—the MetaBLIND study as a case. Res Synth Methods. 2020;11:260–74. https://doi.org/10.1002/jrsm.1392.

    Article  Google Scholar 

  18. Dechartres A, Trinquart L, Faber T, et al. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37. https://doi.org/10.1016/j.jclinepi.2016.04.005.

    Article  Google Scholar 

  19. Moja LP, Telaro E, D’Amico R, et al. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005;330:1053. https://doi.org/10.1136/bmj.38414.515938.8F.

    Article  Google Scholar 

  20. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13:e1002028. https://doi.org/10.1371/journal.pmed.1002028.

    Article  Google Scholar 

  21. Leyrat C, Caille A, Eldridge S, Kerry S, et al. Intervention effect estimates in cluster randomized versus individually randomized trials: a meta-epidemiological study. Int J Epidemiol. 2019:609–19.

  22. Alba AC, Alexander PE, Chang J, et al. High statistical heterogeneity is more frequent in meta-analysis of continuous than binary outcomes. J Clin Epidemiol. 2016;70:129–35.

    Article  Google Scholar 

  23. Letrilliart L, Supper I, Schuers M, et al. ECOGEN : étude des Éléments de la COnsultation en médecine GENérale. Exercer. 2014;114:148–57.

    Google Scholar 

  24. Gandhi GY, Murad MH, Fujiyoshi A, et al. Patient-important outcomes in registered diabetes trials. JAMA. 2008;299:2543–9. https://doi.org/10.1001/jama.299.21.2543.

    Article  CAS  Google Scholar 

  25. Rahimi K, Malhotra A, Banning AP, et al. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707. https://doi.org/10.1136/bmj.c5707.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Laura Smales (BioMedEditing) for English review of the manuscript.

Funding

The authors declare no specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Author information

Authors and Affiliations

Authors

Contributions

ADu, CDD, BG, AC and ADe conceived the study; ADu, AH and CDD participated in data collection; ADu, AH and CDD extracted the data; ET analysed data; ADu wrote the first draft of the manuscript, which was revised by CDD, BG, AC, ET, Ade, and AH. All coauthors read and approved the final version of the manuscript.

Corresponding author

Correspondence to Amandine Dugard.

Ethics declarations

Ethics approval and consent to participate

This study involved analysis of data from published systematic reviews with meta-analyses; we did not collect participant data.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests in relation with this study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. 

List of included meta-analyses (n = 76).

Additional file 2.

Description of the 76 meta-analyses.

Additional file 3.

Characteristics of the primary care setting randomised controlled trials included.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dugard, A., Tavernier, E., Caille, A. et al. Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study. BMC Med Res Methodol 22, 329 (2022). https://doi.org/10.1186/s12874-022-01815-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-022-01815-2

Keywords

  • Meta-epidemiological study
  • Cochrane systematic review
  • Primary care