Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study
BMC Medical Research Methodology volume 22, Article number: 329 (2022)
Many clinical practice guidelines are based on randomised controlled trials conducted in secondary or tertiary care setting and general practitioners frequently question their relevance for primary care patients. Our aim was to compare the intervention effect estimates between primary care setting randomised controlled trials (PC-RCTs) and secondary or tertiary care setting randomised controlled trials (ST-RCTs).
Meta-epidemiological study of meta-analyses (MAs) of a binary outcome including at least one PC-RCT and one ST-RCT. PC-RCTs were defined as trials recruiting patients in general practices, primary care practices, family practices, community centers or community pharmacies. ST-RCTs were defined as trials recruiting in hospitals, including hospitalized patients, hospital outpatients and patients from emergency departments. For each MA, we estimated a ratio of odds ratio (ROR) by using random-effects meta-regression, with an ROR less than 1 indicating lower estimates of the intervention effect in PC-RCTs than ST-RCTs. Finally, we estimated a combined ROR across MAs by using a random-effects meta-analysis. We performed subgroup analyses considering the type of outcomes (objective vs subjective), type of experimental intervention (pharmacological vs non-pharmacological), and control group (active vs inactive) as well as analyses adjusted on items of the Cochrane Risk of Bias tool.
Among 1765 screened reviews, 76 MAs with 230 PC-RCTs and 384 ST-RCTs were selected. The main medical fields were pneumology (13.2%) and psychiatry or addictology (38.2%). Intervention effect estimates did not significantly differ between PC-RCTs and ST-RCTs (ROR = 0.97, 95% confidence interval 0.88 to 1.08), with moderate heterogeneity across MAs (I2 = 45%). Subgroup and adjusted analyses led to consistent results.
We did not observe any significant difference in intervention effect estimates between PC-RCTs and ST-RCTs. Nevertheless, most of the medical fields in this meta-epidemiological study were not representative of the pathologies encountered in primary care. Further studies with pathologies more frequently encountered in primary care are needed.
Many studies have highlighted the lack of adherence to clinical practice guidelines by general practitioners (GPs) [1, 2]. GPs generally argue a lack of relevance or difficulties in applying guidelines [3, 4], and they particularly express concerns about the applicability of clinical trial results used to establish guidelines to their own patients [5, 6]. The first reason is that many randomised controlled trials (RCTs) include highly selected patients, which restricts the generalizability of their results . As an example, a study found that less than half of primary-care patients would have been included in hypertension trials . The second, highly related to the first one reason is that in most RCTs, patients are recruited in secondary- or tertiary-care settings  rather than in primary-care. However, as compared with secondary- or tertiary-care setting patients, primary-care patients have less severe disease and more undifferentiated symptoms  but more multi-morbidity . Such differences may lead to differences in benefits and harms of the intervention. Our hypothesis is that intervention effect estimates differ between primary care setting randomised controlled trials (PC-RCTs) and secondary or tertiary care setting RCTs (ST-RCTs).
We performed a meta-epidemiological study to assess whether intervention effect estimates differ between PC-RCTs and ST-RCTs.
This was a meta-epidemiological study, a study design used to compare intervention effect estimates between trials with and without a characteristic of interest . In this study, we focused on the setting, with the aim of comparing PC-RCTs to ST-RCTs. A meta-epidemiological study is performed by generally using a two-step approach and, contrary to traditional epidemiological studies, units of analysis are studies rather than patients . First, for each selected MA, we assessed the difference in intervention effects between studies with the characteristics of interest (i.e., primary care setting) and without the characteristics of interest. This step involved using a meta-regression for each selected MA (with the intervention effect considered the outcome and the characteristic of interest the independent variable). Second, results from the meta-regression were meta-analysed over the different MAs.
On July 9, 2020, we searched the Cochrane Database of Systematic Reviews to retrieve all systematic reviews with meta-analyses (MAs) that were published, with no restriction on time, by using the following text words in the full text: “primary care” OR “primary healthcare” OR “general practice” OR “family practice”. We only used primary care keywords to ensure that we had PC-RCTs in each MA.
Selection of relevant MAs and trials
We screened the full text of potentially eligible systematic reviews to select MAs of binary outcomes that had at least 3 trials (3 studies is the minimum to perform a meta-regression) and at least one PC-RCT and one ST-RCT. PC-RCTs were defined as trials recruiting patients in general practices, primary care practices, family practices, community centers or community pharmacies according to the definition by Afonso et al. . Trials in nursing homes or at home were not considered PC-RCTs and were excluded. ST-RCTs were defined as trials recruiting in hospitals, including hospitalized patients, hospital outpatients and patients from emergency departments. Hospital-at-home trials were not considered ST-RCTs and were excluded. Trials with both primary and secondary or tertiary care settings were excluded, as were trials with an unclear setting or trials including patients in other than primary or secondary or tertiary care settings, such as schools. For this study, we selected only RCTs and excluded non-randomised or quasi-randomised trials.
If more than one MA was eligible within the same review, we selected the MA for the primary efficacy outcome, then the MA with the highest number of trials. MAs of adverse events were not included because of the uncertainty in the direction of bias. We also discarded MAs when it was impossible to determine which group was the experimental and control group.
When a trial was included several times within the same MA, we kept only the duplicate with the largest sample size. When a trial was included in several selected MAs, we kept the one in the most recent systematic review.
All this selection process was performed independently by two reviewers (A.D., A.H.), with disagreements resolved by discussion, referring to a third opinion (C.D.D.) when necessary.
Two independent reviewers (A.D., A.H.) collected data from all selected MAs and trials by using a standardised data collection form. Disagreements were resolved by discussion, referring to a third opinion (C.D.D.) when necessary. The following characteristics were extracted:
For each meta-analysis:
General information: year of publication, first author name
Type of intervention (pharmacological, non-pharmacological), type of comparator (active control or inactive control, with inactive control defined as placebo or no added intervention to usual care, sham or other)
Outcome and whether it was subjective or objective following the classification provided by Savović et al. 
Objectively assessed (e.g., laboratory results)
Objectively assessed but potentially influenced by a clinician or patient (e.g., smoking cessation)
Subjectively assessed (e.g., pain)
Objective outcomes were the two first categories and subjective outcomes were the two last categories.
Number of included studies
For each eligible RCT included in the MA:
Year of publication
First author name
Trial type (PC-RCT or ST-RCT)
Number of centers (for primary care, a center was defined as a practice, and for secondary and tertiary care, it was a hospital)
For each group, the number of events and the number of patients analysed
Domains of the Cochrane Risk of Bias tool  as was reported by the authors of the systematic reviews
For PC- RCTs, we extracted the following additional information if mentioned in the MA:
Type of primary care setting
Healthcare professionals who recruited patients (when mentioned)
Whether the settings where patients are seen during the study for recruitment, follow-up and primary outcome assessment were the same or not
Country(ies) where the study was performed because of existing differences in healthcare systems between countries in primary-care
All data were collected from the systematic review report except for the number of events, which was collected from the trial reports when missing in the MA report.
We estimated intervention effects as odds ratios (OR). Outcome events were re-coded so that an OR less than 1 indicated a beneficial effect of the experimental intervention. Randomised controlled trials with no event in both groups did not contribute to the analysis.
To compare PC-RCTs and ST-RCTs, we used the two-step approach described by Sterne et al. . First for each meta-analysis, we estimated a ratio of odds ratio (ROR) by using random-effects meta-regression. In our study, the ROR was the ratio of the OR for PC-RCTs to the OR for ST-RCTs, An ROR less than 1 indicates lower intervention effects for PC-RCTs. Second, we estimated a combined ROR across meta-analyses and the 95% confidence interval (CI) by using a random-effects meta-analysis model. The heterogeneity across MAs was assessed with the I2 statistic and its 95% CI and the between–meta-analysis variance τ2.
Subgroup and sensitivity analyses
Subgroup analyses were planned according to the objectivity of outcomes, type of intervention in the experimental group (pharmacological vs non-pharmacological), and control group (active vs inactive). We used an interaction test to assess whether the combined ROR varied between subgroups. We also performed sensitivity analyses by adjusting meta-regression models on each item of the Risk of Bias tool  (high or unclear risk vs low risk).
Study selection and general characteristics
From 1765 identified systematic reviews, we selected 76 MAs with 1685 trials, from which 614 trials were selected (Fig. 1, full references of the 76 MAs in Additional file 1 and 2). Of these 76 MAs, 33 (43.4%) evaluated a pharmacological intervention, 15 (19.7%) had an active control and 32 (42.1%) concerned a subjective outcome. The median number of trials included per MA was 5 (interquartile range [IQR] 3.8–10). MA characteristics are detailed in Table 1.
No eligible MA refers to meta-analyses with several reasons for not selecting it: fewer than 3 trials, without both primary and secondary or tertiary care settings or without a primary efficacy outcome.
MA: meta-analysis; IQR: interquartile range; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.
Characteristics of selected trials
Among 614 trials selected, 230 were PC-RCTs and 384 ST-RCTs (Table 2). The median sample size was 276 (IQR 146–561.5) for PC-RCTs and 139 (IQR 66–284.3) for ST-RCTs. The median number of centers was 13 (IQR 3–40.5) and 1 (IQR 1–3), respectively. A total of 61 (62.9%) PC-RCTs were at low risk of bias for blinding of outcome assessors as compared with 83 (43.4%) ST-RCTs. More characteristics of the PC-RCTs included in this study are available in Additional file 3.
IQR: interquartile range, PC-RCT: primary care setting randomised controlled trial, ST-RCT: secondary or tertiary care setting randomised controlled trial.
Differences in intervention effect estimates between PC-RCTs and ST-RCTs
We found no statistical difference in intervention effect estimates between PC-RCTs and ST-RCTs; the combined ROR was 0.97 (95% CI 0.88 to 1.08) (Fig. 2). The heterogeneity across MAs was moderate (I2 = 45, 95%CI 29 to 58%, between–meta-analysis variance τ2 = 0.07).
PC-RCTs: primary care setting randomised controlled trial.
ST-RCTs: secondary or tertiary care setting randomised controlled trial.
For the 33 MAs assessing pharmacological interventions, the combined ROR was estimated at 0.90 (95% CI 0.76 to 1.07), and for the 43 MAs assessing non-pharmacological interventions, the combined ROR was 1.01 (95% CI 0.90 to 1.14). The interaction test was not statistically significant (P = 0.97). We did not find a significant interaction in the other subgroup analyses (Fig. 3).
ROR: ratio of odds ratio; CI: confidence interval; MA: meta-analysis; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.
Results were consistent when adjusting on each item of the risk of bias (Fig. 4).
ROR: ratio of odds ratio; CI: confidence interval; MA: meta-analysis; PC-RCTs: primary care setting randomised controlled trials; ST-RCTs: secondary or tertiary care setting randomised controlled trials.
In this meta-epidemiological study of 76 MAs, we did not observe a significant difference in intervention effect estimates between randomised controlled trials conducted in primary care versus secondary or tertiary care settings. Consistent results were found in subgroup and sensitivity analyses. Our result and the way we worked to obtain it satisfy the 10 criteria as treated by Moustgaard et al. [ 17].
To our knowledge, our meta-epidemiological study is the first to compare intervention effect estimates between PC-RCTs and ST-RCTs. The number of MAs analysed is the major strength of this study. In 2016, Dechartres et al. found that the median number of MAs per meta-epidemiological study was 29 versus 76 MAs in our study . Nevertheless, we restricted our study to Cochrane reviews, which could limit the generalizability of our results. Cochrane reviews were shown to have better methodological quality  and to be better reported  than non-Cochrane reviews. We performed an analysis of MAs of binary outcomes, so our results cannot be extrapolated to MAs with continuous outcomes. MAs with continuous outcomes have more subjective outcomes, are less frequently blinded and have higher heterogeneity  than those with binary outcomes. Finally, another limitation may be the definition of primary care settings. Primary care is often difficult and poorly defined in the literature and can include a large range of professions and professionals . In Cochrane reviews, the description of the type of healthcare professionals in charge of including, following up and assessing patients was seldom reported. This omission could have led us to misclassify some trials or also to exclude some trials because of uncertainty. Nevertheless, the selection and the extraction process were performed in duplicate by health professionals (i.e., a pharmacy resident, a general medicine resident and a professor of general medicine in case of discrepancy), which helped limit the uncertainty of some terms and reduced the risk of errors.
Our result does not support GPs’ general feeling that results from PC-RCTs differ from those derived from ST-RCTs [3, 5]. Our study is sufficiently powered, and the ROR point estimate was very close to 1, which indeed supports the conclusion of an absence of differences in intervention effect estimates between PC-RCTs and ST-RCTs. However, among the 76 MAs, 29 (38.2%) were related to psychiatry (mainly depression) or addictology (mainly tobacco use) and 10 (13.2%) to pneumology. Very few MAs were dedicated to cardiovascular conditions (n = 4, 5.3%) or diabetes (n = 1, 1.3%). So the medical fields represented in our study do not well match those mostly frequently encountered in primary care, typically cardiovascular diseases (13.6%), muskuloskeletal troubles (12.6%) and pneumology (12.5%), with psychatric diseases representing only 8.5% of the consultations . Therefore, how we selected the MAs for our study led to a sample of diseases that does not perfectly reflect the diseases mainly managed in primary care. Other studies should probably focus on specific medical fields or diseases rather than considering all MAs, whatever the medical field.
Another point is that we focused on binary outcomes, but in cardiovascular diseases or diabetes, most trials use surrogate or intermediate endpoints, which are mainly continuous ones, such as cholesterol level, blood pressure or HbA1c level [24, 25]. This situation may also have led to discarding some MAs focused on diseases that are highly prevalent in primary care.
Finally, although the present study focused on the settings, intervention effect estimates also depend on the target population, defined by selection criteria and/or the type/intensity of the intervention assessed. Due to the limited size of our study, no adjustment on these characteristics could be done while analyzing data. Therefore, one of the remaining questions, which is beyond the scope of the present work, is whether patients included in PC-RCTs are similar to patients included in ST-RCTs.
We found no significant difference in intervention effect estimates between PC-RCTs and ST-RCTs. Nevertheless, the main medical fields selected are not fully representative of primary care consultations. Further studies focusing on medical fields highly prevalent in primary care are needed.
Availability of data and materials
The datasets used and/or analysed during the current study are available in the Additional file.
Ekesbo R, Midlöv P, Gerward S, et al. Lack of adherence to hypertension treatment guidelines among GPs in southern Sweden-a case report-based survey. BMC Fam Pract. 2012;13:34. https://doi.org/10.1186/1471-2296-13-34.
Grol R, Dalhuijsen J, Thomas S, et al. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317:858–61. https://doi.org/10.1136/bmj.317.7162.858.
Carlsen B, Norheim OF. « what lies beneath it all? » – an interview study of GPs’ attitudes to the use of guidelines. BMC Health Serv Res. 2008;8:218. https://doi.org/10.1186/1472-6963-8-218.
Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS. Why don’t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implement Sci. 2009;4:54. https://doi.org/10.1186/1748-5908-4-54.
Cranney M, Warren E, Barton S, et al. Why do GPs not implement evidence-based guidelines? A descriptive study Fam Pract. 2001;18(4):359–63. https://doi.org/10.1093/fampra/18.4.359.
Steel N, Abdelhamid A, Stokes T, et al. A review of clinical practice guidelines found that they were often based on evidence of uncertain relevance to primary care patients. J Clin Epidemiol. 2014;67:1251–7. https://doi.org/10.1016/j.jclinepi.2014.05.020.
Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?”. The Lancet. 2005;365(9453):82–93. https://doi.org/10.1016/S0140-6736(04)17670-8.
Uijen AA, Bakx JC, Mokkink HGA, et al. Hypertension patients participating in trials differ in many aspects from patients treated in general practices. J Clin Epidemiol. 2007:6. https://doi.org/10.1016/j.jclinepi.2006.05.015.
Fahey T. Applying the results of clinical trials to patients to general practice: perceived problems, strengths, assumptions, and challenges for the future. Br J Gen Pract J R Coll Gen Pract. 1998;48:1173–8.
Sonis J. Applicability of clinical trial results to primary care. JAMA. 1998;280(20):1746–a. https://doi.org/10.1001/jama.280.20.1746-a.
Barnett K, Mercer SW, Norbury M, et al. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet. 2012;380:37–43. https://doi.org/10.1016/S0140-6736(12)60240-2.
Sterne JAC, Jüni P, Schulz KF, et al. Statistical methods for assessing the influence of study characteristics on treatment effects in « meta-epidemiological » research. Stat Med. 2002;21:1513–24.
Zhang W. Meta-epidemiology: building the bridge from research evidence to clinical practice. Osteoarthr Cartil. 2010;18(Suppl 2):S1.
Afonso MM, Bénard A, Chapron A, et al. Les soins primaires : une définition du champ pour développer la recherche. Epidemiol Public Health Rev Epidémiologie. Revue d'Épidémiologie et de Santé Publique. 2018;66:157–62 doi : https://doi.org/10.1016/j.respe.2017.09.004.
Savović J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157:429–38. https://doi.org/10.1002/sim.1184.
Cochrane Bias Methods Group Cochrane Statistical Methods Group, Higgins Julian PT, Altman Douglas G, Gøtzsche Peter C, Peter J, David M, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:889–93. https://doi.org/10.1136/bmj.d5928.
Moustgaard H, Jones HE, Savović J, et al. Ten questions to consider when interpreting results of a meta-epidemiological study—the MetaBLIND study as a case. Res Synth Methods. 2020;11:260–74. https://doi.org/10.1002/jrsm.1392.
Dechartres A, Trinquart L, Faber T, et al. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37. https://doi.org/10.1016/j.jclinepi.2016.04.005.
Moja LP, Telaro E, D’Amico R, et al. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005;330:1053. https://doi.org/10.1136/bmj.38414.515938.8F.
Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13:e1002028. https://doi.org/10.1371/journal.pmed.1002028.
Leyrat C, Caille A, Eldridge S, Kerry S, et al. Intervention effect estimates in cluster randomized versus individually randomized trials: a meta-epidemiological study. Int J Epidemiol. 2019:609–19.
Alba AC, Alexander PE, Chang J, et al. High statistical heterogeneity is more frequent in meta-analysis of continuous than binary outcomes. J Clin Epidemiol. 2016;70:129–35.
Letrilliart L, Supper I, Schuers M, et al. ECOGEN : étude des Éléments de la COnsultation en médecine GENérale. Exercer. 2014;114:148–57.
Gandhi GY, Murad MH, Fujiyoshi A, et al. Patient-important outcomes in registered diabetes trials. JAMA. 2008;299:2543–9. https://doi.org/10.1001/jama.299.21.2543.
Rahimi K, Malhotra A, Banning AP, et al. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707. https://doi.org/10.1136/bmj.c5707.
The authors thank Laura Smales (BioMedEditing) for English review of the manuscript.
The authors declare no specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Ethics approval and consent to participate
This study involved analysis of data from published systematic reviews with meta-analyses; we did not collect participant data.
Consent for publication
The authors declare no competing interests in relation with this study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Dugard, A., Tavernier, E., Caille, A. et al. Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study. BMC Med Res Methodol 22, 329 (2022). https://doi.org/10.1186/s12874-022-01815-2