The measurement properties, sample and data collection methods used for this study are presented below. Thereafter, the instruments used to assess convergent validity are described and the data analysis techniques are presented.
Assessment of measurement properties
The measurement properties, structural validity, internal consistency and convergent validity of the SCIROCCO tool were tested in this study. Structural and convergent validity are aspects of construct validity. Construct validity ‘is based on the assumption that the measurement instrument validly measures the construct to be measured and should be assessed in case a gold standard is lacking’ [4]. The first measurement property, structural validity, is defined as ‘the degree to which the scores of a measurement instrument are an adequate reflection of the dimensionality of the construct to be measured’ [6]. This type of validity can be explored by examining the instrument’s factor structure using factor analysis. The second property, convergent validity, refers to the extent to which two instruments capture a corresponding construct [7] and can be assessed by investigating associations between these instruments. Finally, the measurement property, internal consistency, is assessed and is an aspect of reliability. It is a measure of the homogeneity of a scale and indicates the extent to which items in a scale are intercorrelated.
Sample and data collection
Structural validity and internal consistency
To assess the structural validity and internal consistency of the SCIROCCO tool, subjects were invited to fill in the online SCIROCCO tool in three rounds between June 2017 and February 2018. The subjects were recruited according to the following criteria: individuals from European regions involved in the design and deployment of integrated care, including no more than 10 people per region, from several disciplines (i.e. a decision-maker, healthcare professional, an information technology specialist, regulators, payers, users group, and innovation agencies), different sectors (i.e. health care, social care, housing and voluntary sector) and different positions (i.e. senior management, front-line, back-office). In the first round, subjects were recruited from the five regions that participated in the SCIROCCO project and were recruited by SCIROCCO project members. The subjects came from the five participating European regions (Basque Country; Spain, Norrbotten; Sweden, Puglia; Italy, Olomouc; Czech Republic and Scotland). In the second round, subjects that are involved in other relevant EU projects were recruited to fill in the SCIROCCO tool. These subjects were recruited by the project coordinator and by SCIROCCO project members, mainly during dissemination activities that took place within the SCIROCCO project. In the last round, subjects were recruited by the researchers from the Vrije Universiteit Brussel. These subjects were recruited from other European regions (i.e. Denmark, France, Germany, the Netherlands, and United Kingdom) and were derived from a convenience sample (contacts provided by one of the researchers). All those who were identified and selected received a general invitation e-mail that described the purpose and procedure of the study. The invitational e-mail also included a paper providing an overview of the SCIROCCO tool and a web-link to illustrative videos and demonstrations on how to use the online version of the tool.
Convergent validity
To examine the convergent validity of the SCIROCCO tool, the participants who were invited in the first round were also invited to fill in the DMIC Quickscan. This occurred in a period of 6–24 weeks after the participants filled in the SCIROCCO tool. The 22 statements of the Quickscan were presented in an online survey that took about 10 min to complete. Subjects received an invitation by e-mail, including information on the survey, ethical considerations, and the link to the online DMIC Quickscan questionnaire. To construct a general profile of the subjects, data were collected about their professional position, and the name of their organisation, region and service or network.
DMIC Quickscan
The DMIC Quickscan is based on the Development Model of Integrated Care (DMIC) questionnaire, which consists of 89 item [8]. In a recent literature review comparing the B3-MM with existing instruments that focus on assessing the development of integrated care, the DMIC was found to match with all the dimensions of the B3-MM [5]. The elements of the DMIC represent a wide range of activities considered as relevant to the realisation of integrated care which are grouped in nine clusters; ‘patient-centeredness’, ‘delivery system’, ‘performance management’, ‘quality of care’, ‘result-focused learning’, ‘interprofessional teamwork’, ‘roles and tasks’, ‘commitment’ and ‘transparent entrepreneurship’. Implementing the elements of all nine clusters contributes to the further development of integrated care. The DMIC is being used to serve as an assessment tool for health care professionals, managers and integrated care coordinators to support the implementation of improvement activities. The systematic development of the DMIC consisted of a literature study, a Delphi study and several survey studies [9]. The level of evidence on the overall quality of the measurement property content validity for the DMIC was found to be strong [5]. Moreover, the DMIC has been empirically validated in stroke, acute myocardial infarct, and dementia services in the Netherlands [10]. Furthermore, the model has been used, mainly in Europe and Canada, to evaluate and describe a variety of integration contexts [11,12,13].
In this study, to ensure a high response rate, we chose to use the DMIC Quickscan rather than the DMIC, due to a shorter completion time. It takes respondents10 minutes to complete the Quickscan as compared to 45 min for the DMIC. The Quickscan is extracted from the 89 items of the DMIC, of which a total of 22 items were selected based on priority scores [8] .These 22 items are presented as statements in the Quickscan, which reflect the different activities that can be undertaken to implement and develop integrated care. Subjects are asked to rate whether the description on the separate statements matches the current situation of their integrated services/network by using a 5-point scale (which ranges from fully agree-fully disagree). The DMIC Quickscan was translated into Czech, English,, Italian and Spanish by experts in the field of integrated care. Notwithstanding the theoretical validity of the DMIC and the derivation of the DMIC Quickscan from the DMIC, measurement properties including construct validity, internal consistency and convergent validity have not been tested for the DMIC nor the DMIC Quickscan. Since to our knowledge, no other similar instruments to SCIROCCO tool exist, the Quickscan was the most appropriate comparator available to test the construct validity of the SCIROCCO tool.
The convergent validity of the SCIROCCO tool was evaluated by comparing elements of the tool that used an instrument measuring a similar construct, the DMIC Quickscan. This means that the convergent validity of the SCIROCCO tool is based on comparisons between related, but not quite equivalent, concepts. The SCIROCCO tool concentrates on the maturity of elements for integrated care operating in the health care system whereas the DMIC Quickscan focuses on the development of practical elements in integrated care practices or networks. Even though both instruments are considered to operate on different levels, we expected to find a correspondence between the elements of both tools since those elements indicated to be present in the practice/network might also provide an indication of progress on these elements in the healthcare systems of those regions.
Data analysis
Quantitative data-analysis was performed to assess the structural validity, internal consistency and convergent validity of the SCIROCCO tool. Analyses were performed using IBM SPSS Statistics software (SPSS), version 25.0.
Structural validity
A specialist additional module for factor analysis, R V2.4.3 was added to SPSS for the analysis of the structural validity [14]. Conventional methods of exploratory factor analysis (EFA) rely on Pearson correlations and/or maximum likelihood techniques. However, the assumptions for using these methods (item distributions that approach an equal intervals scale and a multivariate normal distribution) were not met in this study. Therefore, the polychoric correlation matrix was analysed to obtain a more accurate reproduction of the correlation structure [15]. Furthermore, EFA using minimum residual method (MINRES) of the polychoric correlation matrix was conducted to explore the structure of the items of the SCIROCCO tool. MINRES is a robust factor extraction method, as it does not require any distributional assumptions, and it can be used with small samples [16].
Multiple methods to determine the numbers of factors to extract for ordinal skewed data exist and the use of a combination of several methods is suggested [14]. In this study, two accurate techniques, Parallel Analysis (PA) [17] and Comparative Data (CD) [18], were chosen as methods to determine the number of factors to retain. Although the accuracy rates of both extraction methods decrease with smaller samples [18, 19], they are the most accurate methods known [20,21,22]. PA was applied using random column permutations of real data matrix, factor estimation, polychoric correlation matrix and mean eigenvalue criterion [19], a 1000 datasets were simulated. For CD, Spearman rank order correlation matrix was used to fit the ordinal scale [14]. The items of the tool relating to ‘maturity for integrated care’ were expected to be correlated, therefore oblique rotation was selected as the rotation technique. A factor loading of > 0.35 was applied.
Descriptive statistics were used to characterize the study sample. To check whether the dataset was suitable for factor analyses, Bartlett’s test for sphericity and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy were assessed [23] Furthermore, the data were screened for any invalid data patterns (e.g., selection of “0”s for all questions), skewness and missing values. We decided to exclude items with an extreme skewed distribution (> 90% of all the responses in one category) for the analyses. Items with a high non-response (> 5% missing values) were also excluded.
Internal consistency
After the factor analysis was completed, the internal consistency of the tool was assessed using Cronbach alpha and ordinal alpha coefficients. Theoretically, the Cronbach alpha is only appropriate when variables are continuous, and it has been shown that Cronbach-α is negatively biased when it is used to measure the reliability of ordinal variables [20]. However, this measure is often used in practice and leads to valid results despite data that are highly skewed. In the event that the assumption of normality is violated, the ordinal alpha coefficient has been recommended as a more appropriate estimate of reliability than Cronbach’s alpha [24]. However, Chalmers indicates that coefficient α has never required continuous item-level data and that ordinal alpha should not be reported as a measure of a tests reliability, but instead should be understood as hypothetical tool [25]. Therefore, the internal consistency of each factor was examined by calculating both the Cronbach’s alpha and the ordinal reliability alpha.
Convergent validity
After the SCIROCCO tool and DMIC Quickscan were administered, quantitative data analysis was used to compare the items of the instruments. The convergent validity of the items of the SCIROCCO tool was evaluated by testing whether scores on the items of the SCIROCCO tool were positively associated with scores on the corresponding items of the DMIC Quickscan. Hypotheses were formulated where we expected moderate correlations between items of the two instruments. This expectation was based on the correspondence between descriptions of items of the SCIROCCO tool and the descriptions of items of the DMIC Quickscan. This resulted in the testing of 23 predefined hypotheses (see Additional file 2). Not all 22 items of the Quickscan were included in the formulated hypotheses, since some item descriptions did not correspond to any of the 12 items of the SCIROCCO tool. Correlations were calculated to test the hypothesized relationships. Strong correlations were not expected a priori because the two instruments do not measure identical constructs. Correlations falling within the range 0.30–0.50 were considered low, within the range 0.50–0.70 were considered moderate and within the range 0.70–0.90 were considered high [26]. Since the distribution of the data was skewed, the agreement between the items of the SCIROCCO tool and the DMIC Quickscan instrument were assessed using Spearman’s ρ correlation coefficients. To provide an indication of the significance and size of a statistical effect, it is recommended to use confidence limit estimation [27]. Therefore, bias-corrected accelerated (BCa) confidence intervals (CI, 95%) were computed using bootstrapping (1000 samples) for all intervals. This technique has been advised in situations where parametric assumptions are not met [28, 29].
