The process of developing the IGEST involved two main phases, each divided into intermediate steps: phase I - tool development (3 steps), where the scope of the IGEST was defined and the item pool was generated and refined, and phase II - content validity (1 step), where the content validity index (CVI) was established [19, 20]. The whole process is illustrated in Fig. 1.
Phase 1 - Tool development
Define the scope (step 1)
The ISS project steering group with experience in CPG development process defined the scope of the IGEST (step 1, Fig. 1) by specifying the domain and its boundaries [21, 22].
Considering the goal of the IGEST as a tool for screening international trustworthy CPGs for their adoption/adaption in the local context, the attention was directed toward aspects known empirically to most affect the guidelines’ trustworthiness [17, 23, 24], such as a rigorous process for assembling, evaluating, summarizing the evidence, and a multidisciplinary guideline development panel that is free from conflicts of interest.
Specifically, the IGEST addresses 1) the CPGs development process and their reporting, without analysing the content of the recommendations, and 2) the extent to which any bias potentially affecting the recommendations (e.g., literature search, selection, assessment, consistency between evidence and recommendations, multidisciplinary process, and conflict of interest management) is minimised during the development process.
Generation of IGEST items (step 2)
To identify appropriate items that fit the IGEST scope, we adopted a deductive approach based on a literature review and assessment of existing tools, followed by an inductive method through focus groups. The combination of both deductive and inductive methods is considered best practice, as a literature review provides the theoretical basis for defining the domain, while the use of qualitative techniques moves the domain from an abstract point to the identification of its manifest forms [21].
Literature review (step 2a, Fig. 1)
The results of a systematic review [25] on the appraisal tools for CPGs represented a starting point to identify tools for generating the IGEST items. Additional studies and tools were identified by examining the review reference lists and from its update. Considering the large number of tools identified, only tools subject to any sort of validation studies were considered [1, 15, 16, 26,27,28,29,30,31,32,33,34,35,36]. In addition, international standards for guideline development, such as those of the Institute of Medicine [17, 23], the Guidelines International Network (G-I-N) [3, 17], and the NICE manual [37], were carefully examined. None of the existing instruments fully met the intended scope of the IGEST. However, as among the existing tools the domain related to validity [23] (e.g., the relationship between the evidence and recommendations, the substance and quality of the scientific and clinical evidence cited, and the means used to evaluate the evidence) was almost always present, the ISS project steering group organised the material from the retrieved tools to identify critical elements useful for the IGEST development. The relevant findings were discussed to identify ambiguous, irrelevant, duplicated, or missing items, and a set of potential items that better reflected the scope was selected to generate the initial IGEST items.
Focus groups for IGEST draft’s refinement (step 2b, Fig. 1)
To refine the initial version of the IGEST, the inductive method of a focus group was used [38]. Thus, a panel of seven experts, selected on the basis of their relevant experience in CPG development with different academic and professional background and geographical origin, was identified and invited by phone by the ISS project steering group (additional file 1). The experts were asked to declare their financial and intellectual interests by using the ISS form for the declaration of interests; these interests were then assessed and managed according to the ISS policy for the disclosure and management of conflict of interests (https://snlg.iss.it/wp-content/uploads/2021/08/MM_v1.3.2_apr_2019.pdf). The initial IGEST version with all instructions was sent them by mail. Afterwards, a focus group aimed at exploring the experts’ opinions, and thoughts on the relevance and congruence of the selected items, the structure and wording of the items, and the type of expected answers to the questions were also discussed. The raised key points were recorded in field notes taken during the session and later collated by the ISS project steering group, and the IGEST vers.1 was then finalised.
Pilot testing (step 3)
To test the feasibility and usability of the tool while identifying its potential practical problems and to establish the approximate time required to complete it, a pilot testing on a sample of seven CPGs was performed by two external researchers (GC, SG) (step 3, Fig. 1).
The piloting CPGs were identified through a rapid search via PubMed using the term: "Practice Guideline" as publication type, then 7 out of the 5498 retrieved records were purposely selected to include different characteristics (e.g., specialties, associations/organizations, target population, countries). Guidelines were considered eligible if they were published after 2016 in English language, and they met the guideline definition proposed by the IOM [17]. We excluded consensus conference, position statement, and any secondary publication of the guidelines.
The researchers were asked to assess the CPGs with the IGEST vers.1 and give suggestions for the IGEST’s clarity and comprehensiveness. The subsequent IGEST vers.2 was tested for content validity.
Phase 2 - Content validity
Content validity index (step 4)
The content validity aims at assessing whether the tool is representative of all aspects of the construct and was established using a rigorous quantification process [19, 20, 39, 40], based on independent consensual judgments (step 4, Fig. 1). The CVI was computed on two levels: item-level CVI (I-CVI) and scale CVI (S-CVI).
Content validity index first round (step 4a, Fig. 1)
As suggested by Polit and Beck [20, 39], the ISS steering group invited 14 already known experts with extensive experience in research methodology and CPG development and evaluation with different academic and professional background and geographical origin (additional file 1). The ISS policy for the disclosure and management of conflict of interests was also applied at this step.
It is worth mentioning that the seven experts who participated in the focus group (tool development) were not the same as the 14 experts who participated in the content validation process.
Then, IGEST vers.2 was emailed along with clear and concise instructions on how to rate each item and a spreadsheet to report judgments on the relevance of the IGEST using a four-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant), and comments and/or suggestions were requested. All the experts were ensured confidentiality throughout the survey process.
To determine the I-CVI, the number of items rated as ‘relevant’ (rating 3 or 4) was divided by the number of experts. Similarly, S-CVI was calculated using the number of items rated or judged as ‘relevant’ by the experts. An I-CVI score of 0.78 and S-CVI of 0.90 were the minimum acceptable indices [39].
Content validity index: second round (step 4b, Fig. 1)
The items that scored a CVI ≥0.78 were retained, while the others were modified or excluded based on the comments provided by the evaluators (step 5, Fig. 1). The experts' comments were appraised by the ISS project steering group, and any differences were resolved by consensus. As a result, the tool was reformulated, and the IGEST vers.3 was sent to the experts or the second round of CVI evaluation. Each expert was asked to express the level of agreement with the amendments made, following the same procedure used in the first CVI evaluation process. At the end of the second round of CVI, the ISS project steering group discussed the results and potential solutions and refined the list of items accordingly.