Table 1 Screening options
Approach | Process | Risk | Mitigating risk |
|---|---|---|---|
1. Stop screening | Change the number of reviews required to 1 and assign the AI tool to exclude the remaining records. If the software does not allow for this, you would leave the remaining records unscreened. There would be no further human screening in this option. | Exclusion of relevant records at title/abstract (i.e., false negatives). | Depending on the threshold that has been used, it may be beneficial to run the AI audit toola to help identify any false negatives. |
2. Single-reviewer screening | Change the include and exclude rules to “1 to include/exclude” and have a single-reviewer screen the remaining records. This may be performed by more than one reviewer, however, only one reviewer will be required to screen any given record. | Over-inclusion of records to be screened at full text (i.e., false positives). Exclusion of relevant records at title/abstract (i.e., false negatives). | Over-inclusion: none Identify false negatives: run AI audit toola |
3. Liberal accelerated screening with AI reviewer, with no conflict resolution | Assign the AI reviewer to exclude the remaining records with human reviewers to screen the remaining records using the liberal accelerated approachb, with no conflict resolution performed. | Over-inclusion of records to be screened at full text (i.e., false positives). Records in conflict will be ignored by the machine learning algorithm and will not contribute to the prediction scores. | Over-inclusion: none Records in conflict: see approach 4. |
4. Liberal accelerated screening with AI reviewer, with conflict resolution | As 3 above, with conflicts resolved. If there is a conflict between the AI reviewer and the human reviewer, a second human reviewer will be required to adjudicate. | Over-inclusion of records to be screened at full text (i.e., false positives). Records in conflict will be ignored by the machine learning algorithm and will not contribute to the prediction scores until conflicts are resolved. | Over-inclusion: none Records in conflict: perform conflict resolution at set intervals (e.g., at the end of each day) so all screened records will contribute to the machine learning. |
5. Liberal accelerated screening, no conflict resolution | Change the include rule to “1 to include”, with no conflict resolution performed. Screening will continue with two or more reviewers. | Over-inclusion of records to be screened at full text (i.e., false positives). Records in conflict will be ignored by the machine learning algorithm and will not contribute to the prediction scores. | Over-inclusion: none Records in conflict: see approach 6 |
6. Liberal accelerated screening, with conflict resolution | As 6 above, with conflicts resolved. | Over-inclusion of records to be screened at full text (i.e., false positives). Records in conflict will be ignored by the machine learning algorithm and will not contribute to the prediction scores until conflicts are resolved. | Over-inclusion: none Records in conflict: perform conflict resolution at set intervals (e.g., at the end of each day) so all screened records will contribute to the machine learning. |
7. Dual-independent with AI reviewer | Assign the AI reviewer to exclude the remaining records with human reviewers to screen the remaining records (i.e., dual-independent screening). Another reviewer would be required in cases where the AI reviewer excluded the record and the human reviewer included the record. | Excluding relevant records (i.e., false negatives), as only a single human reviewer is required to exclude (in addition to the AI reviewer). Records in conflict will be ignored by the machine learning algorithm and will not contribute to the prediction scores until conflicts are resolved. | Identify false negatives: run AI audit toola Records in conflict: perform conflict resolution at set intervals (e.g., at the end of each day) so all screened records will contribute to the machine learning. |
8. Dual-independent, assign some reviewers to full-text screening | Not all reviewers may need to continue title and abstract screening. You may choose to move some of the reviewers to perform full-text screening, while keeping a smaller team of reviewers screening the remaining records at title/abstract. | None, although you may need to be strategic on which reviewers are screening title/abstracts. | Keep at least one senior reviewer (based on experience or clinical expertise) to help ensure high-quality include/exclude decisions. |