Table 5 Summary of the relevant studies. FNR: False-negative rate. SEN Sensitivity, SPE Specificity

Identifying eligible records

Study ID

Tool

Studies

All records

Results

Comments

Olofsson 2017 [16]

Rayyan [4]

3 SRs and 3 literature reviews

7956

SEN of 21-88% after screening 25% of records.

SEN of 86-98% after screening 50% of records.

SEN of 89-100% after screening 75% of records.

Thresholds used were not reported.

SPE rates were not reported.

Rathbone 2015 [17]

Abstrackr [6]

4 SRs

SR1: 1415

SR2: 517

SR3: 1735

SR4: 1042

SR1: Precision of 16.8% and FNR of 10% after screening 18% of records.

SR2: Precision of 24.7% and FNR of 14.5% after screening 23% of records.

SR3: Precision of 29.2% and FNR of 4.7% after screening 7% of records.

SR4: Precision of 45.5% and FNR of 2.4% after screening 12% of records.

SEN and SPE rates were not reported.

Gates 2018 [18]

Abstrackr [6]

3 SRs and 1 descriptive analysis (DA)

SR1: 12763

SR2: 5893

SR3: 47385

DA: 5243

SR1: SPE of 69% and SEN of 79% after screening 2.2% of records.

SR2: SPE of 85% and SEN of 92% after screening 10.3% of records.

SR3: SPE of 90% and SEN of 82% after screening 0.7% of records.

DA: SPE of 19% and SEN of 96% after screening 4% of records.

–

Tsou 2020 [19]

Abstrackr [6] and EPPI-Reviewer [8]

9 SRs

SR1: 9038

SR2: 3181

SR3: 2706

SR4: 889

SR5: 673

SR6: 651

SR7: 500

SR8: 427

SR9: 226

For Abstrackr, SEN of 100% after screening 71.1, 51.5, 96, 95.6, 99, 85.9, 88.2, 99.3, and 93.8% of records for SR1 to SR9 respectively.

For EPPI-Reviewer, SEN of 100% after screening 61.7, 39.9, 91.3, 94.6, 97.9, 86.3, 88.2, 98.8, and 91.6% of records for SR1 to SR9 respectively.

They also reported diagnostic metrics for identifying eligible reports.

Chai 2021 [20]

Research Screener [20]

9 SRs and 2 scoping reviews (SCR)

SR1: 813

SR2: 2249

SR3: 2584

SR4: 368

SR5: 870

SR6: 306

SR7: 23423

SR8: 13376

SR9: 1686

SCR1: 16506

SCR2: 1230

SEN of 100% after screening 32, 13, 6, 5, 4, 4, 5, 13, and 14% of records for SR1 to SR9 respectively.

SEN of 100% after screening 40 and 38% of records for SCR1 to SCR2 respectively.

This tool utilizes deep learning algorithms.

Identifying eligible reports

Study ID

Tool

Studies

Eligible records

Results

Tsou 2020 [19]

Abstrackr [6] and EPPI-Reviewer [8]

9 SRs

SR1: 696

SR2: 200

SR3: 843

SR4: 107

SR5: 267

SR6: 73

SR7: 166

SR8: 149

SR9: 104

For Abstrackr, SEN of 100% after screening 40.7, 39.8, 81.2, 100, 71.6, 56.4, 41.2, 60, and 71.2% of eligible records for SR1 to SR9 respectively.

For EPPI-Reviewer, SEN of 100% after screening 41, 39.8, 97.1, 70, 74, 30.1, 31.8, 59.4, and 51.3% of eligible records for SR1 to SR9 respectively.