COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System

Murphy, Keelin; Smits, Henk; Knoops, Arnoud J. G.; Korst, Michael B. J. M.; Samson, Tijs; Scholten, Ernst T.; Schalekamp, Steven; Schaefer-Prokop, Cornelia M.; Philipsen, Rick H. H. M.; Meijers, Annet; Melendez, Jaime; van Ginneken, Bram; Rutten, Matthieu

BACKGROUND: Chest radiography may play an important role in triage for coronavirus disease 2019 (COVID-19), particularly in low-resource settings. PURPOSE: To evaluate the performance of an artificial intelligence (AI) system for detection of COVID-19 pneumonia on chest radiographs. MATERIALS AND METHODS: An AI system (CAD4COVID-XRay) was trained on 24 678 chest radiographs, including 1540 used only for validation while training. The test set consisted of a set of continuously acquired chest radiographs (n = 454) obtained in patients suspected of having COVID-19 pneumonia between March 4 and April 6, 2020, at one center (223 patients with positive reverse transcription polymerase chain reaction [RT-PCR] results, 231 with negative RT-PCR results). Radiographs were independently analyzed by six readers and by the AI system. Diagnostic performance was analyzed with the receiver operating characteristic curve. RESULTS: For the test set, the mean age of patients was 67 years ± 14.4 (standard deviation) (56% male). With RT-PCR test results as the reference standard, the AI system correctly classified chest radiographs as COVID-19 pneumonia with an area under the receiver operating characteristic curve of 0.81. The system significantly outperformed each reader (P < .001 using the McNemar test) at their highest possible sensitivities. At their lowest sensitivities, only one reader significantly outperformed the AI system (P = .04). CONCLUSION: The performance of an artificial intelligence system in the detection of coronavirus disease 2019 on chest radiographs was comparable with that of six independent readers. © RSNA, 2020