Development and Evaluation of an AI System for COVID-19

Cheng Jin; Weixiang Chen; Yukun Cao; Zhanwei Xu; Xin Zhang; Lei Deng; Chuansheng Zheng; Jie Zhou; Heshui Shi; Jianjiang Feng

The AI system shows good performances and it can be used with different diagnosis thresholds according to different policies or prior probabilities. The Performances of the AI system and five readers in COVID-19 diagnosis on reader study cohort. c. ROC curve for abnormal slice locating. This result was test on 12 COVID-19 positives cases from internal validation cohort which have manual lesion segmentation. d. Metrics of proposed AI system for different cohorts and tasks. e. Discrepancies between the AI system and human readers. L) M) Two COVID-19 cases identified by the AI system but missed by all five readers. R) A COVID-19 case identified by a reader but missed by the AI system. (The yellow circles denote possible lesion area) sensitivity of our system is about 84.76% when specificity is 99.5%, and specificity is 80.02% when sensitivity is 97%. Besides, because patients in our external test cohort have multi-stage CT volumes, some of the stages of positive subjects might be in the recovery state whose CT may have no abnormalities but are still regarded as positive in experiments. Figure 2 a shows the results after roughly filtering out these cases by only keeping the maximum predicted value of multi-stage CTs in the same patient, in which the specificity is about 96.74% at sensitivity of 97%. The decision curve analysis (DCA) for the AI system are presented in Figure 7 b, which indicated that the AI system adds benefit than the "diagnose all" or "diagnose none" strategies when the threshold is within a wide range 1.82-97.6% in COVID-19.