A comprehensive evaluation methodology for the publicly accessible AI services for medical diagnostics

Morozov, S. P.; Gombolevskiy, V. A.; Blokhin, I. A.; Semenov, S. S.; Logunova, T. A.; Andreychenko, A. E.

Online AI in telemedicine for radiology became widely available and valuable in the pandemic, particularly for chest CT analysis. On the other hand, the potentially harmful consequences of such services' inappropriate usage cannot be neglected. Thus, a suitable methodology for quality assurance and quality control has to be established. A study's purpose was to develop and test an original methodology for a complex evaluation of open-access AI services in teleradiology. The approach included assessing the user experience, accessibility, safety, and diagnostic accuracy on the independent reference dataset. The methodology was applied to assess seven AI services for the detection of COVID-19 on a CT scan. A comparative analysis of this assessment is presented in this work. The analysis allowed us to draw conclusions about AI services' quality and their value for different users - patients, physicians, and healthcare data scientists. The originality of the findings, timeliness, and interdisciplinary approach make this quality assurance methodology of particular interest for further application and spreading.