Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans

Xuehai He; Xingyi Yang; Shanghang Zhang; Jinyu Zhao; Yichen Zhang; Eric Xing; Pengtao Xie

Another thing that we are interested in investigating is: given the weights of the feature extractor learned by SSL, when finetuning the overall classification network on the COVID19-CT images and labels, should we just fine-tune the final classifier layer or fine-tune the weights of the feature extractor as well? Table VII shows the results on two backbones: ResNet-50 and DenseNet-169, where "frozen" denotes that the weights of the feature extractor are not fine-tuned during the finetuning process and "unfrozen" denotes that these weights are fine-tuned together with those in the final classification layer. As can be seen, fine-tuning feature extraction weights yields much better performance. This is because using class labels to fine-tune these weights can make the extracted features more discriminative and hence more effective in distinguishing COVID-19 CTs from Non-COVID-19 CTs. Figure 5 shows the Grad-CAM [58] visualizations for DenseNet-169 trained from baseline methods and our proposed Self-Trans. By comparing Column (3) with Column (5), we notice that the DenseNet-169 model trained with random initialization erroneously focuses on some image edges and corners that are not related to COVID-19. In contrast, transfer learning methods generally lead to more accurate diseaserelated visual localization. By comparing Column (5) with Column (7), we can see that our proposed Self-Trans method can have even better localization of the disease region than the ImageNet pretrained method.