User State Detection Using Facial Images with Mask Cover

Bandara, D.

Widespread use of masks was mandated in many countries as a direct result of the covid-19 pandemic. This meant that mask wearing, which was previously restricted to specialized occupations or cities with high levels of pollution became the norm in many places of the world. This has obvious implications for any system that uses facial images to infer user state. This work attempts to gauge the effect of mask wearing on such systems. Arousal classification is used in this study due to its well-studied nature in image processing literature. Using “Affect in the wild” video dataset, the “masks” were synthetically placed on the facial images extracted from videos. A binary classification between high and low arousal shows that there is a drop in accuracy when using masks. However, this drop is larger in across subject classification than within subject classification. The study shows that it is feasible to develop effective user state classification models even with mask cover. © 2021, Springer Nature Switzerland AG.