Deep Convolutional and Recurrent Neural Networks for Emotion Recognition from Human Behaviors

Deng, James J.; Leung, Clement H. C.

Human behaviors and the emotional states that they convey have been studied by psychologist and sociologists. The tracking of behaviors and emotions is becoming more pervasive with the advent of the Internet of Things (IoT), where small and always connected sensors can continuously capture information about human gestures, movements and postures. The captured information about readable behaviors conveys significant information that can be represented as time series. Few studies in emotion recognition and affective computing have explored the connection between the time series sensors data and the emotional behavior they conveys. In this paper, an innovative approach is proposed to study the emotions and behaviors connected to the time series data. A convolutional network augmented with attention-based bidirectional LSTM is introduced to represent the correlations between behaviors and emotions. The advantage of this model is that it can well recognized emotions by exploiting the data captured by sensors. The experimental results show that the proposed deep learning method outperforms separate schemes and achieves a high degree of accuracy for modelling human behaviors and emotions.