Attention‐based sensor fusion for emotion recognition from human motion by combining convolutional neural network and weighted kernel support vector machine and using inertial measurement unit signals

• Published in: IET signal processing Vol. 17; no. 4
• Main Authors: Zhao, Yan, Guo, Ming, Sun, Xuehan, Chen, Xiangyong, Zhao, Feng
• Format: Journal Article
• Language: English
• Published: 01.04.2023
• Subjects: emotion recognition; fuzzy comprehensive evaluation; human motion; neural network; sensor fusion; weighted kernel support vector machine
• ISSN: 1751-9675; 1751-9683
• DOI: 10.1049/sil2.12201

The remarkable development of human–computer interactions has created an urgent need for machines to be able to recognise human emotions. Human motions play a key role in emphasising and conveying emotions to meet the complexity of daily application scenarios, such as medical rehabilitation and social education. Therefore, this paper aims to explore hidden emotional states from human motions. Accordingly, we proposed a novel approach for emotion recognition using multiple inertial measurement unit (IMU) sensors worn on different body parts. First, the mapping relationship between emotion and human motion was established through fuzzy comprehensive evaluation, and data were collected for six emotional states: sleepy, bored, excited, tense, angry, and distressed. Second, the preprocessed data were used as input in a lightweight convolutional neural network to extract discriminative features. Third, an attention‐based sensor fusion module was developed to obtain the importance scores of each IMU sensor for generating a fused feature representation. In the recognition phase, we constructed a weighted kernel support vector machine (SVM) model with an auxiliary fuzzy function to improve the weight calculation method of kernel functions in a multiple kernel SVM. Finally, the results obtained are compared with those of similar state‐of‐the‐art studies, the proposed method showed a higher accuracy (99.02%) for the six emotional states mentioned above. These findings may promote the development of social robots with non‐verbal emotional communication capabilities. This paper aims to explore the hidden emotional states from human motion and make accurate identification, which helps to promote the development of non‐verbal behaviour in human‐computer interaction and equip machines to have emotional communication capabilities. We consider that this is helpful to improve the intelligence and emotionalisation of human‐computer interaction at this stage.