Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

• Published in: IEEE transactions on cybernetics Vol. 51; no. 9; pp. 4386 - 4399
• Main Authors: Zhang, Xiaowei, Liu, Jinyong, Shen, Jian, Li, Shaojie, Hou, Kechen, Hu, Bin, Gao, Jin, Zhang, Tong
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Brain modeling; Computer architecture; Deep neural network; Emotion recognition; Emotions; Feature extraction; Fuses; Kernel; kernel machine; Kernels; multimodal fusion; Optimization; Physiological responses; Physiology; Regularization; Representations; Scientific visualization; Task analysis
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2020.2987575

These days, physiological signals have been studied more broadly for emotion recognition to realize emotional intelligence in human-computer interaction. However, due to the complexity of emotions and individual differences in physiological responses, how to design reliable and effective models has become an important issue. In this article, we propose a regularized deep fusion framework for emotion recognition based on multimodal physiological signals. After extracting the effective features from different types of physiological signals, we construct ensemble dense embeddings of multimodal features using kernel matrices, and then utilize a deep network architecture to learn task-specific representations for each kind of physiological signal from these ensemble dense embeddings. Finally, a global fusion layer with a regularization term, which can efficiently explore the correlation and diversity among all of the representations in a synchronous optimization process, is designed to fuse generated representations. Experiments on two benchmark datasets show that this framework can improve the performance of subject-independent emotion recognition compared to single-modal classifiers or other fusion methods. Data visualization also demonstrates that the final fusion representation exhibits higher class-separability power for emotion recognition.