A Multi-Kernel Embedding Fusion Framework for Physiological Signal Based Emotion Recognition

• Published in: IEEE transactions on affective computing pp. 1 - 14
• Main Authors: He, Xinrun, Huang, Jian, Fu, Zhongzheng, Li, Yixuan, Wu, Dongrui
• Format: Journal Article
• Language: English
• Published: IEEE 2025
• Subjects: Accuracy; Affective computing; Data mining; Emotion recognition; Feature extraction; Kernel; multi-kernel learning; multi-modal fusion; physiological signal; Physiology; Representation learning; Sparse matrices; Vectors
• ISSN: 1949-3045; 1949-3045
• DOI: 10.1109/TAFFC.2025.3562905

Physiological signal-based emotion recognition requires effective fusion of multi-modal physiological signals to improve recognition accuracy. In this paper, a multi-kernel embedding fusion framework (MKEFF) is proposed for multi-modal physiological signal emotion recognition. Specifically, multi-kernel learning and kernel approximation techniques are used to compute the multi-kernel embeddings of the original feature vectors of each modality independently. The embeddings are then fed in parallel to their respective representation learning layer, where the proposed sparse relation learning method is applied to all the modalities to explore the correlation and diversity among them. Finally, a distribution alignment based fusion method is proposed to align each modality in the subspace, and a weighted summation fusion is performed to obtain the fused representations. Extensive cross-subject emotion recognition experiments are conducted on three public datasets, DEAP, DECAF, and SEED-IV, to evaluate the proposed method. The experimental results demonstrate that the proposed method achieves better classification performance and interpretability than the state-of-the-art methods.