A Parallel Multi-Modal Factorized Bilinear Pooling Fusion Method Based on the Semi-Tensor Product for Emotion Recognition

• Published in: Entropy (Basel, Switzerland) Vol. 24; no. 12; p. 1836
• Main Authors: Liu, Fen, Chen, Jianfeng, Li, Kemeng, Tan, Weijie, Cai, Chang, Ayub, Muhammad Saad
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 16.12.2022; MDPI
• Subjects: Costs; Data integration; Emotion recognition; Emotions; low-rank matrix; Matching; Mathematical analysis; multi-modal information fusion; Redundancy; semi-tensor product; Tensors; multi-modal information fusion; emotion recognition; low-rank matrix; semi-tensor product
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e24121836

Multi-modal fusion can exploit complementary information from various modalities and improve the accuracy of prediction or classification tasks. In this paper, we propose a parallel, multi-modal, factorized, bilinear pooling method based on a semi-tensor product (STP) for information fusion in emotion recognition. Initially, we apply the STP to factorize a high-dimensional weight matrix into two low-rank factor matrices without dimension matching constraints. Next, we project the multi-modal features to the low-dimensional matrices and perform multiplication based on the STP to capture the rich interactions between the features. Finally, we utilize an STP-pooling method to reduce the dimensionality to get the final features. This method can achieve the information fusion between modalities of different scales and dimensions and avoids data redundancy due to dimension matching. Experimental verification of the proposed method on the emotion-recognition task using the IEMOCAP and CMU-MOSI datasets showed a significant reduction in storage space and recognition time. The results also validate that the proposed method improves the performance and reduces both the training time and the number of parameters.