Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 3; pp. 1034 - 1047
• Main Authors: Zhang, Ke, Li, Yuanqing, Wang, Jingyu, Cambria, Erik, Li, Xuelong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Brain modeling; Context; Context modeling; Datasets; domain knowledge; Domains; Emotion recognition; Emotions; Machine learning; Multimodal emotion recognition; Real time; Real-time systems; real-time video conversation; Reinforcement learning; Streaming media; Video; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3072412

Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories.