Non-Contact Negative Mood State Detection Using Reliability-Focused Multi-Modal Fusion Model

• Published in: IEEE journal of biomedical and health informatics Vol. 26; no. 9; pp. 4691 - 4701
• Main Authors: Rong, Qian, Ding, Shuai, Yue, Zijie, Wang, Yaping, Wang, Linjie, Zheng, Xi, Li, Yinghui
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Aerospace medicine; Affective computing; Data mining; Data reliability; Datasets; Deep learning; Depression; Fatigue; Feature extraction; Mood; multi-modal fusion; negative mood state; Psychology; Reliability; Videos
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2022.3182357

Negative mood states include tension, depression, anger, fatigue, and confusion, which represent the weak internal emotions of a human. Negative mood states exert adverse impact on individuals' ability to make rational decisions, which entails the practicable method of negative mood state detection. The most commonly used negative mood state detection methods are based on the psychological scale, which requires additional work and brings inconvenience to the subject in the application scenarios. To overcome this challenge, this paper proposes a novel non-contact negative mood state detection method according to the knowledge of affective computing. The POMS-net model is used to extract temporal-spatial features from visible and infrared thermal videos, and the negative mood state detection is realized using data reliability-focused multi-modal fusion. The proposed method is verified using the HDT-BR dataset collected in the aerospace medicine experiment "Earth-Star II" and the VIRI public dataset. The experimental results on the datasets verify that our method outperforms the comparison methods.