Non-Contact Dual-Modality Emotion Recognition System by CW Radar and RGB Camera

• Published in: IEEE sensors journal Vol. 21; no. 20; pp. 23198 - 23212
• Main Authors: Zhang, Li, Fu, Chang-Hong, Hong, Hong, Xue, Biao, Gu, Xuemei, Zhu, Xiaohua, Li, Changzhi
• Format: Journal Article
• Language: English
• Published: New York IEEE 15.10.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Biomedical monitoring; Cameras; Contact stresses; Doppler radar; dual-modality; Emotion recognition; Emotions; Feature extraction; Heart beat; Luminous intensity; Monitoring; Non-contact; Optical communication; Optical flow (image analysis); Optimization; Physiology; radar; Sensors; Signal monitoring; video; Visual signals
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2021.3107429

Emotion recognition has a significant impact on people's health and life quality. Recent studies have shown that emotion recognition can be achieved by analyzing audio-visual emotion channels and physiological signals. However, the main challenges are: 1. the audio-visual techniques can be easily fooled by fake facial expressions; 2. the contact physiological signal monitoring introduces additional stress and is usually unsuitable for long-term monitoring; 3. The non-contact physiological signal monitoring methods are easily affected by complex environmental conditions. In this paper, a non-contact dual-modality emotion recognition system is proposed. First, the respiratory and heartbeat signals are measured by radar and camera simultaneously. Then, a hybrid signal optimization approach is proposed to remove the influence of body motion and light conditions on the physiological signal. It includes a light-intensity-based scheme selecting proper heartbeat signal for different light conditions, and an optical-flow-based algorithm pruning signals with significant radial body motion. Finally, the features extracted from the optimized physiological signals are fused to train an emotion recognition system. As shown in the experimental results, the proposed system could achieve high classification accuracy of 89.6% for 10-fold cross-validation at sample level, and 71.0 % for cross-validation at subject level. The respiratory and heartbeat signals by non-contact approaches are demonstrated to be reliable in emotion recognition.