Recognizing Human Emotional State From Audiovisual Signals

• Published in: IEEE transactions on multimedia Vol. 10; no. 5; pp. 936 - 946
• Main Authors: Wang, Yongjin, Guan, Ling
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.08.2008; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Audio signals; Audiovisual; Audiovisual information; Cepstral analysis; Character recognition; Classification; Classification algorithms; Data mining; Emotion recognition; Emotions; Face detection; Human; Human subjects; Humans; Mel frequency cepstral coefficient; multiclassifier; Recognition; Speech; System testing; Visual
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2008.927665

Machine recognition of human emotional state is an important component for efficient human-computer interaction. The majority of existing works address this problem by utilizing audio signals alone, or visual information only. In this paper, we explore a systematic approach for recognition of human emotional state from audiovisual signals. The audio characteristics of emotional speech are represented by the extracted prosodic, Mel-frequency Cepstral Coefficient (MFCC), and formant frequency features. A face detection scheme based on HSV color model is used to detect the face from the background. The visual information is represented by Gabor wavelet features. We perform feature selection by using a stepwise method based on Mahalanobis distance. The selected audiovisual features are used to classify the data into their corresponding emotions. Based on a comparative study of different classification algorithms and specific characteristics of individual emotion, a novel multiclassifier scheme is proposed to boost the recognition performance. The feasibility of the proposed system is tested over a database that incorporates human subjects from different languages and cultural backgrounds. Experimental results demonstrate the effectiveness of the proposed system. The multiclassifier scheme achieves the best overall recognition rate of 82.14%.