Audio-Visual Affect Recognition

• Published in: IEEE transactions on multimedia Vol. 9; no. 2; pp. 424 - 428
• Main Authors: Zeng, Z., Tu, J., Liu, M., Huang, T.S., Pianfetti, B., Roth, D., Levinson, S.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.02.2007; Institute of Electrical and Electronic Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Affect recognition; affective computing; Application software; Applied sciences; Artificial intelligence; Channels; Classification; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Emotion recognition; Energy use; Exact sciences and technology; Face recognition; Hidden Markov models; Human computer interaction; Multimedia; multimodal human-computer interaction; Pattern recognition. Digital image processing. Computational geometry; Recognition; Smoothing methods; Software; Speech and sound recognition and synthesis. Linguistics; Speech recognition; Streaming media; Streams; Testing; Visual; Voting; Multimedia; Computer vision; Smoothing methods; Image recognition; Man machine system; affective computing; Customization; Emotion emotionality; Markov model; Affect recognition; emotion recognition; Cognitive theory; Prosody; Voting; User interface; Classification; Facies; Speech recognition; multimodal human-computer interaction; Hidden Markov model; Facial expression
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2006.886310

The ability of a computer to detect and appropriately respond to changes in a user's affective state has significant implications to human-computer interaction (HCI). In this paper, we present our efforts toward audio-visual affect recognition on 11 affective states customized for HCI application (four cognitive/motivational and seven basic affective states) of 20 nonactor subjects. A smoothing method is proposed to reduce the detrimental influence of speech on facial expression recognition. The feature selection analysis shows that subjects are prone to use brow movement in face, pitch and energy in prosody to express their affects while speaking. For person-dependent recognition, we apply the voting method to combine the frame-based classification results from both audio and visual channels. The result shows 7.5% improvement over the best unimodal performance. For person-independent test, we apply multistream HMM to combine the information from multiple component streams. This test shows 6.1% improvement over the best component performance