Emotion Recognition Using Multimodal Deep Learning

• Published in: Neural Information Processing Vol. 9948; pp. 521 - 529
• Main Authors: Liu, Wei, Zheng, Wei-Long, Lu, Bao-Liang
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2016; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Auto-encoder; EEG; Emotion recognition; Multimodal deep learning
• ISBN: 3319466712; 9783319466712
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-46672-9_58

To enhance the performance of affective models and reduce the cost of acquiring physiological signals for real-world applications, we adopt multimodal deep learning approach to construct affective models with SEED and DEAP datasets to recognize different kinds of emotions. We demonstrate that high level representation features extracted by the Bimodal Deep AutoEncoder (BDAE) are effective for emotion recognition. With the BDAE network, we achieve mean accuracies of 91.01 % and 83.25 % on SEED and DEAP datasets, respectively, which are much superior to those of the state-of-the-art approaches. By analysing the confusing matrices, we found that EEG and eye features contain complementary information and the BDAE network could fully take advantage of this complement property to enhance emotion recognition.