Generative Listener EEG for Speech Emotion Recognition Using Generative Adversarial Networks With Compressed Sensing

• Published in: IEEE journal of biomedical and health informatics Vol. 28; no. 4; pp. 2025 - 2036
• Main Authors: Chang, Jiang, Zhang, Zhixin, Wang, Zelin, Li, Jiacheng, Meng, Linsheng, Lin, Pan
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Brain modeling; Compressed sensing; Convolutional neural networks; cross-subject; EEG; EEG emotion recognition; EEG generation; Electroencephalography; Emotion recognition; Emotions; Event-related potentials; Feature extraction; Generative adversarial networks; Semantics; Speech; Speech recognition
• DOI: 10.1109/JBHI.2024.3360151
• ISSN: 2168-2194; 2168-2208

Currently, emotional features in speech emotion recognition are typically extracted from the speeches, However, recognition accuracy can be influenced by factors such as semantics, language, and cross-speech datasets. Achieving consistent emotional judgment with human listeners is a key challenge for AI to address. Electroencephalography (EEG) signals prove to be an effective means of capturing authentic and meaningful emotional information in humans. This positions EEG as a promising tool for detecting emotional cues conveyed in speech. In this study, we proposed a novel approach named CS-GAN that generates listener EEGs in response to a speaker's speech, specifically aimed at enhancing cross-subject emotion recognition. We utilized generative adversarial networks (GANs) to establish a mapping relationship between speech and EEGs to generate stimulus-induced EEGs. Furthermore, we integrated compressive sensing theory (CS) into the GAN-based EEG generation method, thereby enhancing the fidelity and diversity of the generated EEGs. The generated EEGs were then processed using a CNN-LSTM model to identify the emotional categories conveyed in the speech. By averaging these EEGs, we obtained the event-related potentials (ERPs) to improve the cross-subject capability of the method. The experimental results demonstrate that the generated EEGs by this method outperform real listener EEGs by 9.31% in cross-subject emotion recognition tasks. Furthermore, the ERPs show an improvement of 43.59%, providing evidence for the effectiveness of this method in cross-subject emotion recognition.