A BiLSTM–Transformer and 2D CNN Architecture for Emotion Recognition from Speech

• Published in: Electronics (Basel) Vol. 12; no. 19; p. 4034
• Main Authors: Kim, Sera, Lee, Seok-Pil
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.10.2023
• Subjects: Accuracy; Algorithms; Artificial intelligence; Artificial neural networks; Deep learning; Emotion recognition; Emotions; Machine learning; Neural networks; Spectrograms; Speech; Speech recognition; Transformers
• ISSN: 2079-9292; 2079-9292
• DOI: 10.3390/electronics12194034

The significance of emotion recognition technology is continuing to grow, and research in this field enables artificial intelligence to accurately understand and react to human emotions. This study aims to enhance the efficacy of emotion recognition from speech by using dimensionality reduction algorithms for visualization, effectively outlining emotion-specific audio features. As a model for emotion recognition, we propose a new model architecture that combines the bidirectional long short-term memory (BiLSTM)–Transformer and a 2D convolutional neural network (CNN). The BiLSTM–Transformer processes audio features to capture the sequence of speech patterns, while the 2D CNN handles Mel-Spectrograms to capture the spatial details of audio. To validate the proficiency of the model, the 10-fold cross-validation method is used. The methodology proposed in this study was applied to Emo-DB and RAVDESS, two major emotion recognition from speech databases, and achieved high unweighted accuracy rates of 95.65% and 80.19%, respectively. These results indicate that the use of the proposed transformer-based deep learning model with appropriate feature selection can enhance performance in emotion recognition from speech.