Subject wise data augmentation based on balancing factor for quaternary emotion recognition through hybrid deep learning model

• Published in: Biomedical signal processing and control Vol. 86; p. 105075
• Main Authors: Singh, Khushboo, Ahirwal, Mitul Kumar, Pandey, Manish
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2023
• Subjects: 1D-CNN; Arousal; BI-LSTM; Classification; Deep learning; EEG signal; Emotion recognition; Valence; Deep learning; Emotion recognition; Valence; Classification; Arousal; 1D-CNN; EEG signal; BI-LSTM
• ISSN: 1746-8094; 1746-8108
• DOI: 10.1016/j.bspc.2023.105075

An electroencephalogram (EEG) identifies neuronal activity as electrical currents produced by a group of specialized pyramidal cells within the brain due to synchronized activity. EEG signal contains essential information about brain activity and is often used to measure the enthusiasm of individual behavioural feelings and polarity. Recently, the researcher focused on classifying EEG emotions from multimodal DEAP data sets into binary (high and low) and ternary (low, medium and high) classes based on valence and arousal scales. However, for deep and intrinsic emotion recognition, multiclass classification is preferred. But due to the limited data sample and improper class distribution, multiclass classification is more challenging. This paper proposes a data augmentation-based hybrid deep learning model to classify EEG signals over four different emotional stages, i.e. happy (H), relaxed (R), anger (A) and sad (S). The hybrid model encapsulates a convolutional neural network (CNN) and bi-directional long short-term memory (Bi-LSTM) to recognize EEG samples’ emotions. Furthermore, the proposed model performs segmentation to enhance EEG data samples, and SMOTE (Synthetic minority oversampling technique) is committed to achieving alleviated skewness in EEG samples among four emotion classes. The experiments are conducted on all the data of 32 subjects before and after processing the data augmentation algorithm on the DEAP data. The average improvement in precision, recall, F1-score, accuracy, and specificity is 6.02%, 9.44%, 7.98%, 6.01%, and 2.66% respectively after data augmentation, and statistically varies with the balancing factor. •A hybrid deep learning model is proposed for multiclass EEG emotion classification.•Incorporating CNN and Bi-LSTM in the hybrid model enhances EEG emotion recognition.•The class imbalance problem is addressed by Segmentation and SMOTE techniques.•The proposed model improves precision, recall, F1 score, accuracy, and specificity.