Development and Validation of an EEG-Based Real-Time Emotion Recognition System Using Edge AI Computing Platform With Convolutional Neural Network System-on-Chip Design

• Published in: IEEE journal on emerging and selected topics in circuits and systems Vol. 9; no. 4; pp. 645 - 657
• Main Authors: Fang, Wai-Chi, Wang, Kai-Yen, Fahier, Nicolas, Ho, Yun-Lung, Huang, Yu-De
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.12.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Acceleration; affective computing; Algorithms; Arousal; Artificial intelligence; Artificial neural networks; Classification; Computer architecture; convolutional neural network (CNN); Convolutional neural networks; Electroencephalography; Emotion recognition; Feature extraction; Field programmable gate arrays; Fourier transforms; Graphical user interface; Hardware; Image reconstruction; Microprocessors; Multiphase; Neural networks; Object recognition; Power consumption; Real time; Real-time systems; System on chip; Training
• ISSN: 2156-3357; 2156-3365
• DOI: 10.1109/JETCAS.2019.2951232

This study proposed an electroencephalogram (EEG)-based real-time emotion recognition hardware system architecture based on multiphase convolutional neural network (CNN) algorithm implemented on a 28-nm technology chip and on field programmable gate array (FPGA) for binary and quaternary classification. Sample entropy, differential asymmetry, short-time Fourier transform, and a channel reconstruction method were used for emotion feature extraction. In this work, six EEG channels were selected (FP1, FP2, F3, F4, F7, and F8), and EEG images were generated from spectrogram fusions. The complete CNN architecture included training and acceleration for efficient artificial intelligence (AI) edge application, and we proposed a multiphase CNN execution method to accommodate hardware resource constraints. Datasets of 32 subjects from the DEAP database were used to validate the proposed design, exhibiting mean accuracies for valance binary classification and valance-arousal quaternary classification of 83.36% and 76.67%, respectively. The core area and total power consumption of the CNN chip were 1.83 x 1.83 mm 2 , respectively, and 76.61 mW. The chip operation was validated using ADVANTEST V93000 PS1600, and the training process and real-time classification processing time took 0.12495 ms and 0.02634 ms for each EEG image, respectively. The proposed EEG-based realtime emotion recognition system included a dry electrode EEG headset, feature extraction processor, CNN chip platform, and graphical user interface, and the execution time costed 450 ms for each emotional state recognition.