Real-time emotion recognition system with multiple physiological signals

• Published in: Journal of Advanced Mechanical Design, Systems, and Manufacturing Vol. 13; no. 4; p. JAMDSM0075
• Main Authors: ZHUANG, Jyun-Rong, GUAN, Ya-Jing, NAGAYOSHI, Hayato, MURAMATSU, Keiichi, WATANUKI, Keiichi, TANAKA, Eiichiro
• Format: Journal Article
• Language: English
• Published: The Japan Society of Mechanical Engineers 01.01.2019
• Subjects: Deep neural networks; Emotional evaluation; Multiple physiological signals; Real-time emotion recognition system; Two-dimensional emotion model
• ISSN: 1881-3054; 1881-3054
• DOI: 10.1299/jamdsm.2019jamdsm0075

Emotion is an internal and subjective experience that plays a significant role in human life. There are several methods of recognizing emotions in people, the most authentic of which is using physiological signals, as they are beyond one’s control and strongly correlated with human emotions. This study aims to develop an emotion recognition system based on three physiological signals, namely, brainwave, heartbeat, and facial muscular activity. It utilizes deep neural network (DNN) and the T method of Mahalanobis-Taguchi system (MTS) to process the multiple physiological signals and further recognize the states of human emotion. As such, nine emotions are effectively recognized on a two-dimensional model through the DNN, then compared against several other algorithms, such as MTS, SVM, Naive Bayes, and K-means, where its superior accuracy is validated. Moreover, although the T method only improves the classification accuracy on the valence state, it rather obtains the intensity of emotion in different states. Furthermore, in this study, the proposed DNN is implemented into a wide range of applications for an accurate understanding of the human emotional states, whereas the T method is utilized to respond to the emotional intensity in different states. Finally, a real-time emotion recognition system is developed with DNN as the classifier; this system can directly monitor the variation of the human emotion through reliable and objective emotion analysis results from the physiological signals. Thus, the method can provide useful treatment effect information for robots or assistive apparatus serving activities of daily living.