Cross-Cultural Emotion Recognition With EEG and Eye Movement Signals Based on Multiple Stacked Broad Learning System

• Published in: IEEE transactions on computational social systems Vol. 11; no. 2; pp. 2014 - 2025
• Main Authors: Gong, Xinrong, Chen, C. L. Philip, Zhang, Tong
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.04.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Affective computing; Brain; Brain modeling; broad learning system (BLS); Cognition; Computational modeling; cross-cultural emotion recognition; Cultural aspects; electroencephalogram (EEG); Electroencephalography; Emotion recognition; Emotions; eye movement (EM); Eye movements; Feature extraction; Frequencies; Globalization; Learning; multimodal fusion; Occipital lobes; Portable equipment
• ISSN: 2329-924X; 2373-7476
• DOI: 10.1109/TCSS.2023.3298324

With increasing social globalization, interaction between people from different cultures has become more frequent. However, there are significant differences in the expression and comprehension of emotions across cultures. Therefore, developing computational models that can accurately identify emotions among different cultures has become a significant research problem. This study aims to investigate the similarities and differences in emotion cognition processes in different cultural groups by employing a fusion of electroencephalography (EEG) and eye movement (EM) signals. Specifically, an effective adaptive region selection method is proposed to investigate the most emotion-related activated brain regions in different groups. By selecting these commonly activated regions, we can eliminate redundant features and facilitate the development of portable acquisition devices. Subsequently, the multiple stacked broad learning system (MSBLS) is designed to explore the complementary information of EEG and EM features and the effective emotional information still contained in the residual value. The intracultural subject-dependent (ICSD), intracultural subject-independent, and cross-cultural subject-independent (CCSI) experiments have been conducted on the SEED-CHN, SEED-GER, and SEED-FRA datasets. Extensive experiments manifest that MSBLS achieves superior performance compared with current state-of-the-art methods. Moreover, we discover that some brain regions (the anterior frontal, temporal, and middle parieto-occipital lobes) and Gamma frequency bands show greater activation during emotion cognition in diverse cultural groups.