A deep perceptual framework for affective video tagging through multiband EEG signals modeling

• Published in: Neural computing & applications Vol. 37; no. 11; pp. 7657 - 7674
• Main Authors: Sharma, Shanu, Dubey, Ashwani Kumar, Ranjan, Priya, Rocha, Alvaro
• Format: Journal Article
• Language: English
• Published: London Springer London 01.04.2025; Springer Nature B.V
• Subjects: Arousal; Artificial Intelligence; Artificial neural networks; Classification; Clips; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Data Mining and Knowledge Discovery; Electroencephalography; Human-computer interface; Image Processing and Computer Vision; Machine learning; Modelling; Multimedia; Neural networks; Power spectral density; Probability and Statistics in Computer Science; Representations; S.I.: Knowledge services: challenges and future prospects; Special Issue on Knowledge Services: Challenges and Future Prospects; Video; CNN; Video; EEG; Implicit tagging; Arousal; Affect; Excitement
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-023-09086-8

Nowadays, multimedia content, such as photographs and movies, is ingrained in every aspect of human lives and has become a vital component of their entertainment. Multimedia content, such as videos or movie clips, is typically created with the intent to evoke certain feelings or emotions in viewers. Thus, by examining the viewer’s cognitive state while watching such content, its affectiveness can be evaluated. Considering the emotional aspect of videos, in this paper, a deep learning-based paradigm for affective tagging of video clips is proposed, in which participants’ irrational EEG responses are used to examine how people perceive videos. The information behind different brain regions, frequency waves, and connections among them play an important role in understanding a human’s cognitive state. Thus, here a contribution is made toward the effective modeling of EEG signals through two different representations, i.e., spatial feature matrix and combined power spectral density maps. The proposed feature representations highlight the spatial features of EEG signals and are therefore used to train a convolution neural network model for implicit tagging of two categories of videos in the Arousal domain, i.e., “Low Arousal” and “High Arousal.” The arousal emotional space represents the excitement level of the viewer; thus, this domain is selected to analyze the viewer’s engagement while watching video clips. The proposed model is developed using the EEG data taken from publicly available datasets “AMIGOS” and “DREAMER.” The model is tested using two different approaches, i.e., single-subject classification and multi-subject classification, and an average accuracy of 90%-95% and 90%-93% is achieved, respectively. The simulations presented in this paper show the pioneering applicability of the proposed framework for the development of brain–computer interface (BCI) devices for affective tagging of videos.