AST-GCN: Augmented Spatial Temporal Graph Convolutional Neural Network for Gait Emotion Recognition

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 6; pp. 4581 - 4595
• Main Authors: Chen, Chuang, Sun, Xiao, Tu, Zhengzheng, Wang, Meng
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: affective computing; Artificial neural networks; Convolutional neural networks; Datasets; Emotion recognition; Emotions; Feature extraction; Gait emotion recognition; graph convolutional; Graph neural networks; Joints; Legged locomotion; Machine learning; Neural networks; Pedestrians; Support vector machines; Three-dimensional displays; Walking
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3341728

Skeleton-based methods have recently achieved good performance in deep learning-based gait emotion recognition (DL-GER). However, the current methods have two drawbacks that limit the ability to learn discriminative emotional features from gait. First, these methods do not exclude the effect of the subject's walking orientation on emotion classification. Second, they do not sufficiently learn the implicit connections between the joints during human walking. In this paper, an augmented spatial-temporal graph convolutional neural network (AST-GCN) is introduced to solve these two problems. The interframe shift encoding (ISE) module acquires interframe shifts of joints to make the network sensitive to changes in emotion-related joint movements regardless of the subject's walking orientation. A multichannel implicit connection inference method learns more implicit connection relations related to emotions. Notably, we unify current skeleton-based methods into a common framework that validates the most powerful feature representation capability of our AST-GCN from a theoretical perspective. In addition, we extend the skeleton-based gait dataset using posture estimation software. Experiments demonstrate that our AST-GCN outperforms state-of-the-art methods on three datasets on two tasks.