Attention adjacency matrix based graph convolutional networks for skeleton-based action recognition

• Published in: Neurocomputing (Amsterdam) Vol. 440; pp. 230 - 239
• Main Authors: Xie, Jun, Miao, Qiguang, Liu, Ruyi, Xin, Wentian, Tang, Lei, Zhong, Sheng, Gao, Xuesong
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 14.06.2021
• Subjects: Attention adjacency matrix; Attention mechanism; Graph convolutional networks; Skeleton-based action recognition; Skeleton-based action recognition; Attention mechanism; Graph convolutional networks; Attention adjacency matrix
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2021.02.001

Recent progress on human action recognition, fueled by the Graph Convolutional Network (GCN), has been substantial. However, two main problems are caused by the design strategy of graph convolution kernels: first, the partitioning strategy of neighbor set for graph vertices relies on the gravity center designed manually, which is limited in generalizability to diverse skeletons in action recognition; second, the existing GCN-based methods can only capture local physical dependencies among joints and result in missing implicit joint correlations due to over-smoothing. In this work, we present (1) a novel attention adjacency matrix (AAM) to design graph convolution kernels and (2) a dimension-attention block to improve the robustness of the model. Specifically, the proposed AAM is designed by a novel partitioning strategy for the neighbor set, through which an adjacency matrix is decomposed into several parametric matrices. Simultaneously, attention mechanism is introduced in the process to generate an attention matrix. Combining the matrix and the parametric matrices into an AAM through ResNet, we further exhibit the AAM based graph convolution network (AAM-GCN). The proposed dimension-attention block strengthens the important information in each dimension of skeleton data by extending the idea of channel-attention. Extensive experiments on two large-scale datasets, NTU-RGB+D and Kinetics, demonstrate that AAM-GCN achieves better performance than the state-of-the-art works.