Linguistic-Driven Partial Semantic Relevance Learning for Skeleton-Based Action Recognition

• Published in: Sensors (Basel, Switzerland) Vol. 24; no. 15; p. 4860
• Main Authors: Chen, Qixiu, Liu, Yingan, Huang, Peng, Huang, Jiani
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 26.07.2024
• Subjects: Algorithms; Analysis; Computational linguistics; cross-modal; Deep learning; Human mechanics; Humans; Language; Language processing; Large language models; Learning - physiology; Linguistics; Movement - physiology; Natural language interfaces; Neural networks; Pattern Recognition, Automated - methods; Semantics; skeleton-based action recognition; transformer; transformer; skeleton-based action recognition; cross-modal
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s24154860

Skeleton-based action recognition, renowned for its computational efficiency and indifference to lighting variations, has become a focal point in the realm of motion analysis. However, most current methods typically only extract global skeleton features, overlooking the potential semantic relationships among various partial limb motions. For instance, the subtle differences between actions such as "brush teeth" and "brush hair" are mainly distinguished by specific elements. Although combining limb movements provides a more holistic representation of an action, relying solely on skeleton points proves inadequate for capturing these nuances. Therefore, integrating detailed linguistic descriptions into the learning process of skeleton features is essential. This motivates us to explore integrating fine-grained language descriptions into the learning process of skeleton features to capture more discriminative skeleton behavior representations. To this end, we introduce a new Linguistic-Driven Partial Semantic Relevance Learning framework (LPSR) in this work. While using state-of-the-art large language models to generate linguistic descriptions of local limb motions and further constrain the learning of local motions, we also aggregate global skeleton point representations and textual representations (which generated from an LLM) to obtain a more generalized cross-modal behavioral representation. On this basis, we propose a cyclic attentional interaction module to model the implicit correlations between partial limb motions. Numerous ablation experiments demonstrate the effectiveness of the method proposed in this paper, and our method also obtains state-of-the-art results.