Jointly Learning Heterogeneous Features for RGB-D Activity Recognition

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 39; no. 11; pp. 2186 - 2200
• Main Authors: Hu, Jian-Fang, Zheng, Wei-Shi, Lai, Jianhuang, Zhang, Jianguo
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: action recognition; Activity recognition; Channels; Datasets; Feature extraction; Feature recognition; Heterogeneous features learning; Image color analysis; Iterative methods; Learning; RGB-D activity recognition; Skeleton; Subspaces; Three-dimensional displays; Transforms; Visualization
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2016.2640292

In this paper, we focus on heterogeneous features learning for RGB-D activity recognition. We find that features from different channels (RGB, depth) could share some similar hidden structures, and then propose a joint learning model to simultaneously explore the shared and feature-specific components as an instance of heterogeneous multi-task learning. The proposed model formed in a unified framework is capable of: 1) jointly mining a set of subspaces with the same dimensionality to exploit latent shared features across different feature channels, 2) meanwhile, quantifying the shared and feature-specific components of features in the subspaces, and 3) transferring feature-specific intermediate transforms (i-transforms) for learning fusion of heterogeneous features across datasets. To efficiently train the joint model, a three-step iterative optimization algorithm is proposed, followed by a simple inference model. Extensive experimental results on four activity datasets have demonstrated the efficacy of the proposed method. Anew RGB-D activity dataset focusing on human-object interaction is further contributed, which presents more challenges for RGB-D activity benchmarking.