Deep learning-based multi-view 3D-human action recognition using skeleton and depth data

• Published in: Multimedia tools and applications Vol. 82; no. 13; pp. 19829 - 19851
• Main Authors: Ghosh, Sampat Kumar, M, Rashmi, Mohan, Biju R, Guddeti, Ram Mohana Reddy
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2023; Springer Nature B.V
• Subjects: Algorithms; Artificial intelligence; Artificial neural networks; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Edge detection; Human activity recognition; Machine learning; Motion perception; Multimedia; Multimedia Information Systems; Neural networks; Special Purpose and Application-Based Systems; Surveillance systems; Deep learning; Human action recognition; Feature fusion; Convolutional neural networks; Score fusion
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-14214-y

Human Action Recognition (HAR) is a fundamental challenge that smart surveillance systems must overcome. With the rising affordability of capturing human actions with more advanced depth cameras, HAR has garnered increased interest over the years, however the majority of these efforts have been on single-view HAR. Recognizing human actions from arbitrary viewpoints is more challenging, as the same action is observed differently from different angles. This paper proposes a multi-stream Convolutional Neural Network (CNN) model for multi-view HAR using depth and skeleton data. We also propose a novel and efficient depth descriptor, Edge Detected-Motion History Image (ED-MHI), based on Canny Edge Detection and Motion History Image. Also, the proposed skeleton descriptor, Motion and Orientation of Joints (MOJ), represent the appropriate action by using joint motion and orientation. Experimental results on two datasets of human actions: NUCLA Multiview Action3D and NTU RGB-D using a Cross-subject evaluation protocol demonstrated that the proposed system exhibits the superior performance as compared to the state-of-the-art works with 93.87% and 85.61% accuracy, respectively.