Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM

• Published in: Applied sciences Vol. 10; no. 18; p. 6293
• Main Authors: Do, Nhu-Tai, Kim, Soo-Hyung, Yang, Hyung-Jeong, Lee, Guee-Sang
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.09.2020
• Subjects: Datasets; Deep learning; Dynamic Hand Gesture Recognition; Feature recognition; Geometric transformation; Gesture recognition; hand shape features; Hands; human-computer interaction; Methods; Posture; Segmentation; Sensors; Sign language; Skeleton; Virtual reality
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app10186293

This study builds robust hand shape features from the two modalities of depth and skeletal data for the dynamic hand gesture recognition problem. For the hand skeleton shape approach, we use the movement, the rotations of the hand joints with respect to their neighbors, and the skeletal point-cloud to learn the 3D geometric transformation. For the hand depth shape approach, we use the feature representation from the hand component segmentation model. Finally, we propose a multi-level feature LSTM with Conv1D, the Conv2D pyramid, and the LSTM block to deal with the diversity of hand features. Therefore, we propose a novel method by exploiting robust skeletal point-cloud features from skeletal data, as well as depth shape features from the hand component segmentation model in order for the multi-level feature LSTM model to benefit from both. Our proposed method achieves the best result on the Dynamic Hand Gesture Recognition (DHG) dataset with 14 and 28 classes for both depth and skeletal data with accuracies of 96.07% and 94.40%, respectively.