3D Hand Gestures Segmentation and Optimized Classification Using Deep Learning

Hand gestures recognition system has massive applications which are mainly utilized in robotics and computer vision specially to control Unmanned Aerial Vehicles (UAV). These methods bypass the presence of electronic control to UAVs and provide an ease to the operators. In this paper, we present a m...

Full description

Saved in:
Bibliographic Details
Published inIEEE access Vol. 9; pp. 131614 - 131624
Main Authors Khan, Fawad Salam, Mohd, Mohd Norzali Haji, Soomro, Dur Muhammad, Bagchi, Susama, Khan, M. Danial
Format Journal Article
LanguageEnglish
Published Piscataway IEEE 2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
3D hand gestures
Classification
Computer vision
Datasets
Deep learning
Degrees of freedom
depth images
Electronic control
Feature extraction
Gesture recognition
grass-hopper optimization
Grasses
Kinematics
Machine learning
Mathematical analysis
Motion segmentation
Optimization
Pose estimation
Robotics
Segmentation
Three-dimensional displays
Unmanned aerial vehicles
Online AccessGet full text

Cover

More Information
Summary:Hand gestures recognition system has massive applications which are mainly utilized in robotics and computer vision specially to control Unmanned Aerial Vehicles (UAV). These methods bypass the presence of electronic control to UAVs and provide an ease to the operators. In this paper, we present a method for 3D hand gestures segmentation and classification by combining MASK-RCNN with Grass Hopper Optimization. We created a private 3D and RGB hand gestures dataset using Intel Kinetic and Intel Real sense d435i camera, then proposed a model for RGB hand gestures to estimate the key points using human kinematics, the key points later then utilize to get the best degree of freedom (DoF). The grass hopper optimization besides minimum distance function was applied to achieve the finest deep features from the 3D hand gestures dataset. The ResNet50 network is used as the backbone to calculate the Overlap Coefficient (OC) for segmentation and the ResNet50, ResNet101 networks to calculate the classification for 3D hand gestures. The classification accuracy achieved on the private dataset is 99.05% and 99.29% on public Microsoft Kinect and Leap Motion dataset where the OC are 88.16%. and 88.19% respectively.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2021.3114871