Simultaneous Hand Gesture Classification and Finger Angle Estimation via a Novel Dual-Output Deep Learning Model

• Published in: Sensors (Basel, Switzerland) Vol. 20; no. 10; p. 2972
• Main Authors: Gao, Qinghua, Jiang, Shuo, Shull, Peter B.
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 24.05.2020; MDPI
• Subjects: Accuracy; Algorithms; Classification; convolutional neural network; Deep Learning; dual-output model; Experiments; Fingers; Fingers & toes; gesture recognition; Gestures; Hand; Humans; Machine learning; modified barometric sensor; Neural networks; Pressure distribution; Sensors; Sign language; Support vector machines; Time series; modified barometric sensor; dual-output model; gesture recognition; convolutional neural network
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s20102972

Hand gesture classification and finger angle estimation are both critical for intuitive human–computer interaction. However, most approaches study them in isolation. We thus propose a dual-output deep learning model to enable simultaneous hand gesture classification and finger angle estimation. Data augmentation and deep learning were used to detect spatial-temporal features via a wristband with ten modified barometric sensors. Ten subjects performed experimental testing by flexing/extending each finger at the metacarpophalangeal joint while the proposed model was used to classify each hand gesture and estimate continuous finger angles simultaneously. A data glove was worn to record ground-truth finger angles. Overall hand gesture classification accuracy was 97.5% and finger angle estimation R 2 was 0.922, both of which were significantly higher than shallow existing learning approaches used in isolation. The proposed method could be used in applications related to the human–computer interaction and in control environments with both discrete and continuous variables.