A depth-based Indian Sign Language recognition using Microsoft Kinect

• Published in: Sadhana (Bangalore) Vol. 45; no. 1
• Main Authors: Raghuveera, T, Deepthi, R, Mangalashri, R, Akshaya, R
• Format: Journal Article
• Language: English
• Published: New Delhi Springer India 01.12.2020; Springer Nature B.V
• Subjects: Algorithms; Color imagery; Engineering; Feature extraction; Histograms; Human communication; Sentences; Sign language; Support vector machines; Translating; SURF; HOG; gesture translation; ISL gesture recognition; multi-class SVM; depth based; LBP
• ISSN: 0256-2499; 0973-7677
• DOI: 10.1007/s12046-019-1250-6

Recognition of sign language by a system has become important to bridge the communication gap between the abled and the Hearing and Speech Impaired people. This paper introduces an efficient algorithm for translating the input hand gesture in Indian Sign Language (ISL) into meaningful English text and speech. The system captures hand gestures through Microsoft Kinect (preferred as the system performance is unaffected by the surrounding light conditions and object colour). The dataset used consists of depth and RGB images (taken using Kinect Xbox 360) with 140 unique gestures of the ISL taken from 21 subjects, which includes single-handed signs, double-handed signs and fingerspelling (signs for alphabets and numbers), totaling to 4600 images. To recognize the hand posture, the hand region is accurately segmented and hand features are extracted using Speeded Up Robust Features, Histogram of Oriented Gradients and Local Binary Patterns. The system ensembles the three feature classifiers trained using Support Vector Machine to improve the average recognition accuracy up to 71.85%. The system then translates the sequence of hand gestures recognized into the best approximate meaningful English sentences. We achieved 100% accuracy for the signs representing 9, A, F, G, H, N and P.