A Gait Energy Image-Based System for Brazilian Sign Language Recognition

• Published in: IEEE transactions on circuits and systems. I, Regular papers Vol. 68; no. 11; pp. 4761 - 4771
• Main Authors: Passos, Wesley L., Araujo, Gabriel M., Gois, Jonathan N., de Lima, Amaro A.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.11.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Assistive technology; Body parts; Cameras; Computer architecture; computer vision; Datasets; Evaluation; gait energy image; Gesture recognition; Image recognition; Libras; Object recognition; Performance evaluation; Sign language; Singular value decomposition; Support vector machines; Training; Video data; Wearable technology
• ISSN: 1549-8328; 1558-0806
• DOI: 10.1109/TCSI.2021.3091001

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with singular value decomposition and support vector machine .