Hand Gesture Recognition for Multi-Culture Sign Language Using Graph and General Deep Learning Network

• Published in: IEEE open journal of the Computer Society Vol. 5; pp. 1 - 12
• Main Authors: Miah, Abu Saleh Musa, Hasan, Md. Al Mehedi, Tomioka, Yoichi, Shin, Jungpil
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: American Sign Language (ASL); Artificial neural networks; Bangla Sign Language (BSL); Convolutional neural networks; Cultural differences; Culture; Deep learning; Feature extraction; GCN; Gesture recognition; Graph meets with Attention and CNN (GmTC); Hand Gesture Recognition (HGR); Hidden Markov models; Japanese Sign Language (JSL); Korean sign language (KSL); Machine learning; McSL; MHSA; Modules; Sign language; SLR; Transformers
• ISSN: 2644-1268; 2644-1268
• DOI: 10.1109/OJCS.2024.3370971

Hand gesture-based Sign Language Recognition (SLR) serves as a crucial communication bridge between deaf and non-deaf individuals. The absence of a universal sign language (SL) leads to diverse nationalities having various cultural SLs, such as Korean, American, and Japanese sign language. Existing SLR systems perform well for their cultural SL but may struggle with other or multi-cultural sign languages (McSL). To address these challenges, this paper introduces a novel end-to-end SLR system called GmTC, designed to translate McSL into equivalent text for enhanced understanding. Here, we employed a Graph and General deep-learning network as two stream modules to extract effective features. In the first stream, produce a graph-based feature by taking advantage of the superpixel values and the graph convolutional network (GCN), aiming to extract distance-based complex relationship features among the superpixel. In the second stream, we extracted long-range and short-range dependency features using attention-based contextual information that passes through multi-stage, multi-head self-attention (MHSA), and CNN modules. Combining these features generates final features that feed into the classification module. Extensive experiments with four culture SL datasets with high-performance accuracy compared to existing state-of-the-art models in individual domains affirming superiority and generalizability.