A vision-based deep learning approach for independent-users Arabic sign language interpretation

• Published in: Multimedia tools and applications Vol. 82; no. 5; pp. 6807 - 6826
• Main Authors: Balaha, Mostafa Magdy, El-Kady, Sara, Balaha, Hossam Magdy, Salama, Mohamed, Emad, Eslam, Hassan, Muhammed, Saafan, Mahmoud M.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2023; Springer Nature B.V
• Subjects: Artificial neural networks; Communication; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Interpreters; Multimedia Information Systems; Neural networks; Recognition; Recurrent neural networks; Sign language; Special Purpose and Application-Based Systems; Arabic Sign Language (ASL); Recurrent Neural Network (RNN); Deep Learning (DL); Video recognition; Sign Language Recognition (SLR); Convolutional Neural Network (CNN)
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-13423-9

More than 5% of the people around the world are deaf and have severe difficulties in communicating with normal people according to the World Health Organization (WHO). They face a real challenge to express anything without an interpreter for their signs. Nowadays, there are a lot of studies related to Sign Language Recognition (SLR) that aims to reduce this gap between deaf and normal people as it can replace the need for an interpreter. However, there are a lot of challenges facing the sign recognition systems such as low accuracy, complicated gestures, high-level noise, and the ability to operate under variant circumstances with the ability to generalize or to be locked to such limitations. Hence, many researchers proposed different solutions to overcome these problems. Each language has its signs and it can be very challenging to cover all the languages’ signs. The current study objectives: (i) presenting a dataset of 20 Arabic words, and (ii) proposing a deep learning (DL) architecture by combining convolutional neural network (CNN) and recurrent neural network (RNN). The suggested architecture reported 98% accuracy on the presented dataset. It also reported 93.4% and 98.8% for the top-1 and top-5 accuracies on the UCF-101 dataset.