Deep motion templates and extreme learning machine for sign language recognition

• Published in: The Visual computer Vol. 36; no. 6; pp. 1233 - 1246
• Main Authors: Imran, Javed, Raman, Balasubramanian
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.06.2020; Springer Nature B.V
• Subjects: Accuracy; Artificial Intelligence; Artificial neural networks; Classification; Computer Graphics; Computer Science; Datasets; Deep learning; Human activity recognition; Human motion; Image Processing and Computer Vision; Machine learning; Motion perception; Neural networks; Original Article; Recognition; Sign language; Sign language recognition; Dynamic image; Late fusion; Extreme learning machine; Motion history image; Convolutional neural network
• DOI: 10.1007/s00371-019-01725-3
• ISSN: 0178-2789

Sign language is a visual language used by persons with hearing and speech impairment to communicate through fingerspellings and body gestures. This paper proposes a framework for automatic sign language recognition without the need of hand segmentation. The proposed method first generates three different types of motion templates: motion history image, dynamic image and our proposed RGB motion image. These motion templates are used to fine-tune three ConvNets trained on ImageNet dataset. Fine-tuning avoids learning all the parameters from scratch, leading to faster network convergence even with a small number of training samples. For combining the output of three ConvNets, we propose a fusion technique based on Kernel-based extreme learning machine (KELM). The features extracted from the last fully connected layer of trained ConvNets are used to train three KELMs, and the final class label is predicted by averaging their scores. The proposed approach is validated on a number of publicly available sign language as well as human action recognition datasets, and state-of-the-art results are achieved. Finally, an Indian sign language dataset is also collected using a thermal camera. The experimental results obtained show that our ConvNet-based deep features along with proposed KELM-based fusion are robust for any type of human motion recognition.