Temporal signed gestures segmentation in an image sequence using deep reinforcement learning

• Published in: Engineering applications of artificial intelligence Vol. 131; p. 107879
• Main Authors: Kalandyk, Dawid, Kapuściński, Tomasz
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.05.2024
• Subjects: Deep reinforcement learning; Gesture database; Gesture spotting; Image sequence segmentation; Deep reinforcement learning; Image sequence segmentation; Gesture database; Gesture spotting
• DOI: 10.1016/j.engappai.2024.107879
• ISSN: 0952-1976

Continuous sign language recognition is challenging due to coarticulatory distortions, which occur at the beginning and end of each gesture. These distortions depend on the temporal context and introduce additional intraclass variability. To address this issue, a new approach is proposed that extracts segments from the image sequence corresponding to undistorted parts of gestures. This should simplify the task by reducing it to the easier problem of isolated gestures recognition. The proposed approach uses deep reinforcement learning for segmentation and a novel image sequence processing scheme to extract gradient changes over time. A dataset recorded by deaf people and annotated according to the proposed approach, was prepared to evaluate the method. The proposed deep learning architectures achieved leave-one-subject-out recognition accuracies in the range of 0.70 to 0.76. Considering the inability to compare with other works, the authors also proposed other evaluation protocols to thoroughly examine the employed approach. This work will be developed, and the main aspiration of the authors will be to create an integrated framework that converts the raw form of RGB video into a string of words representing the Sign Language user’s intentions. •Specialized preprocessing video algorithm provides a comprehensive description.•Dynamic gestures extraction done by Deep Reinforcement Learning.•Dedicated database allows to tackle with Sign Language co-articulation problem.•Different Deep Convolutional Neural Network architectures investigation.•Filtering and Voting for segmentation accuracy improvement.