GaitSTR: Gait Recognition With Sequential Two-Stream Refinement

• Published in: IEEE transactions on biometrics, behavior, and identity science Vol. 6; no. 4; pp. 528 - 538
• Main Authors: Zheng, Wanrong, Zhu, Haidong, Zheng, Zhaoheng, Nevatia, Ram
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.10.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Annotations; Bones; Feature extraction; Gait recognition; Image quality; joint and bone; Joints; Joints (anatomy); Legged locomotion; Pose estimation; Segments; Skeleton; skeletons representation; temporal refinement
• ISSN: 2637-6407; 2637-6407
• DOI: 10.1109/TBIOM.2024.3390626

Gait recognition aims to identify a person based on their walking sequences, serving as a useful biometric modality as it can be observed from long distances without requiring cooperation from the subject. In representing a person's walking sequence, silhouettes and skeletons are the two primary modalities used. Silhouette sequences lack detailed part information when overlapping occurs between different body segments and are affected by carried objects and clothing. Skeletons, comprising joints and bones connecting the joints, provide more accurate part information for different segments; however, they are sensitive to occlusions and low-quality images, causing inconsistencies in frame-wise results within a sequence. In this paper, we explore the use of a two-stream representation of skeletons for gait recognition, alongside silhouettes. By fusing the combined data of silhouettes and skeletons, we refine the two-stream skeletons, joints, and bones through self-correction in graph convolution, along with cross-modal correction with temporal consistency from silhouettes. We demonstrate that with refined skeletons, the performance of the gait recognition model can achieve further improvement on public gait recognition datasets compared with state-of-the-art methods without extra annotations.