ECSNet: Spatio-Temporal Feature Learning for Event Camera

• Published in: IEEE transactions on circuits and systems for video technology Vol. 33; no. 2; pp. 701 - 712
• Main Authors: Chen, Zhiwen, Wu, Jinjian, Hou, Junhui, Li, Leida, Dong, Weisheng, Shi, Guangming
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: action recognition; Brightness; Cameras; Cloud computing; Data mining; Event camera; Feature extraction; Machine learning; Modules; Moving object recognition; object classification; Representation learning; Representations; Robustness; Sparsity; spatio-temporal feature learning; Task analysis; Weight reduction
• DOI: 10.1109/TCSVT.2022.3202659
• ISSN: 1051-8215

The neuromorphic event cameras can efficiently sense the latent geometric structures and motion clues of a scene by generating asynchronous and sparse event signals. Due to the irregular layout of the event signals, how to leverage their plentiful spatio-temporal information for recognition tasks remains a significant challenge. Existing methods tend to treat events as dense image-like or point-serie representations. However, they either suffer from severe destruction on the sparsity of event data or fail to encode robust spatial cues. To fully exploit their inherent sparsity with reconciling the spatio-temporal information, we introduce a compact event representation, namely 2D-1T event cloud sequence (2D-1T ECS). We couple this representation with a novel light-weight spatio-temporal learning framework (ECSNet) that accommodates both object classification and action recognition tasks. The core of our framework is a hierarchical spatial relation module. Equipped with specially designed surface-event-based sampling unit and local event normalization unit to enhance the inter-event relation encoding, this module learns robust geometric features from the 2D event clouds. And we propose a motion attention module for efficiently capturing long-term temporal context evolving with the 1T cloud sequence. Empirically, the experiments show that our framework achieves par or even better state-of-the-art performance. Importantly, our approach cooperates well with the sparsity of event data without any sophisticated operations, hence leading to low computational costs and prominent inference speeds.