Capture the Moment: High-speed Imaging with Spiking Cameras through Short-term Plasticity

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 7; pp. 1 - 16
• Main Authors: Zheng, Yajing, Zheng, Lingxiao, Yu, Zhaofei, Huang, Tiejun, Wang, Song
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Computing time; Error correction; Firing; High speed; High-speed Reconstruction; Image reconstruction; Low latency communication; Motion-dependent; Noise reduction; Plastic properties; Reconstruction algorithms; Short-term Plasticity; Spiking; Spiking Cameras; Streaming media; Vision sensors
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2023.3237856

High-speed imaging can help us understand some phenomena that our eyes cannot capture fast enough. Although ultra-fast frame-based cameras (e.g., Phantom) can record millions of fps at reduced resolution, are too expensive to be widely used. Recently, a retina-inspired vision sensor, spiking camera, has been developed to record external information at 40, 000 Hz. The spiking camera uses the asynchronous binary spike streams to represent visual information. Despite this, how to reconstruct dynamic scenes from asynchronous spikes remains challenging. In this paper, we introduce novel high-speed image reconstruction models based on the short-term plasticity (STP) mechanism of the brain, termed TFSTP and TFMDSTP. We first derive the relationship between states of STP and spike patterns. Then, in TFSTP, by setting up the STP model at each pixel, the scene radiance can be inferred by the states of the models. In TFMDSTP, we use the STP to distinguish the moving and stationary regions, and then use two sets of STP models to reconstruct them respectively. In addition, we present a strategy for correcting error spikes. Experimental results show that the STP-based reconstruction methods can effectively reduce noise with less computing time, and achieve best the performances on both real-world and simulated datasets.