Spike-NeRF: Neural Radiance Field Based On Spike Camera

• Published in: Proceedings (IEEE International Conference on Multimedia and Expo) pp. 1 - 6
• Main Authors: Guo, Yijia, Bai, Yuanxi, Hu, Liwen, Liu, Mianzhi, Guo, Ziyi, Ma, Lei, Huang, Tiejun
• Format: Conference Proceeding
• Language: English
• Published: IEEE 15.07.2024
• Subjects: Cameras; Estimation; Image reconstruction; Neural radiance field; Neuromorphic Vision; Neuromorphics; Optical sensors; Spike Camera; Stability analysis; Streams; Synthetic data; Three-dimensional displays
• ISSN: 1945-788X
• DOI: 10.1109/ICME57554.2024.10687382

As a neuromorphic sensor with high temporal resolution, spike cameras offer notable advantages over traditional cameras in high-speed vision applications such as high-speed optical estimation, depth estimation, and object tracking. Inspired by the success of the spike camera, we proposed Spike-NeRF, the first Neural Radiance Field derived from spike data, to achieve 3D reconstruction and novel viewpoint synthesis for high-speed scenes. Instead of the multi-view images at the same as time of NeRF, the inputs of Spike-NeRF are continuous spike streams captured by a moving spike camera in a very short time. To reconstruct a correct and stable 3D scene from high-frequency but unstable spike data, we devised spike masks along with a distinctive loss function. We evaluate our method qualitatively and quantitatively on several challenging synthetic scenes generated using Blender with the spike camera simulator. Our results demonstrate that Spike-NeRF produces more visually appealing results than the existing methods and the baseline we proposed in high-speed scenes. Our code is available at https://github.com/yijiaguo02/SpikeNerf