DepthInSpace: Exploitation and Fusion of Multiple Video Frames for Structured-Light Depth Estimation

• Published in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 6019 - 6028
• Main Authors: Johari, Mohammad Mahdi, Carta, Camilla, Fleuret, Francois
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2021
• Subjects: 3D from a single image and shape-from-x; 3D from multiview and other sensors; Deep learning; Design methodology; Estimation; Fuses; Lighting; Sensor fusion; Stereo; Training; Transfer/Low-shot/Semi/Unsupervised Learning
• ISSN: 2380-7504
• DOI: 10.1109/ICCV48922.2021.00598

We present DepthInSpace, a self-supervised deep-learning method for depth estimation using a structured-light camera. The design of this method is motivated by the commercial use case of embedded depth sensors in nowadays smartphones. We first propose to use estimated optical flow from ambient information of multiple video frames as a complementary guide for training a single-frame depth estimation network, helping to preserve edges and reduce over-smoothing issues. Utilizing optical flow, we also propose to fuse the data of multiple video frames to get a more accurate depth map. In particular, fused depth maps are more robust in occluded areas and incur less in flying pixels artifacts. We finally demonstrate that these more precise fused depth maps can be used as self-supervision for fine-tuning a single-frame depth estimation network to improve its performance. Our models' effectiveness is evaluated and compared with state-of-the-art models on both synthetic and our newly introduced real datasets. The implementation code, training procedure, and both synthetic and captured real datasets are available at https://www.idiap.ch/paper/depthinspace.