Consistent Direct Time-of-Flight Video Depth Super-Resolution

• Published in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5075 - 5085
• Main Authors: Sun, Zhanghao, Ye, Wei, Xiong, Jinhui, Choe, Gyeongmin, Wang, Jialiang, Su, Shuochen, Ranjan, Rakesh
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2023
• Subjects: 3D from multi-view and sensors; Manufacturing; Mobile handsets; Pattern recognition; Sensor phenomena and characterization; Sensors; Superresolution; Three-dimensional displays
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52729.2023.00491

Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, limited by manufacturing capabilities in a compact module, the dToF data has a low spatial resolution (e.g. \sim 20\times 30 for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches, which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate spatial ambiguity. To evaluate our models on complex dynamic indoor environments and to provide a large-scale dToF sensor dataset, we introduce Dy-DToF, the first synthetic RGB-dToF video dataset that features dynamic objects and a realistic dToF simulator following the physical imaging process. We believe the methods and dataset are beneficial to a broad community as dToF depth sensing is becoming mainstream on mobile devices. Our code and data are publicly available. https://github.com/facebookresearch/DVSR/