Depth from Defocus with Learned Optics for Imaging and Occlusion-aware Depth Estimation

• Published in: IEEE International Conference on Computational Photography pp. 1 - 12
• Main Authors: Ikoma, Hayato, Nguyen, Cindy M., Metzler, Christopher A., Peng, Yifan, Wetzstein, Gordon
• Format: Conference Proceeding
• Language: English
• Published: IEEE 23.05.2021
• Subjects: Apertures; Computational modeling; Computational Optics; Computational Photography; Estimation; Neural networks; Optics; Photography; Prototypes
• DOI: 10.1109/ICCP51581.2021.9466261

Monocular depth estimation remains a challenging problem, despite significant advances in neural network architectures that leverage pictorial depth cues alone. Inspired by depth from defocus and emerging point spread function engineering approaches that optimize programmable optics end-to-end with depth estimation networks, we propose a new and improved framework for depth estimation from a single RGB image using a learned phase-coded aperture. Our optimized aperture design uses rotational symmetry constraints for computational efficiency, and we jointly train the optics and the network using an occlusion-aware image formation model that provides more accurate defocus blur at depth discontinuities than previous techniques do. Using this framework and a custom prototype camera, we demonstrate state-of-the art image and depth estimation quality among end-to-end optimized computational cameras in simulation and experiment.