Boosting Photon-Efficient Image Reconstruction With A Unified Deep Neural Network

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 4; pp. 4180 - 4197
• Main Authors: Peng, Jiayong, Xiong, Zhiwei, Tan, Hao, Huang, Xin, Li, Zheng-Ping, Xu, Feihu
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Background noise; Coders; Computational photography; Correlation; Decoders; deep learning; Detectors; Image reconstruction; Imaging; Neural networks; Photonics; Photons; Single-photon avalanche diodes; single-photon imaging; Three-dimensional displays
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2022.3200745

Photon-efficient imaging, which captures 3D images with single-photon sensors, has enabled a wide range of applications. However, two major challenges limit the reconstruction performance, i.e., the low photon counts accompanied by low signal-to-background ratio (SBR) and the multiple returns. In this paper, we propose a unified deep neural network that, for the first time, explicitly addresses these two challenges, and simultaneously recovers depth maps and intensity images from photon-efficient measurements. Starting from a general image formation model, our network is constituted of one encoder, where a non-local block is utilized to exploit the long-range correlations in both spatial and temporal dimensions of the raw measurement, and two decoders, which are designed to recover depth and intensity, respectively. Meanwhile, we investigate the statistics of the background noise photons and propose a noise prior block to further improve the reconstruction performance. The proposed network achieves decent reconstruction fidelity even under extremely low photon counts / SBR and heavy blur caused by the multiple-return effect, which significantly surpasses the existing methods. Moreover, our network trained on simulated data generalizes well to real-world imaging systems, which greatly extends the application scope of photon-efficient imaging in challenging scenarios with a strict limit on optical flux. Code is available at https://github.com/JiayongO-O/PENonLocal .