Acquiring 360° Light Field by a Moving Dual-Fisheye Camera

• Published in: IEEE transactions on image processing Vol. 32; pp. 4677 - 4688
• Main Authors: Lo, I-Chan, Chen, Homer H.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 360° cameras; 360° light field; Ablation; Angular resolution; Cameras; Consistency; convolutional neural network; Convolutional neural networks; depth estimation; Estimation; Fisheye views; Light; light field generation; Light fields; Matching; Pipelines; Resampling; Three-dimensional displays; Training
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2023.3299444

In this paper, we propose an efficient deep learning pipeline for light field acquisition using a back-to-back dual-fisheye camera. The proposed pipeline generates a light field from a sequence of 360° raw images captured by the dual-fisheye camera. It has three main components: a convolutional network (CNN) that enforces a spatiotemporal consistency constraint on the subviews of the 360° light field, an equirectangular matching cost that aims at increasing the accuracy of disparity estimation, and a light field resampling subnet that produces the 360° light field based on the disparity information. Ablation tests are conducted to analyze the performance of the proposed pipeline using the HCI light field datasets with five objective assessment metrics (MSE, MAE, PSNR, SSIM, and GMSD). We also use real data obtained from a commercially available dual-fisheye camera to quantitatively and qualitatively test the effectiveness, robustness, and quality of the proposed pipeline. Our contributions include: 1) a novel spatiotemporal consistency loss that enforces the subviews of the 360° light field to be consistent, 2) an equirectangular matching cost that combats severe projection distortion of fisheye images, and 3) a light field resampling subnet that retains the geometric structure of spherical subviews while enhancing the angular resolution of the light field.