CMSNet: Deep Color and Monochrome Stereo

• Published in: International journal of computer vision Vol. 130; no. 3; pp. 652 - 668
• Main Authors: Jeon, Hae-Gon, Im, Sunghoon, Choe, Jaesung, Kang, Minjun, Lee, Joon-Young, Hebert, Martial
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2022; Springer; Springer Nature B.V
• Subjects: Artificial Intelligence; Artificial neural networks; Cameras; Color matching; Computer Imaging; Computer Science; Image enhancement; Image Processing and Computer Vision; Luminance; Neural networks; Noise; Pattern Recognition; Pattern Recognition and Graphics; Performance degradation; Quantum efficiency; Special Issue on 3D Computer Vision; Spectral sensitivity; Vision; Disparity estimation; Stereo matching; Convolutional neural network; Image enhancement
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-021-01565-6

In this paper, we propose an end-to-end convolutional neural network for stereo matching with color and monochrome cameras, called CMSNet (Color and Monochrome Stereo Network). Both cameras have the same structure except for the presence of a Bayer filter, but have a fundamental trade-off. The Bayer filter allows capturing chrominance information of scenes, but limits a quantum efficiency of cameras, which causes severe image noise. It seems ideal if we can take advantage of both the cameras so that we obtain noise-free images with their corresponding disparity maps. However, image luminance recorded from a color camera is not consistent with that from a monochrome camera due to spatially-varying illumination and different spectral sensitivities of the cameras. This degrades the performance of stereo matching. To solve this problem, we design CMSNet for disparity estimation from noisy color and relatively clean monochrome images. CMSNet also infers a noise-free image with the estimated disparity map. We leverage a data augmentation to simulate realistic signal-dependent noise and various radiometric distortions between input stereo pairs to train CMSNet effectively. CMSNet is evaluated using various datasets and the performance of our disparity estimation and image enhancement consistently outperforms state-of-the-art methods.