Deep Cross Spectral Stereo Matching Using Multi-Spectral Image Fusion

• Published in: IEEE robotics and automation letters Vol. 7; no. 2; pp. 5373 - 5380
• Main Authors: Liang, Xiaolong, Jung, Cheolkon
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.04.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Color matching; Computer vision; Consistency; Cross spectral; disparity estimation; Estimation; Feature extraction; Image fusion; Image processing; Image reconstruction; Infrared imagery; Material properties; multi-spectral fusion; Near infrared radiation; Parallax; Reconstruction; Skin; Smoothness; Spectra; Spectral bands; stereo matching; Task analysis; Unsupervised learning
• ISSN: 2377-3766; 2377-3766
• DOI: 10.1109/LRA.2022.3155202

Cross spectral stereo matching aims to estimate disparity from color (RGB) and near-infrared (NIR) image pairs. The main difference from traditional stereo matching is that there is a big gap between two spectral bands, which makes cross spectral stereo matching a challenging task. In this letter, we propose deep cross spectral stereo matching using multi-spectral image fusion. We adopt unsupervised learning to consider no ground truth in cross spectral stereo matching. We perform multi-spectral image fusion after cross spectral image translation to bridge the spectral gap between two images. First, we extract features from input RGB and NIR images to get fusion stereo pairs. Second, we get stereo correspondence robust to disparity variation based on parallax attention. In the loss function, we combine four losses: view reconstruction, material aware matching, cycle disparity consistency, and smoothness. We use a view reconstruction loss for spectral translation and fusion to warp stereo image pairs for appearance matching, while we use a material aware matching loss to take material property into consideration. Moreover, we utilize a cycle disparity consistency loss for disparity consistency between left and right predictions, and use a smoothness loss to enforce disparity smoothness. Experimental results show that the proposed network successfully estimates disparity with adaptability to materials and outperforms state-of-the-art models in terms of visual quality and quantitative measurements.