Robust semantic segmentation based on RGB-thermal in variable lighting scenes

• Published in: Measurement : journal of the International Measurement Confederation Vol. 186; p. 110176
• Main Authors: Guo, Zhifeng, Li, Xu, Xu, Qimin, Sun, Zhengliang
• Format: Journal Article
• Language: English
• Published: London Elsevier Ltd 01.12.2021; Elsevier Science Ltd
• Subjects: Artificial neural networks; Color imagery; Computer architecture; Data fusion; Data integration; Fuzzy logic; Image segmentation; Intelligent vehicles; Lighting; Lighting systems; Neural networks; Robustness; Semantic segmentation; Semantics; Thermal data; Urban scene; Variable lighting scenes; Variable lighting scenes; Data fusion; Semantic segmentation; Thermal data; Urban scene
• ISSN: 0263-2241; 1873-412X
• DOI: 10.1016/j.measurement.2021.110176

•Robust semantic segmentation based on RGB-Thermal is proposed.•The two encoders process RGB and thermal images separately and fuse RGB and thermal features hierarchically.•The multi-level skip connections and an auxiliary decoding module are designed to obtain better results.•The proposed method is robust and effective for segmentation under variable lighting environments. Semantic segmentation is an indispensable part of the Intelligent Vehicle-Infrastructure System(IVIS) perception task. It has made considerable progress with the development of convolutional neural networks. However, the current mainstream semantic segmentation networks are designed for 3-channel RGB images captured by visible light cameras, and their accuracy and robustness are still insufficient in variable lighting environments. Therefore, this paper proposes a novel network architecture with RGB and thermal data fusion to improve the accuracy and robustness of semantic segmentation in a variable lighting environment. The experimental results verify that introducing thermal data significantly improves the adaptability and segmentation accuracy of the network to variable lighting environments. Moreover, the results demonstrate that our network is accurate and robust under variable lighting environments, and its overall performance outperforms the state-of-the-art networks.