Multi-Modal Sensor Fusion-Based Semantic Segmentation for Snow Driving Scenarios

• Published in: IEEE sensors journal Vol. 21; no. 15; pp. 16839 - 16851
• Main Authors: Vachmanus, Sirawich, Ravankar, Ankit A., Emaru, Takanori, Kobayashi, Yukinori
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Advanced driver assistance systems; data fusion; Datasets; Feature extraction; Feature maps; Image segmentation; Machine learning; Modules; Neural networks; Roads; Semantic segmentation; Semantics; Sensors; Snow; thermal camera; Thermal mapping; Traffic safety; Training; Weather
• DOI: 10.1109/JSEN.2021.3077029
• ISSN: 1530-437X

In recent years, autonomous vehicle driving technology and advanced driver assistance systems have played a key role in improving road safety. However, weather conditions such as snow pose severe challenges for autonomous driving and are an active research area. Thanks to their superior reliability, the resilience of detection, and improved accuracy, advances in computation and sensor technology have paved the way for deep learning and neural network-based techniques that can replace the classical approaches. In this research, we investigate the semantic segmentation of roads in snowy environments. We propose a multi-modal fused RGB-T semantic segmentation utilizing a color (RGB) image and thermal map (T) as inputs for the network. This paper introduces a novel fusion module that combines the feature map from both inputs. We evaluate the proposed model on a new snow dataset that we collected and on other publicly available datasets. The segmentation results show that the proposed fused RGB-T input can segregate human subjects in snowy environments better than an RGB-only input. The fusion module plays a vital role in improving the efficiency of multiple input neural networks for person detection. Our results show that the proposed network can generate a higher success rate than other state-of-the-art networks. The combination of our fused module and pyramid supervision path generated the best results in both mean accuracy and mean intersection over union in every dataset.