Segmenting Objects in Day and Night: Edge-Conditioned CNN for Thermal Image Semantic Segmentation

• Published in: IEEE transaction on neural networks and learning systems Vol. 32; no. 7; pp. 3069 - 3082
• Main Authors: Li, Chenglong, Xia, Wei, Yan, Yan, Luo, Bin, Tang, Jin
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.07.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Benchmark data set; Benchmark testing; Cameras; Computer architecture; Darkness; edge prior knowledge; Environmental conditions; featurewise affine transform; Haze; Image edge detection; Image processing; Image segmentation; Infrared cameras; Neural networks; Night; Semantic segmentation; Semantics; Smog; Task analysis; thermal infrared image; Visible spectrum
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2020.3009373

Despite much research progress in image semantic segmentation, it remains challenging under adverse environmental conditions caused by imaging limitations of the visible spectrum, while thermal infrared cameras have several advantages over cameras for the visible spectrum, such as operating in total darkness, insensitive to illumination variations, robust to shadow effects, and strong ability to penetrate haze and smog. These advantages of thermal infrared cameras make the segmentation of semantic objects in day and night. In this article, we propose a novel network architecture, called edge-conditioned convolutional neural network (EC-CNN), for thermal image semantic segmentation. Particularly, we elaborately design a gated featurewise transform layer in EC-CNN to adaptively incorporate edge prior knowledge. The whole EC-CNN is end-to-end trained and can generate high-quality segmentation results with edge guidance. Meanwhile, we also introduce a new benchmark data set named "Segmenting Objects in Day And night" (SODA) for comprehensive evaluations in thermal image semantic segmentation. SODA contains over 7168 manually annotated and synthetically generated thermal images with 20 semantic region labels and from a broad range of viewpoints and scene complexities. Extensive experiments on SODA demonstrate the effectiveness of the proposed EC-CNN against state-of-the-art methods.