Exploring Scale-Aware Features for Real-Time Semantic Segmentation of Street Scenes

• Published in: IEEE transactions on intelligent transportation systems Vol. 25; no. 5; pp. 3575 - 3587
• Main Authors: Li, Kaige, Geng, Qichuan, Zhou, Zhong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Autonomous driving; Context; Context modeling; Convolutional neural networks; Encoding; Feature detection; Feature maps; Image analysis; Modules; Pixels; Real time; real-time semantic segmentation; Real-time systems; Redundancy; Road traffic; selective context encoding; selective feature fusion; Semantic segmentation; Semantics; Street scene understanding; Test sets; Transformers
• DOI: 10.1109/TITS.2023.3330498
• ISSN: 1524-9050

Real-time semantic segmentation of street scenes is an essential and challenging task for autonomous driving systems, which needs to achieve both high accuracy and efficiency. Moreover, numerous objects and stuff at different scales in street scenes further increase the difficulty of this task. To address this challenge, we develop a lightweight and high-accuracy network termed Scale-Aware Network (SANet), which aims to selectively aggregate multi-scale features while maintaining high efficiency. In SANet, we first design a Selective Context Encoding (SCE) module, which considers the intrinsic differences of various pixels to selectively encode private contexts for each pixel, thus learning more desirable contextual features while reducing redundancy. With the context embedding in hand, we then design a Selective Feature Fusion (SFF) module to recursively fuses them with multiple features at different levels or scales to generate scale-aware features, where each feature map contains scale-specific information. Extensive experiments on challenging street scene datasets, i.e., Cityscapes and CamVid, illustrate that our SANet achieves a leading trade-off between segmentation accuracy and speed. Concretely, our method yields 78.1% mIoU at 109.0 FPS on the Cityscapes test set and 77.2% mIoU at 250.4 FPS on the CamVid test set. Code will be available at https://github.com/kaigelee/SANet .