SAB Net: A Semantic Attention Boosting Framework for Semantic Segmentation

• Published in: IEEE transaction on neural networks and learning systems Vol. 36; no. 3; pp. 4029 - 4041
• Main Authors: Ding, Xiaofeng, Shen, Chaomin, Zeng, Tieyong, Peng, Yaxin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2025
• Subjects: Aggregates; Boosting; Context modeling; Contextual dependencies; feature fusion; Fuses; Image segmentation; Measurement; semantic attention; semantic attention boosting network (SAB Net); semantic segmentation; Semantics
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2022.3144003

Semantic segmentation has achieved great progress by effectively fusing features of contextual information. In this article, we propose an end-to-end semantic attention boosting (SAB) framework to adaptively fuse the contextual information iteratively across layers with semantic regularization. Specifically, we first propose a pixelwise semantic attention (SAP) block, with a semantic metric representing the pixelwise category relationship, to aggregate the nonlocal contextual information. In addition, we improve the computation complexity of SAP block from <inline-formula> <tex-math notation="LaTeX">O(n^{2}) </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">O(n) </tex-math></inline-formula> for images with size <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. Second, we present a categorywise semantic attention (SAC) block to adaptively balance the nonlocal contextual dependencies and the local consistency with a categorywise weight, overcoming the contextual information confusion caused by the feature imbalance within intra-category. Furthermore, we propose the SAB module to refine the segmentation with SAC and SAP blocks. By applying the SAB module iteratively across layers, our model shrinks the semantic gap and enhances the structure reasoning by fully utilizing the coarse segmentation information. Extensive quantitative evaluations demonstrate that our method significantly improves the segmentation results and achieves superior performance on the PASCAL VOC 2012, Cityscapes, PASCAL Context, and ADE20K datasets.