AMFuse: Add–Multiply-Based Cross-Modal Fusion Network for Multi-Spectral Semantic Segmentation

Multi-spectral semantic segmentation has shown great advantages under poor illumination conditions, especially for remote scene understanding of autonomous vehicles, since the thermal image can provide complementary information for RGB image. However, methods to fuse the information from RGB image a...

Full description

Saved in:
Bibliographic Details
Published inRemote sensing (Basel, Switzerland) Vol. 14; no. 14; p. 3368
Main Authors Liu, Haijun, Chen, Fenglei, Zeng, Zhihong, Tan, Xiaoheng
Format Journal Article
LanguageEnglish
Published Basel MDPI AG 01.07.2022
Subjects
Architecture
Autonomous vehicles
Cameras
Coders
cross-modal feature fusion network
Data integration
Datasets
Feature extraction
Feature maps
Image segmentation
Modules
multi-spectral images
Multiplication
Object recognition
Remote sensing
Salience
salient object detection
Scene analysis
Semantic segmentation
Semantics
Surveillance
Online AccessGet full text

Cover

More Information
Summary:Multi-spectral semantic segmentation has shown great advantages under poor illumination conditions, especially for remote scene understanding of autonomous vehicles, since the thermal image can provide complementary information for RGB image. However, methods to fuse the information from RGB image and thermal image are still under-explored. In this paper, we propose a simple but effective module, add–multiply fusion (AMFuse) for RGB and thermal information fusion, consisting of two simple math operations—addition and multiplication. The addition operation focuses on extracting cross-modal complementary features, while the multiplication operation concentrates on the cross-modal common features. Moreover, the attention module and atrous spatial pyramid pooling (ASPP) modules are also incorporated into our proposed AMFuse modules, to enhance the multi-scale context information. Finally, in the UNet-style encoder–decoder framework, the ResNet model is adopted as the encoder. As for the decoder part, the multi-scale information obtained from our proposed AMFuse modules is hierarchically merged layer-by-layer to restore the feature map resolution for semantic segmentation. The experiments of RGBT multi-spectral semantic segmentation and salient object detection demonstrate the effectiveness of our proposed AMFuse module for fusing the RGB and thermal information.
ISSN:2072-4292
2072-4292
DOI:10.3390/rs14143368