MCAFNet: Multiscale cross-modality adaptive fusion network for multispectral object detection

• Published in: Digital signal processing Vol. 159; p. 104996
• Main Authors: Zheng, Shangpo, Junfeng, Liu, Zeng, Jun
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.04.2025
• Subjects: Attention mechanism; cross-modality; multimodal adaptive feature fusion; multispectral object detection; transformer; multimodal adaptive feature fusion; transformer; Attention mechanism; multispectral object detection; cross-modality
• DOI: 10.1016/j.dsp.2025.104996
• ISSN: 1051-2004

Multispectral object detection techniques integrate data from various spectral modalities, such as combining thermal images with RGB visible light images, to enhance the precision a-nd robustness of object detection under diverse environmental c-onditions. Although this approach has improved detection capab-ilities, significant challenges remain in fully leveraging the specif-ic detail information of each single modality and accurately capt-uring cross-modality shared features information. To address th-ese challenges, we propose a Multiscale Cross-modality Adaptive Fusion Network (MCAFNet). This network incorporates Cross- modality interactive Transformer (CMIT) module, Multimodal Adaptive Weighted Fusion (MAWF) module, and a 3D-Integrated Attention Feature Enhancement (3D-IAFE) module. These components work together to comprehensively extract complementary feature between modalities and specific detailed feature within each modality, thereby enhancing the accuracy and robustness of multimodal object detection. Extensive experimental validation and in-depth ablation studies confirm the effectiveness of the proposed method, achieving state-of-the-art detection performance on multiple public datasets.