Differential Feature Awareness Network Within Antagonistic Learning for Infrared-Visible Object Detection

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 8; pp. 6735 - 6748
• Main Authors: Zhang, Ruiheng, Li, Lu, Zhang, Qi, Zhang, Jin, Xu, Lixin, Zhang, Baomin, Wang, Binglu
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Ablation; Algorithms; Data mining; Feature extraction; Image color analysis; Infrared-visible object detection; Learning; Modules; multi-modal feature fusion; Object detection; Object recognition; Pedestrians; Performance assessment; Semantics; Videos
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3289142

The combination of infrared and visible videos aims to gather more comprehensive feature information from multiple sources and reach superior results on various practical tasks, such as detection and segmentation, over that of a single modality. However, most existing dual-modality object detection algorithms ignore the modal differences and fail to consider the correlation between feature extraction and fusion, which leads to incomplete extraction and inadequate fusion of dual-modality features. Hence, there raises an issue of how to preserve each unique modal feature and fully utilize the complementary infrared and visible information. Facing the above challenges, we propose a novel Differential Feature Awareness Network (DFANet) within antagonistic learning for infrared and visible object detection. The proposed model consists of an Antagonistic Feature Extraction with Divergence (AFED) module used to extract the differential infrared and visible features with unique information, and an Attention-based Differential Feature Fusion (ADFF) module used to fully fuse the extracted differential features. We conduct performance comparisons with existing state-of-the-art models on two benchmark datasets to represent the robustness and superiority of DFANet, and numerous ablation experiments to illustrate its effectiveness.