HIFR-Net: A HRRP-Infrared Fusion Recognition Network Capable of Handling Modality Missing and Multisource Data Misalignment

• Published in: IEEE sensors journal Vol. 25; no. 3; pp. 5769 - 5781
• Main Authors: Zhang, Fan, Bi, Xiaoye, Zhang, Zhaoxiang, Xu, Yuelei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Data mining; Data models; Feature extraction; High-resolution range profile (HRRP); Image recognition; Image resolution; infrared (IR); Infrared detectors; Infrared imaging; Infrared radar; Misalignment; modality missing; multimodal fusion recognition; multisource data misalignment; Robustness; Sensor fusion; Sensors; Spatiotemporal data; Spatiotemporal phenomena; System reliability; Target recognition; Training
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2024.3515204

Radar (RR) and infrared (IR) sensors have different characteristics and applications. Combining these two sensors in complex environments can yield complementary advantages, enhancing the reliability, robustness, and accuracy of detection systems. Some related studies have combined RR's high-resolution range profile (HRRP) data with IR images for target recognition. However, existing recognition frameworks primarily focus on improving recognition accuracy while neglecting practical issues that may arise, such as modality data absence or spatiotemporal misalignment of multisource data, thereby limiting their applicability. To address these challenges, we propose a multimodal fusion recognition network based on HRRP data and IR images, named HIFR-Net, which can effectively handle modality missing and multisource data misalignment. Additionally, we explore a device-cloud distributed collaborative inference approach for deploying HIFR-Net. The design of the modality gating mechanism and cross-modal interaction strategy in HIFR-Net enhances its robustness to modality missing and spatiotemporal differences in multisource data. We evaluate HIFR-Net on a constructed air target dataset containing HRRP data and IR images. Results from multiple experiments demonstrate that HIFR-Net exhibits excellent comprehensive recognition capability, achieving a recognition accuracy of 98.65%, and shows strong robustness and applicability in handling modality missing, multisource data misalignment, and interference such as noise.