SIFusion: Lightweight infrared and visible image fusion based on semantic injection

• Published in: PloS one Vol. 19; no. 11; p. e0307236
• Main Authors: Qian, Song, Yang, Liwei, Xue, Yan, Li, Ping
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 06.11.2024; Public Library of Science (PLoS)
• Subjects: Algorithms; Biology and Life Sciences; Cameras; Computer and Information Sciences; Computer vision; Data visualization; Deep learning; Design; Evaluation; Humans; Image enhancement; Image processing; Image Processing, Computer-Assisted - methods; Information processing; Infrared imagery; Infrared imaging; Infrared Rays; Injection; Light; Lightweight; Machine vision; Methods; Modules; Physical Sciences; Research and Analysis Methods; Semantic networks; Semantics; Social Sciences; Special effects; Task complexity; Visual perception; Visual perception driven algorithms; Visual tasks; Weight reduction; China
• DOI: 10.1371/journal.pone.0307236
• ISSN: 1932-6203

The objective of image fusion is to integrate complementary features from source images to better cater to the needs of human and machine vision. However, existing image fusion algorithms predominantly focus on enhancing the visual appeal of the fused image for human perception, often neglecting their impact on subsequent high-level visual tasks, particularly the processing of semantic information. Moreover, these fusion methods that incorporate downstream tasks tend to be overly complex and computationally intensive, which is not conducive to practical applications. To address these issues, a lightweight infrared and visible light image fusion method known as SIFusion, which is based on semantic injection, is proposed in this paper. This method employs a semantic-aware branch to extract semantic feature information, and then integrates these features into the fused features through a Semantic Injection Module (SIM) to meet the semantic requirements of high-level visual tasks. Furthermore, to simplify the complexity of the fusion network, this method introduces an Edge Convolution Module (ECB) based on structural reparameterization technology to enhance the representational capacity of the encoder and decoder. Extensive experimental comparisons demonstrate that the proposed method performs excellently in terms of visual appeal and advanced semantics, providing satisfactory fusion results for subsequent high-level visual tasks even in challenging scenarios.