Laplacian Pyramid Fusion Network with Hierarchical Guidance for Infrared and Visible Image Fusion

• Published in: IEEE transactions on circuits and systems for video technology Vol. 33; no. 9; p. 1
• Main Authors: Yao, Jiaxin, Zhao, Yongqiang, Bu, Yuanyang, Kong, Seong G., Chan, Jonathan Cheung-Wai
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Computer vision; Deep learning; Global optimization; Hafnium; Image fusion; Image recognition; Imaging; Infrared and visible image fusion; Infrared imagery; Laplace equations; Laplacian Pyramid; Salience; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3245607

The fusion of infrared and visible images combines the information from two complementary imaging modalities for various computer vision tasks. Many existing techniques, however, fail to maintain a uniform overall style and keep salient details of individual modalities simultaneously. This paper presents an end-to-end Laplacian Pyramid Fusion Network with hierarchical guidance (HG-LPFN) that takes advantage of pixel-level saliency reservation of Laplacian Pyramid and global optimization capability of deep learning. The proposed scheme generates hierarchical saliency maps through Laplacian Pyramid decomposition and modal difference calculation. In the pyramid fusion mode, all sub-networks are connected in a bottom-up manner. The sub-network for low-frequency fusion focuses on extracting universal features to produce an opposite style while sub-networks for high-frequency fusion determine how much the details of each modality will be retained. Taking the style, details, and background into consideration, we design a set of novel loss functions to supervise both low-frequency images and full-resolution images under the guidance of saliency maps. Experimental results on public datasets demonstrate that the proposed HG-LPFN outperforms the state-of-the-art image fusion techniques.