General Image Fusion for an Arbitrary Number of Inputs Using Convolutional Neural Networks

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 7; p. 2457
• Main Authors: Xiao, Yifan, Guo, Zhixin, Veelaert, Peter, Philips, Wilfried
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 23.03.2022; MDPI
• Subjects: continual learning; Deep learning; Design; image fusion; Image Processing, Computer-Assisted - methods; Medical imaging equipment; Methods; multiple inputs; Neural networks; Neural Networks, Computer; Night vision; permutation-invariant network; Records; Wavelet transforms; permutation-invariant network; continual learning; multiple inputs; image fusion
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22072457

In this paper, we propose a unified and flexible framework for general image fusion tasks, including multi-exposure image fusion, multi-focus image fusion, infrared/visible image fusion, and multi-modality medical image fusion. Unlike other deep learning-based image fusion methods applied to a fixed number of input sources (normally two inputs), the proposed framework can simultaneously handle an arbitrary number of inputs. Specifically, we use the symmetrical function (e.g., Max-pooling) to extract the most significant features from all the input images, which are then fused with the respective features from each input source. This symmetry function enables permutation-invariance of the network, which means the network can successfully extract and fuse the saliency features of each image without needing to remember the input order of the inputs. The property of permutation-invariance also brings convenience for the network during inference with unfixed inputs. To handle multiple image fusion tasks with one unified framework, we adopt continual learning based on Elastic Weight Consolidation (EWC) for different fusion tasks. Subjective and objective experiments on several public datasets demonstrate that the proposed method outperforms state-of-the-art methods on multiple image fusion tasks.