Regularizing Hyperspectral and Multispectral Image Fusion by CNN Denoiser

• Published in: IEEE transaction on neural networks and learning systems Vol. 32; no. 3; pp. 1124 - 1135
• Main Authors: Dian, Renwei, Li, Shutao, Kang, Xudong
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial neural networks; Coefficients; Computer vision; Convolutional neural network (CNN); Correlation; Dictionaries; Estimation; fusion; Hyperspectral imaging; Image processing; Neural networks; Noise reduction; Retraining; Singular value decomposition; Spatial resolution; Spectral bands; Subspaces; superresolution; Tensile stress
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2020.2980398

Hyperspectral image (HSI) and multispectral image (MSI) fusion, which fuses a low-spatial-resolution HSI (LR-HSI) with a higher resolution multispectral image (MSI), has become a common scheme to obtain high-resolution HSI (HR-HSI). This article presents a novel HSI and MSI fusion method (called as CNN-Fus), which is based on the subspace representation and convolutional neural network (CNN) denoiser, i.e., a well-trained CNN for gray image denoising. Our method only needs to train the CNN on the more accessible gray images and can be directly used for any HSI and MSI data sets without retraining. First, to exploit the high correlations among the spectral bands, we approximate the desired HR-HSI with the low-dimensional subspace multiplied by the coefficients, which can not only speed up the algorithm but also lead to more accurate recovery. Since the spectral information mainly exists in the LR-HSI, we learn the subspace from it via singular value decomposition. Due to the powerful learning performance and high speed of CNN, we use the well-trained CNN for gray image denoising to regularize the estimation of coefficients. Specifically, we plug the CNN denoiser into the alternating direction method of multipliers (ADMM) algorithm to estimate the coefficients. Experiments demonstrate that our method has superior performance over the state-of-the-art fusion methods.