Learning the external and internal priors for multispectral and hyperspectral image fusion

• Published in: Science China. Information sciences Vol. 66; no. 4; p. 140303
• Main Authors: Li, Shutao, Dian, Renwei, Liu, Haibo
• Format: Journal Article
• Language: English
• Published: Beijing Science China Press 01.04.2023; Springer Nature B.V
• Subjects: Artificial neural networks; Computer Science; Computer vision; Decomposition; Exploitation; High resolution; Hyperspectral imaging; Image acquisition; Information Systems and Communication Service; Methods; Neural networks; Optimization; Research Paper; multispectral and hyperspectral image fusion; high-resolution hyperspectral imaging; external and internal prior learning
• ISSN: 1674-733X; 1869-1919
• DOI: 10.1007/s11432-022-3610-5

Recently, multispectral image (MSI) and hyperspectral image (HSI) fusion has been a popular topic in high-resolution HSI acquisition. This fusion leads to a challenging underdetermined problem, which image priors are used to regularize, aiming at improving fusion accuracy. To fully exploit HSI priors, this paper proposes two kinds of priors, i.e., external priors and internal priors, to regularize the fusion problem. An external prior represents the general image characteristics and is learned from abundant training data by using a Gaussian denoising convolutional neural network (CNN) trained in the additional gray images. An internal prior represents the unique characteristics of the HSI and MSI to be fused. To learn the external prior, we first segment the MSI into several superpixels and then enforce a low-rank constraint for each superpixel, which can well model local similarities in the HSI. In addition, to model a low-rank property in the spectral mode, the high-resolution HSI is decomposed into a low-rank spectral basis and abundances. Finally, we formulate the fusion as an external and internal prior-regularized optimization problem, which is efficiently tackled through the alternating direction method of multipliers. Experiments on simulated and real datasets demonstrate the superiority of the proposed method.