Zero-Shot Hyperspectral Sharpening

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 10; pp. 12650 - 12666
• Main Authors: Dian, Renwei, Guo, Anjing, Li, Shutao
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Artificial neural networks; Data models; Hyperspectral image sharpening; Hyperspectral imaging; image fusion; Imaging; imaging model; Pansharpening; Spatial resolution; spectral and spatial response estimation; Tensors; Training; Training data; zero-shot learning
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2023.3279050

Fusing hyperspectral images (HSIs) with multispectral images (MSIs) of higher spatial resolution has become an effective way to sharpen HSIs. Recently, deep convolutional neural networks (CNNs) have achieved promising fusion performance. However, these methods often suffer from the lack of training data and limited generalization ability. To address the above problems, we present a zero-shot learning (ZSL) method for HSI sharpening. Specifically, we first propose a novel method to quantitatively estimate the spectral and spatial responses of imaging sensors with high accuracy. In the training procedure, we spatially subsample the MSI and HSI based on the estimated spatial response and use the downsampled HSI and MSI to infer the original HSI. In this way, we can not only exploit the inherent information in the HSI and MSI, but the trained CNN can also be well generalized to the test data. In addition, we take the dimension reduction on the HSI, which reduces the model size and storage usage without sacrificing fusion accuracy. Furthermore, we design an imaging model-based loss function for CNN, which further boosts the fusion performance. The experimental results show the significantly high efficiency and accuracy of our approach.