Spectral Super-Resolution via Model-Guided Cross-Fusion Network

• Published in: IEEE transaction on neural networks and learning systems Vol. 35; no. 7; pp. 10059 - 10070
• Main Authors: Dian, Renwei, Shan, Tianci, He, Wei, Liu, Haibo
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Cameras; Convexity; Convolutional neural network; Convolutional neural networks; cross fusion (CF); Hyperspectral imaging; Image acquisition; Image reconstruction; Image resolution; Imaging; imaging model; Learning; Modules; Neural networks; Spatial resolution; spectral super-resolution; Superresolution; Training data
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2023.3238506

Spectral super-resolution, which reconstructs a hyperspectral image (HSI) from a single red-green-blue (RGB) image, has acquired more and more attention. Recently, convolution neural networks (CNNs) have achieved promising performance. However, they often fail to simultaneously exploit the imaging model of the spectral super-resolution and complex spatial and spectral characteristics of the HSI. To tackle the above problems, we build a novel cross fusion (CF)-based model-guided network (called SSRNet) for spectral super-resolution. In specific, based on the imaging model, we unfold the spectral super-resolution into the HSI prior learning (HPL) module and imaging model guiding (IMG) module. Instead of just modeling one kind of image prior, the HPL module is composed of two subnetworks with different structures, which can effectively learn the complex spatial and spectral priors of the HSI, respectively. Furthermore, a CF strategy is used to establish the connection between the two subnetworks, which further improves the learning performance of the CNN. The IMG module results in solving a strong convex optimization problem, which adaptively optimizes and merges the two features learned by the HPL module by exploiting the imaging model. The two modules are alternately connected to achieve optimal HSI reconstruction performance. Experiments on both the simulated and real data demonstrate that the proposed method can achieve superior spectral reconstruction results with relatively small model size. The code will be available at https://github.com/renweidian .