Multispectral Image Pan-Sharpening Guided by Component Substitution Model

• Published in: IEEE transactions on geoscience and remote sensing Vol. 61; pp. 1 - 13
• Main Authors: Gao, Huiling, Li, Shutao, Li, Jun, Dian, Renwei
• Format: Journal Article
• Language: English
• Published: New York IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Attention mechanism; Conditioned stimulus; Convolutional neural networks; Deep learning; Fuses; Hadamard product; Heterogeneity; Histograms; Image processing; Image resolution; Linear transformations; multispectral image pan-sharpening; Real numbers; Spatial discrimination; Spatial resolution; Substitutes; Sums; Superresolution; Transforms; two-branch network; Visual perception
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2023.3309863

Multispectral image pan-sharpening aims to increase the spatial details of multispectral images by fusing multispectral and panchromatic (PAN) images. Existing component substitution (CS)-based deep learning pan-sharpening is generally regarded as a black box and fails to mine the image interaction relation with physical significance in each step of pan-sharpening, which not only limits the improvement of image resolution but also ignores the physical interpretability of the models. To improve this situation, according to the traditional CS-based detail injection pan-sharpening model, we consider the matrix calculation in each step as the transformation between image pixel values and carry out linear transformations, and therefore the pan-sharpened multispectral image is represented as the sum of two multispectral images. Then given the spatial and spectral heterogeneity, the two summed images are decomposed based on the fact that any real number can be expressed as the product of two real numbers. Ultimately, the multispectral image pan-sharpening model can be constructed as the sum of two Hadamard products. We design a dual-branch network with attention mechanisms that merges the sum and the Hadamard products into a concise formulation. This method not only enhances physical interpretability but also improves spatial resolution. Experiments on five real-world datasets validate that the proposed multispectral image pan-sharpening model can improve performance.