Remote Sensing Image Rectangling With Iterative Warping Kernel Self-Correction Transformer

• Published in: IEEE transactions on geoscience and remote sensing Vol. 62; pp. 1 - 17
• Main Authors: Qiu, Linwei, Xie, Fengying, Liu, Chang, Wang, Ke, Song, Xuedong, Shi, Zhenwei
• Format: Journal Article
• Language: English
• Published: New York IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Boundaries; Deformation; Distortion; Feature extraction; Image processing; Image segmentation; Indexes; Kernel; Modules; Object recognition; Rectangling; Remote sensing; remote sensing images; Stitching; Transformers; Warping
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2024.3441246

Stitched remote sensing images often exhibit irregular boundaries, which can be frustrating for general users and detrimental to downstream tasks such as object detection and segmentation. However, this issue has received insufficient attention and remains unexplored within the remote sensing domain. In this study, we investigate mesh-based rectangling techniques for remote sensing images, aiming to produce rectangular outputs while preserving the original field-of-view (FoV) and avoiding the introduction of unreliable content. Observing that prior rectangling algorithms tend to generate unsatisfactory boundaries or discernible distortions, that is, under-rectangling or over-rectangling, we propose the concept of a warping kernel associated with mesh deformations to account for these phenomena. Consequently, we introduce the iterative warping kernel self-correction transformer (IWKFormer), designed to enhance warping kernel estimation and generate superior rectangular outcomes. It primarily comprises two components: a mesh feature extractor built upon the partial swin transformer block (PSTB) and a corrector module using the swin transformer block (STB). These modules collaborate to derive warping kernels implicitly. The extractor extracts latent features pertinent to mesh deformation, whereas the corrector iteratively refines the warping kernel estimation to improve the ultimate prediction. Furthermore, to bolster further research, we have constructed an aerial imagery stitching rectangling dataset (AIRD), featuring a wide array of stitching scenes. Extensive experimentation on the AIRD demonstrates that our method yields visually appealing and naturally rectangled images, achieving state-of-the-art performance. The code and data will be available at https://github.com/yyywxk/IWKFormer .