DR-AVIT: Toward Diverse and Realistic Aerial Visible-to-Infrared Image Translation

• Published in: IEEE transactions on geoscience and remote sensing Vol. 62; pp. 1 - 13
• Main Authors: Han, Zonghao, Zhang, Shun, Su, Yuru, Chen, Xiaoning, Mei, Shaohui
• Format: Journal Article
• Language: English
• Published: New York IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Aerial visible-to-infrared image translation (AVIT); Benchmarks; Consistency; Datasets; deep learning; Generative adversarial networks; generative adversarial networks (GANs); Generators; Image representation; image-to-image (I2I) translation; Imaging; Infrared imagery; Learning; Realism; Representations; Semantics; Silver; Task analysis; Training; Translation
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2024.3405989

Image-to-image (I2I) translation methods based on generative adversarial networks (GANs) have shown general solutions for aerial visible-to-infrared image translation (AVIT) task. Though the existing approaches have made impressive results, they still struggle to produce diverse or high-realism translated aerial infrared images (AIIs). In this article, a novel model is proposed to achieve both diverse and realistic AVIT, named DR-AVIT. Specifically, we introduce disentangled representation learning to disentangle the image representation of aerial visible images (AVIs) and AIIs into a domain-invariant semantic structure space and two domain-specific imaging style spaces. By leveraging this disentanglement, our model can perform the translation process conditioned on semantic structure information derived from the input AVI and randomly sampled imaging style features from the AII domain to obtain diverse outputs. Furthermore, a new constraint is present to encourage GANs to learn efficient mappings between AVI and AII domains by integrating geometry-consistency constraint and a dual learning framework, named dual geometry-consistency constraint. Coping with these two designs, our method exhibits superiority in both realism and diversity of the translation results over several state-of-the-art I2I translation methods on AVIID dataset and two new benchmark datasets for AVIT, which are obtained by extracting data from publicly available datasets. Codes of DR-AVIT and proposed benchmark datasets are available at https://github.com/silver-hzh/DR-AVIT .