Exploring the Potential of Conditional Adversarial Networks for Optical and SAR Image Matching

• Published in: IEEE journal of selected topics in applied earth observations and remote sensing Vol. 11; no. 6; pp. 1811 - 1820
• Main Authors: Merkle, Nina, Auer, Stefan, Muller, Rupert, Reinartz, Peter
• Format: Journal Article
• Language: English
• Published: IEEE 01.06.2018
• Subjects: Adaptive optics; Artificial image generation; Biomedical optical imaging; conditional generative adversarial networks (cGANs); Geology; multisensor image matching; Optical distortion; Optical imaging; optical satellite images; Optical sensors; Synthetic aperture radar; synthetic aperture radar (SAR)
• ISSN: 1939-1404
• DOI: 10.1109/JSTARS.2018.2803212

Tasks such as the monitoring of natural disasters or the detection of change highly benefit from complementary information about an area or a specific object of interest. The required information is provided by fusing high accurate coregistered and georeferenced datasets. Aligned high-resolution optical and synthetic aperture radar (SAR) data additionally enable an absolute geolocation accuracy improvement of the optical images by extracting accurate and reliable ground control points (GCPs) from the SAR images. In this paper, we investigate the applicability of a deep learning based matching concept for the generation of precise and accurate GCPs from SAR satellite images by matching optical and SAR images. To this end, conditional generative adversarial networks (cGANs) are trained to generate SAR-like image patches from optical images. For training and testing, optical and SAR image patches are extracted from TerraSAR-X and PRISM image pairs covering greater urban areas spread over Europe. The artificially generated patches are then used to improve the conditions for three known matching approaches based on normalized cross-correlation (NCC), scale-invariant feature transform (SIFT), and binary robust invariant scalable key (BRISK), which are normally not usable for the matching of optical and SAR images. The results validate that a NCC-, SIFT-, and BRISK-based matching greatly benefit, in terms of matching accuracy and precision, from the use of the artificial templates. The comparison with two state-of-the-art optical and SAR matching approaches shows the potential of the proposed method but also revealed some challenges and the necessity for further developments.