Multisensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery

• Published in: IEEE transactions on geoscience and remote sensing Vol. 59; no. 7; pp. 5866 - 5878
• Main Authors: Ebel, Patrick, Meraner, Andrea, Schmitt, Michael, Zhu, Xiao Xiang
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cloud computing; Cloud removal; Clouds; Data; data fusion; Data integration; Datasets; deep learning; Earth; generative adversarial network (GAN); Image acquisition; Image quality; Image reconstruction; Imagery; Multisensor fusion; optical imagery; Optical imaging; Optical sensors; Performance evaluation; Performance measurement; Radar; Regions; Removal; synthetic aperture radar (SAR)-optical; Training
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2020.3024744

The majority of optical observations acquired via spaceborne Earth imagery are affected by clouds. While there is numerous prior work on reconstructing cloud-covered information, previous studies are, oftentimes, confined to narrowly defined regions of interest, raising the question of whether an approach can generalize to a diverse set of observations acquired at variable cloud coverage or in different regions and seasons. We target the challenge of generalization by curating a large novel data set for training new cloud removal approaches and evaluate two recently proposed performance metrics of image quality and diversity. Our data set is the first publically available to contain a global sample of coregistered radar and optical observations, cloudy and cloud-free. Based on the observation that cloud coverage varies widely between clear skies and absolute coverage, we propose a novel model that can deal with either extreme and evaluate its performance on our proposed data set. Finally, we demonstrate the superiority of training models on real over synthetic data, underlining the need for a carefully curated data set of real observations. To facilitate future research, our data set is made available online.