Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion

• Published in: ISPRS journal of photogrammetry and remote sensing Vol. 166; pp. 333 - 346
• Main Authors: Meraner, Andrea, Ebel, Patrick, Zhu, Xiao Xiang, Schmitt, Michael
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.08.2020; Elsevier
• Subjects: climate change; cloud cover; Cloud removal; cropland; data collection; Data fusion; Deep learning; image analysis; land use and land cover maps; monitoring; Optical imagery; remote sensing; Residual network; SAR-optical; satellites; Deep learning; Data fusion; Cloud removal; Optical imagery; SAR-optical; Residual network
• ISSN: 0924-2716; 1872-8235
• DOI: 10.1016/j.isprsjprs.2020.05.013

Optical remote sensing imagery is at the core of many Earth observation activities. The regular, consistent and global-scale nature of the satellite data is exploited in many applications, such as cropland monitoring, climate change assessment, land-cover and land-use classification, and disaster assessment. However, one main problem severely affects the temporal and spatial availability of surface observations, namely cloud cover. The task of removing clouds from optical images has been subject of studies since decades. The advent of the Big Data era in satellite remote sensing opens new possibilities for tackling the problem using powerful data-driven deep learning methods. In this paper, a deep residual neural network architecture is designed to remove clouds from multispectral Sentinel-2 imagery. SAR-optical data fusion is used to exploit the synergistic properties of the two imaging systems to guide the image reconstruction. Additionally, a novel cloud-adaptive loss is proposed to maximize the retainment of original information. The network is trained and tested on a globally sampled dataset comprising real cloudy and cloud-free images. The proposed setup allows to remove even optically thick clouds by reconstructing an optical representation of the underlying land surface structure.