Joint Depth and Density Guided Single Image De-Raining

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 7; pp. 4108 - 4121
• Main Authors: Cai, Lei, Fu, Yuli, Zhu, Tao, Xiang, Youjun, Zhang, Ying, Zeng, Huanqiang
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive systems; Density; depth-density inference network; depth-density-based conditional generative adversarial network; Discriminators; Generative adversarial networks; global-local discriminators; Machine vision; Predictive models; Rain; Shape; Single image de-raining; Task analysis; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3121012

Single image de-raining is an important and highly challenging problem. To address this problem, some depth or density guided single-image de-raining methods have been developed with encouraging performance. However, these methods individually use the depth or the density to guide the network to conduct image de-raining. In this paper, a novel joint depth and density guided de-raining (JDDGD) method is technically developed. The JDDGD starts with a depth-density inference network (DDINet) to extract the depth and density information from an input rainy image, followed by a depth-density-based conditional generative adversarial network (DD-CGAN) to exploit the depth and density information provided by DDINet to achieve adaptive rain streak and fog removal. To prevent the spatially-varying local artifacts, an effective global-local discriminators structure is introduced in the proposed DD-CGAN to globally and locally inspect the generated images. In addition, multiple loss functions including multi-scale pixel loss , multi-scale perceptual loss , and global-local generative adversarial loss are also jointly used to train our model to achieve the best performance. Both quantitative and qualitative results show that the proposed JDDGD method achieves superior performance than previous non-guided , density-guided , and depth-guided de-raining methods.