Toward High-Quality HDR Deghosting With Conditional Diffusion Models

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 5; pp. 4011 - 4026
• Main Authors: Yan, Qingsen, Hu, Tao, Sun, Yuan, Tang, Hao, Zhu, Yu, Dong, Wei, Van Gool, Luc, Zhang, Yanning
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; diffusion model; Diffusion processes; Dynamic range; Ghosting; ghosting artifacts; High dynamic range image; Image color analysis; Image processing; multi-exposed imaging; Noise prediction; Noise reduction; Semantics; Steering; Training; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3326293

High Dynamic Range (HDR) images can be recovered from several Low Dynamic Range (LDR) images by existing Deep Neural Networks (DNNs) techniques. Despite the remarkable progress, DNN-based methods still generate ghosting artifacts when LDR images have saturation and large motion, which hinders potential applications in real-world scenarios. To address this challenge, we formulate the HDR deghosting problem as an image generation that leverages LDR features as the diffusion model's condition, consisting of the feature condition generator and the noise predictor. Feature condition generator employs attention and Domain Feature Alignment (DFA) layer to transform the intermediate features to avoid ghosting artifacts. With the learned features as conditions, the noise predictor leverages a stochastic iterative denoising process for diffusion models to generate an HDR image by steering the sampling process. Furthermore, to mitigate semantic confusion caused by the saturation problem of LDR images, we design a sliding window noise estimator to sample smooth noise in a patch-based manner. In addition, an image space loss is proposed to avoid the color distortion of the estimated HDR results. We empirically evaluate our model on benchmark datasets for HDR imaging. The results demonstrate that our approach achieves state-of-the-art performances and well generalization to real-world images.