Training-free diffusion for controlling illumination conditions in images

• Published in: Computer vision and image understanding Vol. 260; p. 104450
• Main Authors: Xing, Xiaoyan, Hu, Tao, Metzen, Jan Hendrik, Groh, Konrad, Karaoglu, Sezer, Gevers, Theo
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.10.2025
• Subjects: Deep learning; Diffusion models; Illumination control; Intrinsic images; Low-level vision; Deep learning; Diffusion models; Low-level vision; Intrinsic images; Illumination control
• ISSN: 1077-3142
• DOI: 10.1016/j.cviu.2025.104450

This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. While most of methods employ ControlNet and its variants to address the illumination-aware guidance in diffusion models. In contrast, We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-related properties during the generative process. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections. Remarkably, it achieves this without the necessity for learning intrinsic decomposition, finding directions in latent space, or undergoing additional training with new datasets. •Diffusion models can be treated as black-box renderers.•Diffusion energy function can affect lighting change in image synthesis.•Training-free relighting can be achieved with proper physics-based constraints.