Training-free diffusion for controlling illumination conditions in images - Xing, Xiaoyan; Hu, Tao; Metzen, Jan Hendrik; Groh, Konrad; Karaoglu, Sezer; Gevers, Theo (Computer vision and image understanding, 2025)
This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. While most of methods employ ControlNet and its variants to address the illumination-aware guidance in diffusion models. In contrast, We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-related properties during the generative process. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections. Remarkably, it achieves this without the necessity for learning intrinsic decomposition, finding directions in latent space, or undergoing additional training with new datasets. •Diffusion models can be treated as black-box renderers.•Diffusion energy function can affect lighting change in image synthesis.•Training-free relighting can be achieved with proper physics-based constraints.