Check, Locate, Rectify: A Training-Free Layout Calibration System for Text- to- Image Generation

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 6624 - 6634
• Main Authors: Gong, Biao, Huang, Siteng, Feng, Yutong, Zhang, Shiwei, Li, Yuyuan, Liu, Yu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Benchmark testing; Diffusion models; Generators; Image synthesis; Layout; layout calibration; Pipelines; Text to image; text-to-image generation; training-free
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.00633

Diffusion models have recently achieved remarkable progress in generating realistic images. However, chal-lenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system S imM that inter-venes in the generative process on the fly during infer-ence time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM.