DisenDreamer: Subject-Driven Text-to-Image Generation With Sample-Aware Disentangled Tuning

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 8; pp. 6860 - 6873
• Main Authors: Chen, Hong, Zhang, Yipeng, Wang, Xin, Duan, Xuguang, Zhou, Yuwei, Zhu, Wenwu
• Format: Journal Article
• Language: English
• Published: IEEE 01.08.2024
• Subjects: Circuits and systems; Controllability; Diffusion model; disentangled finetuning; Image synthesis; Noise reduction; subject-driven text-to-image generation; Training; Tuning; Visualization
• DOI: 10.1109/TCSVT.2024.3369757
• ISSN: 1051-8215

Subject-driven text-to-image generation aims to generate customized images of the given subject based on the text descriptions, which has drawn increasing attention recently. Existing methods mainly resort to finetuning a pretrained generative model, where the identity-relevant information (e.g., the boy) and the identity-irrelevant sample-specific information (e.g., the background or the pose of the boy) are entangled in the latent embedding space. However, the highly entangled latent embedding may lead to low subject identity fidelity and text prompt fidelity. To tackle the problems, we propose DisenDreamer, a sample-aware disentangled tuning framework for subject-driven text-to-image generation in this paper. Specifically, DisenDreamer finetunes the pretrained diffusion model in the denoising process. Different from previous works that utilize an entangled embedding to denoise, DisenDreamer instead utilizes a common text embedding to capture the identity-relevant information and a sample-specific visual embedding to capture the identity-irrelevant information. To disentangle the two embeddings, we further design the novel weak common denoising, weak sample-aware denoising, and the contrastive embedding auxiliary tuning objectives. Extensive experiments show that our proposed DisenDreamer framework outperforms baseline models for subject-driven text-to-image generation. Additionally, by combining the identity-relevant and the identity-irrelevant embedding, DisenDreamer demonstrates more generation flexibility and controllability.