BATINeT: Background-Aware Text to Image Synthesis and Manipulation Network

• Published in: 2023 IEEE International Conference on Image Processing (ICIP) pp. 765 - 769
• Main Authors: Morita, Ryugo, Zhang, Zhiqiang, Zhou, Jinjia
• Format: Conference Proceeding
• Language: English
• Published: IEEE 08.10.2023
• Subjects: Background-induced text to image synthesis; Generative Adversarial Networks; Image reconstruction; Image synthesis; Impedance matching; Shape; Task analysis; Text to image; Text-guided image manipulation
• DOI: 10.1109/ICIP49359.2023.10223174

Background-Induced Text2Image (BIT2I) aims to generate foreground content according to the text on the given background image. Most studies focus on generating high-quality foreground content, although they ignore the relationship between the two contents. In this study, we analyzed a novel Background-Aware Text2Image (BAT2I) task in which the generated content matches the input background. We proposed a Background-Aware Text to Image synthesis and manipulation Network (BATINet), which contains two key components: Position Detect Network (PDN) and Harmonize Network (HN). The PDN detects the most plausible position of the text-relevant object in the background image. The HN harmonizes the generated content referring to background style information. Finally, we reconstructed the generation network, which consists of the multi-GAN and attention module to match more user preferences. Moreover, we can apply BATINet to text-guided image manipulation. It solves the most challenging task of manipulating the shape of an object. We demonstrated through qualitative and quantitative evaluations on the CUB dataset that the proposed model outperforms other state-of-the-art methods.