GenesisTex: Adapting Image Denoising Diffusion to Texture Space

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4620 - 4629
• Main Authors: Gao, Chenjian, Jiang, Boyan, Li, Xinghui, Zhang, Yingpeng, Yu, Qian
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Computer vision; Diffusion models; Geometry; Noise; Pattern recognition; Rendering (computer graphics); Three-dimensional displays
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.00442

We present GenesisTex, a novel method for synthesizing textures for 3D geometries from text descriptions. GenesisTex adapts the pretrained image diffusion model to texture space by texture space sampling. Specifically, we maintain a latent texture map for each viewpoint, which is updated with predicted noise on the rendering of the corresponding viewpoint. The sampled latent texture maps are then decoded into a final texture map. During the sampling process, we focus on both global and local consistency across multiple viewpoints: global consistency is achieved through the integration of style consistency mechanisms within the noise prediction network, and low-level consistency is achieved by dynamically aligning latent textures. Finally, we apply reference-based inpainting and img2img on denser views for texture refinement. Our approach overcomes the limitations of slow optimization in distillation-based methods and instability in inpainting-based methods. Experiments on meshes from various sources demonstrate that our method surpasses the baseline methods quantitatively and qualitatively.