Wonder3D: Single Image to 3D Using Cross-Domain Diffusion

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 9970 - 9980
• Main Authors: Long, Xiaoxiao, Guo, Yuan-Chen, Lin, Cheng, Liu, Yuan, Dou, Zhiyang, Liu, Lingjie, Ma, Yuexin, Zhang, Song-Hai, Habermann, Marc, Theobalt, Christian, Wang, Wenping
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: 3D generation; Color; Computer vision; diffusion; Diffusion models; Geometry; MVS; NeuS; normal; Pattern recognition; Surface reconstruction; Three-dimensional displays
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.00951

In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works di-rectly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of single-view reconstruction tasks, we pro-pose a cross-domain diffusion model that generates multi-view normal maps and the corresponding color images. To ensure the consistency of generation, we employ a multi-view cross-domain attention mechanism that facilitates information exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D representations in only 2 r-;» 3 minutes. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, robust generalization, and good efficiency compared to prior works.