Pixel2Mesh: 3D Mesh Model Generation via Image Guided Deformation

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 43; no. 10; pp. 3600 - 3613
• Main Authors: Wang, Nanyang, Zhang, Yinda, Li, Zhuwen, Fu, Yanwei, Yu, Hang, Liu, Wei, Xue, Xiangyang, Jiang, Yu-Gang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D shape generation; Adaptation models; Artificial neural networks; coarse-to-fine; Color; Color imagery; Deep learning; Deformation; end-to-end framework; Feature extraction; Finite element method; Geometry; graph convolutional neural network; Mesh generation; mesh reconstruction; Shape; Solid modeling; Strain; Three dimensional models; Three-dimensional displays
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2020.2984232

In this paper, we propose an end-to-end deep learning architecture that generates 3D triangular meshes from single color images. Restricted by the nature of prevalent deep learning techniques, the majority of previous works represent 3D shapes in volumes or point clouds. However, it is non-trivial to convert these representations to compact and ready-to-use mesh models. Unlike the existing methods, our network represents 3D shapes in meshes, which are essentially graphs and well suited for graph-based convolutional neural networks. Leveraging perceptual features extracted from an input image, our network produces the correct geometry by progressively deforming an ellipsoid. To make the whole deformation procedure stable, we adopt a coarse-to-fine strategy, and define various mesh/surface related losses to capture properties of various aspects, which benefits producing the visually appealing and physically accurate 3D geometry. In addition, our model by nature can be adapted to objects in specific domains, e.g., human faces, and be easily extended to learn per-vertex properties, e.g., color. Extensive experiments show that our method not only qualitatively produces the mesh model with better details, but also achieves the higher 3D shape estimation accuracy compared against the state-of-the-arts.