Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

• Published in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 2690 - 2698
• Main Authors: Xie, Haozhe, Yao, Hongxun, Sun, Xiaoshuai, Zhou, Shangchen, Zhang, Shengping
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2019
• Subjects: Decoding; Feature extraction; Image reconstruction; Kernel; Shape; Solid modeling; Three-dimensional displays
• ISSN: 2380-7504
• DOI: 10.1109/ICCV.2019.00278

Recovering the 3D representation of an object from single-view or multi-view RGB images by deep neural networks has attracted increasing attention in the past few years. Several mainstream works (e.g., 3D-R2N2) use recurrent neural networks (RNNs) to fuse multiple feature maps extracted from input images sequentially. However, when given the same set of input images with different orders, RNN-based approaches are unable to produce consistent reconstruction results. Moreover, due to long-term memory loss, RNNs cannot fully exploit input images to refine reconstruction results. To solve these problems, we propose a novel framework for single-view and multi-view 3D reconstruction, named Pix2Vox. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. Then, a context-aware fusion module is introduced to adaptively select high-quality reconstructions for each part (e.g., table legs) from different coarse 3D volumes to obtain a fused 3D volume. Finally, a refiner further refines the fused 3D volume to generate the final output. Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-of-the-arts by a large margin. Furthermore, the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. The experiments on ShapeNet unseen 3D categories have shown the superior generalization abilities of our method.