A Dataset-Dispersion Perspective on Reconstruction Versus Recognition in Single-View 3D Reconstruction Networks

• Published in: 2021 International Conference on 3D Vision (3DV) pp. 1331 - 1340
• Main Authors: Zhou, Yefan, Shen, Yiru, Yan, Yujun, Feng, Chen, Yang, Yaoqing
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2021
• Subjects: 3D vision; Artificial neural networks; Image recognition; Reconstruction metric; Shape; Single view 3D reconstruction; Systematics; Three-dimensional displays; Training; Training data
• ISSN: 2475-7888
• DOI: 10.1109/3DV53792.2021.00140

Neural networks (NN) for single-view 3D reconstruction (SVR) have gained in popularity. Recent work points out that for SVR, most cutting-edge NNs have limited performance on reconstructing unseen objects because they rely primarily on recognition (i.e., classification-based methods) rather than shape reconstruction. To understand this issue in depth, we provide a systematic study on when and why NNs prefer recognition to reconstruction and vice versa. Our finding shows that a leading factor in determining recognition versus reconstruction is how "dispersed" the training data is. Thus, we introduce the dispersion score, a new data-driven metric, to quantify this leading factor and study its effect on NNs. We hypothesize that NNs are biased toward recognition when training images are more dispersed and training shapes are less dispersed. Our hypothesis is supported and the dispersion score is proved effective through our experiments on synthetic and benchmark datasets. We show that the proposed metric is a principal way to analyze reconstruction quality and provides novel information in addition to the conventional reconstruction score. We have open-sourced our code. 1