Deep-MDS framework for recovering the 3D shape of 2D landmarks from a single image

• Published in: Journal of visual communication and image representation Vol. 98; p. 104032
• Main Authors: Kamyab, Shima, Azimifar, Zohreh
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.02.2024
• Subjects: Deep learning human face shape recovery; Single view 3D human face shape recovery; Using multidimensional scaling for 2D shape recovery from a singe image; Single view 3D human face shape recovery; Using multidimensional scaling for 2D shape recovery from a singe image; Deep learning human face shape recovery
• ISSN: 1047-3203; 1095-9076
• DOI: 10.1016/j.jvcir.2023.104032

Using 3D reconstruction techniques within computer vision frameworks can result in more robust and accurate solutions. However, the main challenge lies in the high computation and memory resources required by such methods. To reduce the complexity of these frameworks, a practical solution is to use geometric landmarks instead of the entire image. Therefore, in this paper the problem of 3D shape recovery of a set of standard 2D landmarks, on a single human face image is faced using multi-dimensional scaling (MDS) approach to find a 3D embedding for a set of standard 2D points. Hence, MDS approach is used for the first time in this study to establish an unbiased mapping from 2D landmark space to the corresponding 3D shape space. A deep neural network learns the pairwise 3D dissimilarity among standard 2D landmarks. This scheme leads to find a symmetric dissimilarity matrix, to be fed into the MDS approach to appropriately recovering the 3D shape of corresponding 2D landmarks. In the case of complex input image formations like posedness or perspective projection causing occlusion in the input image, an autoencoder component is used in the proposed framework, as an occlusion removal part, which turns different input views of the human face into a profile view. The results of performance evaluation using variety of synthetic and real-world human face datasets, including NoW dataset, Besel Face Model (BFM), CelebA, CoMA - FLAME, and CASIA-3D, indicates the superiority of the proposed framework, despite its small number of training parameters, with the related state-of-the-art and recent 3D shape recovery of landmark methods from the literature, in terms of unbiasedness, independent projection, efficiency and accuracy. Further, ablation study is performed to find the best training scheme of the deep learning components. All codes and public data of our paper are publicly available for research purposes at https://github.com/s2kamyab/DeepMDS.