Monocular 3D facial shape reconstruction from a single 2D image with coupled-dictionary learning and sparse coding

• Published in: Pattern recognition Vol. 81; pp. 515 - 527
• Main Authors: Dou, Pengfei, Wu, Yuhang, Shah, Shishir K., Kakadiaris, Ioannis A.
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2018
• Subjects: 3D face modeling; 3D face reconstruction; 3D super-resolution; Dictionary learning; Sparse coding; 3D super-resolution; 3D face reconstruction; 3D face modeling; Sparse coding; Dictionary learning
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2018.03.002

•Proposing a dictionary-based parametric model of 3D facial shape that achieves more than 50 and 20% improvement in reconstruction accuracy for seen and unseen data, respectively, over the widely applied PCA-based subspace model.•Developing an efficient algorithm for estimating the sparse 3D facial shape from 2D facial landmarks that is generalizable to different types of data, including facial images, portraits, and facial sketches.•Developing an algorithm for 3D super-resolution that reconstructs the dense 3D face based on the estimated sparse 3D facial shape and achieves an average of 10% reduction in reconstruction error over four state-of-the-art algorithms. [Display omitted] Monocular 3D face reconstruction from a single image has been an active research topic due to its wide applications. It has been demonstrated that the 3D face can be reconstructed efficiently using a PCA-based subspace model for facial shape representation and facial landmarks for model parameter estimation. However, due to the limited expressiveness of the subspace model and the inaccuracy of landmark detection, most existing methods are not robust to pose and illumination variation. To overcome this limitation, this work proposes a coupled-dictionary model for parametric facial shape representation and a two-stage framework for 3D face reconstruction from a single 2D image by using facial landmarks. Motivated by image super-resolution, the proposed coupled-model consists of two dictionaries for the sparse and the dense 3D facial shapes, respectively. In the first stage, the sparse 3D face is estimated from facial landmarks by using partial least-squares regression. In the second stage, the dense 3D face is reconstructed by 3D super-resolution on the estimated sparse 3D face. Comprehensive experimental evaluations demonstrate that the proposed coupled-dictionary model outperforms the PCA-based subspace model in 3D face modeling accuracy and that the proposed framework achieves much lower reconstruction error on facial images with pose and illumination variations compared to state-of-the-art algorithms. Moreover, qualitative analysis demonstrates that the proposed method is generalizable to different types of data, including facial images, portraits, and facial sketches.