Multi-view 3D face reconstruction with deep recurrent neural networks

• Published in: Image and vision computing Vol. 80; pp. 80 - 91
• Main Authors: Dou, Pengfei, Kakadiaris, Ioannis A.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2018
• Subjects: 3D face reconstruction; Face recognition; Long-short term memory; Recurrent neural network; 3D face reconstruction; Long-short term memory; Recurrent neural network; Face recognition
• ISSN: 0262-8856; 1872-8138
• DOI: 10.1016/j.imavis.2018.09.004

Image-based 3D face reconstruction has great potential in different areas, such as facial recognition, facial analysis, and facial animation. Due to the variations in image quality, single-image-based 3D face reconstruction might not be sufficient to accurately reconstruct a 3D face. To overcome this limitation, multi-view 3D face reconstruction uses multiple images of the same subject and aggregates complementary information for better accuracy. Though appealing, there are multiple challenges in practice. Among these challenges, the most significant is the difficulty to establish coherent and accurate correspondence among a set of images, especially when these images are captured under unconstrained in-the-wild condition. This work proposes a method, Deep Recurrent 3D FAce Reconstruction (DRFAR), to solve the task of multi-view 3D face reconstruction using a subspace representation of the 3D facial shape and a deep recurrent neural network that consists of both a deep convolutional neural network (DCNN) and a recurrent neural network (RNN). The DCNN disentangles the facial identity and the facial expression components for each single image independently, while the RNN fuses identity-related features from the DCNN and aggregates the identity specific contextual information, or the identity signal, from the whole set of images to estimate the facial identity parameter, which is robust to variations in image quality and is consistent over the whole set of images. Experimental results indicate significant improvement over state-of-the-art in both the accuracy and the consistency of 3D face reconstruction. Moreover, face recognition results on IJB-A with the UR2D face recognition pipeline indicate that, compared to single-view 3D face reconstruction, the proposed multi-view 3D face reconstruction algorithm can improve the face identification accuracy of UR2D by two percentage points in Rank-1 identification rate. •Recurrent neural network is effective for feature fusion in MV3DR.•End-to-end MV3DR is achieved with a CNN and a RNN for feature extraction and fusion.•Fusion by averaging over shapes suppresses facial identity information in 3D face.•Fusion by aggregating facial features helps preserve facial identity information.•Performance of 3D-aided face recognition is improved with MV3DR.