Identity-Preserving Face Recovery from Stylized Portraits

• Published in: International journal of computer vision Vol. 127; no. 6-7; pp. 863 - 883
• Main Authors: Shiri, Fatemeh, Yu, Xin, Porikli, Fatih, Hartley, Richard, Koniusz, Piotr
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.06.2019; Springer; Springer Nature B.V
• Subjects: Artificial Intelligence; Computer Imaging; Computer Science; Distance measurement; Feature extraction; Feature maps; Ground truth; Image Processing and Computer Vision; Pattern Recognition; Pattern Recognition and Graphics; Portraits; Recovery; Sketches; Vision; Face recovery; Face synthesis; Image stylization; Generative models; Destylization
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-019-01169-1

Given an artistic portrait, recovering the latent photorealistic face that preserves the subject’s identity is challenging because the facial details are often distorted or fully lost in artistic portraits. We develop an Identity-preserving Face Recovery from Portraits method that utilizes a Style Removal network (SRN) and a Discriminative Network (DN). Our SRN, composed of an autoencoder with residual block-embedded skip connections, is designed to transfer feature maps of stylized images to the feature maps of the corresponding photorealistic faces. Owing to the Spatial Transformer Network, SRN automatically compensates for misalignments of stylized portraits to output aligned realistic face images. To ensure the identity preservation, we promote the recovered and ground truth faces to share similar visual features via a distance measure which compares features of recovered and ground truth faces extracted from a pre-trained FaceNet network. DN has multiple convolutional and fully-connected layers, and its role is to enforce recovered faces to be similar to authentic faces. Thus, we can recover high-quality photorealistic faces from unaligned portraits while preserving the identity of the face in an image. By conducting extensive evaluations on a large-scale synthesized dataset and a hand-drawn sketch dataset, we demonstrate that our method achieves superior face recovery and attains state-of-the-art results. In addition, our method can recover photorealistic faces from unseen stylized portraits, artistic paintings, and hand-drawn sketches.