Joint Face Image Restoration and Frontalization for Recognition

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 3; pp. 1285 - 1298
• Main Authors: Tu, Xiaoguang, Zhao, Jian, Liu, Qiankun, Ai, Wenjie, Guo, Guodong, Li, Zhifeng, Liu, Wei, Feng, Jiashi
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D based face normalization; Coders; Encoders-Decoders; Face recognition; Feature extraction; Generators; Image quality; Image restoration; Lighting; multi-degradation face restoration; Object recognition; Task analysis; Training; unconstrained face recognition
• DOI: 10.1109/TCSVT.2021.3078517
• ISSN: 1051-8215

In real-world scenarios, many factors may harm face recognition performance, e.g ., large pose, bad illumination, low resolution, blur and noise. To address these challenges, previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. However, most of these methods are stage-wise, which is sub-optimal and deviates from the reality. In this paper, we address all these challenges jointly for unconstrained face recognition. We propose an M ulti- D egradation F ace R estoration (MDFR) model to restore frontalized high-quality faces from the given low-quality ones under arbitrary facial poses, with three distinct novelties. First, MDFR is a well-designed encoder-decoder architecture which extracts feature representation from an input face image with arbitrary low-quality factors and restores it to a high-quality counterpart. Second, MDFR introduces a pose residual learning strategy along with a 3D-based P ose N ormalization M odule (PNM), which can perceive the pose gap between the input initial pose and its real-frontal pose to guide the face frontalization. Finally, MDFR can generate frontalized high-quality face images by a single unified network, showing a strong capability of preserving face identity. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks demonstrate the superiority of MDFR over state-of-the-art methods on both face frontalization and face restoration.