Multi-Task Convolutional Neural Network for Pose-Invariant Face Recognition

• Published in: IEEE transactions on image processing Vol. 27; no. 2; pp. 964 - 975
• Main Authors: Yin, Xi, Liu, Xiaoming
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial neural networks; Biometric Identification - methods; CNN; Databases, Factual; disentangled representation; Estimation; Face; Face - diagnostic imaging; Face recognition; Facial recognition technology; Feature extraction; Humans; Image Processing, Computer-Assisted - methods; Invariants; Multi-task learning; Neural networks; Neural Networks (Computer); Pattern Recognition, Automated - methods; pose-invariant face recognition; Testing; Training; Weight analysis
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2017.2765830

This paper explores multi-task learning (MTL) for face recognition. First, we propose a multi-task convolutional neural network (CNN) for face recognition, where identity classification is the main task and pose, illumination, and expression (PIE) estimations are the side tasks. Second, we develop a dynamic-weighting scheme to automatically assign the loss weights to each side task, which solves the crucial problem of balancing between different tasks in MTL. Third, we propose a pose-directed multi-task CNN by grouping different poses to learn pose-specific identity features, simultaneously across all poses in a joint framework. Last but not least, we propose an energy-based weight analysis method to explore how CNN-based MTL works. We observe that the side tasks serve as regularizations to disentangle the PIE variations from the learnt identity features. Extensive experiments on the entire multi-PIE dataset demonstrate the effectiveness of the proposed approach. To the best of our knowledge, this is the first work using all data in multi-PIE for face recognition. Our approach is also applicable to in-the-wild data sets for pose-invariant face recognition and achieves comparable or better performance than state of the art on LFW, CFP, and IJB-A datasets.