Facial optical flow estimation via neural non-rigid registration

• Published in: Computational visual media (Beijing) Vol. 9; no. 1; pp. 109 - 122
• Main Authors: Peng, Zhuang, Jiang, Boyi, Xu, Haofei, Feng, Wanquan, Zhang, Juyong
• Format: Journal Article
• Language: English
• Published: Beijing Tsinghua University Press 01.03.2023; Springer Nature B.V; SpringerOpen
• Subjects: Accuracy; Artificial Intelligence; Cognitive tasks; Computer Graphics; Computer Science; Datasets; facial priors; human face; Image Processing and Computer Vision; Methods; Neural networks; non-rigid registration; Object recognition; optical flow; Optical flow (image analysis); Optimization; Pedestrians; Registration; Research Article; self-supervised; User Interfaces and Human Computer Interaction; human face; non-rigid registration; facial priors; optical flow; self-supervised; neural networks
• ISSN: 2096-0433; 2096-0662
• DOI: 10.1007/s41095-021-0267-z

Optical flow estimation in human facial video, which provides 2D correspondences between adjacent frames, is a fundamental pre-processing step for many applications, like facial expression capture and recognition. However, it is quite challenging as human facial images contain large areas of similar textures, rich expressions, and large rotations. These characteristics also result in the scarcity of large, annotated real-world datasets. We propose a robust and accurate method to learn facial optical flow in a self-supervised manner. Specifically, we utilize various shape priors, including face depth, landmarks, and parsing, to guide the self-supervised learning task via a differentiable nonrigid registration framework. Extensive experiments demonstrate that our method achieves remarkable improvements for facial optical flow estimation in the presence of significant expressions and large rotations.