Detecting Facial Action Units From Global-Local Fine-Grained Expressions

• Published in: IEEE transactions on circuits and systems for video technology Vol. 34; no. 2; pp. 983 - 994
• Main Authors: Zhang, Wei, Li, Lincheng, Ding, Yu, Chen, Wei, Deng, Zhigang, Yu, Xin
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Action units; Annotations; Datasets; deep learning; Detection algorithms; Embedding; expression embedding; Face recognition; facial expression; Feature extraction; Image reconstruction; Labels; Three-dimensional displays; Training; Transformers
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3288903

Since Facial Action Unit (AU) annotations require domain expertise, common AU datasets only contain a limited number of subjects. As a result, a crucial challenge for AU detection is addressing identity overfitting. We find that AUs and facial expressions are highly associated, and existing facial expression datasets often contain a large number of identities. In this paper, we aim to utilize the expression datasets without AU labels to facilitate AU detection. Specifically, we develop a novel AU detection framework aided by the Global-Local facial Expressions Embedding, dubbed GLEE-Net. Our GLEE-Net consists of three branches to extract identity-independent expression features for AU detection. We introduce a global branch for modeling the overall facial expression while eliminating the impacts of identities. We also design a local branch focusing on specific local face regions. The combined output of global and local branches is firstly pre-trained on an expression dataset as an identity-independent expression embedding, and then finetuned on AU datasets. Therefore, we significantly alleviate the issue of limited identities. Furthermore, we introduce a 3D global branch that extracts expression coefficients through 3D face reconstruction to consolidate 2D AU descriptions. Finally, a Transformer-based multi-label classifier is employed to fuse all the representations for AU detection. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art on the widely-used DISFA, BP4D and BP4D+ datasets.