A convolution-transformer dual branch network for head-pose and occlusion facial expression recognition

• Published in: The Visual computer Vol. 39; no. 6; pp. 2277 - 2290
• Main Authors: Liang, Xingcan, Xu, Linsen, Zhang, Wenxiang, Zhang, Yan, Liu, Jinfu, Liu, Zhipeng
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.06.2023; Springer Nature B.V
• Subjects: Artificial Intelligence; Computer Graphics; Computer Science; Convolution; Deep learning; Face recognition; Image Processing and Computer Vision; Modules; Natural language processing; Neural networks; Occlusion; Original Article; Robustness; Teaching methods; Transformers; Facial expression recognition; Transformers; Feature fusion; Robust on occlusions and head-pose variations; CNNs
• ISSN: 0178-2789; 1432-2315
• DOI: 10.1007/s00371-022-02413-5

Facial expression recognition (FER) has attracted much more attention due to its broad range of applications. Occlusions and head-pose variations are two major obstacles for automatic FER. In this paper, we propose a convolution-transformer dual branch network (CT-DBN) that takes advantage of local and global facial information to tackle the real-word occlusions and head-pose variant robust FER. The CT-DBN contains two branches. Taking into account local modeling ability of CNN, the first branch utilizes CNN to capture local edge information. Inspired by transformers’ successful application in natural language processing, we employ transformer to the second branch to be responsible for obtaining better global representation. Then, a local–global feature fusion module is proposed to adaptively integrate both features to hybrid features and model the relationship between them. With the help of feature fusion module, our network not only integrates local and global features in an adaptive weighting manner but can also learn the corresponding distinguishable features autonomously. Experimental results under inner-database and cross-database evaluation on four leading facial expression databases illustrate that our proposed CT-DBN outperforms other state-of-the-art methods and achieves robust performance under in-the-wild condition.