RES-CapsNet: an improved capsule network for micro-expression recognition

• Published in: Multimedia systems Vol. 29; no. 3; pp. 1593 - 1601
• Main Authors: Shu, Xin, Li, Jia, Shi, Liang, Huang, Shucheng
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.06.2023; Springer Nature B.V
• Subjects: Computer Communication Networks; Computer Graphics; Computer Science; Cryptology; Data Storage Representation; Feature extraction; Machine learning; Multimedia Information Systems; Operating Systems; Recognition; Regular Paper; Deep learning; SENet; Micro-expression recognition; Capsule network; Res2Net
• DOI: 10.1007/s00530-023-01068-z
• ISSN: 0942-4962

Micro-expression is a type of facial expression that reveals the deepest feeling held within the human heart. Despite the substantial improvement that has been achieved, micro-expression recognition remains a significant challenge considering its low intensity and short duration. In this paper, we investigate the recognition of micro-expression using deep learning techniques and present the RES-CapsNet, which is an improved capsule network that employs Res2Net as the backbone to extract multi-level and multi-scale characteristics. Furthermore, RES-CapsNet adds a squeeze-excitation (SE) block to the primary capsule layer (PrimaryCaps). Benefiting from a SE block, the valuable micro-expression features are highlighted and the useless ones are suppressed. In addition, between the first convolutional layer and the PrimaryCaps in RES-CapsNet, we introduce an effective channel attention (ECA) module that simply includes a few parameters while dramatically improving the performance. The proposed architecture initially obtains apex frames from the micro-expression sequence to capture the most distinct facial muscle movements and then feeds the pre-processed images into RES-CapsNet for further feature extraction and classification. The Leave-One-Subject-Out (LOSO) cross-validation strategy is implemented on three prevalent spontaneous micro-expression databases (i.e., CASME II, SMIC, and SAMM) to assess the feasibility of our RES-CapsNet. Extensive experiments demonstrate that our RES-CapsNet describes considerable details of micro-expression effectively and achieves superiorly higher performance than the baseline CapsuleNet.