Your heart rate betrays you: multimodal learning with spatio-temporal fusion networks for micro-expression recognition

• Published in: International journal of multimedia information retrieval Vol. 11; no. 4; pp. 553 - 566
• Main Authors: Zhang, Ren, He, Ning, Liu, Shengjie, Wu, Ying, Yan, Kang, He, Yuzhe, Lu, Ke
• Format: Journal Article
• Language: English
• Published: London Springer London 01.12.2022; Springer Nature B.V
• Subjects: Accuracy; Age; Blood; Cameras; Classification; Computer Science; Data Mining and Knowledge Discovery; Database Management; Heart rate; Image Processing and Computer Vision; Information Storage and Retrieval; Information Systems Applications (incl.Internet); Multimedia Information Systems; Neural networks; Physiology; Recognition; Regular Paper; Signal processing; Spatial data; Micro-expression; Spatio-temporal feature fusion; Optical flow; Multimodal learning
• ISSN: 2192-6611; 2192-662X
• DOI: 10.1007/s13735-022-00250-9

Micro-expressions can convey feelings that people are trying to hide. At present, some studies on micro-expression, most of which only use the temporal or spatial information in the image to recognize micro-expressions, neglect the intrinsic features of the image. To solve this problem, we detect the subject’s heart rate in the long micro-expression videos; we extract the image’s spatio-temporal feature through a spatio-temporal network and then extract the heart rate feature using a heart rate network. A multimodal learning method that combines the heart rate and spatio-temporal features is used to recognize micro-expressions. The experimental results on CASMEII , SAMM , and SMIC show that the proposed methods’ unweighted F1-score and unweighted average recall are 0.8867 and 0.8962, respectively. The spatio-temporal fusion network combined with heart rate information provides an essential reference for multimodal approaches to micro-expression recognition.