Micro‐expression recognition by two‐stream difference network

• Published in: IET computer vision Vol. 15; no. 6; pp. 440 - 448
• Main Authors: Pan, Hang, Xie, Lun, Li, Juan, Lv, Zeping, Wang, Zhiliang
• Format: Journal Article
• Language: English
• Published: Stevenage John Wiley & Sons, Inc 01.09.2021; Wiley
• Subjects: Artificial neural networks; convolutional neural nets; Ethnicity; face recognition; Feature recognition; Gender; Identity; image coding; image sequences; Neural networks
• ISSN: 1751-9632; 1751-9640
• DOI: 10.1049/cvi2.12030

Facial micro‐expression is a superposition of micro‐expression features and identity information of a subject. For the problem of identity information interference in micro‐expression recognition, this study proposes a new method for facial micro‐expression recognition by de‐identity information, called two‐stream difference network (TSDN). First, a two‐stream encoder‐decoder network is trained by a convolutional neural network, where the input of the micro‐expression stream is a micro‐expression image, and the identity stream is a facial identity image. The micro‐expression image is the apex image, and the identity image is the onset image in the micro‐expression sequence. The identity information and micro‐expression features are recorded in the intermediate layer of the micro‐expression stream, while the intermediate layer of the identity stream contains only the identity information of a subject. Then, the identity information is removed by the difference network, but micro‐expression features are stored in the intermediate layer of the micro‐expression stream. Given the sequence of the micro‐expressions, the TSDN model of de‐identity information learns the difference that stores in the expression stream. Two public spontaneous facial micro‐expression data sets (SMIC and CASME II) are employed in our experiments. The experiment results show that our model can achieve a superior performance in micro‐expression recognition.