Analysis and Recognition of Voluntary Facial Expression Mimicry Based on Depressed Patients

• Published in: IEEE journal of biomedical and health informatics Vol. 27; no. 8; pp. 3698 - 3709
• Main Authors: Ye, Jiayu, Yu, Yanhong, Fu, Gang, Zheng, Yunshao, Liu, Yang, Zhu, Yitao, Wang, Qingxiang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adolescent; Adult; Algorithms; Anger; Attention; Bioinformatics; Cognitive ability; Correlation analysis; Correlation of Data; Deep learning; Depression; Disgust; E-ResNet; E-ViT; Emotions; Face recognition; Facial Expression; Facial features; Fear; Feature extraction; Feature selection; FER; Happiness; Humans; Imitative Behavior; Logistic Models; Middle Aged; Mimicry; Pattern recognition; Random Forest; Sadness; Support vector machines; VFEM; Young Adult
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2023.3260816

Many clinical studies have shown that facial expression recognition and cognitive function are impaired in depressed patients. Different from spontaneous facial expression mimicry (SFEM), 164 subjects (82 in a case group and 82 in a control group) participated in our voluntary facial expression mimicry (VFEM) experiment using expressions of neutrality, anger, disgust, fear, happiness, sadness and surprise. Our research is as follows. First, we collected a large amount of subject data for VFEM. Second, we extracted the geometric features of subject facial expression images for VFEM and used Spearman correlation analysis, a random forest, and logistic regression-based recursive feature elimination (LR-RFE) to perform feature selection. The features selected revealed the difference between the case group and the control group. Third, we combined geometric features with the original images and improved the advanced deep learning facial expression recognition (FER) algorithms in different systems. We propose the E-ViT and E-ResNet based on VFEM. The accuracies and F1 scores were higher than those of the baseline models, respectively. Our research proved that it is effective to use feature selection to screen geometric features and combine them with a deep learning model for depression facial expression recognition.