Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli

• Published in: Journal of affective disorders Vol. 381; pp. 44 - 54
• Main Authors: Liu, Yang, Li, Xingyun, Wang, Mengqi, Bi, Jianlu, Lin, Shaoqin, Wang, Qingxiang, Yu, Yanhong, Ye, Jiayu, Zheng, Yunshao
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 15.07.2025
• Subjects: Adult; Body posture; Deep Learning; Depression - diagnosis; Depression recognition; Depressive Disorder - diagnosis; Emotional stimuli; Emotions - physiology; Facial Expression; Facial expressions; Female; Humans; Male; Middle Aged; Posture - physiology; Psychiatric/Mental Health; Body posture; Emotional stimuli; Depression recognition; Facial expressions
• ISSN: 0165-0327; 1573-2517; 1573-2517
• DOI: 10.1016/j.jad.2025.03.155

Clinical studies have shown that facial expressions and body posture in depressed patients differ significantly from those of healthy individuals. Combining relevant behavioral features with artificial intelligence technology can effectively improve the efficiency of depression detection, thereby assisting doctors in early identification of patients. This study aims to develop an end-to-end multimodal recognition model combining facial expressions and body posture via deep learning techniques, enabling rapid preliminary screening of depression. We invited 146 subjects (73 in the patient group and 73 in the control group) to participate in an emotion-stimulus experiment for depression recognition. We focused on differentiating depression patients from the control group by analyzing changes in body posture and facial expressions under emotional stimuli. We first extracted images of body position and facial emotions from the video, then used a pre-trained ResNet-50 network to extract features. Additionally, we analyzed facial expression features using OpenFace for sequence analysis. Subsequently, various deep learning frameworks were combined to assess the severity of depression. We found that under different stimuli, facial expression units AU04, AU07, AU10, AU12, AU17, and AU26 had significant effects in the emotion-stimulus experiment, with these features generally being negative. The decision-level fusion model based on facial expressions and body posture achieved excellent results, with the highest accuracy of 0.904 and an F1 score of 0.901. The experimental results suggest that depression patients exhibit predominantly negative facial expressions. This study validates the emotion-stimulus experiment, demonstrating that combining facial expressions and body posture enables accurate preliminary depression screening. •In the area of depression recognition, we are the first to suggest an experiment on how body posture changes in response to emotional stimuli.•We accomplished single-mode and multimodal depression recognition using body postures and facial expressions.•We found that AU04, AU07, AU10, AU12, AU17 and AU26 showed significant differences in emotional stimulation experiments.•The emotional stimulation experiment was easier to carry out because of its simple operation.