Dual-stream Multiple Instance Learning for Depression Detection with Facial Expression Videos

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 31; p. 1
• Main Authors: Shangguan, Zixuan, Liu, Zhenyu, Li, Gang, Chen, Qiongqiong, Ding, ZhiJie, Hu, Bin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Algorithms; Classification; Deep learning; Depression; Depression - diagnosis; Depression detection; Facial Expression; Feature extraction; Humans; Illnesses; Image Interpretation, Computer-Assisted - methods; Learning; Mental depression; Mental disorders; Mental Recall; multiple instance learning; Recall; Rehabilitation; Signs and symptoms; Supervised learning; Task analysis; Three-dimensional displays; Video; Videos; weakly supervised learning
• DOI: 10.1109/TNSRE.2022.3204757
• ISSN: 1534-4320; 1558-0210

Depression is a common mental illness which has brought great harm to the individuals. With recent evidence that many objective physiological signals are associated with depression, automated detection of depression is urgent and important for the growing concern of mental illness. We investigate the problem of classifying depression by facial expressions, which may aid in online diagnosis and rehabilitation engineering of depression. In this work, We propose a weakly supervised learning approach employing multiple instance learning (MIL) on 150 videos data from 75 depressed and 75 healthy subjects. In addition, we present a novel MIL dual-stream aggregator that considers both the instance-level and the bag-level in order to emphasize the information with symptoms. Specifically, our method named ADDMIL uses max-pooling at the instance level to capture symptom information and further integrates the contribution of each instance at the bag level using attention weights. Our method achieves 74.7% accuracy and 74.5% recall on the collected dataset, which not only improves 10.1% accuracy and 9.8% recall over the baseline but also exceeds the best accuracy result of MIL-based method by 2.1%. Our work achieves results that are comparable to the state-of-the-art methods and demonstrates that multiple instance learning has great potential for depression classification. We present for the first time a weakly supervised learning approach in the detection of depression through raw facial expressions, which may provide a new framework for other psychiatric disorders detection methods.