Multimodal temporal machine learning for Bipolar Disorder and Depression Recognition

• Published in: Pattern analysis and applications : PAA Vol. 25; no. 3; pp. 493 - 504
• Main Authors: Ceccarelli, Francesco, Mahmoud, Marwa
• Format: Journal Article
• Language: English
• Published: London Springer London 01.08.2022; Springer Nature B.V
• Subjects: Bipolar disorder; Computer architecture; Computer Science; Evolution; Feature extraction; Machine learning; Mental depression; Mental disorders; Neural networks; Original Article; Pattern Recognition; Public health; Recognition; Recurrent neural networks; Special Issue on Computer Vision and Machine Learning for Healthcare Applications; Mental disorder; Multimodal; Recurrent neural network; Machine learning
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-021-01001-y

Mental disorder is a serious public health concern that affects the life of millions of people throughout the world. Early diagnosis is essential to ensure timely treatment and to improve the well-being of those affected by a mental disorder. In this paper, we present a novel multimodal framework to perform mental disorder recognition from videos. The proposed approach employs a combination of audio, video and textual modalities. Using recurrent neural network architectures, we incorporate the temporal information in the learning process and model the dynamic evolution of the features extracted for each patient. For multimodal fusion, we propose an efficient late fusion strategy based on a simple feed-forward neural network that we call adaptive nonlinear judge classifier . We evaluate the proposed framework on two mental disorder datasets. On both, the experimental results demonstrate that the proposed framework outperforms the state-of-the-art approaches. We also study the importance of each modality for mental disorder recognition and infer interesting conclusions about the temporal nature of each modality. Our findings demonstrate that careful consideration of the temporal evolution of each modality is of crucial importance to accurately perform mental disorder recognition.