MS ^ -GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection

• Published in: IEEE transactions on cybernetics pp. 1 - 11
• Main Authors: Chen, Tao, Hong, Richang, Guo, Yanrong, Hao, Shijie, Hu, Bin
• Format: Journal Article
• Language: English
• Published: IEEE 01.12.2023
• Subjects: Data mining; Depression; Depression detection; Electroencephalography; Feature extraction; graph neural network (GNN); Graph neural networks; multimodal fusion; Semantics; Task analysis
• ISSN: 2168-2267
• DOI: 10.1109/TCYB.2022.3197127

Major depressive disorder (MDD) is one of the most common and severe mental illnesses, posing a huge burden on society and families. Recently, some multimodal methods have been proposed to learn a multimodal embedding for MDD detection and achieved promising performance. However, these methods ignore the heterogeneity/homogeneity among various modalities. Besides, earlier attempts ignore interclass separability and intraclass compactness. Inspired by the above observations, we propose a graph neural network (GNN)-based multimodal fusion strategy named modal-shared modal-specific GNN, which investigates the heterogeneity/homogeneity among various psychophysiological modalities as well as explores the potential relationship between subjects. Specifically, we develop a modal-shared and modal-specific GNN architecture to extract the inter/intramodal characteristics. Furthermore, a reconstruction network is employed to ensure fidelity within the individual modality. Moreover, we impose an attention mechanism on various embeddings to obtain a multimodal compact representation for the subsequent MDD detection task. We conduct extensive experiments on two public depression datasets and the favorable results demonstrate the effectiveness of the proposed algorithm.