Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

• Published in: IEEE transaction on neural networks and learning systems Vol. 35; no. 6; pp. 7627 - 7641
• Main Authors: Liu, Rui, Huang, Zhi-An, Hu, Yao, Zhu, Zexuan, Wong, Ka-Chun, Tan, Kay Chen
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adult; Algorithms; Attention; Attention Deficit Disorder with Hyperactivity - diagnosis; Attention Deficit Disorder with Hyperactivity - diagnostic imaging; Attention deficit hyperactivity disorder; Autism; autism spectrum disorder (ASD); Autism Spectrum Disorder - diagnosis; Autism Spectrum Disorder - diagnostic imaging; Biomarkers; Brain - diagnostic imaging; computer-aided diagnosis; Data augmentation; Data models; Diagnosis; Female; Functional magnetic resonance imaging; Generative adversarial networks; Humans; Image Interpretation, Computer-Assisted - methods; Machine learning; Magnetic resonance imaging; Magnetic Resonance Imaging - methods; Male; Medical imaging; Mental disorders; Mental Disorders - diagnosis; Mental Disorders - diagnostic imaging; Multimodal Imaging - methods; multimodality data generation; multimodality fusion; Neural Networks, Computer; Neuroimaging; Neuroimaging - methods; Public health; Schizophrenia; Schizophrenia - diagnosis; Schizophrenia - diagnostic imaging; Solid modeling; Spatial discrimination; Spatial resolution; Structure-function relationships; Support vector machines; Urban areas; Young Adult
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2022.3219551

The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients' suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of <inline-formula> <tex-math notation="LaTeX">75.1~\pm ~1.5 </tex-math></inline-formula>%, <inline-formula> <tex-math notation="LaTeX">72.9~\pm ~1.1 </tex-math></inline-formula>%, and <inline-formula> <tex-math notation="LaTeX">87.2~\pm ~1.5 </tex-math></inline-formula>% for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.