A multi-modal approach for identifying schizophrenia using cross-modal attention

• Published in: 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) Vol. 2024; pp. 1 - 5
• Main Authors: Premananth, Gowtham, Siriwarden, Yashish M., Resnik, Philip, Espy-Wilson, Carol
• Format: Conference Proceeding Journal Article
• Language: English
• Published: United States IEEE 01.07.2024
• Subjects: Adult; Algorithms; Attention; Audio-visual systems; Biological system modeling; Computational modeling; Engineering in medicine and biology; Facial Action units; Feature extraction; Female; Humans; Male; Multi-modal model; Schizophrenia; Schizophrenia - diagnosis; Schizophrenia - physiopathology; Text Embeddings; Vocal tract variables

This study focuses on how different modalities of human communication can be used to distinguish between healthy controls and subjects with schizophrenia who exhibit strong positive symptoms. We developed a multi-modal schizophrenia classification system using audio, video, and text. Facial action u...