Artificial intelligence for multimodal data integration in oncology

• Published in: Cancer cell Vol. 40; no. 10; pp. 1095 - 1110
• Main Authors: Lipkova, Jana, Chen, Richard J., Chen, Bowen, Lu, Ming Y., Barbieri, Matteo, Shao, Daniel, Vaidya, Anurag J., Chen, Chengkuan, Zhuang, Luoting, Williamson, Drew F.K., Shaban, Muhammad, Chen, Tiffany Y., Mahmood, Faisal
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 10.10.2022
• Subjects: AI in oncology; Artificial Intelligence; deep learning; deep learning in oncology; Electronic Health Records; Genomics; Humans; Medical Oncology; multimodal AI; multimodal fusion; multimodal integration; Radiology; deep learning in oncology; deep learning; AI in oncology; multimodal fusion; multimodal integration; multimodal AI
• ISSN: 1535-6108; 1878-3686; 1878-3686
• DOI: 10.1016/j.ccell.2022.09.012

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, histology, and genomics to electronic health records. Current artificial intelligence (AI) models operate mainly in the realm of a single modality, neglecting the broader clinical context, which inevitably diminishes their potential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. AI models are also capable of discovering novel patterns within and across modalities suitable for explaining differences in patient outcomes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion and association discovery. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges in clinical adoption and discuss emerging solutions. This review summarizes the potential of AI methods to leverage and utilize complex multimodal data to provide improved patient outcome predictions and accelerate the discovery of novel biomarkers via multimodal interconnection.