Task-Induced Pyramid and Attention GAN for Multimodal Brain Image Imputation and Classification in Alzheimer's Disease

• Published in: IEEE journal of biomedical and health informatics Vol. 26; no. 1; pp. 36 - 43
• Main Authors: Gao, Xingyu, Shi, Feng, Shen, Dinggang, Liu, Manhua
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Alzheimer Disease - diagnostic imaging; Alzheimer's disease; Attention; Attention task; Brain; Brain - diagnostic imaging; Classification; Convolution; Deep learning; dense convolution network; Diagnosis; Diseases; Feature extraction; generative adversarial network; Generative adversarial networks; Humans; Image classification; Machine learning; Magnetic resonance imaging; Magnetic Resonance Imaging - methods; Medical imaging; Missing data; Multimodal brain images; Neurodegenerative diseases; Neuroimaging; Positron emission; Positron emission tomography; Positron-Emission Tomography - methods; Structure-function relationships; Task analysis; Tomography
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2021.3097721

With the advance of medical imaging technologies, multimodal images such as magnetic resonance images (MRI) and positron emission tomography (PET) can capture subtle structural and functional changes of brain, facilitating the diagnosis of brain diseases such as Alzheimer's disease (AD). In practice, multimodal images may be incomplete since PET is often missing due to high financial costs or availability. Most of the existing methods simply excluded subjects with missing data, which unfortunately reduced the sample size. In addition, how to extract and combine multimodal features is still challenging. To address these problems, we propose a deep learning framework to integrate a task-induced pyramid and attention generative adversarial network (TPA-GAN) with a pathwise transfer dense convolution network (PT-DCN) for imputation and classification of multimodal brain images. First, we propose a TPA-GAN to integrate pyramid convolution and attention module as well as disease classification task into GAN for generating the missing PET data with their MRI. Then, with the imputed multimodal images, we build a dense convolution network with pathwise transfer blocks to gradually learn and combine multimodal features for final disease classification. Experiments are performed on ADNI-1/2 datasets to evaluate our method, achieving superior performance in image imputation and brain disease diagnosis compared to state-of-the-art methods.