Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks

• Published in: NeuroImage (Orlando, Fla.) Vol. 145; no. Pt B; pp. 314 - 328
• Main Authors: Jang, Hojin, Plis, Sergey M., Calhoun, Vince D., Lee, Jong-Hwan
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 15.01.2017; Elsevier Limited
• Subjects: Adult; Brain; Brain - diagnostic imaging; Brain - physiology; Classification; Functional Neuroimaging - methods; Humans; Magnetic Resonance Imaging - methods; Motor Activity - physiology; Neural networks; Neural Networks, Computer; Pattern Recognition, Automated - methods; Perception - physiology; Sparsity
• ISSN: 1053-8119; 1095-9572; 1095-9572
• DOI: 10.1016/j.neuroimage.2016.04.003

Feedforward deep neural networks (DNNs), artificial neural networks with multiple hidden layers, have recently demonstrated a record-breaking performance in multiple areas of applications in computer vision and speech processing. Following the success, DNNs have been applied to neuroimaging modalities including functional/structural magnetic resonance imaging (MRI) and positron-emission tomography data. However, no study has explicitly applied DNNs to 3D whole-brain fMRI volumes and thereby extracted hidden volumetric representations of fMRI that are discriminative for a task performed as the fMRI volume was acquired. Our study applied fully connected feedforward DNN to fMRI volumes collected in four sensorimotor tasks (i.e., left-hand clenching, right-hand clenching, auditory attention, and visual stimulus) undertaken by 12 healthy participants. Using a leave-one-subject-out cross-validation scheme, a restricted Boltzmann machine-based deep belief network was pretrained and used to initialize weights of the DNN. The pretrained DNN was fine-tuned while systematically controlling weight-sparsity levels across hidden layers. Optimal weight-sparsity levels were determined from a minimum validation error rate of fMRI volume classification. Minimum error rates (mean±standard deviation; %) of 6.9 (±3.8) were obtained from the three-layer DNN with the sparsest condition of weights across the three hidden layers. These error rates were even lower than the error rates from the single-layer network (9.4±4.6) and the two-layer network (7.4±4.1). The estimated DNN weights showed spatial patterns that are remarkably task-specific, particularly in the higher layers. The output values of the third hidden layer represented distinct patterns/codes of the 3D whole-brain fMRI volume and encoded the information of the tasks as evaluated from representational similarity analysis. Our reported findings show the ability of the DNN to classify a single fMRI volume based on the extraction of hidden representations of fMRI volumes associated with tasks across multiple hidden layers. Our study may be beneficial to the automatic classification/diagnosis of neuropsychiatric and neurological diseases and prediction of disease severity and recovery in (pre-) clinical settings using fMRI volumes without requiring an estimation of activation patterns or ad hoc statistical evaluation. •Deep neural network (DNN) was proposed to classify fMRI volume of sensorimotor tasks.•DNN weights were optimized via non-zero value percentage in cross-validation (CV) framework.•Classification performance was superior from three-layer DNN than one-/two-/four-layer DNNs.•Weight/hidden representations were highly task-specific from higher than lower hidden layers.•Sparsity of weights between the input and first hidden layer enhanced the performance.