Multi-subject data augmentation for target subject semantic decoding with deep multi-view adversarial learning

• Published in: Information sciences Vol. 547; pp. 1025 - 1044
• Main Authors: Li, Dan, Du, Changde, Wang, Shengpei, Wang, Haibao, He, Huiguang
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 08.02.2021
• Subjects: Data augmentation; Multi-view adversarial learning; Semantic decoding; Sparse reconstruction relation; Data augmentation; Multi-view adversarial learning; Semantic decoding; Sparse reconstruction relation
• ISSN: 0020-0255; 1872-6291
• DOI: 10.1016/j.ins.2020.09.012

•We provide a new insight into the problem of brain semantic decoding. That is, we introduce a multi-subject fMRI data augmentation method to improve the performance of the target subject.•A latent space is introduced to solve the problem of feature mismatch and multiple GAN architectures are introduced to solve the problem of distribution mismatch between distinct subjects.•The experimental results show that our method is better than the baseline methods, especially when the size of the training data is small. Functional magnetic resonance imaging (fMRI) is widely used in the field of brain semantic decoding. However, as fMRI data acquisition is time-consuming and expensive, the number of samples is usually small in the existing fMRI datasets. It is difficult to build an accurate brain decoding model for a subject with insufficient fMRI data. The majority of semantic decoding methods focus on designing predictive model with limited samples, while less attention is paid to fMRI data augmentation. Leveraging data from related but different subjects can be regarded as a new strategy to improve the performance of predictive model. There are two challenges when using information from different subjects: 1) feature mismatch; 2) distribution mismatch. In this paper, we propose a multi-subject fMRI data augmentation method to address the above two challenges, which can improve the decoding accuracy of the target subject. Specifically, the subject information can be translated from one to another by using multiple subject-specific encoders, decoders and discriminators. The encoder maps each subject to a shared latent space, solving the feature mismatch problem. The decoders and discriminators form multiple generative adversarial network architectures, which solves the distribution mismatch problem. Meanwhile, to ensure that the representation of the latent space preserves information of the input space, our method not only minimizes the local data reconstruction loss, but also preserves the sparse reconstruction (semantic) relation over the whole dataset of the input space. Extensive experiments on three fMRI datasets demonstrate the effectiveness of the proposed method.