MindBridge: A Cross-Subject Brain Decoding Framework

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 11333 - 11342
• Main Authors: Wang, Shizun, Liu, Songhua, Tan, Zhenxiong, Wang, Xinchao
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Accuracy; Adaptation models; Biological system modeling; Brain decoding; Brain modeling; Cross-subject; diffusion model; Functional magnetic resonance imaging; Neuroscience; Representation learning
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.01077

Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle re-construction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Fur-thermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge