BTMF-GAN: A multi-modal MRI fusion generative adversarial network for brain tumors

• Published in: Computers in biology and medicine Vol. 157; p. 106769
• Main Authors: Liu, Xiao, Chen, Hongyi, Yao, Chong, Xiang, Rui, Zhou, Kun, Du, Peng, Liu, Weifan, Liu, Jie, Yu, Zekuan
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.05.2023; Elsevier Limited
• Subjects: Adaptive transformer; Brain; Brain cancer; Brain Neoplasms - diagnostic imaging; Brain tumors; Computer vision; Deep learning; Generative adversarial networks; Humans; Image contrast; Image fusion; Image Processing, Computer-Assisted; Image quality; Internal Medicine; Machine learning; Magnetic Resonance Imaging; Medical imaging; Multi-modal MRI; Neuroimaging; Other; Qualitative analysis; Salient loss; Tumors; Adaptive transformer; Multi-modal MRI; Salient loss; Image fusion
• DOI: 10.1016/j.compbiomed.2023.106769
• ISSN: 0010-4825; 1879-0534

Image fusion techniques have been widely used for multi-modal medical image fusion tasks. Most existing methods aim to improve the overall quality of the fused image and do not focus on the more important textural details and contrast between the tissues of the lesion in the regions of interest (ROIs). This can lead to the distortion of important tumor ROIs information and thus limits the applicability of the fused images in clinical practice. To improve the fusion quality of ROIs relevant to medical implications, we propose a multi-modal MRI fusion generative adversarial network (BTMF-GAN) for the task of multi-modal MRI fusion of brain tumors. Unlike existing deep learning approaches which focus on improving the global quality of the fused image, the proposed BTMF-GAN aims to achieve a balance between tissue details and structural contrasts in brain tumor, which is the region of interest crucial to many medical applications. Specifically, we employ a generator with a U-shaped nested structure and residual U-blocks (RSU) to enhance multi-scale feature extraction. To enhance and recalibrate features of the encoder, the multi-perceptual field adaptive transformer feature enhancement module (MRF-ATFE) is used between the encoder and the decoder instead of a skip connection. To increase contrast between tumor tissues of the fused image, a mask-part block is introduced to fragment the source image and the fused image, based on which, we propose a novel salient loss function. Qualitative and quantitative analysis of the results on the public and clinical datasets demonstrate the superiority of the proposed approach to many other commonly used fusion methods. •The BTMF-GAN is proposed for the multi-modal MRI fusion of brain tumors.•Uses a generator with a U-shaped nested structure to improve multi-scale feature extraction.•Uses MRF-AFTE to enhance and calibrate the features of the encoder.•Proposes a novel salient loss function to preserve tissue contrasts and textural details.