A multiscale double-branch residual attention network for anatomical–functional medical image fusion

• Published in: Computers in biology and medicine Vol. 141; p. 105005
• Main Authors: Li, Weisheng, Peng, Xiuxiu, Fu, Jun, Wang, Guofen, Huang, Yuping, Chao, Feifei
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.02.2022; Elsevier Limited
• Subjects: Algorithms; Artificial neural networks; Attention; Clustering; Computer vision; Decomposition; Deep learning; Dictionaries; Disease Progression; Double branches; Explosions; Feature extraction; Frequency dependence; Humans; Image fusion; Image processing; Image Processing, Computer-Assisted; Image reconstruction; Kernels; Magnetic resonance imaging; Medical imaging; Metabolism; Methods; Modules; Multiscale; Neural networks; Neural Networks, Computer; Optimization algorithms; Optimization techniques; Residual; Technology; Tomography; Wavelet transforms; Double branches; Residual; Image fusion; Multiscale; Attention
• ISSN: 0010-4825; 1879-0534
• DOI: 10.1016/j.compbiomed.2021.105005

Medical image fusion technology synthesizes complementary information from multimodal medical images. This technology is playing an increasingly important role in clinical applications. In this paper, we propose a new convolutional neural network, which is called the multiscale double-branch residual attention (MSDRA) network, for fusing anatomical–functional medical images. Our network contains a feature extraction module, a feature fusion module and an image reconstruction module. In the feature extraction module, we use three identical MSDRA blocks in series to extract image features. The MSDRA block has two branches. The first branch uses a multiscale mechanism to extract features of different scales with three convolution kernels of different sizes, while the second branch uses six 3 × 3 convolutional kernels. In addition, we propose the Feature L1-Norm fusion strategy to fuse the features obtained from the input images. Compared with the reference image fusion algorithms, MSDRA consumes less fusion time and achieves better results in visual quality and the objective metrics of Spatial Frequency (SF), Average Gradient (AG), Edge Intensity (EI), Quality-Aware Clustering (QAC), Variance (VAR), and Visual Information Fidelity for Fusion (VIFF). •A new convolutional neural network (MSDRA) is applied to extract image features.•The Feature L1-Norm fusion strategy is implemented in the fusion process.•Our fusion results have better performance of objective metrics.•Our fusion results provide clearer details in fusion images.