Multi-Modal Tumor Segmentation with Deformable Aggregation and Uncertain Region Inpainting

• Published in: IEEE transactions on medical imaging Vol. 42; no. 10; p. 1
• Main Authors: Zhang, Yue, Peng, Chengtao, Tong, Ruofeng, Lin, Lanfen, Chen, Yen-Wei, Chen, Qingqing, Hu, Hongjie, Kevin Zhou, S.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Computed tomography; deformable feature aggregation; Formability; Fuses; Image segmentation; Liver; Medical diagnosis; Medical diagnostic imaging; Misalignment; Modules; Multi-modal tumor segmentation; Segmentation; Three-dimensional displays; Tumors; uncertain region inpainting
• ISSN: 0278-0062; 1558-254X; 1558-254X
• DOI: 10.1109/TMI.2023.3275592

Multi-modal tumor segmentation exploits complementary information from different modalities to help recognize tumor regions. Known multi-modal segmentation methods mainly have deficiencies in two aspects: First, the adopted multi-modal fusion strategies are built upon well-aligned input images, which are vulnerable to spatial misalignment between modalities (caused by respiratory motions, different scanning parameters, registration errors, etc). Second, the performance of known methods remains subject to the uncertainty of segmentation, which is particularly acute in tumor boundary regions. To tackle these issues, in this paper, we propose a novel multi-modal tumor segmentation method with deformable feature fusion and uncertain region refinement. Concretely, we introduce a deformable aggregation module, which integrates feature alignment and feature aggregation in an ensemble, to reduce inter-modality misalignment and make full use of cross-modal information. Moreover, we devise an uncertain region inpainting module to refine uncertain pixels using neighboring discriminative features. Experiments on two clinical multi-modal tumor datasets demonstrate that our method achieves promising tumor segmentation results and outperforms state-of-the-art methods.