NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

• Published in: Medical Image Computing and Computer Assisted Intervention - MICCAI 2022 Vol. 13435; pp. 140 - 150
• Main Authors: Xing, Zhaohu, Yu, Lequan, Wan, Liang, Han, Tong, Zhu, Lei
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer 2022; Springer Nature Switzerland
• Series: Lecture Notes in Computer Science
• Subjects: Brain tumor segmentation; Modality-sensitive gating; Multi-modal MRI; Nested Modality-aware Feature Aggregation
• ISBN: 9783031164422; 3031164423
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-16443-9_14

Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities and apply modality-sensitive gating (MSG) at lower scales for more effective skip connections. Specifically, the multi-modal fusion is conducted in our proposed Nested Modality-aware Feature Aggregation (NMaFA) module, which enhances long-term dependencies within individual modalities via a tri-orientated spatial-attention transformer, and further complements key contextual information among modalities via a cross-modality attention transformer. Extensive experiments on BraTS2020 benchmark and a private meningiomas segmentation (MeniSeg) dataset show that the NestedFormer clearly outperforms the state-of-the-arts. The code is available at https://github.com/920232796/NestedFormer.