Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

• Published in: IEEE transactions on radiation and plasma medical sciences Vol. 3; no. 2; pp. 162 - 169
• Main Authors: Guo, Zhe, Li, Xiang, Huang, Heng, Guo, Ning, Li, Quanzheng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Biomedical imaging; Clinical decision making; Computed tomography; Computed tomography (CT); Computer vision; convolutional neural network (CNN); Decision making; Deep learning; Empirical analysis; Feature extraction; Image analysis; Image processing; Image segmentation; Imaging techniques; Machine learning; Magnetic resonance imaging; magnetic resonance imaging (MRI); Medical imaging; Medical research; multimodal image; Neural networks; Positron emission; Positron emission tomography; positron emission tomography (PET); Soft tissues; Tomography; Tumors; Convolutional Neural Network; Computed Tomography (CT); Magnetic Resonance Imaging (MRI); Multi-modal Image; Positron Emission Tomography (PET)
• ISSN: 2469-7311; 2469-7303
• DOI: 10.1109/TRPMS.2018.2890359

Multimodality medical imaging techniques have been increasingly applied in clinical practice and research studies. Corresponding multimodal image analysis and ensemble learning schemes have seen rapid growth and bring unique value to medical applications. Motivated by the recent success of applying deep learning methods to medical image processing, we first propose an algorithmic architecture for supervised multimodal image analysis with cross-modality fusion at the feature learning level, classifier level, and decision-making level. We then design and implement an image segmentation system based on deep convolutional neural networks to contour the lesions of soft tissue sarcomas using multimodal images, including those from magnetic resonance imaging, computed tomography, and positron emission tomography. The network trained with multimodal images shows superior performance compared to networks trained with single-modal images. For the task of tumor segmentation, performing image fusion within the network (i.e., fusing at convolutional or fully connected layers) is generally better than fusing images at the network output (i.e., voting). This paper provides empirical guidance for the design and application of multimodal image analysis.