Detecting Anatomical Landmarks From Limited Medical Imaging Data Using Two-Stage Task-Oriented Deep Neural Networks

• Published in: IEEE transactions on image processing Vol. 26; no. 10; pp. 4753 - 4764
• Main Authors: Zhang, Jun, Liu, Mingxia, Shen, Dinggang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Anatomical landmark detection; Artificial neural networks; Biological neural networks; Biomedical imaging; Brain; Computed tomography; Computer architecture; deep convolutional neural networks; Landmarks; limited medical imaging data; Machine learning; Magnetic resonance imaging; Medical imaging; Neural networks; Patches (structures); Prostate; Real time; Target recognition; task-oriented; Testing; Three-dimensional displays; Training; Training data
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2017.2721106

One of the major challenges in anatomical landmark detection, based on deep neural networks, is the limited availability of medical imaging data for network learning. To address this problem, we present a two-stage task-oriented deep learning method to detect large-scale anatomical landmarks simultaneously in real time, using limited training data. Specifically, our method consists of two deep convolutional neural networks (CNN), with each focusing on one specific task. Specifically, to alleviate the problem of limited training data, in the first stage, we propose a CNN based regression model using millions of image patches as input, aiming to learn inherent associations between local image patches and target anatomical landmarks. To further model the correlations among image patches, in the second stage, we develop another CNN model, which includes a) a fully convolutional network that shares the same architecture and network weights as the CNN used in the first stage and also b) several extra layers to jointly predict coordinates of multiple anatomical landmarks. Importantly, our method can jointly detect large-scale (e.g., thousands of) landmarks in real time. We have conducted various experiments for detecting 1200 brain landmarks from the 3D T1-weighted magnetic resonance images of 700 subjects, and also 7 prostate landmarks from the 3D computed tomography images of 73 subjects. The experimental results show the effectiveness of our method regarding both accuracy and efficiency in the anatomical landmark detection.