DeepOrganNet: On-the-Fly Reconstruction and Visualization of 3D / 4D Lung Models from Single-View Projections by Deep Deformation Network

• Published in: IEEE transactions on visualization and computer graphics Vol. 26; no. 1; pp. 960 - 970
• Main Authors: Wang, Yifan, Zhong, Zichun, Hua, Jing
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 2D projections; 3D / 4D shapes; Accuracy; Algorithms; Apexes; Artificial neural networks; Biomedical imaging; Computational efficiency; Computed tomography; Computing time; Cone-Beam Computed Tomography; Deep deformation network; Deep Learning; Deformation; Human subjects; Humans; Image degradation; Image quality; Image reconstruction; Imaging, Three-Dimensional - methods; Lung - diagnostic imaging; Lung Neoplasms - diagnostic imaging; Lungs; Machine learning; Medical imaging; Neural Networks, Computer; organ meshes; Phantoms, Imaging; Radiation dosage; Radiation therapy; Real time; Shape; single-view; Solid modeling; Strain; Tensors; Three dimensional models; Three-dimensional displays; Two dimensional displays; Visualization
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2019.2934369

This paper introduces a deep neural network based method, i.e., DeepOrganNet, to generate and visualize fully high-fidelity 3D / 4D organ geometric models from single-view medical images with complicated background in real time. Traditional 3D / 4D medical image reconstruction requires near hundreds of projections, which cost insufferable computational time and deliver undesirable high imaging / radiation dose to human subjects. Moreover, it always needs further notorious processes to segment or extract the accurate 3D organ models subsequently. The computational time and imaging dose can be reduced by decreasing the number of projections, but the reconstructed image quality is degraded accordingly. To our knowledge, there is no method directly and explicitly reconstructing multiple 3D organ meshes from a single 2D medical grayscale image on the fly. Given single-view 2D medical images, e.g., 3D / 4D-CT projections or X-ray images, our end-to-end DeepOrganNet framework can efficiently and effectively reconstruct 3D / 4D lung models with a variety of geometric shapes by learning the smooth deformation fields from multiple templates based on a trivariate tensor-product deformation technique, leveraging an informative latent descriptor extracted from input 2D images. The proposed method can guarantee to generate high-quality and high-fidelity manifold meshes for 3D / 4D lung models; while, all current deep learning based approaches on the shape reconstruction from a single image cannot. The major contributions of this work are to accurately reconstruct the 3D organ shapes from 2D single-view projection, significantly improve the procedure time to allow on-the-fly visualization, and dramatically reduce the imaging dose for human subjects. Experimental results are evaluated and compared with the traditional reconstruction method and the state-of-the-art in deep learning, by using extensive 3D and 4D examples, including both synthetic phantom and real patient datasets. The efficiency of the proposed method shows that it only needs several milliseconds to generate organ meshes with 10K vertices, which has great potential to be used in real-time image guided radiation therapy (IGRT).