Real-Time Globally Consistent 3D Reconstruction With Semantic Priors

• Published in: IEEE transactions on visualization and computer graphics Vol. 29; no. 4; pp. 1977 - 1991
• Main Authors: Huang, Shi-Sheng, Chen, Haoxiang, Huang, Jiahui, Fu, Hongbo, Hu, Shi-Min
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D reconstruction; Bundle adjustment; Cameras; Consistency; Continuity (mathematics); Geometry; Metric space; Pose estimation; Real time; Real-time systems; Reconstruction; semantic fusion; semantic pose graph; semantic tracker; Semantics; Simultaneous localization and mapping; Three-dimensional displays
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2021.3137912

Maintaining global consistency continues to be critical for online 3D indoor scene reconstruction. However, it is still challenging to generate satisfactory 3D reconstruction in terms of global consistency for previous approaches using purely geometric analysis, even with bundle adjustment or loop closure techniques. In this article, we propose a novel real-time 3D reconstruction approach which effectively integrates both semantic and geometric cues. The key challenge is how to map this indicative information, i.e., semantic priors, into a metric space as measurable information, thus enabling more accurate semantic fusion leveraging both the geometric and semantic cues. To this end, we introduce a semantic space with a continuous metric function measuring the distance between discrete semantic observations. Within the semantic space, we present an accurate frame-to-model semantic tracker for camera pose estimation, and semantic pose graph equipped with semantic links between submaps for globally consistent 3D scene reconstruction. With extensive evaluation on public synthetic and real-world 3D indoor scene RGB-D datasets, we show that our approach outperforms the previous approaches for 3D scene reconstruction both quantitatively and qualitatively, especially in terms of global consistency.