Self-supervised learning of monocular 3D geometry understanding with two- and three-view geometric constraints

• Published in: The Visual computer Vol. 40; no. 2; pp. 1193 - 1204
• Main Authors: Liu, Xiaoliang, Shen, Furao, Zhao, Jian, Nie, Changhai
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.02.2024
• Subjects: Artificial Intelligence; Computer Graphics; Computer Science; Image Processing and Computer Vision; Original Article; Depth estimation; Optical flow estimation; Dynamic scenes; 3D geometry understanding; Self-supervised learning; Visual odometry
• ISSN: 0178-2789; 1432-2315
• DOI: 10.1007/s00371-023-02840-y

The 3D geometry understanding of dynamic scenes captured by moving cameras is one of the cornerstones of 3D scene understanding. Optical flow estimation, visual odometry, and depth estimation are the three most basic tasks in 3D geometry understanding. In this work, we present a unified framework for joint self-supervised learning of optical flow estimation, visual odometry, and depth estimation with two- and three-view geometric constraints. As we all know, visual odometry and depth estimation are more sensitive to dynamic objects, while optical flow estimation is more difficult to estimate the boundary area moved out of the image. To this end, we use estimated optical flow to help visual odometry and depth estimation process dynamic objects and use a rigid flow synthesized by the estimated pose and depth to help learn the optical flow of the area that moves out of the boundary due to camera motion. In order to further improve the consistency of cross-tasks, we introduce three-view geometric constraints and propose a three-view consistency loss. Finally, experiments on the KITTI data set show that our method can effectively improve the performance of the occluded boundary area and the dynamic object area. Moreover, our method achieves comparable or better performance than other monocular self-supervised state-of-the-art methods in these three subtasks.