Bi-ClueMVSNet: Learning Bidirectional Occlusion Clues for Multi-View Stereo

• Published in: 2023 International Joint Conference on Neural Networks (IJCNN) pp. 1 - 8
• Main Authors: Zhang, Zhe, Hu, Yuxi, Gao, Huachen, Wang, Ronggang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 18.06.2023
• Subjects: Costs; Neural networks; Pipelines; Point cloud compression; Solid modeling; Three-dimensional displays; Training
• ISSN: 2161-4407
• DOI: 10.1109/IJCNN54540.2023.10191325

Deep learning-based multi-view stereo (MVS) meth-ods have achieved promising results in recent years. However, very few existing works take the occlusion issues into consideration, leading to poor reconstruction results on the boundaries and occluded areas. In this paper, the Bidirectional Occlusion Clues-based Multi-View Stereo Network (Bi-ClueMVSNet) is proposed as an end-to-end MVS framework that explicitly models the occlusion obstacle for depth map inference and 3D modeling. To this end, we use bidirectional projection for the first time to reduce the propagation and accumulation of incorrect matches and build the occlusion-enhanced network to further advance the representational ability from 2D visibility maps to 3D occlusion clues. As for depth map estimation, we combine the characteristics of both regression and classification approaches to propose the adaptive depth map inference strategy. Besides, the robustness of the training process is further guaranteed and elevated by the occlusion clues-based loss function. The proposed method significantly improves the accuracy of depth map inference in boundaries and heavily occluded areas and brings the overall quality of the reconstructed point cloud to a new altitude. Extensive experiments are performed on DTU, Tanks and Temples, and BlendedMVS datasets to demonstrate the persuasiveness of the proposed framework.