Brain Inspired Keypoint Matching for 3D Scene Reconstruction

• Published in: 2022 8th International Conference on Virtual Reality (ICVR) pp. 33 - 40
• Main Authors: Zaman, Anam, Yangyu, Fan, Ayub, Muhammad Saad, Guoyun, LV, Shiva, Liu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 26.05.2022
• Subjects: 3D scene reconstruction; Benchmark testing; Continual learning; Graph Neural Networks; Image keypoint matching; Image matching; Knowledge engineering; Learning systems; Memory management; Performance Evaluation; Performance gain; Three-dimensional displays
• ISSN: 2331-9569
• DOI: 10.1109/ICVR55215.2022.9847807

In this paper, we investigate the keypoint matching problem in a 3D scene reconstruction system. 3D scene reconstruction using a sequential set of images or video is an essential component in various virtual reality(VR) and augmented reality(AR) solutions. Keypoint matching is necessary for achieving a close to reality model of the scene using varying views. Although deep learning-based methods have been readily proposed for image matching using the keypoints and respective descriptors. These methods do not take into account the previous image matches when performing correspondence on the current pair of images. This is crucial in the presence of sequential images or frames from a video. A continual learning-based image matching framework is proposed that replicates the working of a human brain. The method efficiently extracts knowledge, stores the knowledge in its memory, and reuses it for future matches. The proposed method increases the expressiveness of the descriptors to be used for keypoint matching in the pair of images. Specifically, the methodology using a continual graph attention network to find the correspondence among keypoints in a pair of images. The methodology is thoroughly validated on a challenging benchmark dataset namely HPatches. The methodology is evaluated along with present state-of-the-art handcrafted and learning-based image matching methods under varying confidence thresholds. The experimental results reveal that the proposed methodology outperforms all the underlying methods while achieving significant improvement.