PredStereo: An Accurate Real-time Stereo Vision System

• Published in: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp. 4078 - 4087
• Main Authors: Moolchandani, Diksha, Shrivastava, Nivedita, Kumar, Anshul, Sarangi, Smruti R.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.01.2022
• Subjects: Computer vision; Control systems; Estimation error; Heuristic algorithms; Prediction algorithms; Switches; Three-dimensional displays; Vision Systems and Applications 3D Computer Vision; Stereo Processing; Vision for Aerial/Drone/Underwater/Ground Vehicles
• ISSN: 2642-9381
• DOI: 10.1109/WACV51458.2022.00413

Stereo vision algorithms are important building blocks of self-driving applications. The two primary requirements of a self-driving vehicle are real-time operation and nearly 100% accuracy in constructing the 3D scene regardless of the weather conditions and the degree of ambient light. Sadly, most real-time systems as of today provide a level of accuracy that is inadequate and this endangers the life of the passengers; consequently, it is necessary to supplement such systems with expensive LiDAR-based sensors. We observe that for a given scene, different stereo matching algorithms can have vastly different accuracies, and among these algorithms there is no clear winner. This makes the case for a hybrid stereo vision system where the best stereo vision algorithm for a stereo image pair is chosen by a predictor dynamically, in real-time.We implement such a system called PredStereo in ASIC 1 that combines two diametrically different stereo vision algorithms, CNN-based and traditional, and chooses the best one at runtime. In addition, it associates a confidence with the chosen algorithm, such that the higher-level control system can be switched on in case of a low confidence value. We show that designing a predictor that is explainable and a system that respects soft real-time constraints is non-trivial. Hence, we propose a variety of hardware optimizations that enable our system to work in real-time. Overall, PredStereo improves the disparity estimation error over a state-of-the-art CNN-based stereo vision system by up to 18% (on average 6.25%) with a negligible area overhead (0.003mm 2 ) while respecting real-time constraints.