SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

• Published in: arXiv.org
• Main Authors: Ahmed Tawfik Aboukhadra, Robertini, Nadia, Malik, Jameel, Elhayek, Ahmed, Reis, Gerd, Stricker, Didier
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 02.10.2024
• Subjects: Mixed reality; Neural networks; Pose estimation; Real time; Robotic surgery; Surgical instruments; Synthetic data

Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.